openpitstop 1.9.0 → 1.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (105) hide show
  1. package/PRIVACY.md +77 -90
  2. package/README.md +365 -887
  3. package/dist/agentBudget.d.ts +33 -0
  4. package/dist/agentBudget.js +165 -0
  5. package/dist/agentBudget.js.map +1 -0
  6. package/dist/analyzers/ledger/index.d.ts +2 -1
  7. package/dist/analyzers/ledger/index.js +10 -1
  8. package/dist/analyzers/ledger/index.js.map +1 -1
  9. package/dist/analyzers/types.d.ts +2 -0
  10. package/dist/candidate.d.ts +20 -0
  11. package/dist/candidate.js +87 -0
  12. package/dist/candidate.js.map +1 -0
  13. package/dist/cli.js +1 -1
  14. package/dist/cli.js.map +1 -1
  15. package/dist/commands/ask.d.ts +0 -8
  16. package/dist/commands/ask.js +36 -34
  17. package/dist/commands/ask.js.map +1 -1
  18. package/dist/commands/budget.d.ts +1 -0
  19. package/dist/commands/budget.js +12 -3
  20. package/dist/commands/budget.js.map +1 -1
  21. package/dist/commands/drive.js +77 -51
  22. package/dist/commands/drive.js.map +1 -1
  23. package/dist/commands/fix.d.ts +7 -0
  24. package/dist/commands/fix.js +90 -108
  25. package/dist/commands/fix.js.map +1 -1
  26. package/dist/commands/gate.js +6 -2
  27. package/dist/commands/gate.js.map +1 -1
  28. package/dist/commands/pen.d.ts +3 -1
  29. package/dist/commands/pen.js +31 -42
  30. package/dist/commands/pen.js.map +1 -1
  31. package/dist/commands/readyCheck.d.ts +1 -1
  32. package/dist/commands/readyCheck.js +6 -36
  33. package/dist/commands/readyCheck.js.map +1 -1
  34. package/dist/commands/scan.d.ts +6 -5
  35. package/dist/commands/scan.js +26 -28
  36. package/dist/commands/scan.js.map +1 -1
  37. package/dist/commands/verify.d.ts +3 -1
  38. package/dist/commands/verify.js +5 -1
  39. package/dist/commands/verify.js.map +1 -1
  40. package/dist/evidence.js +4 -0
  41. package/dist/evidence.js.map +1 -1
  42. package/dist/installer/targets.js +1 -1
  43. package/dist/installer/targets.js.map +1 -1
  44. package/dist/intent.d.ts +4 -7
  45. package/dist/intent.js +41 -77
  46. package/dist/intent.js.map +1 -1
  47. package/dist/pen/dynamic.d.ts +1 -24
  48. package/dist/pen/dynamic.js +10 -1
  49. package/dist/pen/dynamic.js.map +1 -1
  50. package/dist/pen/fix.js +5 -1
  51. package/dist/pen/fix.js.map +1 -1
  52. package/dist/repro/generate.js +1 -0
  53. package/dist/repro/generate.js.map +1 -1
  54. package/dist/repro/pen.js +1 -0
  55. package/dist/repro/pen.js.map +1 -1
  56. package/dist/repro/run.d.ts +1 -0
  57. package/dist/repro/run.js +17 -1
  58. package/dist/repro/run.js.map +1 -1
  59. package/dist/sandbox/container.d.ts +9 -0
  60. package/dist/sandbox/container.js +129 -0
  61. package/dist/sandbox/container.js.map +1 -0
  62. package/dist/sandbox/worker.d.ts +1 -0
  63. package/dist/sandbox/worker.js +41 -0
  64. package/dist/sandbox/worker.js.map +1 -0
  65. package/dist/scanCache.d.ts +27 -0
  66. package/dist/scanCache.js +89 -0
  67. package/dist/scanCache.js.map +1 -0
  68. package/dist/verify/acceptance.d.ts +5 -1
  69. package/dist/verify/acceptance.js +29 -16
  70. package/dist/verify/acceptance.js.map +1 -1
  71. package/dist/verify/architecture.d.ts +5 -1
  72. package/dist/verify/architecture.js +6 -1
  73. package/dist/verify/architecture.js.map +1 -1
  74. package/dist/verify/baseline.d.ts +5 -1
  75. package/dist/verify/baseline.js +5 -1
  76. package/dist/verify/baseline.js.map +1 -1
  77. package/dist/verify/chain.js +17 -0
  78. package/dist/verify/chain.js.map +1 -1
  79. package/dist/verify/flow.d.ts +2 -0
  80. package/dist/verify/flow.js +18 -3
  81. package/dist/verify/flow.js.map +1 -1
  82. package/dist/verify/gateMatrix.d.ts +4 -0
  83. package/dist/verify/gateMatrix.js +23 -12
  84. package/dist/verify/gateMatrix.js.map +1 -1
  85. package/dist/verify/holdout.d.ts +5 -1
  86. package/dist/verify/holdout.js +8 -2
  87. package/dist/verify/holdout.js.map +1 -1
  88. package/dist/verify/regression.d.ts +5 -1
  89. package/dist/verify/regression.js +6 -1
  90. package/dist/verify/regression.js.map +1 -1
  91. package/dist/verify/stack.d.ts +5 -1
  92. package/dist/verify/stack.js +6 -1
  93. package/dist/verify/stack.js.map +1 -1
  94. package/dist/verify/state.d.ts +5 -1
  95. package/dist/verify/state.js +6 -1
  96. package/dist/verify/state.js.map +1 -1
  97. package/dist/verify/verifier.d.ts +5 -1
  98. package/dist/verify/verifier.js +6 -1
  99. package/dist/verify/verifier.js.map +1 -1
  100. package/docs/release-controls.md +107 -0
  101. package/package.json +70 -69
  102. package/templates/pitstop.command.md +17 -5
  103. package/templates/pitstop.prompt.md +390 -390
  104. package/scripts/__pycache__/gen-gifs.cpython-313.pyc +0 -0
  105. package/scripts/gen-gifs.py +0 -231
package/README.md CHANGED
@@ -1,887 +1,365 @@
1
- <p align="center">
2
- <img src="docs/media/pitstop-logo.png" alt="OpenPitStop" width="380">
3
- </p>
4
-
5
- # OpenPitStop CLI
6
-
7
- **The agent finally has a referee it can't cheat.** OpenPitStop is a CLI that scans your repo,
8
- scores it, and checks everything your AI coding agent does — so when it says "done", you
9
- know it's actually done.
10
-
11
- [![npm version](https://img.shields.io/npm/v/openpitstop)](https://www.npmjs.com/package/openpitstop)
12
- [![CI](https://github.com/Krish-1507/OpenPitStop/actions/workflows/ci.yml/badge.svg)](https://github.com/Krish-1507/OpenPitStop/actions/workflows/ci.yml)
13
- [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
14
-
15
- > AI coding agents are brilliant at fixing things, and just as brilliant at *saying they
16
- > did* when they didn't. OpenPitStop measures your repo with scans, seals every number so it
17
- > can't be edited later, attacks your own app with a live penetration test, and checks every
18
- > change your agent makes. The exit codes tell you the truth: `0` clean, `1` suspicious,
19
- > `2` confirmed cheat.
20
-
21
- ---
22
-
23
- ## Quick install
24
-
25
- **Start here:** `npx openpitstop try .` scores any repo in about two seconds with zero
26
- install, or `npm i -g openpitstop` for daily use. No account, no config, no SaaS.
27
-
28
- **Zero install, try it now:**
29
- ```bash
30
- npx openpitstop try .
31
- ```
32
- Scores any repo in about two seconds of scanning. The first run downloads the
33
- package once, after that it is instant.
34
-
35
- **Install it globally (recommended for daily use):**
36
- ```bash
37
- npm install -g openpitstop
38
- openpitstop --help
39
- ```
40
- Now every command starts with `openpitstop` instead of `npx openpitstop`.
41
-
42
- Requires **Node.js 22+**, that is the only dependency. For the full setup
43
- (slash command, pre-commit hook, CI), see [Install](#install).
44
-
45
- ## Use-Cases
46
-
47
- - **You ship with an AI agent and want proof it actually finished.** The gate and
48
- `verify` turn "the agent says done" into a pass or fail you can block a build on.
49
- - **You want a security scan that proves findings, not just guesses.** `pen` attacks
50
- your app in a sandbox and labels each issue PROVEN, indicated, or unproven.
51
- - **You are tired of agents quietly deleting a failing test.** `integrity` and the
52
- gate catch focused, deleted, or rewritten tests and exit 2 (confirmed cheat).
53
- - **You want proof a fix is real, not just a green suite.** `baseline-verify` proves the
54
- verification FAILED on the broken state and PASSES after the fix; `state-verify` proves the
55
- changes your agent *claimed* actually landed on disk and in git; `verifier-check` proves the
56
- referee itself can still say NO.
57
- - **You want CI to fail on a regression, not just a new bug.** Drift compares every
58
- `pen` run to the last sealed one and goes red on a NEW finding.
59
- - **You need a score you can show your team or an auditor.** `report` and `honesty`
60
- produce a sealed HTML report and an honesty certificate.
61
- - **You already pay for a pen tool and want continuous proof for free.** OpenPitStop
62
- keeps a running ledger of proof in your repo. See the OpenPitStop vs Strix
63
- comparison above.
64
-
65
- **Ready to try it?** Run `npx openpitstop try .` on any repo and get your score in
66
- about two seconds.
67
-
68
- ---
69
-
70
- ## Why I built this
71
-
72
- I spend my days running coding agents on real repos. They're brilliant at fixing things —
73
- and equally brilliant at *telling me they did* when they didn't: focusing tests to hide
74
- failures, deleting the failing test, editing an assertion to match the buggy output. I got
75
- tired of auditing my agent's work by hand, so I built a referee.
76
-
77
- OpenPitStop is my own workflow tool, not a showcase: every repo I touch gets the loop, every
78
- change gets the gate, and the numbers in this README are the same numbers I trust. It's
79
- dogfooded hard — OpenPitStop's own CI scans a real repo with OpenPitStop on every push (Linux and
80
- Windows), and the evidence chain is regression-tested because a bug in it once made OpenPitStop
81
- cry `TAMPERED` at baselines it had just written. If it can referee itself, it can referee
82
- your agent.
83
-
84
- ---
85
-
86
- ## Quick Nav
87
-
88
- | Jump to | |
89
- |---|---|
90
- | [Feature tour](#feature-tour) — every feature, in plain English | [Install](#install) · [Usage](#usage) · [Tool support](#tool-support) |
91
- | [Architecture](#architecture) | [Known limitations](#known-limitations) · [Contributing](#contributing) · [License](#license) |
92
-
93
- **Straight to one feature:** [The scan](#the-scan) · [Security fixes](#security-fixes) · [Try it on your repo](#try-it-on-your-repo) · [The test pyramid](#the-test-pyramid) · [The gate](#the-gate) · [Integrity](#integrity) · [Baseline-aware verification](#baseline-aware-verification) · [State verification](#state-verification-dont-trust-the-claim) · [Verifier health](#verifier-health-falsifiability) · [Holdout verification](#holdout-verification-anti-overfitting) · [Acceptance verification](#acceptance-verification-did-the-agent-satisfy-the-requirement) · [Regression verification](#regression-verification-dont-break-what-already-worked) · [The evidence chain](#the-evidence-chain-why-should-i-trust-this-verdict) · [Repo discipline](#repo-discipline-understand--plan--architecture--stack--flow) · [The pen test](#the-pen-test) · [Honesty](#honesty) · [Verify](#verify) · [Trends](#trends) · [Inspect](#inspect) · [Repro](#repro) · [Report](#report) · [Share](#share) · [The live shield](#the-live-shield) · [The GitHub Action](#the-github-action) · [The pre-commit hook](#the-pre-commit-hook)
94
-
95
- **Receipts:** [Caught in the wild](docs/caught-in-the-wild.md) — real gate output, screenshot-ready.
96
-
97
- ---
98
-
99
- ## OpenPitStop vs Strix — why teams pick the referee
100
-
101
- Strix is a great *finder*. OpenPitStop is the *referee*. Both pen-test your app; the
102
- difference is what happens after a vulnerability is found:
103
-
104
- > **Strix finds. OpenPitStop proves.**
105
-
106
- | | [Strix](https://github.com/usestrix/strix) | **OpenPitStop** |
107
- |---|---|---|
108
- | What it produces | Findings + a PoC *report* | Findings + a **failing-first regression test** (`pitstop pen --fix`) |
109
- | Trust in the fix | One-click auto-fix PR | **Honesty Score (0–100)** + integrity gate that catches deleted tests, hardcoded passes, reverted baselines |
110
- | How it runs | Docker + LLM key, non-deterministic | **Zero infra, deterministic, no LLM bill** — runs in any CI |
111
- | Evidence | `strix_runs/` logs | **Tamper-evident, signed `.pitstop/` artifacts** you can audit |
112
- | Secret exfiltration | not emphasized | **Ledger mode** proves the app doesn't phone home with your keys |
113
- | DevSecOps | Cloud platform (paid tiers) | **Free SARIF → GitHub Security tab** + one-number `pitstop gate` |
114
- | Proof coverage | — | **`PITSTOP_PROOF` badge**: % of findings that ship a permanent repro test |
115
- | Continuous proof (drift) | report only — re-run and hope | **Drift gate**: every `pitstop pen` compares to the last sealed run, *proves a fix* (finding gone) and *fails the CI gate* on a new high/critical regression or a hypothesis the live attack just confirmed |
116
- | Prove-my-fix loop | manual | **`pitstop repro <id>`** re-runs the exact attack and asserts the safe outcome — a PASS means the fix is real, a deleted repro test is flagged as a cheat |
117
- | Bug classes covered | strong general set | **30+ vulnerability classes** — race/TOCTOU, IDOR/BOLA, price-tampering, XXE, insecure deserialization, JWT alg-confusion/weak-secret, SSRF, SQL/NoSQLi, command-injection, path traversal, XSS, secrets, CORS, missing headers, rate-limit, and more (plus an optional Semgrep engine you can bolt on) |
118
-
119
- The honest pitch: a pen-test that only reports is a list of things to argue about. A
120
- pen-test that ships the regression test, signs the evidence, and scores the fix on a
121
- gate your CI can block on is something you can actually ship. That's the OpenPitStop
122
- loop — and it's the reason to choose the referee over the hacker.
123
-
124
- ---
125
-
126
- ## Feature tour
127
-
128
- Every feature below is explained in plain English: what it does, and how it
129
- works. Most of it needs nothing more than a `pitstop scan` first.
130
-
131
- ### The scan
132
-
133
- `pitstop scan` runs every check at once, in parallel, and prints one box with a
134
- single **OpenPitStop Score** (0 to 100, A to F). It looks at circular imports,
135
- security issues, duplicated code, test results, build speed, accessibility and
136
- code quality. Each check gives a real number or prints `skipped` with a hint on
137
- how to install the tool it needs. It never makes up a number.
138
-
139
- ### Security fixes
140
-
141
- Under the scan box, every security finding ships with a concrete `fix:` line, so
142
- you get a worklist, not just a list of problems. Findings are labeled
143
- `[indicated]` with the exact code and the fix. The full matrix is in
144
- [docs/security.md](docs/security.md).
145
-
146
- ### Try it on your repo
147
-
148
- `pitstop try .` scores any repo in about two seconds, no setup, no config. It is
149
- the fastest first look, and it seeds a baseline your later runs build on.
150
-
151
- ### The test pyramid
152
-
153
- `pitstop test` runs your **unit, integration and e2e** layers separately, so a
154
- suite that passes cannot hide a missing layer. It names every failing test. One
155
- failing layer and it prints **DO NOT SHIP** and exits 1.
156
-
157
- ### The gate
158
-
159
- `pitstop gate` is the final independent verification authority. It does not ask the agent
160
- whether it's done — it reasons over the **evidence model**: the live scan (score, tests,
161
- regression risk), the integrity diff, the baseline evidence signature, and every sealed
162
- deep-verification layer (baseline-aware, state, acceptance, regression, security/pen drift,
163
- holdout, verifier health). A deterministic decision matrix — evaluated strictly in order —
164
- produces the verdict: **CHEAT** (manipulated evidence, CONFIRMED_CHEAT) → **BLOCKED**
165
- (critical regression, proven critical security, drift) → **FAILED** (required verification
166
- failed) → **UNPROVEN** (insufficient evidence) → **VERIFIED** (strong evidence, all clear).
167
- Layers that never ran are rendered `NOT_CONFIGURED` — never faked into a pass. Configure
168
- mandatory layers with `--require baseline,acceptance,holdout`. Exit codes: `0` pass, `1`
169
- fail, `2` cheat. The old single-box output and `gateOutcome` remain for backward
170
- compatibility.
171
-
172
- ### Integrity
173
-
174
- `pitstop integrity` diffs your change against the sealed baseline and hunts cheat
175
- patterns: focused tests, deleted tests, rewritten tests, swallowed errors,
176
- hardcoded-to-pass values. It exits `0/1/2` the same way.
177
-
178
- ### Baseline-aware verification
179
-
180
- `pitstop baseline-verify` closes the hole in "the test passed, so it's fixed." A passing
181
- verification proves nothing unless the *same* verification demonstrably FAILED on the
182
- broken state — otherwise the agent is grading its own homework. So OpenPitStop runs the
183
- verification against a known-bad baseline commit (must FAIL, evidence sealed), then against
184
- the candidate (must PASS), and only calls it `VERIFIED` when both hold, the verification
185
- identity (command + file hashes) is identical on both sides, and nothing was tampered with.
186
- Anything less is honestly `FAILED`, `UNPROVEN`, or `INTEGRITY_FAILURE`. It runs in isolated
187
- git worktrees, so your working tree is never touched. Full semantics:
188
- [docs/baseline-verify.md](docs/baseline-verify.md).
189
-
190
- ### State verification (don't trust the claim)
191
-
192
- `pitstop state-verify` never reads the agent's natural-language summary — it inspects the
193
- actual filesystem and git. You give it structured claims (`--claim modified:src/auth.ts`,
194
- `--claim created:x`, `--claim deleted:y`) and it independently checks existence, content
195
- hashes, line counts, `git status`/HEAD, catching the classic failures: the tool returned
196
- HTTP 200 but the file never changed, was written empty, was reverted, or a *different*
197
- file changed. Verdicts: `STATE_VERIFIED`, `STATE_MISMATCH`, `UNPROVEN`,
198
- `INTEGRITY_FAILURE`. This proves **that** a change occurred — never whether the code is
199
- correct. Full semantics: [docs/state-verify.md](docs/state-verify.md).
200
-
201
- ### Verifier health (falsifiability)
202
-
203
- `pitstop verifier-check` asks the referee's own question: **can this verification actually
204
- say NO?** It runs the verification on a known-good state (must PASS) and a controlled
205
- known-bad state — an explicit bad ref or your declared mutation, applied in a temp worktree
206
- (must FAIL). `VERIFIER_VALID` means the verification demonstrated falsifiability and its
207
- PASS carries information; `VERIFIER_WEAK` means the seeded fault sailed through;
208
- `VERIFIER_BROKEN` means it fails even when things are correct. A referee that cannot fail
209
- is not a referee. Full semantics: [docs/verifier-check.md](docs/verifier-check.md).
210
-
211
- ### Holdout verification (anti-overfitting)
212
-
213
- `pitstop holdout-verify` is the final, hidden exam. An agent that iterates against visible
214
- checks learns the *evaluator*, not the task — so OpenPitStop separates the two: the checks
215
- it iterates against stay visible, and a **holdout suite defined OUTSIDE the repository**
216
- runs once, at the final stage, in a fresh isolated worktree of the candidate commit. The
217
- agent cannot read the holdout, cannot modify it (every suite file is hashed before and
218
- after execution), and sees only redacted ids + verdicts — never commands, expectations, or
219
- output. With `--baseline`, the suite must FAIL on the known-bad baseline and PASS on the
220
- candidate, so a suite that only ever says PASS is exposed as `HOLDOUT_UNPROVEN`. Verdicts:
221
- `HOLDOUT_PASS`, `HOLDOUT_FAIL`, `HOLDOUT_UNPROVEN`, `HOLDOUT_INTEGRITY_FAILURE`. Full
222
- semantics: [docs/holdout-verify.md](docs/holdout-verify.md).
223
-
224
- ### Acceptance verification (did the agent satisfy the requirement?)
225
-
226
- `pitstop acceptance-verify` answers the question everything else orbits: **did the agent
227
- actually satisfy the original task requirements?** A structured acceptance contract —
228
- requirements with deterministic criteria (`command`, real `http` requests against the
229
- booted app, `fileExists`, `fileContains`) — is the source of truth, not the agent's
230
- self-report. In-repo contracts are hash-pinned: if the agent quietly redefines success,
231
- OpenPitStop reports `INTEGRITY_FAILURE` until a human re-authorizes. With `--baseline`,
232
- criteria that already passed there don't count as the agent's work — a contract that
233
- passes on both sides is `UNPROVEN`. This catches the classic failure: green unit test,
234
- plausible diff, real user flow broken. Verdicts: `SATISFIED`, `NOT_SATISFIED`,
235
- `UNPROVEN`, `INTEGRITY_FAILURE`. Deterministic and observable by design — never an LLM
236
- judge. Full semantics: [docs/acceptance-verify.md](docs/acceptance-verify.md).
237
-
238
- ### Regression verification (don't break what already worked)
239
-
240
- `pitstop regression-check` compares check-level results (per-test names where the runner
241
- exposes them) between a baseline and the candidate, both in isolated worktrees — and is
242
- honest about what each difference means: only previously **verified passing** behavior
243
- that now fails is a `REGRESSION`; already-broken checks that stay broken are `UNCHANGED`;
244
- fixes are `FIXED`; new checks are `NEW_PASS`/`NEW_FAILURE`; flaky checks (with `--runs >1`)
245
- and vanished checks are `UNPROVEN` rather than guessed. Regressions hard-block the gate.
246
- Full semantics: [docs/regression-check.md](docs/regression-check.md).
247
-
248
- ### The evidence chain (why should I trust this verdict?)
249
-
250
- `pitstop explain` aggregates every sealed verification document in `.pitstop/` into one
251
- explainable chain — baseline → state → tests → acceptance → security → regression →
252
- integrity → holdout — re-verifies each seal, and derives the verdict instead of asserting
253
- it. Components that ran show their real status with evidence references and digests;
254
- never-run components are listed as **NOT_CONFIGURED** (never rendered as a pass); skipped
255
- tools are **SKIPPED**; tampered evidence is **TAMPERED** and blocks. `BLOCKED` verdicts
256
- quote the underlying reasons ("previously passing check(s) now failing — check B").
257
- Full semantics: [docs/explain.md](docs/explain.md).
258
-
259
- ### Repo discipline (understand → plan → architecture → stack → flow)
260
-
261
- The stages that make the referee *repo-aware*:
262
-
263
- - **`pitstop understand`** — builds the sealed repo-awareness artifact: languages,
264
- frameworks, package manager, verification commands (test/typecheck/lint/build), test
265
- layers, CI, module map, entry points, CODEOWNERS ownership, and the architecture config.
266
- - **`pitstop plan`** — plan before patching: goal, steps, the paths the change may touch
267
- (`expectedPaths`), and the verification commands that will judge it. A sealed contract.
268
- - **`pitstop architecture-check`** — does the change FIT THE SYSTEM? Declared import
269
- boundaries, protected paths (auth/deploy/CI require explicit `--approved`), forbidden
270
- paths (secrets), CODEOWNERS routing, the AI-cheat detectors as shortcut findings, and —
271
- with `--against-plan` — scope-creep detection against the plan.
272
- - **`pitstop verify-stack`** — beyond "did the test pass": unit/integration/e2e +
273
- typecheck + lint + build, whatever the repo has, with a deterministic **failure
274
- diagnosis** per layer (type-error TSxxxx, missing-dependency, assertion-failure,
275
- lint rule, environment, timeout) so fixes are targeted instead of random edits.
276
- - **`pitstop flow`** — the whole pipeline in one command: understand → contract? →
277
- plan-scope → verify-stack → architecture → baseline? → regression? → holdout? → GATE.
278
- Unconfigured stages are SKIPPED honestly.
279
-
280
- Full semantics: [docs/repo-discipline.md](docs/repo-discipline.md).
281
-
282
- ### The pen test
283
-
284
- `pitstop pen` boots your app in a sandbox and fires real attack traffic, so a
285
- finding is **PROVEN** by a live attack, not just guessed. With `--fix` it writes a
286
- failing-first repro test and a safe patch. Nothing reaches the real network.
287
-
288
- ### Drift (the permanent referee)
289
-
290
- `pitstop pen` remembers. Every run seals its verdicts and compares them to the last one, so you
291
- see exactly what changed between today and last week:
292
-
293
- - **NEW** — a finding appeared (or escalated from indicated to proven). This is a regression, so the
294
- gate exits `1` and your CI goes red.
295
- - **RESOLVED** — a finding is gone because the fix worked. This is the "prove my fix" loop, and it is
296
- the most satisfying thing here: run `pitstop repro <id>` to turn a finding into a failing test, ship
297
- the patch, run `pitstop pen` again, and watch it flip to resolved.
298
- - **ESCALATIONS** — something that was only *indicated* by static analysis is now *proven* by a live
299
- attack.
300
-
301
- Strix, the enterprise tool, runs a one-off scan. OpenPitStop keeps a running ledger of proof, so a
302
- fix can never silently rot back into a bug.
303
-
304
- ### Honesty
305
-
306
- `pitstop honesty` prints an honest self-assessment of what the tool cannot do, with
307
- the evidence chain behind every number. No SaaS, no telemetry, no dashboard, no
308
- fixing your code: it tells you its limits in plain words.
309
-
310
- ### Verify
311
-
312
- `pitstop verify` re-scans after a change and shows exactly how the score moved, and
313
- it checks your diff for cheat patterns. The numbers cannot be argued with.
314
-
315
- ### Trends
316
-
317
- `pitstop trends` turns your saved scan history into per-category sparklines and a
318
- score trend, so you can watch a repo actually improve over time.
319
-
320
- ### Inspect
321
-
322
- `pitstop inspect <finding-id>` opens one finding: the code snippet, the root
323
- cause, whether a repro test exists, and what OpenPitStop remembers about these
324
- files.
325
-
326
- ### Repro
327
-
328
- `pitstop repro <finding-id>` turns any finding into a regression test that FAILS
329
- while the bug is live and must PASS after the fix. Proof first, fix second.
330
-
331
- ### Report
332
-
333
- `pitstop report --html` writes one self-contained HTML report, sealed with an
334
- evidence signature, plus a README-ready score badge (`PITSTOP_BADGE.svg`).
335
-
336
- ### Share
337
-
338
- `pitstop share` renders a single share card (score, trend, top findings) you can
339
- screenshot and post, or paste into a PR.
340
-
341
- ### The live shield
342
-
343
- `pitstop watch` sits in a terminal and re-scans the moment you save a file, printing
344
- the score delta so you see problems as you type.
345
-
346
- ### Drive the agent
347
-
348
- `pitstop drive <finding-id>` hands one finding to your own agent with explicit orders:
349
- write the failing repro first, fix it, make the repro pass, then verify.
350
- OpenPitStop referees the result and never edits your code.
351
-
352
- ### The next step
353
-
354
- `pitstop next` reads the sealed artifacts and prints the single best next command plus
355
- a checklist of everything still open, so you always know where you are.
356
-
357
- ### Ask in plain English
358
-
359
- `pitstop ask "make this safe"` (or `/pitstop make this safe`) maps a plain-English
360
- request to the right command. No need to memorize flags.
361
-
362
- ### Autopilot fix
363
-
364
- `pitstop fix` chains **scan to pen --fix to verify to gate** and shows the `next` card
365
- after each hop, so a clean repo is reachable without touching the agent.
366
-
367
- ### Memory and budget
368
-
369
- `pitstop memory` is a repo scratchpad for decisions and rejected approaches that
370
- survive across sessions. `pitstop budget` shows the token and compute bill of your
371
- scans and reproves, so a fix loop stays cheap.
372
-
373
- ### The slash command
374
-
375
- `/pitstop` in Claude Code, Cursor, OpenCode, Codex and more runs the full loop
376
- immediately. `pitstop install` writes it into your tools; `pitstop prompt` shows the
377
- exact prompt it expands to. See [Install](#install).
378
-
379
- ### The GitHub Action
380
-
381
- `uses: openpitstop/action` (or `Krish-1507/OpenPitStop@main`) puts the gate on every PR
382
- as a comment and a failing check when it matters. No wiring by hand. See
383
- [docs/github-action.md](docs/github-action.md).
384
-
385
- ### The pre-commit hook
386
-
387
- `npx openpitstop install --hooks` installs the gate one step earlier: the commit
388
- cannot land until the gate passes. See
389
- [docs/caught-in-the-wild.md](docs/caught-in-the-wild.md).
390
-
391
- ### Ledger mode (payment proof)
392
-
393
- `pitstop scan --ledger` boots your app with every outbound HTTP call rerouted to a mock
394
- gateway, then replays the classic payment bugs (duplicate webhook, concurrent
395
- double-submit, delayed retry). If the mock shows more than one charge per idempotency
396
- key, that is a **proven double-charge**, not a guess.
397
-
398
- ### CI reports
399
-
400
- `pitstop ci` runs a CI-friendly scan plus verify against the base branch and writes a
401
- PR-ready markdown report, the gate as a PR comment. This is the engine behind the
402
- GitHub Action.
403
-
404
- ### Ready-check and doctor
405
-
406
- `pitstop ready-check` answers "is it worth scanning again?" and reuses the baseline when
407
- nothing changed. `pitstop doctor` explains why a category shows `skipped` and prints
408
- copy-paste install hints for the tools you are missing.
409
-
410
- ### Digest (progress story)
411
-
412
- `pitstop digest` turns your history into a plain-English progress story: how the score
413
- moved, what got fixed, what regressed, and every cheat it caught.
414
-
415
- ---
416
-
417
- ## Install
418
-
419
- One command, that's it:
420
-
421
- ```bash
422
- npx openpitstop
423
- ```
424
-
425
- No arguments needed: the CLI detects your AI tools, and asks what you want —
426
- install `/pitstop` into them, or score *this* repo (`try .`). Pick, and it does it.
427
- (In a non-interactive terminal it skips the questions and prints the one-line menu
428
- instead.)
429
-
430
- Or go straight to the files:
431
-
432
- ```bash
433
- npx openpitstop@latest install
434
- ```
435
-
436
- Run it from inside any project directory. It writes the `/pitstop` command into every
437
- supported tool below — project-level for the current repo, user-level so it works in any
438
- repo on your machine. Re-run with `-y` to refresh after updates (it's safe to re-run):
439
-
440
- ```bash
441
- npx openpitstop install -y
442
- ```
443
-
444
- Re-installing overwrites each tool's `/pitstop` command file with the latest prompt
445
- (say, a new mode or an updated loop) — your tool picks it up on its next use.
446
-
447
- Want the gate *before* the commit, not just on the PR? One extra flag installs the
448
- pre-commit hook — every commit is checked (SUSPICIOUS/CONFIRMED_CHEAT → blocked) before
449
- it can land:
450
-
451
- ```bash
452
- npx openpitstop install --hooks
453
- ```
454
-
455
- The hook runs the same `pitstop gate` (exit 0 = PASS · 1 = FAIL · 2 = CONFIRMED_CHEAT),
456
- never blocks the first commit of a repo, never jails a repo that hasn't been scanned yet
457
- (it warns instead), and can be bypassed once with `git commit --no-verify`. Remove it with
458
- `npx openpitstop install --uninstall --hooks`. To point the hook at a local build, export
459
- `PITSTOP_CLI` (e.g. `PITSTOP_CLI="node /path/to/dist/cli.js"`).
460
-
461
- Speed tip: the `try`/`scan` itself takes ~2 seconds — but the **first** `npx openpitstop …`
462
- on a machine has to download the package first (a few seconds on a fast connection, more on a
463
- slow one). For an instant first run on machines you own, install once:
464
-
465
- ```bash
466
- npm i -g openpitstop
467
- openpitstop try .
468
- ```
469
-
470
- Requires Node.js 22+ (npm will warn on older versions).
471
-
472
- ## Usage
473
-
474
- Open your repo in any supported tool and type:
475
-
476
- ```
477
- /pitstop
478
- ```
479
-
480
- Bare `/pitstop` runs the full quality loop **immediately** — scan, one confirmation pause,
481
- fix, verify, repeat. No menu, no waiting. Everything below is the power paths on top of
482
- that:
483
-
484
- | Invocation | Mode | What it does |
485
- |---|---|---|
486
- | `/pitstop` (bare) | **default full loop** | Scans right away, prints the boxed report, one confirmation pause, then the autonomous fix loop — repeat until clean. |
487
- | `/pitstop --menu` | menu | Prints the full mode list below and **waits** — handy if you forgot the flags. |
488
- | `/pitstop --scan-only` | scan-only | Runs `openpitstop scan`, prints the entire boxed report verbatim, and stops — no fixes, no commentary. |
489
- | `/pitstop --ledger` | ledger | Runs `openpitstop scan --ledger` (boots the app with every outbound HTTP call intercepted and replays duplicate-webhook / double-submit / retry traffic), then runs the loop restricted to the payment findings. |
490
- | `/pitstop --integrity-only` | integrity-only | Runs `openpitstop integrity`, prints the boxed verdict verbatim, and stops — no scanning, no fixes. |
491
- | `/pitstop --pen` | pen | Live penetration test with proof — see [The pen test](#the-pen-test). |
492
- | `/pitstop <your question>` | custom ask | Any free-form text (e.g. `check the security of this app`, `are our tests flaky?`, `did my agent cheat on the last commit?`) is scoped to exactly that ask: the agent maps it to the right command (`pen` for security, `integrity` for cheats, `scan` for health/tests…), states its interpretation in one line, confirms before fixing, and fixes only what you asked. |
493
-
494
- For reference, `/pitstop --menu` shows this list:
495
-
496
- ```
497
- OpenPitStop modes:
498
- (enter) — full autonomous loop (scan, confirm, fix, verify, repeat)
499
- --scan-only — scan and report, no fixes
500
- --ledger — payment idempotency fuzzing only
501
- --integrity-only — re-check the last commit for cheat patterns, no scanning
502
- --pen — penetration test: live attacks + proof + fixes (regression tests, patches)
503
- (your own ask) — reply with anything else, e.g. "check the security of this app"
504
- ```
505
-
506
- A flag after `/pitstop` picks a specific mode; any free-form text after it becomes a scoped
507
- custom ask; bare `/pitstop` is the full loop. If a tool ever fails to substitute arguments,
508
- `/pitstop` behaves as bare — the default full loop — rather than guessing.
509
-
510
- ## Tool support
511
-
512
- | Tool | Installed to | Status |
513
- |------|--------------|--------|
514
- | Claude Code | `.claude/commands/pitstop.md` (project + user), plus a Skill at `.claude/skills/pitstop/SKILL.md | Full support |
515
- | Cursor | `.cursor/commands/pitstop.md` (project + user) | Full support |
516
- | OpenCode | `.opencode/commands/pitstop.md` (project), `~/.config/opencode/commands/` (user) | Full support |
517
- | Kilo Code | `.kilo/commands/pitstop.md` (project), `~/.config/kilo/commands/` (user) | Full support |
518
- | Antigravity | `.agent/workflows/pitstop.md` (project + user) | Full support |
519
- | Gemini CLI | `.gemini/commands/pitstop.toml` (project + user) | Full support |
520
- | Codex CLI | `~/.codex/prompts/pitstop.md` | Full support |
521
- | FreeBuff CLI | portable `pitstop.md` in your tool's commands folder | Full support (portable) |
522
- | Grok Build CLI | portable `pitstop.md` in your tool's commands folder | Full support (portable) |
523
- | MUSE Code CLI | portable `pitstop.md` in your tool's commands folder | Full support (portable) |
524
- | Any other agent CLI | portable `pitstop.md` (drop it in the commands folder) | Full support (portable) — see below |
525
- | Codex App / VS Code extension | — (no file written) | **Not supported** — OpenAI hasn't shipped custom slash commands there; install prints a manual-copy note instead |
526
- | GitHub Action (PRs) | `uses: Krish-1507/OpenPitStop@main` | **Full support** — gate verdict as a PR comment + failing check; see [docs/github-action.md](docs/github-action.md) |
527
- | git pre-commit hook | `.git/hooks/pre-commit` (installed with `--hooks`) | **Full support** — the gate blocks the commit before it lands |
528
-
529
- **Works with any agent CLI.** OpenPitStop's `/pitstop` is a portable command file:
530
- run `pitstop prompt` to print the exact instruction text, then paste it as a custom
531
- slash command in any coding CLI that supports them (FreeBuff, Grok Build, MUSE Code and
532
- others included above). The CLIs listed by name also get a dedicated path written
533
- automatically by `pitstop install` when their commands-folder convention is known. Tell
534
- us your CLI and we'll add it to the auto-install list. Legacy/alternate locations are
535
- also written where tool docs are inconsistent across versions (see
536
- `src/installer/targets.ts`). Existing files are never overwritten unless you pass
537
- `-y`/`--force`; `npx openpitstop install --uninstall` removes everything.
538
-
539
- ## What OpenPitStop actually does
540
-
541
- ### The loop at a glance
542
-
543
- <p align="center">
544
- <img src="docs/media/pitstop-loop.png" alt="OpenPitStop's autonomous loop: scan → report → you confirm → repro (must FAIL first) → fix → verify → repeat until a fresh scan shows zero clusters" width="900">
545
- </p>
546
-
547
- OpenPitStop never touches your code. It checks, scores, and referees — your AI agent does the
548
- editing, knowing it's being watched.
549
-
550
- ### The scan
551
-
552
- `pitstop scan` runs a bunch of checks on your repo: circular imports, known security
553
- issues, duplicated code (`jscpd`), test results
554
- (jest/vitest/pytest plus native suites for Go, Rust, Flutter/Dart, .NET and Java
555
- (Maven/Gradle) — pass/fail, duration, coverage), build speed, accessibility, flaky-test
556
- and race-condition heuristics, and developer-experience checks (unused exports, duplicate
557
- functions).
558
-
559
- Security is two layers:
560
-
561
- 1. **Dependency audits** — `npm audit`, plus `pip-audit`/`osv-scanner` for Python and
562
- other stacks, and `gitleaks` for committed secrets. A failed audit is reported as
563
- `skipped` with a repair hint — deps that were never scanned are never reported as clean.
564
- 2. **The static vulnerability pass** (fully offline, every language, no tooling needed) —
565
- the classic classes plus the full posture: **SQL injection** (concatenated/
566
- interpolated queries, ORM raw builders, `$where`, Python f-strings), **authentication**
567
- (cleartext password compares, missing hashing, `Math.random` tokens, inline JWT
568
- secrets), **authorization** (unprotected data routes, admin routes without role
569
- checks), **input validation** (unrestricted uploads, unvalidated money fields,
570
- `eval`, XSS sinks), **secret management** (known credential formats, inline secret
571
- literals, committed `.env` files) — plus command injection, path traversal, SSRF,
572
- **rate limiting** (missing limiters on state-changing routes, limits set so high they
573
- are decorations), **database lockdown** (privileged accounts in committed connection
574
- strings, `GRANT ALL`/`SUPERUSER`, TLS-free connections, missing row-level security),
575
- **data exposure** (credentials/PII in API responses, full DB rows shipped to the
576
- client, PII in logs), **hidden vulnerabilities** (disabled TLS verification, `alg:
577
- none` JWTs, security TODOs, lint/type bypasses, tokens in `localStorage`, committed
578
- minified bundles, backup files), CORS+credentials, missing security headers, CSRF
579
- exposure, stack leaks and sensitive logging.
580
-
581
- Every static finding is labeled `[indicated]` and ships with its exact **fix**; `scan`
582
- and `try` print the complete identify-and-solve list under the score box, so the report
583
- is a worklist, not a scare. The full matrix — every detection and every fix — is in
584
- [docs/security.md](docs/security.md).
585
-
586
- **Optional deeper SAST (Semgrep).** OpenPitStop's built-in static pass needs no extra
587
- tooling. For a second, cross-language engine you can bolt on [Semgrep](https://semgrep.dev)
588
- — it is **off by default** and only runs when you opt in. Install it (`pip install
589
- semgrep`) and set one variable:
590
-
591
- ```bash
592
- export PITSTOP_SEMGREP_CONFIG=auto # the free Semgrep Registry rules
593
- pitstop scan # Semgrep is picked up automatically
594
- ```
595
-
596
- Point it at your own rules any time: `PITSTOP_SEMGREP_CONFIG="p/security-audit p/owasp-top-ten ./my-rules"`.
597
- With nothing set, no Semgrep process ever runs — no surprise network calls, no slow scans.
598
-
599
- Each check either contributes a real number, or prints `skipped` with a one-line hint on
600
- how to install the tool it needs — it never makes up a number. Everything adds up to one
601
- box that always opens with the **OpenPitStop Score**: a single 0–100 health number (with an
602
- A–F grade) across the categories that actually ran.
603
-
604
- Scans are fast by design: the checks run **in parallel**, flaky detection runs the suite
605
- **twice** by default (`--reliability-runs <n>` to tune; `1` disables it), and `npm audit`
606
- (plus `osv-scanner`) results are cached for 24 hours (keyed on the lockfile hash) so
607
- repeated scans inside one fix loop never hit the registry again.
608
-
609
- ### The test pyramid
610
-
611
- `pitstop test [path] [--unit] [--integration] [--e2e]` runs your **unit, integration and
612
- e2e** layers the way a senior dev would — it discovers each layer (`test`/`test:unit`,
613
- `test:integration`/`test:it`, `test:e2e`/`e2e` npm scripts first, then vitest/jest/
614
- pytest/playwright/cypress by config), executes them, and reports per-layer pass/fail
615
- counts with the **names of the failing tests**, so the fix list is actionable. Layers it
616
- cannot find are reported as `skipped — no suite discovered`, never invented; the command
617
- exits 1 the moment any layer fails, so CI can trust it. (Add `"test:e2e": "playwright
618
- test"` to a repo and it is picked up automatically on the next run.)
619
-
620
- ## Every command
621
-
622
- All 39 commands, grouped by job. Run them from inside a repo as `pitstop …` (CLI) or
623
- `npx openpitstop …` (one-off); `/pitstop` in a tool drives the loop, the rest are
624
- one-shot.
625
-
626
- **Measure — the numbers**
627
-
628
- | Command | What it does |
629
- |---|---|
630
- | `pitstop scan [path] [--json] [--reuse] [--ledger]` | The big one. Runs every check in parallel and prints one box with a single **OpenPitStop Score** (0–100, A–F). `--json` for scripts and pipelines; `--reuse` returns the saved baseline when nothing changed; `--ledger` also fuzzes payment idempotency. |
631
- | `pitstop verify` | Re-scans after a change and shows exactly how the score moved — the numbers can't be argued with. Also checks your diff for agent-cheat patterns. |
632
- | `pitstop try [path]` | Get a score on **any** repo in ~2 seconds of scanning — no install, no config, no setup (the first `npx` run on a machine downloads the package once). Saves a sealed baseline so verify and gate can build on it later. |
633
- | `pitstop ready-check [path]` | Quick "is it worth scanning again?" — nothing changed → exit 0 and reuse the baseline; something changed → exit 1. |
634
- | `pitstop watch [path] [--interval ms]` | The live shield. Sits in a terminal and re-checks the moment you save a file, printing how the score moved. |
635
- | `pitstop trends` | Turns your saved scan history into per-category sparklines and a score trend — watch a repo actually improve. |
636
- | `pitstop budget [path]` | The token bill: how many scans/verifies/pens/repros you've run and the compute-seconds, plus advice on what to reuse in a fix loop. |
637
- | `pitstop test [path] [--unit] [--integration] [--e2e]` | The test pyramid: discovers and runs the **unit, integration and e2e** layers, reports per-layer pass/fail with the failing test names. Any failing layer → exit 1. See [The test pyramid](#the-test-pyramid). |
638
-
639
- **The fix loop — what the agent is told to do**
640
-
641
- | Command | What it does |
642
- |---|---|
643
- | `pitstop drive <finding-id> [path]` | Hands one finding to *your own agent* (`PITSTOP_AGENT` or `--agent '…{prompt}'`) with explicit orders: write a failing repro first, fix it, make the repro pass, then verify. OpenPitStop referees the whole thing and never edits your code. |
644
- | `pitstop next [path]` | Reads the sealed artifacts and prints the single best next command plus a checklist of everything still open — repo-aware (detects language, framework, tests, CI). |
645
- | `pitstop fix [path]` | Autopilot without an agent: chains **scan → pen --fix → verify → gate**, showing the `next` card after each hop. |
646
- | `pitstop ask "<your ask>"` | Maps a plain-English request ("check the security of this app", "are our tests flaky?") to the right command, states its interpretation, and confirms before fixing. |
647
- | `pitstop memory add/list/relevant` | A scratchpad inside the repo: record a decision (`add`), see them newest-first (`list`), or pull up anything related to a file (`relevant`) — so past fixes and rejected approaches survive across sessions. |
648
- | `pitstop inspect <finding-id>` | Opens up one finding: the code snippet, the root cause, whether a repro test exists, and what OpenPitStop remembers about these files. |
649
-
650
- **Integrity & anti-cheat — the referee**
651
-
652
- | Command | What it does |
653
- |---|---|
654
- | `pitstop gate [--score 60]` | A commit gate for CI, pre-commit hooks or PRs: score threshold + regression risk + diff integrity + evidence signature + the newest baseline-verify / state-verify / verifier-check reports. **Exit 0 = PASS · 1 = FAIL · 2 = CONFIRMED_CHEAT.** |
655
- | `pitstop integrity [path]` | Checks the latest commit or working tree for cheat patterns *without* a full scan: deleted or neutered tests, swallowed errors, suppression comments, hardcoded-to-pass values, mocked modules, forced exits. **Exit 0 = CLEAN · 1 = SUSPICIOUS · 2 = CONFIRMED_CHEAT.** |
656
- | `pitstop ci [path]` | CI-friendly scan + verify against the base branch → a PR-ready markdown report — the gate as a PR comment. It only reports; fixes stay local via `/pitstop`. Wired into the [GitHub Action](docs/github-action.md), which comments the gate on every PR and fails the check when it fails. |
657
-
658
- **Deep verification — can the referee say NO?**
659
-
660
- | Command | What it does |
661
- |---|---|
662
- | `pitstop baseline-verify --baseline <ref> --command <cmd> …` | Proves a fix is real: runs the SAME verification on a known-baseline commit (must FAIL) and the candidate (must PASS), in isolated git worktrees, with sealed tamper-evident evidence and a verification-identity hash. `VERIFIED` only when both hold and nothing changed; otherwise `FAILED` / `UNPROVEN` / `INTEGRITY_FAILURE`. Exit 0/1/2/3. See [docs/baseline-verify.md](docs/baseline-verify.md). |
663
- | `pitstop state-verify --claim modified:src/auth.ts …` | Independent external state check: verifies the agent's structured claims against the actual filesystem + git (existence, content hashes, line counts, porcelain status, HEAD). Catches "HTTP 200 but nothing changed", empty writes, reverts, wrong-file changes, whitespace-only edits. `STATE_VERIFIED` / `STATE_MISMATCH` / `UNPROVEN` / `INTEGRITY_FAILURE`. Exit 0/1/2/3. See [docs/state-verify.md](docs/state-verify.md). |
664
- | `pitstop verifier-check --command <cmd> --mutate …` | Verifier self-test: runs the verification on a known-good state (must PASS) and a controlled known-bad state (must FAIL) in temp worktrees. `VERIFIER_VALID` = falsifiable; `VERIFIER_WEAK` = the seeded fault sailed through; `VERIFIER_BROKEN` = fails a correct state. Never mutates your working tree. See [docs/verifier-check.md](docs/verifier-check.md). |
665
- | `pitstop holdout-verify --suite <dir-or-id> [--baseline <ref>]` | Final hidden exam against verifier overfitting: a holdout suite defined OUTSIDE the repo runs once in a fresh isolated worktree of the candidate commit — the agent never saw it and cannot modify it (files hashed before/after); output is redacted to ids + verdicts. With `--baseline` the suite must FAIL there and PASS on the candidate. `HOLDOUT_PASS` / `HOLDOUT_FAIL` / `HOLDOUT_UNPROVEN` / `HOLDOUT_INTEGRITY_FAILURE`. See [docs/holdout-verify.md](docs/holdout-verify.md). |
666
- | `pitstop acceptance-verify --contract <dir\|file\|id> [--baseline <ref>]` | Requirement verification: a structured acceptance contract (deterministic `command`/`http`/`fileExists`/`fileContains` criteria — never an LLM judge) is the source of truth for "did the agent satisfy the original requirement?". Boots the app when the contract declares a start command; in-repo contracts are hash-pinned so the agent cannot redefine success without `--authorize`; `--baseline` exposes contracts that pass on both sides. `SATISFIED` / `NOT_SATISFIED` / `UNPROVEN` / `INTEGRITY_FAILURE`. See [docs/acceptance-verify.md](docs/acceptance-verify.md). |
667
- | `pitstop regression-check --command <cmd> --baseline <ref>` | Per-check regression comparison: previously **verified passing** checks that now fail are `REGRESSION` (hard-blocks the gate); already-broken stays `UNCHANGED`; fixes are `FIXED`; new checks are `NEW_PASS`/`NEW_FAILURE`; flaky (`--runs >1`) and vanished checks are `UNPROVEN` rather than guessed. Per-test names parsed from TAP/spec/jest/pytest/go output, suite-level fallback. See [docs/regression-check.md](docs/regression-check.md). |
668
- | `pitstop explain [--verbose]` | The unified evidence chain: aggregates every sealed verification document, re-verifies each seal, and derives VERIFIED / BLOCKED / UNPROVEN with per-item evidence references, digests and reasons. Never-run components are NOT_CONFIGURED — never rendered as passes. See [docs/explain.md](docs/explain.md). |
669
- | `pitstop understand [path]` | Repo awareness: builds the sealed understanding artifact — languages, frameworks, verification commands (test/typecheck/lint/build), test layers, CI, module map, entry points, CODEOWNERS ownership, architecture config. Stage 1 of the pipeline. See [docs/repo-discipline.md](docs/repo-discipline.md). |
670
- | `pitstop plan --goal … --path … --verify-command …` | Plan before patching: a sealed change contract (goal, steps, `expectedPaths` the change may touch, verification commands). `--show` renders the latest plan with live scope status. See [docs/repo-discipline.md](docs/repo-discipline.md). |
671
- | `pitstop architecture-check [--against-plan] [--approved]` | Does the change FIT THE SYSTEM? Declared import boundaries, protected paths (auth/deploy/CI need explicit `--approved`), forbidden paths, CODEOWNERS routing, shortcut detectors on the diff, and scope-creep detection against the plan. `CONFORMS` / `APPROVAL_REQUIRED` / `VIOLATIONS` / `INTEGRITY_FAILURE`. See [docs/repo-discipline.md](docs/repo-discipline.md). |
672
- | `pitstop verify-stack [--only …]` | The full verification stack — unit/integration/e2e + typecheck + lint + build, whatever the repo has — with a deterministic failure DIAGNOSIS per layer (type-error TSxxxx, missing-dependency, assertion-failure, lint rule, environment, timeout) so fixes are targeted, not random edits. See [docs/repo-discipline.md](docs/repo-discipline.md). |
673
- | `pitstop flow [--baseline …] [--contract …] [--suite …]` | The whole pipeline in one command: understand → contract? → plan-scope → verify-stack → architecture → baseline? → regression? → holdout? → GATE. Unconfigured stages are SKIPPED honestly. See [docs/repo-discipline.md](docs/repo-discipline.md). |
674
-
675
- **Penetration test — attack your own app**
676
-
677
- | Command | What it does |
678
- |---|---|
679
- | `pitstop pen [path] [--fix] [--html] [--json]` | A real pen test of your own app. Static heuristics find candidates (secrets, routes, injection/SSRF/XSS), then it **boots the app in a sandbox** and attacks it live, recording every outbound HTTP call and spawned process. Findings are labeled `PROVEN` only when the sandbox saw real evidence; everything else is honestly `indicated`/`unproven`. Nothing reaches the real network; raw sockets are blocked. `--fix` writes failing-then-passing repro tests + `git apply`-able patches. Exit 0 = clean · 1 = high/critical · 2 = aborted. |
680
- | `pitstop inspect <pen-id>` | Deep-dives a pen finding: exactly what attack was fired, what the app responded with, the sandbox evidence lines, the fix, the repro. |
681
- | `pitstop repro <pen-id>` | Turns a pen finding into a regression test that boots the app — fails now, must pass after the fix. |
682
-
683
- **Proof & reports — what you show people**
684
-
685
- | Command | What it does |
686
- |---|---|
687
- | `pitstop report --html` | One self-contained `PITSTOP_REPORT.html` (inline SVG trends, integrity timeline, zero external assets). Also writes `PITSTOP_BADGE.svg` — a README-ready shield: `![OpenPitStop score](PITSTOP_BADGE.svg)`. |
688
- | `pitstop share` | Renders a 1200×630 share card (`PITSTOP_CARD.html`) — score, trend, integrity/evidence chips, top findings. Screenshot it and post it. |
689
- | `pitstop digest [--days N] [--md]` | The progress story: how the score moved, what got fixed and what regressed, gate results, cheat catches, flakies, open findings. |
690
- | `pitstop honesty [--html]` | The proof that the numbers are real: evidence chain + integrity history + verify deltas + committed repro tests → one verdict, or a shareable HTML certificate. |
691
-
692
- **Setup & transparency**
693
-
694
- | Command | What it does |
695
- |---|---|
696
- | `pitstop install` / `install --uninstall` | Writes `/pitstop` into every supported tool (project + user level). `--uninstall` removes it all. `--hooks` also installs (or with `--uninstall`, removes) the git pre-commit gate. |
697
- | `pitstop` (no args) | The guided first-run: detects your AI tools and git repo, then offers to install or score this repo (`try .`). Non-TTY prints the one-line menu instead. |
698
- | `pitstop demo` | Full loop rehearsal in a throwaway temp dir — walk the slash-command loop by hand against a seeded-broken repo, ending with a `PITSTOP_REPORT.md`. |
699
- | `pitstop doctor` | Explains why categories show `skipped`: checks your toolchain (Node, git, jscpd, gitleaks, pa11y) and prints copy-paste install hints. Semgrep is optional and opt-in, so doctor won't flag its absence. |
700
- | `pitstop prompt [--args …]` | Prints the exact prompt your AI tool expands `/pitstop` into, with your arguments filled in — full transparency into what the agent was told. |
701
-
702
- ### The score & badge
703
-
704
- Skipped categories are excluded and the weights re-adjusted, so a missing `jscpd` never
705
- silently drags the number down. The verify Δ compares against the last scan with the *exact
706
- same categories measured* — a category skipped on both sides can't move the score. The
707
- score only moves when your code does.
708
-
709
- ### Tamper-evident evidence
710
-
711
- Every scan, verify and integrity document OpenPitStop writes gets a `sha256` fingerprint of
712
- its own contents (`pitstop-sha256-canonical-v1`, deterministic key-sorted JSON). Edit the
713
- JSON after the fact — inflate a score, delete a finding — and the next
714
- `pitstop verify`/`pitstop gate` recomputes the fingerprint, sees the mismatch, and
715
- reports the chain as broken. OpenPitStop can't be tricked into endorsing a baseline it didn't
716
- write; the `gate` exit code treats a broken chain as a hard fail.
717
-
718
- ### Prompt transparency
719
-
720
- Some AI tools show you the expanded slash-command prompt in their UI, some don't. OpenPitStop
721
- keeps your chat clean either way: the `/pitstop` agent acknowledges with a single short line
722
- and gets straight to work — the full instruction set stays out of your window. And
723
- `pitstop prompt` lets you preview the raw prompt before anyone types anything.
724
-
725
- ### Root-cause correlation
726
-
727
- Findings that touch the same files get grouped into one root cause, so the box shows
728
- `1 root cause → 2 symptoms` instead of a flat list. Every cluster gets a stable id (e.g.
729
- `security-19c390c6`) that the repro step can reference.
730
-
731
- ### Confirm, then loop
732
-
733
- The agent prints the boxed summary, then **waits for your one-time OK**. After that it
734
- works through each cluster on a `pitstop/*` branch: capture the bug as a failing test
735
- first (`pitstop repro <id>`), make the smallest fix, pass the same repro test, run
736
- `pitstop verify`, and commit. It re-scans after every fix and stops when a fresh scan
737
- shows zero clusters (hard limits: 10 fix rounds or 45 minutes), ending with a
738
- `PITSTOP_REPORT.md`.
739
-
740
- ### Ledger mode (opt-in)
741
-
742
- `pitstop scan --ledger` boots your app with **every outbound HTTP call rerouted to a mock
743
- gateway**, then replays the three classic payment bugs: duplicate webhook, concurrent
744
- double-submit, delayed retry. If the mock gateway's own receipt log shows more than one
745
- charge per idempotency key, that's a **proven double-charge** — not a guess. The shipped
746
- If the sandbox can't intercept some traffic,
747
- the run aborts (`exit 77`); nothing ever reaches a real gateway.
748
-
749
- **Which stacks are covered?** Node/JS apps run under the nock preload, which intercepts
750
- every outbound call in-process. Go, Python, Rust and .NET apps run under a recording
751
- `HTTP_PROXY` sandbox that answers the known payment-gateway hosts with mocked receipts
752
- and 502s everything else. Java and Dart are refused (their HTTP clients don't honor
753
- `HTTP_PROXY`, so interception could not be guaranteed). HTTPS stays blocked (502): without
754
- a trusted CA the proxy cannot terminate a CONNECT tunnel, so an HTTPS double-charge is
755
- reported as *indicated*, never *proven*. Native binaries and raw sockets bypass the proxy
756
- and are not observed. Set `PITSTOP_START` to override start-command guessing for
757
- non-Node repos.
758
-
759
- ### Integrity gate
760
-
761
- Every `pitstop verify` also diffs your change against HEAD and checks for the classic
762
- agent-cheat moves: deleted or loosened tests, tests focused to hide failures
763
- (`fit`/`test.only`), swallowed exceptions, suppression comments, hardcoded-to-pass values,
764
- a mocked module-under-test, a forced `exit(0)` in app code, or an assertion's expected
765
- value edited to match the buggy output. A caught cheat looks like this: change
766
- `assert.equal(round2(8.075), 8.08)` to expect `8.07` with nothing else in the diff →
767
- `CONFIRMED_CHEAT`, the change is blocked, and a human reviews it (verified against
768
- `fixtures/assertion-literal-tamper/`). An honest app-side fix sails through `CLEAN`.
769
-
770
- ## Architecture
771
-
772
- OpenPitStop is two pieces that never mix: a **CLI that measures**, and **your host agent that
773
- reasons and edits**. The CLI produces the scan/verify numbers and the gate verdicts; the
774
- model in whichever tool you're using reads them, decides what to change, and does the
775
- editing through the `/pitstop` prompt template. This is deliberately *not* one monolithic
776
- agent — the numbers can't be talked into looking better, and the agent can't silently
777
- cheat its own referee. That separation is the product.
778
-
779
- ## Known limitations
780
-
781
- - **Windows** is a first-class, CI-verified platform (build + smoke on `ubuntu-latest`
782
- and `windows-latest` every push). `watch`, `pen`, `pen --fix` and `scan --ledger` were
783
- each run end-to-end on a real Windows host against sample apps, with a live watch
784
- delta, PROVEN ledger double-charges (sealed evidence) and honest pen verdicts
785
- (including the honest "0 patches" case) all verified. The only open caveat is breadth,
786
- not correctness: not every exotic repo shape has been hand-exercised on Windows yet.
787
- - **Codex App / VS Code extension** isn't supported and won't be until OpenAI ships custom
788
- slash commands; use Codex CLI for `/pitstop`.
789
- - **Graceful degradation:** duplication (`jscpd`), secret scanning (`gitleaks`),
790
- dependency CVEs (`pip-audit`, `osv-scanner`), and accessibility runtime checks
791
- (`pa11y`/`axe`) run only when that tool is installed locally. The scan reports
792
- `skipped` for those categories and works fine without them. **Semgrep is an opt-in
793
- deeper engine** (see [Security fixes](#security-fixes)): it only runs when you set
794
- `PITSTOP_SEMGREP_CONFIG`, so it never runs just because the binary happens to be on
795
- PATH — no surprise network calls or slow scans.
796
- - Requires **Node.js 22+** (the CLI depends on execa 10, which uses ES2024 `Set.union`).
797
- - **Multi-stack honesty:** test runs (JS, Python, Go, Rust, Flutter, .NET, Java via Maven
798
- or Gradle), dependency CVEs, and the `pen`/`ledger` sandboxes are real for Node/JS and
799
- best-effort elsewhere. Go/Rust/Python/.NET apps run under the `HTTP_PROXY` recording
800
- sandbox (see Ledger mode); Java and Dart are refused for ledger, and proxy-mode results
801
- are labelled `indicated`, never `proven`, when the proxy cannot observe the traffic. The
802
- native test runners parse each toolchain's real output (`go test -json`, `cargo test
803
- --format json`, `flutter test --machine`, dotnet/maven/gradle summaries) and report
804
- `skipped` when a toolchain isn't on PATH.
805
- - **Pen-test honesty:** `pitstop pen` reports each finding with a **runtime-proof
806
- verdict**: **proven** (the live dynamic attack confirmed it under the sandbox),
807
- **indicated** (static rule fired but the dynamic phase couldn't confirm), **unproven**
808
- (the dynamic phase ran and found no evidence for that rule), or **not-tested** (dynamic
809
- phase aborted). Proven findings are real; everything else is a hypothesis until you
810
- replay the attack yourself. The sandbox records outbound connections and spawned
811
- processes instead of blocking them (so real bytes never leave your machine for canaries,
812
- but a compromised app could still run commands locally); raw socket APIs are blocked
813
- outright. `pen --fix` writes deterministic patches **only** for findings fixable by pure
814
- insertion (e.g. missing `helmet()`, `x-powered-by` leaks) — anything else gets a failing
815
- repro test and fix guidance, which is your contract for the fix.
816
- - **`drive` verdicts** for runtime pen findings come from the repro test (FAIL first, PASS
817
- after the fix), not from the static score — the static gate has nothing to say about a
818
- runtime-only finding.
819
- - **Baseline-aware verification honesty:** without an explicit `expectedFailure` predicate,
820
- a non-zero baseline exit cannot prove the failure was the *intended* bug rather than a
821
- broken environment — such results are downgraded to `UNPROVEN`, never `VERIFIED`. The
822
- verification identity covers only the files you declare (`--test-file`/`--config`).
823
- See [docs/baseline-verify.md](docs/baseline-verify.md).
824
- - **State verification is not semantic verification.** A content hash proves content
825
- changed, not that the change is correct or complete; untracked files without a snapshot
826
- have no before-state and are reported `UNPROVEN` rather than guessed. Files over 8 MB are
827
- recorded but not hashed. See [docs/state-verify.md](docs/state-verify.md).
828
- - **Verifier health is per fault class.** Passing one known-bad case proves the verifier can
829
- fail, not that it covers every regression; choosing a meaningful known-bad state is the
830
- caller's responsibility, and the evidence records exactly what was seeded.
831
- See [docs/verifier-check.md](docs/verifier-check.md).
832
- - **A holdout is a sample, not a guarantee.** It proves the candidate satisfies checks it
833
- could not see or modify — not correctness in general. Suites must live outside the
834
- agent's modifiable workspace (a hidden file inside the repo is not a holdout), and a
835
- suite that cannot fail on a known-bad baseline is reported `HOLDOUT_UNPROVEN` rather than
836
- trusted. See [docs/holdout-verify.md](docs/holdout-verify.md).
837
- - **Acceptance is only as strong as its contract.** `acceptance-verify` proves the
838
- candidate satisfies the contract's *observable* criteria — not that the contract captures
839
- the whole requirement, and not unobservable qualities (UX, load, security beyond the
840
- asserted properties). It is deterministic by design, never an LLM judge; a contract that
841
- passes on the baseline is reported `UNPROVEN` rather than trusted.
842
- See [docs/acceptance-verify.md](docs/acceptance-verify.md).
843
- - **Regression detection vs flaky tests:** with the default single run, a flaky candidate
844
- check is indistinguishable from a regression and IS reported as one; `--runs <n>` detects
845
- candidate-side inconsistency (`UNPROVEN`), but the baseline is a single run, so a flaky
846
- baseline check can still mislabel a fix or hide a regression. Per-test granularity depends
847
- on the runner's output; unparseable output degrades to one suite-level check.
848
- See [docs/regression-check.md](docs/regression-check.md).
849
-
850
- ## Privacy
851
-
852
- **Zero telemetry, zero SaaS, zero accounts — nothing leaves your machine unless you ask
853
- it to.** OpenPitStop is a local CLI with no server and no phone-home: scans, gates and
854
- pen tests run entirely on your machine, and the only network calls in the entire codebase
855
- are the dependency audits you can see and opt out of (plus the `npx` download you
856
- initiated). The full, auditable list — every connection, every cache, every file stored —
857
- is in [PRIVACY.md](PRIVACY.md). The honesty brand is the product; that statement is the
858
- receipt.
859
-
860
- ## Contributing
861
-
862
- OpenPitStop is built to be extended — adding a whole new analyzer is a small, well-scoped
863
- change. See [CONTRIBUTING.md](CONTRIBUTING.md) for the analyzer interface, conventions, and
864
- how to open a PR. For the launch notes and the "why", read [LAUNCH.md](LAUNCH.md).
865
-
866
- ## License
867
-
868
- [MIT](LICENSE)
869
-
870
- ## Support the project
871
-
872
- If OpenPitStop saved you from shipping a bug your agent swore was fixed, the best
873
- support is a star and a real repo:
874
-
875
- - **Star** it: https://github.com/Krish-1507/OpenPitStop
876
- - **Report issues or ideas**: https://github.com/Krish-1507/OpenPitStop/issues
877
- - **Contribute** an analyzer (small, well-scoped): see [CONTRIBUTING.md](CONTRIBUTING.md)
878
-
879
- No donation, no paywall, no telemetry.
880
-
881
- ---
882
-
883
- <p align="center">
884
- <img src="docs/media/pitstop-icon.png" alt="OpenPitStop icon" width="64">
885
- </p>
886
-
887
- <p align="center">Built by <b>Krish J</b> — if it can referee itself, it can referee your agent.</p>
1
+ <p align="center">
2
+ <img src="docs/media/pitstop-logo.png" alt="OpenPitStop" width="380">
3
+ </p>
4
+
5
+ # OpenPitStop
6
+
7
+ **OpenPitStop gives your AI coding CLI a repo-aware engineering workflow: understand, plan, check architecture, test security, and verify.**
8
+ **Use it to turn fast code generation into reviewable changes, with regression evidence, explicit boundaries, and local checks that use zero model credits.**
9
+
10
+ [![npm version](https://img.shields.io/npm/v/openpitstop)](https://www.npmjs.com/package/openpitstop)
11
+ [![CI](https://github.com/Krish-1507/OpenPitStop/actions/workflows/ci.yml/badge.svg)](https://github.com/Krish-1507/OpenPitStop/actions/workflows/ci.yml)
12
+ [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
13
+
14
+ Working code can still be wrong for the system. A change can pass a test while bypassing
15
+ authentication, crossing a module boundary, weakening an assertion, or breaking a real user
16
+ flow. PitStop makes those questions part of a repeatable workflow, with inspectable evidence
17
+ and explicit `UNPROVEN` results when the checks cannot establish an answer.
18
+
19
+ **Local CLI · Bring your own coding agent · No PitStop account · No telemetry · MIT**
20
+
21
+ ## Start with one command:
22
+
23
+ Requires **Node.js 22+** and npm. Git is needed for change comparison and isolated worktrees.
24
+
25
+ ```bash
26
+ npm install -g openpitstop
27
+ pitstop ask "understand this repo"
28
+ ```
29
+
30
+ Or get a quick first scan without a global install:
31
+
32
+ ```bash
33
+ npx openpitstop try .
34
+ ```
35
+
36
+ `try` runs the lighter checks and saves a baseline. It does not run the full test/build
37
+ stack; duration depends on the repository and installed tools.
38
+
39
+ Install the slash command into your coding CLI, then speak naturally:
40
+
41
+ ```bash
42
+ pitstop install
43
+ ```
44
+
45
+ ```text
46
+ /pitstop find vulnerabilities in this repo
47
+ /pitstop find and fix vulnerabilities in this repo
48
+ /pitstop review my changes
49
+ ```
50
+
51
+ PitStop executes the matching workflow internally. The host CLI handles the slash
52
+ command; the terminal equivalent is `pitstop ask "find vulnerabilities in this repo"`.
53
+ Static vulnerability searches need no model call and do not boot the app.
54
+
55
+ Then say what you need:
56
+
57
+ | Your request | PitStop runs internally |
58
+ |---|---|
59
+ | `pitstop ask "check the security of this app"` | Static security review; no app boot |
60
+ | `pitstop ask "check tests types lint and build"` | Discovered verification layers, with failure diagnosis |
61
+ | `pitstop ask "review my changes"` | Repo understanding → architecture → verification stack → evidence gate |
62
+ | `pitstop ask "did my agent cheat"` | Test and verification integrity checks |
63
+ | `pitstop ask "can I ship this"` | Strict gate; `UNPROVEN` returns a failure exit code |
64
+ | `pitstop ask "show my budget"` | Recorded scan/test/build activity |
65
+ | `pitstop ask "what should I do next"` | Next action and remaining work |
66
+
67
+ The router runs locally and makes **no LLM call**. It recognizes common requests; it is
68
+ not a general language model. Unknown requests and unsupported constraints run nothing.
69
+ Use `--dry-run` to preview, `--repo ./app` to select a repository, or `--json` for a
70
+ machine-readable routing preview. The executed command's exit code is preserved.
71
+
72
+ Actions that write source, launch a paid agent, install integrations, or actively attack
73
+ an app are previewed until you add `--execute`:
74
+
75
+ ```bash
76
+ pitstop ask "make this safe" --dry-run
77
+ pitstop ask "make this safe" --execute
78
+ pitstop ask "attack my app" --execute
79
+ ```
80
+
81
+ `make this safe` runs the supported deterministic fix workflow. It does not promise to
82
+ repair every issue. `attack my app` runs the live security checks without silently applying
83
+ fixes. Test and build checks execute your repository's scripts; use trusted repositories.
84
+
85
+ ## The engineering loop
86
+
87
+ ```mermaid
88
+ flowchart LR
89
+ A[Understand the repo] --> B[Plan and boundaries]
90
+ B --> C[Make a scoped change]
91
+ C --> D[Tests, types, lint, build]
92
+ D --> E[Architecture and security]
93
+ E --> F[Regression and acceptance evidence]
94
+ F --> G[Gate and next action]
95
+ G -->|More work needed| B
96
+ ```
97
+
98
+ | Engineering question | Evidence PitStop provides |
99
+ |---|---|
100
+ | What system am I changing? | Languages, frameworks, entry points, module map, verification commands, CI and CODEOWNERS |
101
+ | What may this change touch? | A sealed plan, declared import boundaries, protected/forbidden paths and scope checks |
102
+ | Why did verification fail? | Per-layer results and diagnoses: assertion, type error, missing dependency, lint, environment or timeout |
103
+ | Does the change fit this repo? | Declared architecture rules, ownership routing and suspicious shortcuts in the diff |
104
+ | Did the fix address the actual bug? | The same verification against a known-bad baseline and the candidate |
105
+ | Did it meet the requirement? | An acceptance contract with command, HTTP and file criteria |
106
+ | Did it break existing behavior? | Check-level baseline/candidate comparison and optional repeated runs |
107
+ | Why trust the verdict? | Inspectable artifacts, content digests, evidence references and explicit missing checks |
108
+
109
+ Repo awareness is structural analysis and declared rules. Semantic design judgment still
110
+ belongs to the developer and host agent. A plan or gate checks behavior within its scope;
111
+ it does not prevent a separately running agent from editing files.
112
+
113
+ <p align="center">
114
+ <img src="docs/media/flow.gif" alt="OpenPitStop verification pipeline" width="880">
115
+ </p>
116
+
117
+ ## Feature tour
118
+
119
+ ### Understand, plan and protect the architecture
120
+
121
+ `understand` discovers the repository's structure and tools. `plan` records the goal,
122
+ steps, expected paths and verification commands before source patching.
123
+ `architecture-check --against-plan` checks scope, declared import boundaries, protected
124
+ paths, forbidden paths, CODEOWNERS routing and shortcuts in the diff.
125
+
126
+ Configure boundaries in `openpitstop.architecture.json` or `.pitstop/architecture.json`.
127
+ Protected-path approval is recorded through an explicit option; it is not an authenticated
128
+ human identity. [Configuration and examples](docs/repo-discipline.md).
129
+
130
+ ### Scan and score
131
+
132
+ `scan` combines dependency graphs, security, duplication, tests, performance, accessibility,
133
+ reliability and developer-experience checks. Findings include locations and fix guidance;
134
+ related findings are grouped. Optional missing tools are reported as skipped.
135
+
136
+ The **0–100 score** summarizes the categories that ran. It is not a probability of
137
+ correctness or security: always read the coverage and skipped categories alongside it.
138
+
139
+ <p align="center">
140
+ <img src="docs/media/scan.gif" alt="Repository score and findings" width="880">
141
+ </p>
142
+
143
+ ### Test pyramid and failure diagnosis
144
+
145
+ `test` discovers unit, integration and E2E layers separately and reports failing test names
146
+ when the runner exposes them. `verify-stack` adds typecheck, lint and build with targeted
147
+ failure diagnosis. Missing layers are visible, not fabricated passes. A failing layer
148
+ returns a nonzero exit. [Verification stack](docs/repo-discipline.md).
149
+
150
+ ### Security testing and fixes
151
+
152
+ `pen --static` reviews supported patterns without booting the app. `pen` also starts the
153
+ local application and sends supported attack probes, recording requests, responses and
154
+ intercepted outbound activity. Findings distinguish static indications from runtime evidence.
155
+ The [security matrix](docs/security.md) describes the built-in checks; optional Semgrep
156
+ adds a separate SAST engine.
157
+
158
+ `pen --fix` writes repro tests for supported findings and patch files for a small set of
159
+ deterministic fixes, including Express header hardening. It does not apply source patches.
160
+ `fix` runs understand → scan → generate → plan → apply → recheck security → stack and gate.
161
+ Automatic source patching requires a clean git tree, creates a `pitstop/fix-*` branch,
162
+ respects protected paths and the plan, and never commits or pushes. `--no-apply` keeps
163
+ source patches for review but still writes repro tests and reports. Unsupported fixes
164
+ remain explicit work for you or your agent.
165
+
166
+ **Execution boundary:** live pen, payment attacks and their generated repros run in a
167
+ disposable Linux Docker container: no external network, read-only root and input mounts,
168
+ non-root user, no capabilities, and bounded CPU/memory/processes/time. Only a sanitized
169
+ copy is exposed; `.git`, `.env*`, common credential files and host environment secrets
170
+ are excluded. There is no automatic host-execution fallback.
171
+
172
+ Install/start Docker once, then prepare the default Node image:
173
+
174
+ ```bash
175
+ docker pull node:22-bookworm-slim
176
+ pitstop ask "attack my app" --execute
177
+ ```
178
+
179
+ Missing Docker, missing dependencies or an unbootable app produces an aborted check,
180
+ never a clean security verdict. Non-Node/native dependencies need a prepared Linux image
181
+ selected with `PITSTOP_SANDBOX_IMAGE`; use a trusted image with Node and your runtime.
182
+ Dependencies must already work on Linux; the isolated run cannot download them.
183
+ See [execution and spending controls](docs/release-controls.md).
184
+
185
+ <p align="center">
186
+ <img src="docs/media/pen.gif" alt="Live security probes and evidence" width="880">
187
+ </p>
188
+
189
+ ### Verification beyond a green suite
190
+
191
+ | Layer | What it checks | Guide |
192
+ |---|---|---|
193
+ | `baseline-verify` | Verification fails on a known-bad commit and passes on the candidate; verification identity stays consistent. An intended-failure predicate distinguishes the bug from environment failure. | [Baseline](docs/baseline-verify.md) |
194
+ | `state-verify` | Structured file-change claims against actual filesystem and git state | [State](docs/state-verify.md) |
195
+ | `verifier-check` | A known-good case passes and an explicit known-bad case fails | [Falsifiability](docs/verifier-check.md) |
196
+ | `holdout-verify` | An externally maintained suite checks the committed candidate with redacted results | [Holdout](docs/holdout-verify.md) |
197
+ | `acceptance-verify` | Observable requirements defined in a pinned contract | [Acceptance](docs/acceptance-verify.md) |
198
+ | `regression-check` | Previously passing checks versus the candidate; new, missing and flaky checks remain distinguishable | [Regression](docs/regression-check.md) |
199
+ | `integrity` | Deleted/focused/weakened tests, hardcoded passes, suppression creep and other suspicious diff patterns | [Real examples](docs/caught-in-the-wild.md) |
200
+
201
+ These layers need meaningful contracts, baselines or suites. `flow` runs configured
202
+ stages and reports others as skipped. Git worktrees isolate checkouts, not process
203
+ permissions. A holdout is hidden only when the host's filesystem permissions actually
204
+ keep it outside the editing agent's reach.
205
+
206
+ ### Gate and evidence chain
207
+
208
+ `gate` combines live measurements and saved verification layers. Its decision order is
209
+ `CHEAT → BLOCKED → FAILED → UNPROVEN → VERIFIED`. `explain` shows the evidence chain;
210
+ `honesty` exposes limitations and supporting measurements.
211
+
212
+ ```bash
213
+ pitstop gate --strict --require stack,architecture,acceptance,regression
214
+ ```
215
+
216
+ In strict mode only `VERIFIED` exits `0`; insufficient or failed evidence exits `1`,
217
+ and detected integrity manipulation exits `2`. Legacy `gate` without `--strict` can
218
+ exit `0` with `UNPROVEN` for compatibility. Required layers never silently disappear.
219
+ Other commands have their own exit contracts; inspect `--help` and the linked guides.
220
+
221
+ Artifacts use canonical SHA-256 content digests. They detect edits that do not update
222
+ the digest; they are **not cryptographic signatures from a trusted external signer**.
223
+ Someone with write access can recompute them. Keep authoritative verification and
224
+ holdouts in trusted CI or a separate permission boundary. Every passing historical gate
225
+ layer must match the current source snapshot, Git state, policy and PitStop engine.
226
+ Missing/old bindings or changed inputs become `UNPROVEN`; commit-based checks also
227
+ require that exact clean candidate. Contract and holdout changes invalidate their results.
228
+ [Evidence semantics](docs/explain.md).
229
+
230
+ ### Payment proofs, drift and repros
231
+
232
+ `scan --ledger` replays duplicate webhooks, concurrent submissions and delayed retries
233
+ through a mock payment gateway. Duplicate charge receipts provide evidence for the
234
+ tested scenario. Proxy-only stacks have weaker observation coverage, while Docker still enforces the
235
+ network boundary. This does not prove all payment behavior.
236
+
237
+ `pen` compares comparable runs to identify new, disappeared and escalated findings.
238
+ A disappeared finding is a signal to investigate; preserve and replay the regression
239
+ test to establish that the fix holds. `repro <id>` generates and runs supported repros;
240
+ unsupported findings need a developer-written check. [Demo](docs/demo.md).
241
+
242
+ ### Keep the loop affordable
243
+
244
+ - **No model required** for intent routing, structural discovery, built-in scans and
245
+ verification. Your chosen agent uses its own provider credits when it reasons or edits.
246
+ - **Content-aware reuse:** `scan --reuse` checks the baseline digest, file contents and
247
+ paths, git HEAD, scan options and selected engine/environment settings. Deleted files,
248
+ backdated edits and dot-directory changes invalidate reuse. Quick scans cannot replace
249
+ full scans. Snapshots are checked before and after measurement.
250
+ - **Bounded freshness:** cached scans expire after one hour; live ledger checks are not
251
+ reused. Symlinks, unreadable inputs and snapshot size limits force a fresh scan.
252
+ Generated/dependency directories are excluded. Reuse is an iteration optimization,
253
+ not release evidence: run fresh checks after changing installed tools, dependencies,
254
+ services or environment variables outside the recorded inputs.
255
+ - **One shared agent budget:** `drive` defaults to three total launches, ten minutes
256
+ and 48,000 total prompt characters across every finding. Change these with
257
+ `--max-agent-calls`, `--max-seconds` and `--max-prompt-chars`. Repeated identical
258
+ failures stop retries; concurrent/nested drive sessions are refused.
259
+ - **Optional dollar ceiling:** `drive --max-cost-usd 1` divides one allowance across
260
+ all launches using Claude print mode’s native budget flag. Providers without an
261
+ enforceable adapter refuse this request before an agent starts.
262
+ - **Honest accounting:** `budget` reports recorded test/build time and activity counts.
263
+ It cannot read your agent's token bill, prove cache savings or promise a dollar amount.
264
+
265
+ `memory` keeps decisions and rejected approaches; `ready-check` explains reuse eligibility;
266
+ `doctor` explains missing tools. `watch` rescans during editing, while `trends` and `digest`
267
+ show the repository's recorded progress.
268
+
269
+ ## Install into your coding CLI
270
+
271
+ ```bash
272
+ pitstop install
273
+ ```
274
+
275
+ The installer writes agent instruction files. Check its displayed destinations; use
276
+ `--force`/`-y` deliberately when replacing existing files. `pitstop prompt` prints the
277
+ full workflow for inspection or manual installation. In a host that supports the
278
+ installed command, use `/pitstop` or `/pitstop check the security of this app`.
279
+
280
+ | Bundled target | Command location |
281
+ |---|---|
282
+ | Claude Code | `.claude/commands/pitstop.md`, plus a skill template |
283
+ | OpenCode | `.opencode/commands/pitstop.md` and configured user locations |
284
+ | Kilo | `.kilo/commands/pitstop.md` and legacy workflow locations |
285
+ | Antigravity | `.agent/workflows/pitstop.md` and legacy workflow locations |
286
+ | Gemini CLI | `.gemini/commands/pitstop.toml` |
287
+ | Codex CLI | `~/.codex/prompts/pitstop.md` |
288
+ | Cursor command files | `.cursor/commands/pitstop.md` |
289
+ | Other agent CLIs | Export `pitstop prompt` into their supported custom-command format |
290
+
291
+ Portable instructions can be used with FreeBuff, Grok Build, MUSE Code and other CLIs
292
+ that accept them; those integrations are not automatically validated by the presence of
293
+ a Markdown file. Host versions determine command invocation and argument syntax.
294
+ The shipped installer does not provide a dedicated Codex desktop-app or Codex VS Code
295
+ extension integration. [Installer targets](src/installer/targets.ts).
296
+
297
+ ## Every command, by job
298
+
299
+ | Job | Commands |
300
+ |---|---|
301
+ | Start and navigate | `ask`, `next`, `try`, `demo`, bare `pitstop` onboarding |
302
+ | Understand and plan | `understand`, `plan`, `architecture-check`, `memory` |
303
+ | Measure and diagnose | `scan`, `test`, `verify-stack`, `doctor`, `ready-check` |
304
+ | Fix and inspect | `inspect`, `repro`, `drive`, `fix` |
305
+ | Security | `pen`, `pen --static`, `pen --fix`, `scan --ledger` |
306
+ | Verify | `verify`, `flow`, `gate`, `integrity`, `baseline-verify`, `state-verify`, `verifier-check`, `holdout-verify`, `acceptance-verify`, `regression-check` |
307
+ | Explain and share | `explain`, `honesty`, `report`, `share`, `trends`, `digest`, `budget` |
308
+ | Integrate | `install`, `prompt`, `watch`, `ci` |
309
+
310
+ Use `pitstop <command> --help` for exact options. `report --html` produces a self-contained
311
+ HTML report and a README badge; `share` creates a screenshot-ready card. `ci` produces
312
+ PR-ready reports, and `install --hooks` installs a pre-commit gate. See the
313
+ [GitHub Action guide](docs/github-action.md) and [repository workflow guide](docs/repo-discipline.md).
314
+
315
+ ## Where OpenPitStop fits
316
+
317
+ OpenPitStop's focus is the engineering workflow around a change. Specialized tools
318
+ remain useful alongside it; the project does not claim universal detection superiority.
319
+
320
+ | Tool | Documented focus | Relationship to OpenPitStop |
321
+ |---|---|---|
322
+ | [Strix](https://github.com/usestrix/strix) | Agentic security investigation, exploit validation and remediation; also documents CI and continuous testing | A deeper security-testing option. PitStop concentrates on local checks, architecture rules, regression contracts and evidence gates across the change lifecycle. |
323
+ | [Semgrep](https://docs.semgrep.dev/getting-started/quickstart) | Rule-based static security analysis and broader security products | Optional SAST integration already exists through `PITSTOP_SEMGREP_CONFIG`. |
324
+ | [CodeQL](https://docs.github.com/en/code-security/concepts/code-scanning/codeql/codeql-code-scanning) | Queries over a code database for security analysis | Complementary analysis; no built-in CodeQL execution integration is claimed. |
325
+ | [ZAP](https://www.zaproxy.org/docs/automate/automation-framework/) | Web security automation, authenticated contexts and active/passive scanning | Broader web-testing infrastructure; a future adapter would complement the built-in probes. |
326
+ | [Aider](https://aider.chat/docs/repomap.html) | AI coding with repository maps and relevant context | A host editing agent; PitStop adds independently executed checks around its changes. |
327
+
328
+ Research checked **2026-09-25** against the linked primary sources. This is a capability
329
+ comparison, not a benchmark. [Detailed audit, research and next milestones](docs/engineering-review.md).
330
+
331
+ ## Architecture and limits
332
+
333
+ The TypeScript CLI dispatches commands to analyzers, repository-understanding modules,
334
+ security probes and verification engines. Results live in `.pitstop/`; the report and
335
+ gate layers consume them. Templates instruct the host coding agent, and `drive` can invoke
336
+ that agent. `fix` is the separate, limited deterministic patch path.
337
+
338
+ The strongest built-in coverage is JS/TS and Node application workflows. Native test
339
+ runners and optional tools extend other stacks, but support varies by feature. Architecture
340
+ checks rely on declared rules and supported import patterns; they do not infer a complete
341
+ system design. Static findings can be false positives, and live probes can miss vulnerabilities.
342
+ Acceptance contracts and tests are only as strong as the behavior they assert.
343
+
344
+ Linux and Windows are included in CI. Passing these fixtures does not establish correctness
345
+ for every repository, framework or environment. No tool can promise to remove all engineering
346
+ problems or make every application secure.
347
+
348
+ ## Privacy
349
+
350
+ OpenPitStop has no telemetry service or hosted account requirement. Dependency audits,
351
+ opt-in external tools, repository scripts and a host coding agent may access the network.
352
+ Reports can contain code locations and security evidence; review them before sharing.
353
+ See [PRIVACY.md](PRIVACY.md) for the connection and storage details.
354
+
355
+ ## Contribute
356
+
357
+ The most useful contributions are reproducible bugs, meaningful verification fixtures,
358
+ framework adapters and measured false-positive reductions. Start with
359
+ [CONTRIBUTING.md](CONTRIBUTING.md) and the [engineering roadmap](docs/engineering-review.md).
360
+
361
+ If PitStop catches a real failure, share the minimal reproduction and its evidence.
362
+ [Star the project](https://github.com/Krish-1507/OpenPitStop) or
363
+ [open an issue](https://github.com/Krish-1507/OpenPitStop/issues).
364
+
365
+ [MIT License](LICENSE) · Built by **Krish J**