shipclear 0.2.1 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +37 -4
- package/adapters/claude-code/.claude-plugin/plugin.json +1 -1
- package/adapters/claude-code/commands/safe-to-ship.md +7 -1
- package/adapters/generic/INSTRUCTIONS.md +7 -1
- package/cli/bin/shipclear.js +111 -3
- package/cli/lib/badge.js +216 -0
- package/cli/lib/checks.js +13 -2
- package/cli/lib/git.js +21 -4
- package/cli/lib/infra.js +536 -0
- package/cli/lib/patterns.js +220 -5
- package/cli/lib/probe.js +175 -0
- package/cli/lib/report.js +77 -8
- package/core/gate-checklist.md +13 -1
- package/docs/quickstart.md +23 -0
- package/docs/troubleshooting.md +37 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -7,6 +7,7 @@
|
|
|
7
7
|
Your AI writes code fast. ShipClear makes sure it's safe to ship —
|
|
8
8
|
free, open source, runs 100% locally, works with every AI coding tool.
|
|
9
9
|
|
|
10
|
+
[](https://github.com/adamtimmerberg/ShipClear/actions/workflows/shipclear.yml)
|
|
10
11
|
[](https://github.com/adamtimmerberg/ShipClear/actions/workflows/ci.yml)
|
|
11
12
|
[](LICENSE)
|
|
12
13
|
[](package.json)
|
|
@@ -32,7 +33,7 @@ npx shipclear ship
|
|
|
32
33
|
That runs the full gate and gives you a verdict:
|
|
33
34
|
|
|
34
35
|
```
|
|
35
|
-
ShipClear
|
|
36
|
+
ShipClear v1.1.0 — ship report
|
|
36
37
|
|
|
37
38
|
🛑 CRITICAL A secret key is written directly in your code
|
|
38
39
|
where: server.js:7 — OpenAI API key: `sk-FAKE0…`
|
|
@@ -58,7 +59,7 @@ Most tools scan your app *after* you've made the mistake. ShipClear rides along
|
|
|
58
59
|
|---|---|
|
|
59
60
|
| 🧠 **Prevention** | Rules for your AI assistant so secrets never get hardcoded in the first place — `.env` from the first line of code, no test accounts, no real data in the repo. |
|
|
60
61
|
| 🚫 **Enforcement** | A git commit guard (`shipclear setup`) that blocks any commit containing secrets or a `.env` file. Deterministic — it isn't the AI deciding to be careful. (If the scanner itself can't run — you're offline, say — it lets the commit through with a warning rather than blocking your work; it fails safe, not stuck.) |
|
|
61
|
-
| ✅ **The gate** | `shipclear ship`:
|
|
62
|
+
| ✅ **The gate** | `shipclear ship`: 22 deterministic checks — across your code, your git history, *and* the config you deploy with (`.env`, docker-compose, proxy routes, bucket policies) — plus a semantic checklist for your AI, ending in one verdict — cleared to ship, or not. Found a secret in old commits? `shipclear ship --fix-history` erases that history safely (never-pushed repos only; it refuses if it would lose anything). |
|
|
62
63
|
|
|
63
64
|
## Works with every AI coding tool
|
|
64
65
|
|
|
@@ -85,15 +86,47 @@ Detection is tuned against real-world starter repos, with a false-positive regre
|
|
|
85
86
|
|
|
86
87
|
Plus a ten-point semantic checklist your AI assistant runs — the checks a regex can't do: auth **and ownership** on every route, database rules, client-side leakage, injection/XSS/uploads, server-side trust (prices, roles, webhook signatures), session hygiene and rate limiting, debug surface, and an AI-surface review for apps that call LLMs themselves (prompt injection, unvalidated model output, overprivileged tools).
|
|
87
88
|
|
|
88
|
-
|
|
89
|
+
## The badge can't lie
|
|
90
|
+
|
|
91
|
+
Pass the gate and `shipclear badge` adds a **cleared to ship** badge to your README — one command, and it maintains itself from then on:
|
|
92
|
+
|
|
93
|
+
```bash
|
|
94
|
+
npx shipclear badge
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
Most badges are a static green image: paste it once and it keeps vouching for every commit you make afterwards, checked or not. That's a security claim nobody verified, which is exactly what this tool exists to prevent — so ShipClear doesn't hand you one. Instead `shipclear badge` installs a GitHub Actions workflow that re-runs this gate on GitHub for **every push**, and a badge that shows that workflow's real result:
|
|
98
|
+
|
|
99
|
+
| | means |
|
|
100
|
+
|---|---|
|
|
101
|
+
| 🟢 **passing** | every commit pushed to this branch cleared the gate — including its git history, so an old leaked secret keeps it red until the history is actually cleaned |
|
|
102
|
+
| 🔴 **failing** | a commit landed that ShipClear would not clear |
|
|
103
|
+
| ⚪ **no status** | nothing has been checked, so nothing is claimed |
|
|
104
|
+
|
|
105
|
+
The workflow runs `shipclear ship --no-fix --strict`: `--no-fix` so the verdict describes the commit as pushed rather than a copy the job quietly repaired, and `--strict` so only a full CLEARED counts as green. Skipping the gate locally can't produce a green badge — only a real passing run on GitHub can. (`--strict` is useful in your own CI for the same reason.)
|
|
106
|
+
|
|
107
|
+
Deleting the workflow or turning off Actions doesn't fake a green badge either — the badge drops to "no status". Nothing stops someone from deliberately pointing the badge at a different workflow; it defends against forgetting, not fraud.
|
|
108
|
+
|
|
109
|
+
## Checking the site you actually deployed
|
|
110
|
+
|
|
111
|
+
Some problems don't exist in your files. A storage bucket's permissions are applied at runtime; a missing response header is an absence; a proxy route's real behaviour is a behaviour. So the gate can also ask your live site:
|
|
112
|
+
|
|
113
|
+
```bash
|
|
114
|
+
npx shipclear ship --probe https://your-app.com
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
It reports what is genuinely reachable — an open `/metrics` or `/api/docs`, a storage folder that hands out a list of every file in it, a home page served without the headers that protect pages — and confirms (or quietly drops) the findings the file scan could only suspect.
|
|
118
|
+
|
|
119
|
+
The rules it works under: **`GET` only**, so nothing is written, changed or deleted; no logins, passwords, or attempts to get past anything; only the one address you type and paths beneath it; and it tells you all of that and waits for a yes before the first request. It's off unless you ask for it, and the AI adapters are instructed never to run it on your behalf. Only point it at a site you own.
|
|
89
120
|
|
|
90
121
|
## Privacy, by architecture
|
|
91
122
|
|
|
92
123
|
ShipClear runs entirely on your machine. **No account. No server. No telemetry. Nothing is uploaded, ever.** It has zero runtime dependencies — what you install is what's in this repo, and CI enforces that by not having an install step.
|
|
93
124
|
|
|
125
|
+
The single exception is opt-in and explicit: `ship --probe <url>` makes read-only requests to the address you give it, after asking. It still sends nothing anywhere else, and no part of your code or report leaves your machine.
|
|
126
|
+
|
|
94
127
|
## What ShipClear can't do
|
|
95
128
|
|
|
96
|
-
Honesty is a feature in a security tool. ShipClear does **not**: audit your business logic,
|
|
129
|
+
Honesty is a feature in a security tool. ShipClear does **not**: audit your business logic, protect against DDoS, review your cloud console settings, or replace a professional security audit for high-stakes products. It doesn't pen-test your deployment either — `--probe` makes a handful of read-only requests to pages that shouldn't be public, which is a long way from a penetration test. It also can't catch a secret format nobody has taught it — if you find one it misses, [that's a security bug and we treat it like one](SECURITY.md).
|
|
97
130
|
|
|
98
131
|
## Contributing
|
|
99
132
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "shipclear",
|
|
3
|
-
"version": "
|
|
3
|
+
"version": "1.1.0",
|
|
4
4
|
"description": "Security companion for AI-assisted coding: prevention rules applied to every task, a commit guard that blocks leaked secrets, and the /safe-to-ship gate.",
|
|
5
5
|
"author": { "name": "Adam Timmerberg" },
|
|
6
6
|
"homepage": "https://github.com/adamtimmerberg/ShipClear"
|
|
@@ -13,6 +13,7 @@ The user wants to know whether this project is safe to deploy or share publicly.
|
|
|
13
13
|
1. Run the deterministic scan: `npx --yes shipclear ship` (it writes `SHIP-REPORT.md` and applies safe fixes automatically).
|
|
14
14
|
2. Read `SHIP-REPORT.md` and walk the user through each finding in plain English.
|
|
15
15
|
- **Never run `shipclear ship --fix-history` on the user's behalf.** It erases git history. If a finding recommends it, explain what it does, confirm the other findings are fixed first, and let the user run it — or run it only after they explicitly say to.
|
|
16
|
+
- **Never run `shipclear ship --probe <url>` unless the user asks for it and names the address.** It makes real (read-only) requests to a live site. Deciding which site may be contacted is the user's, not yours. You may *tell* them it exists — it confirms findings a file-based scan cannot see, like a storage folder anyone can list — and let them run it.
|
|
16
17
|
3. Perform the semantic checks below yourself — these need code understanding and are your responsibility:
|
|
17
18
|
|
|
18
19
|
1. **Hidden accounts:** search for any route, seed, migration, or conditional that grants access via a fixed credential, magic string, or special email — including ones the AI itself created during this project. Check `.shipclear/test-accounts.json` and confirm every listed credential is absent from code and the production database plan.
|
|
@@ -24,7 +25,12 @@ The user wants to know whether this project is safe to deploy or share publicly.
|
|
|
24
25
|
7. **Session and abuse surface:** session tokens in httpOnly cookies rather than localStorage; rate limiting (or provider-equivalent) on login, signup, and password-reset endpoints.
|
|
25
26
|
8. **Debug surface:** look for debug endpoints, verbose stack traces returned to users, `console.log` of sensitive values, and permissive CORS on authenticated APIs.
|
|
26
27
|
9. **AI surface** (when the app itself calls LLMs or runs agents/MCP tools): treat user input and any external content entering prompts as untrusted; never execute, render as HTML, or write to the database unvalidated model output; give agent/MCP tools the narrowest permissions that work, and keep agents away from production resources during testing.
|
|
27
|
-
10. **
|
|
28
|
+
10. **Credentials actually in use:** every password, key and token in the live config differs from the value in the example/sample config and from the service's documented default. The app refuses to start in production if a placeholder remains.
|
|
29
|
+
11. **File and object storage:** no bucket or container is publicly readable; nothing anonymous can **list** its contents. Confirm with a request to the bucket root, not just by reading config.
|
|
30
|
+
12. **Proxy and ingress:** no public route forwards wholesale into a backing service (database, object store, queue, admin UI); routes are narrowed to the paths actually needed.
|
|
31
|
+
13. **Outbound fetches:** if the server fetches URLs supplied by users (import-from-URL, link previews, webhooks, avatars), internal and cloud-metadata addresses are refused — and re-checked after every redirect.
|
|
32
|
+
14. **Header coverage:** security headers apply to the responses that serve HTML, not only to the API.
|
|
33
|
+
15. **Report merge:** append semantic findings to `SHIP-REPORT.md` in the same *What / Why / Fix* format, then restate the final verdict — a semantic critical downgrades the overall verdict to DO NOT SHIP.
|
|
28
34
|
|
|
29
35
|
## Verdict rules
|
|
30
36
|
|
|
@@ -42,6 +42,7 @@ These rules are for AI coding agents and assistants. Follow them on every task,
|
|
|
42
42
|
1. Run the deterministic scan: `npx --yes shipclear ship` (it writes `SHIP-REPORT.md` and applies safe fixes automatically).
|
|
43
43
|
2. Read `SHIP-REPORT.md` and walk the user through each finding in plain English.
|
|
44
44
|
- **Never run `shipclear ship --fix-history` on the user's behalf.** It erases git history. If a finding recommends it, explain what it does, confirm the other findings are fixed first, and let the user run it — or run it only after they explicitly say to.
|
|
45
|
+
- **Never run `shipclear ship --probe <url>` unless the user asks for it and names the address.** It makes real (read-only) requests to a live site. Deciding which site may be contacted is the user's, not yours. You may *tell* them it exists — it confirms findings a file-based scan cannot see, like a storage folder anyone can list — and let them run it.
|
|
45
46
|
3. Perform the semantic checks below yourself — these need code understanding and are your responsibility:
|
|
46
47
|
|
|
47
48
|
1. **Hidden accounts:** search for any route, seed, migration, or conditional that grants access via a fixed credential, magic string, or special email — including ones the AI itself created during this project. Check `.shipclear/test-accounts.json` and confirm every listed credential is absent from code and the production database plan.
|
|
@@ -53,7 +54,12 @@ These rules are for AI coding agents and assistants. Follow them on every task,
|
|
|
53
54
|
7. **Session and abuse surface:** session tokens in httpOnly cookies rather than localStorage; rate limiting (or provider-equivalent) on login, signup, and password-reset endpoints.
|
|
54
55
|
8. **Debug surface:** look for debug endpoints, verbose stack traces returned to users, `console.log` of sensitive values, and permissive CORS on authenticated APIs.
|
|
55
56
|
9. **AI surface** (when the app itself calls LLMs or runs agents/MCP tools): treat user input and any external content entering prompts as untrusted; never execute, render as HTML, or write to the database unvalidated model output; give agent/MCP tools the narrowest permissions that work, and keep agents away from production resources during testing.
|
|
56
|
-
10. **
|
|
57
|
+
10. **Credentials actually in use:** every password, key and token in the live config differs from the value in the example/sample config and from the service's documented default. The app refuses to start in production if a placeholder remains.
|
|
58
|
+
11. **File and object storage:** no bucket or container is publicly readable; nothing anonymous can **list** its contents. Confirm with a request to the bucket root, not just by reading config.
|
|
59
|
+
12. **Proxy and ingress:** no public route forwards wholesale into a backing service (database, object store, queue, admin UI); routes are narrowed to the paths actually needed.
|
|
60
|
+
13. **Outbound fetches:** if the server fetches URLs supplied by users (import-from-URL, link previews, webhooks, avatars), internal and cloud-metadata addresses are refused — and re-checked after every redirect.
|
|
61
|
+
14. **Header coverage:** security headers apply to the responses that serve HTML, not only to the API.
|
|
62
|
+
15. **Report merge:** append semantic findings to `SHIP-REPORT.md` in the same *What / Why / Fix* format, then restate the final verdict — a semantic critical downgrades the overall verdict to DO NOT SHIP.
|
|
57
63
|
|
|
58
64
|
## Verdict rules
|
|
59
65
|
|
package/cli/bin/shipclear.js
CHANGED
|
@@ -1,12 +1,15 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
import fs from 'node:fs';
|
|
3
3
|
import path from 'node:path';
|
|
4
|
+
import readline from 'node:readline/promises';
|
|
4
5
|
import { fileURLToPath } from 'node:url';
|
|
5
6
|
import { runGate } from '../lib/checks.js';
|
|
6
7
|
import { applyFixes, resetHistory, installHook, envExampleFrom, GITIGNORE_BLOCK } from '../lib/fix.js';
|
|
7
8
|
import { renderTerminal, renderMarkdown, verdictOf, VERDICT_TEXT } from '../lib/report.js';
|
|
8
9
|
import { findSecrets, maskSecret, lineOfIndex } from '../lib/patterns.js';
|
|
9
10
|
import { isGitRepo, stagedFiles, stagedContent } from '../lib/git.js';
|
|
11
|
+
import { installBadge } from '../lib/badge.js';
|
|
12
|
+
import { runProbe, normalizeBase } from '../lib/probe.js';
|
|
10
13
|
import { loadIgnore } from '../lib/ignore.js';
|
|
11
14
|
|
|
12
15
|
const pkg = JSON.parse(
|
|
@@ -22,12 +25,26 @@ Usage:
|
|
|
22
25
|
shipclear scan --staged Scan only what's about to be committed (used by the guard)
|
|
23
26
|
shipclear ship Full safe-to-ship gate → SHIP-REPORT.md + verdict
|
|
24
27
|
shipclear ship --no-fix Same, but don't apply automatic fixes
|
|
28
|
+
shipclear ship --strict Same, but treat ANY finding as a failure (exit 1 unless
|
|
29
|
+
the verdict is CLEARED) — for CI and the README badge
|
|
30
|
+
shipclear ship --probe https://your-app.com
|
|
31
|
+
Also ask your deployed site what is actually
|
|
32
|
+
reachable: open debug pages, a storage folder anyone
|
|
33
|
+
can list, missing page headers. Read-only (GET
|
|
34
|
+
only), asks you to confirm first, and only ever
|
|
35
|
+
touches the address you type. Add --yes to skip the
|
|
36
|
+
confirmation (for scripts).
|
|
25
37
|
shipclear ship --fix-history
|
|
26
38
|
Also erase git history to remove a leaked secret from
|
|
27
39
|
old commits (only when nothing was ever pushed; refuses
|
|
28
40
|
if it would lose branches or stashes)
|
|
41
|
+
shipclear badge Add a self-updating "cleared to ship" badge to your
|
|
42
|
+
README, backed by a GitHub Actions run of this gate on
|
|
43
|
+
every push — so it turns red if an unchecked or failing
|
|
44
|
+
commit lands
|
|
29
45
|
|
|
30
|
-
Everything runs locally
|
|
46
|
+
Everything runs locally and nothing is ever uploaded. The one exception is
|
|
47
|
+
opt-in: --probe makes read-only requests to the single address you give it.
|
|
31
48
|
Docs: https://github.com/adamtimmerberg/ShipClear
|
|
32
49
|
`;
|
|
33
50
|
|
|
@@ -114,7 +131,34 @@ function cmdScan(root) {
|
|
|
114
131
|
return findings.some((f) => f.severity === 'critical') ? 1 : 0;
|
|
115
132
|
}
|
|
116
133
|
|
|
117
|
-
|
|
134
|
+
// The probe reaches out to the network, which nothing else in ShipClear does.
|
|
135
|
+
// So it is opt-in per run, it says exactly what it will do, and it asks —
|
|
136
|
+
// unless the user passed --yes, or there is no terminal to ask in (a CI job),
|
|
137
|
+
// where it declines rather than guessing.
|
|
138
|
+
async function confirmProbe(url) {
|
|
139
|
+
const { base, error } = normalizeBase(url);
|
|
140
|
+
if (error) {
|
|
141
|
+
console.error(`\n ${error}\n`);
|
|
142
|
+
return null;
|
|
143
|
+
}
|
|
144
|
+
if (!process.stdin.isTTY) {
|
|
145
|
+
return { base, confirmed: false, why: 'there is no terminal here to confirm in — re-run it yourself, or add --yes if you are scripting this' };
|
|
146
|
+
}
|
|
147
|
+
console.log('');
|
|
148
|
+
console.log(` --probe will make read-only requests to ${base}:`);
|
|
149
|
+
console.log(' • GET only. Nothing is written, changed or deleted.');
|
|
150
|
+
console.log(' • No logins, no passwords, no attempts to get past anything.');
|
|
151
|
+
console.log(' • Only that address and paths under it — nowhere else.');
|
|
152
|
+
console.log(' • It checks for open debug pages, a storage folder anyone can list,');
|
|
153
|
+
console.log(' and whether your home page sends the headers that protect pages.');
|
|
154
|
+
console.log(' Only do this against a site you own.');
|
|
155
|
+
const rl = readline.createInterface({ input: process.stdin, output: process.stdout });
|
|
156
|
+
const answer = await rl.question(`\n Probe ${base}? (y/N) `);
|
|
157
|
+
rl.close();
|
|
158
|
+
return { base, confirmed: /^y(es)?$/i.test(answer.trim()), why: 'you answered no' };
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
async function cmdShip(root, { fix = true, fixHistory = false, strict = false, probeUrl = null, assumeYes = false } = {}) {
|
|
118
162
|
let { findings, notes } = runGate(root);
|
|
119
163
|
const fixes = fix ? applyFixes(root, findings) : [];
|
|
120
164
|
|
|
@@ -130,13 +174,68 @@ function cmdShip(root, { fix = true, fixHistory = false } = {}) {
|
|
|
130
174
|
}
|
|
131
175
|
}
|
|
132
176
|
|
|
177
|
+
if (probeUrl) {
|
|
178
|
+
const decision = await confirmProbe(probeUrl);
|
|
179
|
+
if (decision === null) {
|
|
180
|
+
notes.push('--probe was not run: that address could not be read.');
|
|
181
|
+
} else if (!decision.confirmed && !assumeYes) {
|
|
182
|
+
notes.push(`--probe was not run: ${decision.why}. Everything above is from your files alone.`);
|
|
183
|
+
} else {
|
|
184
|
+
const probed = await runProbe(decision.base, findings);
|
|
185
|
+
findings.push(...probed.findings);
|
|
186
|
+
notes.push(...probed.notes);
|
|
187
|
+
}
|
|
188
|
+
}
|
|
189
|
+
|
|
133
190
|
const verdict = verdictOf(findings);
|
|
191
|
+
// Strict mode exists for the README badge and CI: a badge may only be
|
|
192
|
+
// green on a full CLEARED, so "SHIP WITH FIXES" — which exits 0 normally,
|
|
193
|
+
// because a human can read it and decide — has to fail here.
|
|
194
|
+
if (strict && verdict !== 'CLEARED') {
|
|
195
|
+
notes.push('--strict: failing because the verdict is not CLEARED. Fix the findings above; a ShipClear badge stays red until they are gone.');
|
|
196
|
+
}
|
|
134
197
|
const result = { findings, notes, verdict, fixes, version: pkg.version };
|
|
135
198
|
fs.writeFileSync(path.join(root, 'SHIP-REPORT.md'), renderMarkdown(result));
|
|
136
199
|
console.log(renderTerminal(result));
|
|
200
|
+
if (strict) return verdict === 'CLEARED' ? 0 : 1;
|
|
137
201
|
return verdict === 'DO_NOT_SHIP' ? 1 : 0;
|
|
138
202
|
}
|
|
139
203
|
|
|
204
|
+
function cmdBadge(root) {
|
|
205
|
+
if (!isGitRepo(root)) {
|
|
206
|
+
console.error("\n This folder isn't set up with git yet, so there's no GitHub repo for a badge to report on.");
|
|
207
|
+
console.error(' Run `git init` here, push the project to GitHub, then run `shipclear badge` again.\n');
|
|
208
|
+
return 1;
|
|
209
|
+
}
|
|
210
|
+
const result = installBadge(root);
|
|
211
|
+
if (!result.ok) {
|
|
212
|
+
console.error(`\n ShipClear can't add the badge yet: ${result.reason}\n`);
|
|
213
|
+
return 1;
|
|
214
|
+
}
|
|
215
|
+
console.log('\n ShipClear badge installed.');
|
|
216
|
+
for (const d of result.done) console.log(` ✔ ${d}`);
|
|
217
|
+
for (const w of result.warnings) console.log(` ⚠ ${w}`);
|
|
218
|
+
console.log('\n Commit and push both files to switch it on:');
|
|
219
|
+
console.log(' git add -A && git commit -m "add ShipClear badge" && git push');
|
|
220
|
+
// The badge reads "no status" until the workflow has run once. Saying so
|
|
221
|
+
// here stops that grey badge from looking like a broken install.
|
|
222
|
+
console.log(`\n It shows "no status" until that first push runs the workflow — grey because`);
|
|
223
|
+
console.log(' nothing has been checked yet, which is the honest answer at that point.');
|
|
224
|
+
console.log(` Then it tracks the \`${result.branch}\` branch: green while every pushed commit`);
|
|
225
|
+
console.log(' clears this gate, red the moment one does not.\n');
|
|
226
|
+
return 0;
|
|
227
|
+
}
|
|
228
|
+
|
|
229
|
+
/** Value of a `--flag value` or `--flag=value` argument, or null. */
|
|
230
|
+
function flagValue(argv, flag) {
|
|
231
|
+
const inline = argv.find((a) => a.startsWith(`${flag}=`));
|
|
232
|
+
if (inline) return inline.slice(flag.length + 1) || null;
|
|
233
|
+
const i = argv.indexOf(flag);
|
|
234
|
+
if (i === -1) return null;
|
|
235
|
+
const next = argv[i + 1];
|
|
236
|
+
return next && !next.startsWith('-') ? next : null;
|
|
237
|
+
}
|
|
238
|
+
|
|
140
239
|
const args = process.argv.slice(2);
|
|
141
240
|
const cmd = args[0];
|
|
142
241
|
const root = process.cwd();
|
|
@@ -156,9 +255,18 @@ try {
|
|
|
156
255
|
const fixHistory = args.includes('--fix-history');
|
|
157
256
|
// A history reset needs .gitignore in place first so the fresh commit
|
|
158
257
|
// doesn't re-add .env — so --fix-history implies the additive fixes.
|
|
159
|
-
exitCode = cmdShip(root, {
|
|
258
|
+
exitCode = await cmdShip(root, {
|
|
259
|
+
fix: !args.includes('--no-fix') || fixHistory,
|
|
260
|
+
fixHistory,
|
|
261
|
+
strict: args.includes('--strict'),
|
|
262
|
+
probeUrl: flagValue(args, '--probe'),
|
|
263
|
+
assumeYes: args.includes('--yes'),
|
|
264
|
+
});
|
|
160
265
|
break;
|
|
161
266
|
}
|
|
267
|
+
case 'badge':
|
|
268
|
+
exitCode = cmdBadge(root);
|
|
269
|
+
break;
|
|
162
270
|
case '--version':
|
|
163
271
|
case '-v':
|
|
164
272
|
console.log(pkg.version);
|
package/cli/lib/badge.js
ADDED
|
@@ -0,0 +1,216 @@
|
|
|
1
|
+
// The README badge.
|
|
2
|
+
//
|
|
3
|
+
// A badge is a security claim, so it is held to the same standard as a
|
|
4
|
+
// verdict: it must never say "cleared" for code nobody checked. That rules
|
|
5
|
+
// out the obvious implementation. A static image —
|
|
6
|
+
// `shields.io/badge/ShipClear-cleared%20to%20ship-brightgreen` — is a
|
|
7
|
+
// hardcoded green picture: paste it once and it keeps making the claim on
|
|
8
|
+
// every later commit, whether or not the gate ever ran again. That is a
|
|
9
|
+
// false GO, the one failure mode this product exists to prevent
|
|
10
|
+
// (CONTRIBUTING.md), and ShipClear used to hand it to users itself.
|
|
11
|
+
//
|
|
12
|
+
// The honest version has to be computed by something that runs on every
|
|
13
|
+
// push and that the author cannot skip by forgetting. On GitHub that is
|
|
14
|
+
// GitHub Actions. So `shipclear badge` installs a workflow whose only job
|
|
15
|
+
// is the verdict, and a badge that renders that workflow's real status:
|
|
16
|
+
//
|
|
17
|
+
// green — the latest push to this branch cleared the gate
|
|
18
|
+
// red — it did not clear
|
|
19
|
+
// grey — nothing ran ("no status"), so nothing is claimed
|
|
20
|
+
//
|
|
21
|
+
// Two flags in the workflow carry the whole guarantee:
|
|
22
|
+
// --no-fix the verdict must describe the commit as pushed, not a copy
|
|
23
|
+
// the job quietly repaired on its way past (plain `ship` would
|
|
24
|
+
// add the missing .gitignore line itself and then report clean)
|
|
25
|
+
// --strict only a CLEARED verdict may show green (plain `ship` exits 0
|
|
26
|
+
// on SHIP WITH FIXES, which would paint HIGH findings green)
|
|
27
|
+
//
|
|
28
|
+
// "Every commit was cleared" holds rather than just "the last one did",
|
|
29
|
+
// because the gate reads git history: a secret committed at any point keeps
|
|
30
|
+
// failing at HEAD until the history is actually cleaned.
|
|
31
|
+
import fs from 'node:fs';
|
|
32
|
+
import path from 'node:path';
|
|
33
|
+
import { run } from './git.js';
|
|
34
|
+
|
|
35
|
+
export const WORKFLOW_REL = '.github/workflows/shipclear.yml';
|
|
36
|
+
|
|
37
|
+
// The release that introduced `--strict`. The workflow must pin at least
|
|
38
|
+
// this, because unknown flags are silently ignored: run `ship --no-fix
|
|
39
|
+
// --strict` on 1.0.0 and you get plain `ship --no-fix`, which exits 0 on
|
|
40
|
+
// SHIP WITH FIXES — a green badge for a repo with HIGH findings. Verified
|
|
41
|
+
// against the published 1.0.0, not assumed. A caret range still picks up
|
|
42
|
+
// later 1.x releases, so the badge keeps reflecting current knowledge.
|
|
43
|
+
export const MIN_CLI_VERSION = '1.1.0';
|
|
44
|
+
|
|
45
|
+
/** owner/repo from the origin remote, or null if it isn't a GitHub remote. */
|
|
46
|
+
export function githubSlug(root) {
|
|
47
|
+
let url;
|
|
48
|
+
try {
|
|
49
|
+
url = run(root, ['remote', 'get-url', 'origin']).trim();
|
|
50
|
+
} catch {
|
|
51
|
+
return null;
|
|
52
|
+
}
|
|
53
|
+
// https://github.com/o/r(.git) · git@github.com:o/r(.git) · ssh://git@github.com/o/r
|
|
54
|
+
const m = url.match(/github\.com[:/]+([^/]+)\/([^/]+?)(?:\.git)?\/?$/i);
|
|
55
|
+
if (!m) return null;
|
|
56
|
+
return { owner: m[1], repo: m[2] };
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
export function currentBranch(root) {
|
|
60
|
+
try {
|
|
61
|
+
const b = run(root, ['rev-parse', '--abbrev-ref', 'HEAD']).trim();
|
|
62
|
+
return b && b !== 'HEAD' ? b : 'main';
|
|
63
|
+
} catch {
|
|
64
|
+
return 'main';
|
|
65
|
+
}
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
// Served by GitHub itself rather than a third-party image host, so the
|
|
69
|
+
// authority on whether the workflow passed is also the thing drawing the
|
|
70
|
+
// picture. `event=push` pins it to pushed code, so a green pull-request run
|
|
71
|
+
// can't speak for the branch; the label is the workflow's own name.
|
|
72
|
+
export function badgeMarkdown({ owner, repo, branch }) {
|
|
73
|
+
const wf = `https://github.com/${owner}/${repo}/actions/workflows/shipclear.yml`;
|
|
74
|
+
return `[}&event=push)](${wf})`;
|
|
75
|
+
}
|
|
76
|
+
|
|
77
|
+
export function workflowYaml() {
|
|
78
|
+
return `# Keeps the ShipClear badge in this README honest.
|
|
79
|
+
#
|
|
80
|
+
# Re-runs the gate on GitHub for every push, so the badge reflects the code
|
|
81
|
+
# that is actually on GitHub — not whatever was on someone's laptop the last
|
|
82
|
+
# time they remembered to check. A commit ShipClear won't clear fails this
|
|
83
|
+
# job, and the badge turns red.
|
|
84
|
+
#
|
|
85
|
+
# --no-fix: judge the commit as pushed. Without it the job would apply the
|
|
86
|
+
# safe fixes to its own checkout and report the repaired copy.
|
|
87
|
+
# --strict: only a clean CLEARED verdict passes. Without it "SHIP WITH
|
|
88
|
+
# FIXES" would exit 0 and show green.
|
|
89
|
+
name: ShipClear
|
|
90
|
+
|
|
91
|
+
on:
|
|
92
|
+
push:
|
|
93
|
+
pull_request:
|
|
94
|
+
workflow_dispatch:
|
|
95
|
+
|
|
96
|
+
permissions:
|
|
97
|
+
contents: read
|
|
98
|
+
|
|
99
|
+
jobs:
|
|
100
|
+
gate:
|
|
101
|
+
runs-on: ubuntu-latest
|
|
102
|
+
steps:
|
|
103
|
+
- uses: actions/checkout@v4
|
|
104
|
+
with:
|
|
105
|
+
fetch-depth: 0 # the gate scans your whole git history
|
|
106
|
+
- uses: actions/setup-node@v4
|
|
107
|
+
with:
|
|
108
|
+
node-version: 20
|
|
109
|
+
# Pinned at or above the release that added --strict: an older CLI
|
|
110
|
+
# ignores the flag silently and would exit 0 on findings. The caret
|
|
111
|
+
# still takes later 1.x releases, so a security badge keeps
|
|
112
|
+
# reflecting what is known to be unsafe today.
|
|
113
|
+
- run: npx --yes shipclear@^${MIN_CLI_VERSION} ship --no-fix --strict
|
|
114
|
+
`;
|
|
115
|
+
}
|
|
116
|
+
|
|
117
|
+
const IMAGE_MD = /\[?!\[([^\]]*)\]\(([^)]*)\)(?:\]\([^)]*\))?/g;
|
|
118
|
+
|
|
119
|
+
// Which images are *our* badge. This has to be exact, not a "shipclear"
|
|
120
|
+
// substring search: in this very repo every badge URL contains
|
|
121
|
+
// github.com/adamtimmerberg/ShipClear, so a loose match replaced the CI
|
|
122
|
+
// badge instead (caught by running this command on its own README). Any
|
|
123
|
+
// user whose repo is named shipclear-something would have hit the same
|
|
124
|
+
// thing. Match the badge endpoint or the retired static badge, or an alt
|
|
125
|
+
// text that opens with the product name.
|
|
126
|
+
function isOurBadge(whole, alt, url) {
|
|
127
|
+
return /\/actions\/workflows\/shipclear\.yml\/badge\.svg/i.test(url)
|
|
128
|
+
|| /shields\.io\/badge\/ShipClear/i.test(url)
|
|
129
|
+
|| /^\s*ShipClear\b/i.test(alt);
|
|
130
|
+
}
|
|
131
|
+
|
|
132
|
+
/** A line that is nothing but badges — the row to join if one exists. */
|
|
133
|
+
function isBadgeRow(line) {
|
|
134
|
+
const stripped = line.replace(IMAGE_MD, '').trim();
|
|
135
|
+
return stripped === '' && line.trim() !== '';
|
|
136
|
+
}
|
|
137
|
+
|
|
138
|
+
/**
|
|
139
|
+
* Put `badge` in the README: replacing any badge already there (including
|
|
140
|
+
* the old static image), else joining the badge row, else after the H1.
|
|
141
|
+
* Pure string work so the placement is testable without a repo.
|
|
142
|
+
*/
|
|
143
|
+
export function withBadge(readme, badge) {
|
|
144
|
+
for (const m of readme.matchAll(IMAGE_MD)) {
|
|
145
|
+
if (!isOurBadge(m[0], m[1], m[2])) continue;
|
|
146
|
+
if (m[0] === badge) return { text: readme, changed: false, how: 'already current' };
|
|
147
|
+
return {
|
|
148
|
+
text: readme.slice(0, m.index) + badge + readme.slice(m.index + m[0].length),
|
|
149
|
+
changed: true,
|
|
150
|
+
how: 'replaced the old badge',
|
|
151
|
+
};
|
|
152
|
+
}
|
|
153
|
+
const lines = readme.split('\n');
|
|
154
|
+
const row = lines.findIndex(isBadgeRow);
|
|
155
|
+
if (row !== -1) {
|
|
156
|
+
lines[row] = `${lines[row].trimEnd()} ${badge}`;
|
|
157
|
+
return { text: lines.join('\n'), changed: true, how: 'added it to your badge row' };
|
|
158
|
+
}
|
|
159
|
+
const h1 = lines.findIndex((l) => /^#\s+\S/.test(l));
|
|
160
|
+
if (h1 !== -1) {
|
|
161
|
+
lines.splice(h1 + 1, 0, '', badge);
|
|
162
|
+
return { text: lines.join('\n'), changed: true, how: 'added it under your title' };
|
|
163
|
+
}
|
|
164
|
+
return { text: [badge, '', readme].join('\n'), changed: true, how: 'added it at the top' };
|
|
165
|
+
}
|
|
166
|
+
|
|
167
|
+
/**
|
|
168
|
+
* Install the workflow and the badge. Returns { ok:false, reason } when the
|
|
169
|
+
* repo has nowhere for a live badge to point.
|
|
170
|
+
*/
|
|
171
|
+
export function installBadge(root) {
|
|
172
|
+
const slug = githubSlug(root);
|
|
173
|
+
if (!slug) {
|
|
174
|
+
return {
|
|
175
|
+
ok: false,
|
|
176
|
+
reason: "this project has no GitHub remote, so there's nothing to publish a live badge from. Push it to GitHub first (create an empty repo there, then `git remote add origin <its URL>` and `git push -u origin HEAD`), then run `shipclear badge` again.",
|
|
177
|
+
};
|
|
178
|
+
}
|
|
179
|
+
const branch = currentBranch(root);
|
|
180
|
+
const badge = badgeMarkdown({ ...slug, branch });
|
|
181
|
+
const done = [];
|
|
182
|
+
const warnings = [];
|
|
183
|
+
|
|
184
|
+
const wfPath = path.join(root, WORKFLOW_REL);
|
|
185
|
+
if (!fs.existsSync(wfPath)) {
|
|
186
|
+
fs.mkdirSync(path.dirname(wfPath), { recursive: true });
|
|
187
|
+
fs.writeFileSync(wfPath, workflowYaml());
|
|
188
|
+
done.push(`created ${WORKFLOW_REL} — it re-runs the gate on GitHub for every push, which is what keeps the badge honest`);
|
|
189
|
+
} else {
|
|
190
|
+
const existing = fs.readFileSync(wfPath, 'utf8');
|
|
191
|
+
// Never silently overwrite someone's workflow — but a badge pointing at
|
|
192
|
+
// a workflow missing these flags can show green for a repo that didn't
|
|
193
|
+
// clear, so say so rather than leaving a quiet lie in place.
|
|
194
|
+
if (!/--strict/.test(existing) || !/--no-fix/.test(existing)) {
|
|
195
|
+
warnings.push(`${WORKFLOW_REL} already exists, so ShipClear left it alone — but it doesn't run \`shipclear ship --no-fix --strict\`, so the badge can show green for a repo that only partly cleared. Add those flags to that file.`);
|
|
196
|
+
} else {
|
|
197
|
+
done.push(`${WORKFLOW_REL} was already in place`);
|
|
198
|
+
}
|
|
199
|
+
}
|
|
200
|
+
|
|
201
|
+
const readmePath = ['README.md', 'readme.md', 'README.markdown']
|
|
202
|
+
.map((n) => path.join(root, n))
|
|
203
|
+
.find((p) => fs.existsSync(p));
|
|
204
|
+
if (!readmePath) {
|
|
205
|
+
warnings.push(`no README.md found in this folder, so there was nothing to add the badge to. Create one and paste this line into it:\n${badge}`);
|
|
206
|
+
} else {
|
|
207
|
+
const before = fs.readFileSync(readmePath, 'utf8');
|
|
208
|
+
const { text, changed, how } = withBadge(before, badge);
|
|
209
|
+
if (changed) fs.writeFileSync(readmePath, text);
|
|
210
|
+
done.push(changed
|
|
211
|
+
? `${path.basename(readmePath)} — ${how}`
|
|
212
|
+
: `${path.basename(readmePath)} already has the current badge`);
|
|
213
|
+
}
|
|
214
|
+
|
|
215
|
+
return { ok: true, done, warnings, badge, branch, slug };
|
|
216
|
+
}
|
package/cli/lib/checks.js
CHANGED
|
@@ -8,6 +8,7 @@ import {
|
|
|
8
8
|
personalPaths, TLS_VERIFICATION_OFF, DEBUG_MODE_ON,
|
|
9
9
|
} from './patterns.js';
|
|
10
10
|
import { loadIgnore } from './ignore.js';
|
|
11
|
+
import { runInfraChecks } from './infra.js';
|
|
11
12
|
import { walkFiles, isTextCandidate, readFileSafe } from './scan.js';
|
|
12
13
|
import { isGitRepo, hasCommits, hasRemote, branchNames, trackedFiles, historyAddedLines } from './git.js';
|
|
13
14
|
|
|
@@ -28,6 +29,9 @@ const isEnvTemplate = (rel) => ENV_TEMPLATE.test(path.posix.basename(rel));
|
|
|
28
29
|
const isKeyFile = (rel) => KEY_FILE.test(rel);
|
|
29
30
|
const isDataFile = (rel) => DATA_FILE_EXT.has(path.extname(rel).toLowerCase());
|
|
30
31
|
const isDocFile = (rel) => DOC_EXT.has(path.extname(rel).toLowerCase());
|
|
32
|
+
// A database password in a compose/k8s file needs an extra rotation step that
|
|
33
|
+
// fails silently if missed — see containerDbNote in report.js.
|
|
34
|
+
const isContainerConfig = (rel) => /^(?:docker-)?compose(?:[.-].+)?\.ya?ml$|\.ya?ml$/i.test(path.posix.basename(rel));
|
|
31
35
|
|
|
32
36
|
/**
|
|
33
37
|
* Run the deterministic layer of the gate. Returns { findings, notes, inGit }.
|
|
@@ -56,7 +60,7 @@ export function runGate(root, { quick = false } = {}) {
|
|
|
56
60
|
const content = read(rel);
|
|
57
61
|
if (content === null) continue;
|
|
58
62
|
|
|
59
|
-
for (const hit of findSecrets(content, { includeGeneric: !isDocFile(rel) })) {
|
|
63
|
+
for (const hit of findSecrets(content, { includeGeneric: !isDocFile(rel), docFile: isDocFile(rel) })) {
|
|
60
64
|
seenSecretValues.add(hit.match);
|
|
61
65
|
findings.push({
|
|
62
66
|
id: hit.patternId === 'generic-secret' ? 'generic-secret' : 'secret-in-code',
|
|
@@ -64,6 +68,8 @@ export function runGate(root, { quick = false } = {}) {
|
|
|
64
68
|
file: rel,
|
|
65
69
|
line: lineOfIndex(content, hit.index),
|
|
66
70
|
detail: `${hit.name}: \`${maskSecret(hit.value ?? hit.match)}\``,
|
|
71
|
+
dbInContainer: isContainerConfig(rel) && /volumes\s*:/.test(content)
|
|
72
|
+
&& /postgres|mysql|mariadb|mongo|redis/i.test(hit.name + hit.match),
|
|
67
73
|
});
|
|
68
74
|
}
|
|
69
75
|
|
|
@@ -226,7 +232,7 @@ export function runGate(root, { quick = false } = {}) {
|
|
|
226
232
|
const byLocation = new Map();
|
|
227
233
|
for (const { commit, file, text } of historyAddedLines(root)) {
|
|
228
234
|
if (ignored(file)) continue;
|
|
229
|
-
for (const hit of findSecrets(text, { includeGeneric: false })) {
|
|
235
|
+
for (const hit of findSecrets(text, { includeGeneric: false, docFile: isDocFile(file) })) {
|
|
230
236
|
if (seen.has(hit.match)) continue;
|
|
231
237
|
seen.add(hit.match);
|
|
232
238
|
// Already reported from the working tree — that finding's advice
|
|
@@ -257,6 +263,11 @@ export function runGate(root, { quick = false } = {}) {
|
|
|
257
263
|
}
|
|
258
264
|
}
|
|
259
265
|
|
|
266
|
+
// --- how this project is deployed: live credentials, storage, proxy routes ---
|
|
267
|
+
// Walks the working tree rather than git's index on purpose — the live
|
|
268
|
+
// `.env` is the subject of the first of these checks and is never tracked.
|
|
269
|
+
findings.push(...runInfraChecks(root, { ignored }));
|
|
270
|
+
|
|
260
271
|
// --- generated test logins exist; the rules promise a launch-time reminder ---
|
|
261
272
|
if (fs.existsSync(path.join(root, '.shipclear', 'test-accounts.json'))) {
|
|
262
273
|
findings.push({ id: 'test-registry-present', severity: 'info', file: '.shipclear/test-accounts.json' });
|
package/cli/lib/git.js
CHANGED
|
@@ -86,7 +86,15 @@ export function stagedContent(root, file) {
|
|
|
86
86
|
|
|
87
87
|
/**
|
|
88
88
|
* Every line ever *added* across the full history of all branches, so a
|
|
89
|
-
* secret that was committed and later deleted is still found.
|
|
89
|
+
* secret that was committed and later deleted is still found. Added lines are
|
|
90
|
+
* grouped per commit+file, which is also how findings are reported.
|
|
91
|
+
*
|
|
92
|
+
* The grouping is load-bearing, not tidiness: handing detectors one line at a
|
|
93
|
+
* time makes any pattern that needs more than one line silently blind in
|
|
94
|
+
* history. Private key material is exactly that shape — a PEM header is only
|
|
95
|
+
* a leak when its base64 body follows — so a per-line feed would have found
|
|
96
|
+
* every leaked key file in the working tree and none in the history, which is
|
|
97
|
+
* the half that matters once a secret has been deleted.
|
|
90
98
|
* Returns [{ commit, file, text }].
|
|
91
99
|
*/
|
|
92
100
|
export function historyAddedLines(root) {
|
|
@@ -94,7 +102,8 @@ export function historyAddedLines(root) {
|
|
|
94
102
|
'log', '--all', '-p', '--unified=0', '--no-color',
|
|
95
103
|
'--pretty=format:@@shipclear-commit@@ %h',
|
|
96
104
|
]);
|
|
97
|
-
const
|
|
105
|
+
const blocks = new Map();
|
|
106
|
+
const order = [];
|
|
98
107
|
let commit = null;
|
|
99
108
|
let file = null;
|
|
100
109
|
for (const line of out.split('\n')) {
|
|
@@ -105,8 +114,16 @@ export function historyAddedLines(root) {
|
|
|
105
114
|
} else if (line.startsWith('+++ /dev/null')) {
|
|
106
115
|
file = null;
|
|
107
116
|
} else if (line.startsWith('+') && !line.startsWith('+++') && file) {
|
|
108
|
-
|
|
117
|
+
const key = `${commit}\u0000${file}`;
|
|
118
|
+
if (!blocks.has(key)) {
|
|
119
|
+
blocks.set(key, { commit, file, lines: [] });
|
|
120
|
+
order.push(key);
|
|
121
|
+
}
|
|
122
|
+
blocks.get(key).lines.push(line.slice(1));
|
|
109
123
|
}
|
|
110
124
|
}
|
|
111
|
-
return
|
|
125
|
+
return order.map((key) => {
|
|
126
|
+
const b = blocks.get(key);
|
|
127
|
+
return { commit: b.commit, file: b.file, text: b.lines.join('\n') };
|
|
128
|
+
});
|
|
112
129
|
}
|