rvlib-shipkit 0.36.0 → 0.37.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +7 -0
- package/package.json +1 -1
- package/src/hooks/leak-guard.ts +313 -38
- package/src/hooks/run-hooks.ts +55 -7
- package/src/repo-config.ts +7 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,12 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.37.0
|
|
4
|
+
|
|
5
|
+
- **The leak guard grew an allowlist, because it had no way to be wrong.** Every keyword matched as a case-insensitive substring over the staged added lines, so a four letter row landed inside a real npm dependency name and inside a base64 integrity hash in `bun.lock`, and that lockfile became uncommittable forever with no per repo override of any kind. The only way past was `--no-verify`, which disarms every other gate too, so the guard was training people to switch it off. Two answers ship, in this order. A `w:<word>` keyword row now matches on word boundaries spelled in unicode rather than with `\b`, which is ascii only and would let an accented name match inside a longer accented word: that makes the PATTERN smarter rather than the guard looser, and it is opt in, so a keyword file that carries no `w:` row behaves exactly as before. Then `repo.config.ts hooks.leakAllow` (with the machine-global twin `~/.rv-private/leak-allow.json`) declares a known-good match: `paths` names the files, `patterns` names the keywords, and `inHashes` narrows to a match that falls inside a content hash, where the bytes are an address rather than prose. `paths` plus `patterns` is the shape to reach for, because it waives one keyword in named files while the same keyword still blocks everywhere else, and a single occurrence outside a hash refuses an `inHashes` entry outright.
|
|
6
|
+
- **A waiver is never silent.** Each entry carries a required `reason`, every waiver that fires PRINTS with that reason and the entry that owns it (on the FAIL path too, so a partial waive is visible beside what still blocks), and `hooks list` counts the declared waivers. An entry with no reason, with neither `paths` nor `patterns`, or with an unknown key dies at config load through `defineRepo`, and an allowlist file that will not parse waives nothing and WARNs, so a broken waiver leaves the guard stricter rather than looser.
|
|
7
|
+
- **A tracked allowlist never becomes a copy of the private keyword list.** An entry names a keyword by FINGERPRINT (`fp:` plus 12 hex of its sha256), printed beside every hit, which is how a waiver gets written without anyone typing the private word into git. `repo.config.ts` takes the fingerprint form alone and refuses a literal by name. The private machine-global file may use either. The honest limit is stated in the source: a fingerprint stops the list being read off a tracked file and is no defence against a targeted guess.
|
|
8
|
+
- The staged diff is now parsed per FILE (`addedLinesByFile`), so a hit says which file it came from and a waiver can name one. Which file a line belongs to is decided by COUNTING each hunk body from the size its header declares, never by trusting a `+++ …` line wherever it appears: an added line whose own text starts with `++` reaches the parser looking exactly like a file header, and believing one inside a hunk would let ordinary looking text drop every added line after it (`++ /dev/null`) or file the rest under a name a waiver covers (`++ b/bun.lock`). Substring rows match through a case-insensitive regex instead of `indexOf` over a lowercased copy, because one unicode character lowercases into two and slides every offset after it, and those offsets are what decide whether a match sits inside a content hash. A content-hash span is consulted ONLY by an entry that asked for `inHashes`: an api token is also a long base64 run, so skipping those spans by default would silence the guard on the exact shape it exists to catch.
|
|
9
|
+
|
|
3
10
|
## 0.36.0
|
|
4
11
|
|
|
5
12
|
- **`shipkit smoke` — verify the way the end user does, after a deploy.** A read only Playwright pass: it logs in through the app's OWN login page, visits every route `repo.config.ts smoke` declares, samples the body text until it stops growing (a `waitFor` selector first, because a spinner holds the length constant), counts the selector's matches, measures whether the PAGE scrolls sideways, screenshots each route under `tmp/smoke/`, and collects console errors, page errors, failed requests and 4xx/5xx responses. The exit code is the number of failed routes. The password is a secret NAME, read through the repo's new `credsReadCommand` template (the READ twin of `credsCommand`, `{name}` and `{field}` substituted, no shell), never argv, env or config, and `defineRepo` refuses a value typed where the name belongs and a login declared without the reader. Playwright is resolved from the CONSUMER repo at run time, never carried by this package: a repo without it gets one line naming `bun add -d playwright && npx playwright install chromium`. When the host sends `x-pockethost-ratelimit-ip-hourly-*` headers, the report prints the budget and labels a CORS-shaped failure as the backend refusing rather than the app failing, telling the hourly quota from the concurrent cap by the remaining count. Between routes it pauses `paceMs` (default 3000, measured against a 15-concurrent cap: 0 reported 8 of 12 routes, 3000 reports 12 of 12). A route with `open` (a link selector) follows the first match's href after it and judges the landed page as its own row, which is how a detail page with a real id gets visited without the config knowing one. The login form lives at `login.path`, default `/login`. A problem collected during the login is a failed row of its own, and the pause runs before the row label moves, so a previous page's late failure lands on the page that caused it. `emailSecret` is refused when the reader template carries no `{field}`: the account read would answer the value, and the run would type the password into the identity box, where a failed-login journal keeps it.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "rvlib-shipkit",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.37.0",
|
|
4
4
|
"funding": "https://github.com/sponsors/rvion",
|
|
5
5
|
"description": "All-in-one repo companion: ci gate (task DAG), auto-discovered health checks, release-contract fulfillness, asks queue, md-wiki tools, asset atelier, live tui",
|
|
6
6
|
"license": "MIT",
|
package/src/hooks/leak-guard.ts
CHANGED
|
@@ -4,10 +4,23 @@
|
|
|
4
4
|
//
|
|
5
5
|
// the keyword file is deliberately untracked (a list of things to grep for
|
|
6
6
|
// is itself private). absent file → loud skip, never a silent pass.
|
|
7
|
+
//
|
|
8
|
+
// a keyword matches as a SUBSTRING, which is what catches a private name
|
|
9
|
+
// glued into a hostname or an identifier. The cost is a false positive a repo
|
|
10
|
+
// cannot escape: a four letter row lands inside a dependency name or inside a
|
|
11
|
+
// base64 integrity hash in a lockfile, and that lockfile is uncommittable
|
|
12
|
+
// forever. A guard that cries wolf is a guard people answer with --no-verify,
|
|
13
|
+
// which protects nothing. Two answers, in this order. `w:<word>` rows match on
|
|
14
|
+
// word boundaries, which makes the PATTERN smarter rather than the guard
|
|
15
|
+
// looser. An ALLOWLIST waives a known-good match in a named context, and every
|
|
16
|
+
// waiver that fires is PRINTED with its reason, so a reader sees what was let
|
|
17
|
+
// through and why.
|
|
18
|
+
import { createHash } from 'crypto'
|
|
7
19
|
import { existsSync, readFileSync } from 'fs'
|
|
8
|
-
import { join } from 'path'
|
|
20
|
+
import { basename, join } from 'path'
|
|
9
21
|
|
|
10
|
-
export type LeakHit = { where: string; pattern: string }
|
|
22
|
+
export type LeakHit = { where: string; pattern: string; file?: string }
|
|
23
|
+
export type LeakWaiver = LeakHit & { reason: string; source: string }
|
|
11
24
|
|
|
12
25
|
/** what a hit REPORTS. The matched string is private by construction, and a
|
|
13
26
|
* gate that prints it verbatim puts it in a terminal, a CI log and any paste
|
|
@@ -15,16 +28,66 @@ export type LeakHit = { where: string; pattern: string }
|
|
|
15
28
|
export const maskPattern = (p: string): string =>
|
|
16
29
|
p.startsWith('re:') || p.length <= 6 ? p : `${p.slice(0, 3)}…${p.slice(-2)} (${p.length} chars)`
|
|
17
30
|
|
|
31
|
+
/** a stable name for a pattern that does not CARRY it, so a tracked allowlist
|
|
32
|
+
* points at a private keyword without becoming a copy of the private keyword
|
|
33
|
+
* list. Printed beside every hit, which is how a waiver gets written without
|
|
34
|
+
* anyone typing the word.
|
|
35
|
+
*
|
|
36
|
+
* Honest scope: this stops the list being READ off a tracked file. It is no
|
|
37
|
+
* defence against a targeted guess, because a short keyword has a small
|
|
38
|
+
* search space and whoever already suspects the word can confirm it. */
|
|
39
|
+
export const patternFingerprint = (pattern: string): string =>
|
|
40
|
+
'fp:' + createHash('sha256').update(pattern.toLowerCase()).digest('hex').slice(0, 12)
|
|
41
|
+
|
|
42
|
+
/** one known-good match a repo (or this machine) waives.
|
|
43
|
+
*
|
|
44
|
+
* `paths` alone waives every pattern in those files. `patterns` alone waives
|
|
45
|
+
* those patterns everywhere. The PAIR is the shape to reach for: this
|
|
46
|
+
* pattern, in these files, while the same pattern still blocks anywhere else.
|
|
47
|
+
* `inHashes` narrows further, to a match that lands inside a content hash. */
|
|
48
|
+
export type LeakAllowEntry = {
|
|
49
|
+
/** why this match is known good. Printed every time the waiver fires, so
|
|
50
|
+
* a reader months later can judge it. Required. */
|
|
51
|
+
reason: string
|
|
52
|
+
/** staged paths this entry covers: a repo-relative path, a bare basename
|
|
53
|
+
* ('bun.lock'), or a 'dir/' prefix. Absent = every target, the commit
|
|
54
|
+
* message included. */
|
|
55
|
+
paths?: string[]
|
|
56
|
+
/** the patterns this entry waives, named by FINGERPRINT (`fp:` + 12 hex,
|
|
57
|
+
* printed beside every hit). Absent = every pattern, on `paths` alone.
|
|
58
|
+
* A TRACKED source (repo.config.ts) takes the fingerprint form only: a
|
|
59
|
+
* literal there puts the private keyword into git. */
|
|
60
|
+
patterns?: string[]
|
|
61
|
+
/** waive only a match that falls inside a content hash (an npm `sha512-…`
|
|
62
|
+
* integrity field, a 40+ char base64 run). The bytes of a content address
|
|
63
|
+
* are arithmetic, not prose, so a keyword inside one is noise. The same
|
|
64
|
+
* keyword on a normal line of the same file still blocks. */
|
|
65
|
+
inHashes?: boolean
|
|
66
|
+
}
|
|
67
|
+
|
|
18
68
|
export type LeakGuardResult =
|
|
19
69
|
| { status: 'skipped'; reason: string }
|
|
20
|
-
| { status: 'clean'; patterns: number }
|
|
21
|
-
| { status: 'hit'; hits: LeakHit[]; patterns: number }
|
|
70
|
+
| { status: 'clean'; patterns: number; waived: LeakWaiver[]; notes: string[] }
|
|
71
|
+
| { status: 'hit'; hits: LeakHit[]; patterns: number; waived: LeakWaiver[]; notes: string[] }
|
|
72
|
+
|
|
73
|
+
export type ParsedPatterns = {
|
|
74
|
+
substrings: string[]
|
|
75
|
+
words: { word: string; re: RegExp }[]
|
|
76
|
+
regexes: { row: string; re: RegExp }[]
|
|
77
|
+
}
|
|
78
|
+
|
|
79
|
+
const escapeRegex = (s: string): string => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')
|
|
22
80
|
|
|
23
81
|
/** Parse the keyword file: blank lines and `#` comments dropped, `re:<pattern>`
|
|
24
|
-
* rows become case-insensitive regexes,
|
|
25
|
-
* substring.
|
|
26
|
-
|
|
82
|
+
* rows become case-insensitive regexes, `w:<word>` rows match on word
|
|
83
|
+
* boundaries, everything else a case-insensitive substring.
|
|
84
|
+
*
|
|
85
|
+
* A `w:` row REPORTS as the bare word, so its mask and its fingerprint are
|
|
86
|
+
* the same with or without the prefix: adding the prefix to a row never
|
|
87
|
+
* invalidates a waiver already written against it. */
|
|
88
|
+
export function parsePatterns(text: string): ParsedPatterns {
|
|
27
89
|
const substrings: string[] = []
|
|
90
|
+
const words: { word: string; re: RegExp }[] = []
|
|
28
91
|
const regexes: { row: string; re: RegExp }[] = []
|
|
29
92
|
for (const raw of text.split('\n')) {
|
|
30
93
|
const row = raw.trim()
|
|
@@ -32,24 +95,134 @@ export function parsePatterns(text: string): { substrings: string[]; regexes: {
|
|
|
32
95
|
if (row.startsWith('re:')) {
|
|
33
96
|
try { regexes.push({ row, re: new RegExp(row.slice(3), 'i') }) }
|
|
34
97
|
catch { /* a malformed row must not disable the whole guard */ }
|
|
98
|
+
} else if (row.startsWith('w:')) {
|
|
99
|
+
const word = row.slice(2).trim().toLowerCase()
|
|
100
|
+
// \b is ascii-only in JS, which lets an accented name match inside
|
|
101
|
+
// a longer accented word: the boundary is spelled in unicode
|
|
102
|
+
if (word !== '') words.push({ word, re: new RegExp(`(?<![\\p{L}\\p{N}_])${escapeRegex(word)}(?![\\p{L}\\p{N}_])`, 'giu') })
|
|
35
103
|
} else substrings.push(row.toLowerCase())
|
|
36
104
|
}
|
|
37
|
-
return { substrings, regexes }
|
|
105
|
+
return { substrings, words, regexes }
|
|
38
106
|
}
|
|
39
107
|
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
108
|
+
type Span = [start: number, end: number]
|
|
109
|
+
|
|
110
|
+
// a substring row is matched by a case-insensitive REGEX rather than by
|
|
111
|
+
// indexOf over a lowercased copy: one unicode character (İ) lowercases into
|
|
112
|
+
// two, which slides every offset after it, and the offsets are what decide
|
|
113
|
+
// whether a match sits inside a content hash. Compiled once per keyword.
|
|
114
|
+
const subRe = new Map<string, RegExp>()
|
|
115
|
+
const substringRegex = (s: string): RegExp => {
|
|
116
|
+
const hit = subRe.get(s)
|
|
117
|
+
if (hit != null) return hit
|
|
118
|
+
const re = new RegExp(escapeRegex(s), 'gi')
|
|
119
|
+
subRe.set(s, re)
|
|
120
|
+
return re
|
|
44
121
|
}
|
|
45
122
|
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
123
|
+
/** every occurrence of every pattern, keyed by the id a hit reports. */
|
|
124
|
+
function occurrences(haystack: string, p: ParsedPatterns): Map<string, Span[]> {
|
|
125
|
+
const found = new Map<string, Span[]>()
|
|
126
|
+
const push = (id: string, span: Span): void => {
|
|
127
|
+
const prev = found.get(id)
|
|
128
|
+
if (prev == null) found.set(id, [span])
|
|
129
|
+
else prev.push(span)
|
|
130
|
+
}
|
|
131
|
+
for (const s of p.substrings) for (const m of haystack.matchAll(substringRegex(s))) push(s, [m.index, m.index + m[0].length])
|
|
132
|
+
for (const w of p.words) for (const m of haystack.matchAll(w.re)) push(w.word, [m.index, m.index + m[0].length])
|
|
133
|
+
for (const r of p.regexes) {
|
|
134
|
+
const g = new RegExp(r.re.source, r.re.flags.includes('g') ? r.re.flags : r.re.flags + 'g')
|
|
135
|
+
for (const m of haystack.matchAll(g)) push(r.row, [m.index, m.index + m[0].length])
|
|
136
|
+
}
|
|
137
|
+
return found
|
|
138
|
+
}
|
|
139
|
+
|
|
140
|
+
// an npm/yarn integrity field, and any long base64 run. ONLY an entry that
|
|
141
|
+
// asked for `inHashes` consults this: an api token is also a long base64 run,
|
|
142
|
+
// so skipping these by default would be the guard going quiet on the exact
|
|
143
|
+
// shape it exists to catch.
|
|
144
|
+
const HASH_SPAN = /(?:sha\d{1,3}|md5)-[A-Za-z0-9+/=]{16,}|[A-Za-z0-9+/]{40,}={0,2}/gi
|
|
145
|
+
|
|
146
|
+
const hashSpans = (text: string): Span[] => [...text.matchAll(HASH_SPAN)].map((m): Span => [m.index, m.index + m[0].length])
|
|
147
|
+
|
|
148
|
+
const inside = (span: Span, spans: Span[]): boolean => spans.some(([a, b]) => span[0] >= a && span[1] <= b)
|
|
149
|
+
|
|
150
|
+
/** a path selector: a repo-relative path, a bare basename, or a 'dir/' prefix. */
|
|
151
|
+
function pathCovered(file: string | undefined, selectors: string[]): boolean {
|
|
152
|
+
if (file == null) return false
|
|
153
|
+
return selectors.some((s) => s === file || (s.endsWith('/') && file.startsWith(s)) || (!s.includes('/') && basename(file) === s))
|
|
154
|
+
}
|
|
155
|
+
|
|
156
|
+
export type LoadedAllow = LeakAllowEntry & { source: string }
|
|
157
|
+
|
|
158
|
+
/** what the guard reads: one file's added lines, one staged path, or a text
|
|
159
|
+
* the caller handed in (a commit message). */
|
|
160
|
+
export type ScanTarget = { where: string; content: string; file?: string }
|
|
161
|
+
|
|
162
|
+
/** the whole decision, with no git and no disk in it: which matches block,
|
|
163
|
+
* which are waived, by whom, and why. */
|
|
164
|
+
export function evaluateLeak(targets: ScanTarget[], p: ParsedPatterns, allow: LoadedAllow[]): { hits: LeakHit[]; waived: LeakWaiver[] } {
|
|
165
|
+
const hits: LeakHit[] = []
|
|
166
|
+
const waived: LeakWaiver[] = []
|
|
167
|
+
// one row per distinct (where, file, pattern): a keyword hit 40 times in
|
|
168
|
+
// one file is one finding
|
|
169
|
+
const seen = new Set<string>()
|
|
170
|
+
for (const t of targets) {
|
|
171
|
+
const found = occurrences(t.content, p)
|
|
172
|
+
if (found.size === 0) continue
|
|
173
|
+
let hashes: Span[] | null = null
|
|
174
|
+
for (const [id, spans] of found) {
|
|
175
|
+
const key = `${t.where} ${t.file ?? ''} ${id}`
|
|
176
|
+
if (seen.has(key)) continue
|
|
177
|
+
seen.add(key)
|
|
178
|
+
const fp = patternFingerprint(id)
|
|
179
|
+
const entry = allow.find((a) => {
|
|
180
|
+
if (a.paths != null && !pathCovered(t.file, a.paths)) return false
|
|
181
|
+
if (a.patterns != null && !a.patterns.some((x) => x === fp || x.toLowerCase() === id)) return false
|
|
182
|
+
if (a.inHashes === true) {
|
|
183
|
+
hashes ??= hashSpans(t.content)
|
|
184
|
+
if (!spans.every((s) => inside(s, hashes!))) return false
|
|
185
|
+
}
|
|
186
|
+
return true
|
|
187
|
+
})
|
|
188
|
+
if (entry == null) hits.push({ where: t.where, pattern: id, file: t.file })
|
|
189
|
+
else waived.push({ where: t.where, pattern: id, file: t.file, reason: entry.reason, source: entry.source })
|
|
190
|
+
}
|
|
191
|
+
}
|
|
192
|
+
return { hits, waived }
|
|
193
|
+
}
|
|
194
|
+
|
|
195
|
+
const ALLOW_KEYS = ['reason', 'paths', 'patterns', 'inHashes']
|
|
196
|
+
const FINGERPRINT = /^fp:[0-9a-f]{12}$/
|
|
197
|
+
|
|
198
|
+
/** why an entry cannot be used, or null. `tracked` = the source is committed
|
|
199
|
+
* (repo.config.ts), where a literal pattern publishes the private word the
|
|
200
|
+
* guard exists to keep out of git. */
|
|
201
|
+
export function leakAllowProblem(entry: unknown, where: string, tracked: boolean): string | null {
|
|
202
|
+
if (entry == null || typeof entry !== 'object' || Array.isArray(entry)) return `${where}: every entry is an object`
|
|
203
|
+
const e = entry as Record<string, unknown>
|
|
204
|
+
for (const k of Object.keys(e)) if (!ALLOW_KEYS.includes(k)) return `${where}: unknown key '${k}' (known: ${ALLOW_KEYS.join(', ')})`
|
|
205
|
+
if (typeof e['reason'] !== 'string' || e['reason'].trim() === '') return `${where}: needs a 'reason' — a waiver nobody can judge later is a hole`
|
|
206
|
+
if (e['paths'] == null && e['patterns'] == null)
|
|
207
|
+
return `${where}: needs 'paths' or 'patterns' — an entry with neither waives every pattern in every file, which is the guard switched off`
|
|
208
|
+
for (const k of ['paths', 'patterns'] as const) {
|
|
209
|
+
const v = e[k]
|
|
210
|
+
if (v == null) continue
|
|
211
|
+
if (!Array.isArray(v) || v.length === 0 || v.some((x) => typeof x !== 'string' || x.trim() === ''))
|
|
212
|
+
return `${where}: '${k}' is a non-empty array of non-empty strings`
|
|
213
|
+
}
|
|
214
|
+
if (e['inHashes'] != null && typeof e['inHashes'] !== 'boolean') return `${where}: 'inHashes' is a boolean`
|
|
215
|
+
if (tracked)
|
|
216
|
+
for (const x of (e['patterns'] as string[] | undefined) ?? [])
|
|
217
|
+
if (!FINGERPRINT.test(x))
|
|
218
|
+
return `${where}: '${x}' — a tracked allowlist names a pattern by FINGERPRINT (fp:…, printed beside every hit), never by value, or the private keyword lands in git`
|
|
219
|
+
return null
|
|
220
|
+
}
|
|
221
|
+
|
|
222
|
+
/** every problem in a declared allowlist. `defineRepo` throws on the first. */
|
|
223
|
+
export function leakAllowProblems(entries: unknown, where: string, tracked: boolean): string[] {
|
|
224
|
+
if (!Array.isArray(entries)) return [`${where}: leakAllow is an array of entries`]
|
|
225
|
+
return entries.map((e, i) => leakAllowProblem(e, `${where}[${i}]`, tracked)).filter((x): x is string => x != null)
|
|
53
226
|
}
|
|
54
227
|
|
|
55
228
|
/** the machine-global list every repo's guard also reads: one private list
|
|
@@ -62,36 +235,138 @@ export function globalKeywordFile(): string | null {
|
|
|
62
235
|
return existsSync(p) ? p : null
|
|
63
236
|
}
|
|
64
237
|
|
|
238
|
+
/** the machine-global allowlist, the twin of the keyword file. It is private,
|
|
239
|
+
* so it may name a pattern by value as well as by fingerprint. */
|
|
240
|
+
export function globalAllowFile(): string | null {
|
|
241
|
+
const home = process.env.HOME ?? ''
|
|
242
|
+
if (home === '') return null
|
|
243
|
+
const p = join(home, '.rv-private', 'leak-allow.json')
|
|
244
|
+
return existsSync(p) ? p : null
|
|
245
|
+
}
|
|
246
|
+
|
|
247
|
+
/** read an allowlist file (`LeakAllowEntry[]` as JSON). A file that will not
|
|
248
|
+
* parse, or an entry that cannot be used, is a NOTE and never a failure: a
|
|
249
|
+
* waiver that does not load leaves the guard stricter, never looser. */
|
|
250
|
+
export function loadAllowFile(path: string, tracked: boolean): { entries: LoadedAllow[]; notes: string[] } {
|
|
251
|
+
let parsed: unknown
|
|
252
|
+
try { parsed = JSON.parse(readFileSync(path, 'utf8')) }
|
|
253
|
+
catch (e) { return { entries: [], notes: [`${path} is not readable JSON (${e instanceof Error ? e.message.split('\n')[0] : String(e)}) — its waivers are IGNORED`] } }
|
|
254
|
+
if (!Array.isArray(parsed)) return { entries: [], notes: [`${path} holds an array of allowlist entries — its waivers are IGNORED`] }
|
|
255
|
+
const entries: LoadedAllow[] = []
|
|
256
|
+
const notes: string[] = []
|
|
257
|
+
parsed.forEach((raw, i) => {
|
|
258
|
+
const problem = leakAllowProblem(raw, `${path}[${i}]`, tracked)
|
|
259
|
+
if (problem != null) notes.push(`${problem} — entry IGNORED`)
|
|
260
|
+
else entries.push({ ...(raw as LeakAllowEntry), source: `${basename(path)}[${i}]` })
|
|
261
|
+
})
|
|
262
|
+
return { entries, notes }
|
|
263
|
+
}
|
|
264
|
+
|
|
265
|
+
export type LeakGuardOpts = {
|
|
266
|
+
/** path to the keyword file, relative to root (default '.rv-private/banned-keywords.txt') */
|
|
267
|
+
keywordFile?: string
|
|
268
|
+
/** extra text to scan (a commit message, a release note) */
|
|
269
|
+
text?: { label: string; content: string }[]
|
|
270
|
+
/** scan the staged diff + staged paths (default true) */
|
|
271
|
+
staged?: boolean
|
|
272
|
+
/** the repo's declared waivers (repo.config.ts hooks.leakAllow). Tracked,
|
|
273
|
+
* so fingerprints only. */
|
|
274
|
+
allow?: LeakAllowEntry[]
|
|
275
|
+
/** override the machine-global allowlist path; null reads none */
|
|
276
|
+
allowFile?: string | null
|
|
277
|
+
}
|
|
278
|
+
|
|
279
|
+
/** git c-quotes a path that carries a space or a non-ascii byte. */
|
|
280
|
+
function unquotePath(p: string): string {
|
|
281
|
+
if (!p.startsWith('"')) return p
|
|
282
|
+
try { return JSON.parse(p) as string } catch { return p.slice(1, -1) }
|
|
283
|
+
}
|
|
284
|
+
|
|
285
|
+
/** `@@ -old,n +new,m @@`, the only line that says how long a hunk body is. */
|
|
286
|
+
const HUNK_HEADER = /^@@ -\d+(?:,(\d+))? \+\d+(?:,(\d+))? @@/
|
|
287
|
+
|
|
288
|
+
/** ADDED lines only, grouped by the file that receives them: what this commit
|
|
289
|
+
* puts INTO the object store. Removed and context lines are already in
|
|
290
|
+
* history, and a commit that DELETES private content must never be blocked by
|
|
291
|
+
* that content. The grouping is what lets a waiver name a file.
|
|
292
|
+
*
|
|
293
|
+
* A `+++ x` line is a file header only OUTSIDE a hunk body. Inside one it is
|
|
294
|
+
* an added line whose own text starts with `++`, and reading it as a header
|
|
295
|
+
* is a hole in both directions: `++ /dev/null` would drop every added line
|
|
296
|
+
* after it in that file, and `++ b/bun.lock` would file them under a name a
|
|
297
|
+
* waiver covers. So the hunk body is COUNTED, from the size its header
|
|
298
|
+
* declares. A header this does not recognise (a combined `@@@` diff) opens an
|
|
299
|
+
* unbounded body that only the next file closes, which is the safe direction:
|
|
300
|
+
* content is still scanned, and no line inside it can name a file. */
|
|
301
|
+
export function addedLinesByFile(diff: string): { file: string; content: string }[] {
|
|
302
|
+
const out: { file: string; content: string }[] = []
|
|
303
|
+
let current: { file: string; lines: string[] } | null = null
|
|
304
|
+
let oldLeft = 0
|
|
305
|
+
let newLeft = 0
|
|
306
|
+
const flush = (): void => {
|
|
307
|
+
if (current != null && current.lines.length > 0) out.push({ file: current.file, content: current.lines.join('\n') })
|
|
308
|
+
current = null
|
|
309
|
+
}
|
|
310
|
+
const add = (line: string): void => { if (current != null) current.lines.push(line.slice(1)) }
|
|
311
|
+
for (const line of diff.split('\n')) {
|
|
312
|
+
// a `diff --git ` at column 0 can never be diff CONTENT: a content line
|
|
313
|
+
// always carries its ' ', '+' or '-' prefix first
|
|
314
|
+
if (line.startsWith('diff --git ')) {
|
|
315
|
+
flush()
|
|
316
|
+
oldLeft = 0
|
|
317
|
+
newLeft = 0
|
|
318
|
+
} else if (oldLeft > 0 || newLeft > 0) {
|
|
319
|
+
if (line.startsWith('+')) { newLeft--; add(line) }
|
|
320
|
+
else if (line.startsWith('-')) oldLeft--
|
|
321
|
+
else if (!line.startsWith('\\')) { oldLeft--; newLeft-- } // '' counts for neither
|
|
322
|
+
} else if (line.startsWith('@@')) {
|
|
323
|
+
const h = HUNK_HEADER.exec(line)
|
|
324
|
+
oldLeft = h == null ? Infinity : h[1] == null ? 1 : Number(h[1])
|
|
325
|
+
newLeft = h == null ? Infinity : h[2] == null ? 1 : Number(h[2])
|
|
326
|
+
} else if (line.startsWith('+++ ')) {
|
|
327
|
+
flush()
|
|
328
|
+
const p = unquotePath(line.slice(4).trim())
|
|
329
|
+
current = p === '/dev/null' ? null : { file: p.startsWith('b/') ? p.slice(2) : p, lines: [] }
|
|
330
|
+
} else if (line.startsWith('+')) add(line) // a stray added line outside any hunk is still content
|
|
331
|
+
}
|
|
332
|
+
flush()
|
|
333
|
+
return out
|
|
334
|
+
}
|
|
335
|
+
|
|
65
336
|
export function runLeakGuard(root: string, o: LeakGuardOpts = {}): LeakGuardResult {
|
|
66
337
|
const rel = o.keywordFile ?? '.rv-private/banned-keywords.txt'
|
|
67
338
|
const file = join(root, rel)
|
|
68
339
|
const sources = [existsSync(file) ? file : null, globalKeywordFile()].filter((f): f is string => f != null)
|
|
69
340
|
if (sources.length === 0) return { status: 'skipped', reason: `${rel} not found and no ~/.rv-private/banned-keywords.txt — no keyword list, no leak guard` }
|
|
70
341
|
const patterns = parsePatterns(sources.map((f) => readFileSync(f, 'utf8')).join('\n'))
|
|
71
|
-
const count = patterns.substrings.length + patterns.regexes.length
|
|
342
|
+
const count = patterns.substrings.length + patterns.words.length + patterns.regexes.length
|
|
72
343
|
if (count === 0) return { status: 'skipped', reason: `keyword file(s) have no patterns (${sources.join(', ')})` }
|
|
73
344
|
|
|
74
|
-
const
|
|
345
|
+
const notes: string[] = []
|
|
346
|
+
const allow: LoadedAllow[] = []
|
|
347
|
+
;(o.allow ?? []).forEach((e, i) => {
|
|
348
|
+
const problem = leakAllowProblem(e, `repo.config.ts hooks.leakAllow[${i}]`, true)
|
|
349
|
+
if (problem != null) notes.push(`${problem} — entry IGNORED`)
|
|
350
|
+
else allow.push({ ...e, source: `repo.config.ts hooks.leakAllow[${i}]` })
|
|
351
|
+
})
|
|
352
|
+
const globalAllow = o.allowFile === undefined ? globalAllowFile() : o.allowFile
|
|
353
|
+
if (globalAllow != null && existsSync(globalAllow)) {
|
|
354
|
+
const g = loadAllowFile(globalAllow, false)
|
|
355
|
+
allow.push(...g.entries)
|
|
356
|
+
notes.push(...g.notes)
|
|
357
|
+
}
|
|
358
|
+
|
|
359
|
+
const targets: ScanTarget[] = []
|
|
75
360
|
if (o.staged !== false) {
|
|
76
|
-
// staged CONTENT (what lands in the object store) and staged PATHS
|
|
77
|
-
// (a filename can leak as loudly as a line of code)
|
|
78
361
|
const diff = Bun.spawnSync(['git', 'diff', '--cached'], { cwd: root, maxBuffer: 256 * 1024 * 1024 })
|
|
79
|
-
if (diff.exitCode === 0)
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
// deletes private content must never be blocked by that content
|
|
83
|
-
const added = diff.stdout.toString().split('\n')
|
|
84
|
-
.filter((l) => l.startsWith('+') && !l.startsWith('+++'))
|
|
85
|
-
.join('\n')
|
|
86
|
-
scan(added, 'staged content', patterns, hits)
|
|
87
|
-
}
|
|
362
|
+
if (diff.exitCode === 0)
|
|
363
|
+
for (const f of addedLinesByFile(diff.stdout.toString())) targets.push({ where: 'staged content', file: f.file, content: f.content })
|
|
364
|
+
// a filename can leak as loudly as a line of code
|
|
88
365
|
const names = Bun.spawnSync(['git', 'diff', '--cached', '--name-only'], { cwd: root })
|
|
89
|
-
if (names.exitCode === 0)
|
|
366
|
+
if (names.exitCode === 0) for (const n of names.stdout.toString().split('\n')) if (n.trim() !== '') targets.push({ where: 'staged path', file: n, content: n })
|
|
90
367
|
}
|
|
91
|
-
for (const t of o.text ?? [])
|
|
368
|
+
for (const t of o.text ?? []) targets.push({ where: t.label, content: t.content })
|
|
92
369
|
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
const unique = hits.filter((h) => { const k = h.where + ' ' + h.pattern; if (seen.has(k)) return false; seen.add(k); return true })
|
|
96
|
-
return unique.length > 0 ? { status: 'hit', hits: unique, patterns: count } : { status: 'clean', patterns: count }
|
|
370
|
+
const { hits, waived } = evaluateLeak(targets, patterns, allow)
|
|
371
|
+
return hits.length > 0 ? { status: 'hit', hits, patterns: count, waived, notes } : { status: 'clean', patterns: count, waived, notes }
|
|
97
372
|
}
|
package/src/hooks/run-hooks.ts
CHANGED
|
@@ -7,7 +7,7 @@ import { existsSync, mkdirSync, readFileSync, realpathSync, writeFileSync } from
|
|
|
7
7
|
import { dirname, join } from 'path'
|
|
8
8
|
import { fileURLToPath } from 'url'
|
|
9
9
|
import { commitMessageOffenses } from './commit-message.ts'
|
|
10
|
-
import { globalKeywordFile, maskPattern, runLeakGuard } from './leak-guard.ts'
|
|
10
|
+
import { globalAllowFile, globalKeywordFile, maskPattern, patternFingerprint, runLeakGuard, type LeakAllowEntry, type LeakHit, type LeakWaiver } from './leak-guard.ts'
|
|
11
11
|
import { runRootIdentity } from './root-identity.ts'
|
|
12
12
|
import { spawnWithTimeout } from '../core/spawn-timeout.ts'
|
|
13
13
|
import type { RepoConfig } from '../repo-config.ts'
|
|
@@ -21,6 +21,14 @@ export type HooksConfig = {
|
|
|
21
21
|
commitMsg?: { id: string; cmd: string[] }[]
|
|
22
22
|
/** keyword file for the leak guard (default '.rv-private/banned-keywords.txt') */
|
|
23
23
|
keywordFile?: string
|
|
24
|
+
/** known-good leak-guard matches this repo waives: a lockfile whose
|
|
25
|
+
* dependency names or integrity hashes collide with a keyword is the
|
|
26
|
+
* motivating case. Each entry carries a reason, and a waiver that fires
|
|
27
|
+
* PRINTS, so nothing passes silently. This file is tracked, so an entry
|
|
28
|
+
* names a pattern by fingerprint (`fp:…`, printed beside every hit) and
|
|
29
|
+
* never by value. The machine-global twin is ~/.rv-private/leak-allow.json.
|
|
30
|
+
* hooks/leak-guard.ts */
|
|
31
|
+
leakAllow?: LeakAllowEntry[]
|
|
24
32
|
/** staged-path prefixes that may never enter git (local state like
|
|
25
33
|
* '.comfy-ts/'). gitignore is advisory and editable; this gate is not. */
|
|
26
34
|
blockedPaths?: string[]
|
|
@@ -148,6 +156,29 @@ function seedKeywordFile(root: string, keywordFile?: string): string | null {
|
|
|
148
156
|
|
|
149
157
|
export type GateOutcome = { id: string; ok: boolean; lines: string[] }
|
|
150
158
|
|
|
159
|
+
/** a hit line: the masked pattern plus its FINGERPRINT, which is the token a
|
|
160
|
+
* waiver is written against, so nobody has to type the private word to
|
|
161
|
+
* exempt it. The file rides along for a content hit; a staged-PATH hit adds
|
|
162
|
+
* nothing, because there the path IS what matched. */
|
|
163
|
+
const hitLine = (x: LeakHit): string =>
|
|
164
|
+
` ${x.where}${x.where === 'staged content' && x.file != null ? ` ${x.file}` : ''}: ${maskPattern(x.pattern)} [${patternFingerprint(x.pattern)}]`
|
|
165
|
+
|
|
166
|
+
/** what the allowlist let through, one line per waiver. An allowlist that
|
|
167
|
+
* fires invisibly is a silent weakening, so a waived match is as loud as a
|
|
168
|
+
* blocked one and carries the reason its author wrote. */
|
|
169
|
+
function waivedLines(waived: LeakWaiver[]): string[] {
|
|
170
|
+
const rows = new Map<string, { pattern: string; reason: string; source: string; targets: Set<string> }>()
|
|
171
|
+
for (const w of waived) {
|
|
172
|
+
const key = `${w.pattern} ${w.source}`
|
|
173
|
+
const row = rows.get(key) ?? { pattern: w.pattern, reason: w.reason, source: w.source, targets: new Set<string>() }
|
|
174
|
+
row.targets.add(w.file ?? w.where)
|
|
175
|
+
rows.set(key, row)
|
|
176
|
+
}
|
|
177
|
+
return [...rows.values()].map(
|
|
178
|
+
(r) => ` waived ${maskPattern(r.pattern)} [${patternFingerprint(r.pattern)}] in ${r.targets.size} target(s) — ${r.reason} (${r.source})`,
|
|
179
|
+
)
|
|
180
|
+
}
|
|
181
|
+
|
|
151
182
|
/** the pre-commit gate: built-in leak guard, then the repo's own commands.
|
|
152
183
|
* A commit takes ~1s (build/shipkit.md): the gate times itself, and going
|
|
153
184
|
* over budget WARNs naming the fix, so drift back to slow commits is loud. */
|
|
@@ -156,16 +187,27 @@ export async function runPreCommit(root: string, config: RepoConfig): Promise<Ga
|
|
|
156
187
|
const h = config.hooks ?? {}
|
|
157
188
|
const out: GateOutcome[] = []
|
|
158
189
|
if (!(h.disable ?? []).includes('leak-guard')) {
|
|
159
|
-
const r = runLeakGuard(root, { keywordFile: h.keywordFile })
|
|
190
|
+
const r = runLeakGuard(root, { keywordFile: h.keywordFile, allow: h.leakAllow })
|
|
160
191
|
if (r.status === 'skipped') out.push({ id: 'leak-guard', ok: true, lines: [`skip: leak-guard (${r.reason})`] })
|
|
161
|
-
else if (r.status === 'clean') out.push({
|
|
192
|
+
else if (r.status === 'clean') out.push({
|
|
193
|
+
id: 'leak-guard',
|
|
194
|
+
ok: true,
|
|
195
|
+
lines: [
|
|
196
|
+
`ok: leak-guard clean (${r.patterns} pattern(s)${r.waived.length === 0 ? '' : `, ${r.waived.length} waived`})`,
|
|
197
|
+
...waivedLines(r.waived),
|
|
198
|
+
...r.notes.map((n) => ` WARN ${n}`),
|
|
199
|
+
],
|
|
200
|
+
})
|
|
162
201
|
else out.push({
|
|
163
202
|
id: 'leak-guard',
|
|
164
203
|
ok: false,
|
|
165
204
|
lines: [
|
|
166
205
|
`FAIL: leak-guard matched ${r.hits.length} banned pattern(s) — this commit would leak private content`,
|
|
167
|
-
...r.hits.map(
|
|
206
|
+
...r.hits.map(hitLine),
|
|
207
|
+
...waivedLines(r.waived),
|
|
208
|
+
...r.notes.map((n) => ` WARN ${n}`),
|
|
168
209
|
` fix: remove the content, or edit the keyword file if the pattern is wrong`,
|
|
210
|
+
` a known-good match: repo.config.ts hooks.leakAllow += { reason, paths, patterns: ['<the fp: above>'] }`,
|
|
169
211
|
],
|
|
170
212
|
})
|
|
171
213
|
}
|
|
@@ -242,9 +284,11 @@ export async function runCommitMsg(root: string, config: RepoConfig, messageFile
|
|
|
242
284
|
// the leak guard also reads the MESSAGE, a private name pasted into a
|
|
243
285
|
// commit body is as public as one pasted into a file
|
|
244
286
|
if (!(h.disable ?? []).includes('leak-guard')) {
|
|
245
|
-
const r = runLeakGuard(root, { keywordFile: h.keywordFile, staged: false, text: [{ label: 'commit message', content: message }] })
|
|
287
|
+
const r = runLeakGuard(root, { keywordFile: h.keywordFile, staged: false, allow: h.leakAllow, text: [{ label: 'commit message', content: message }] })
|
|
246
288
|
if (r.status === 'hit')
|
|
247
|
-
out.push({ id: 'leak-guard', ok: false, lines: [`FAIL: leak-guard matched in the commit message`, ...r.hits.map(
|
|
289
|
+
out.push({ id: 'leak-guard', ok: false, lines: [`FAIL: leak-guard matched in the commit message`, ...r.hits.map(hitLine), ...waivedLines(r.waived)] })
|
|
290
|
+
else if (r.status === 'clean' && r.waived.length > 0)
|
|
291
|
+
out.push({ id: 'leak-guard', ok: true, lines: [`ok: leak-guard clean in the commit message (${r.waived.length} waived)`, ...waivedLines(r.waived)] })
|
|
248
292
|
}
|
|
249
293
|
for (const c of h.commitMsg ?? []) {
|
|
250
294
|
const r = await spawnWithTimeout([...c.cmd, messageFile], { cwd: root, timeoutMs: 60_000 })
|
|
@@ -265,8 +309,12 @@ export function listGates(config: RepoConfig, root = process.cwd()): string[] {
|
|
|
265
309
|
// a gate listed as active but unable to fire is worse than no listing:
|
|
266
310
|
// this surface exists so nobody has to guess
|
|
267
311
|
const leakState = off.has('leak-guard') ? ' (DISABLED by config)' : existsSync(join(root, kw)) || globalKeywordFile() != null ? '' : ` (INACTIVE, ${kw} absent and no ~/.rv-private list)`
|
|
312
|
+
// a waiver nobody can enumerate is a hole nobody can audit: the count of
|
|
313
|
+
// declared exemptions belongs on the same surface as the gate itself
|
|
314
|
+
const waivers = (h.leakAllow ?? []).length + (globalAllowFile() != null ? 1 : 0)
|
|
315
|
+
const leakAllowState = waivers === 0 ? '' : `, ${(h.leakAllow ?? []).length} repo waiver(s)${globalAllowFile() == null ? '' : ' + ~/.rv-private/leak-allow.json'}`
|
|
268
316
|
return [
|
|
269
|
-
`pre-commit leak-guard${leakState} — staged added lines + paths vs ${kw} + ~/.rv-private/banned-keywords.txt`,
|
|
317
|
+
`pre-commit leak-guard${leakState} — staged added lines + paths vs ${kw} + ~/.rv-private/banned-keywords.txt${leakAllowState}`,
|
|
270
318
|
...((h.blockedPaths ?? []).length > 0 ? [`pre-commit blocked-paths — nothing under ${h.blockedPaths!.join(', ')} enters git`] : []),
|
|
271
319
|
`pre-commit root-identity${off.has('root-identity') ? ' (DISABLED by config)' : ''} — the repo's own name in package.json + repo.config.ts, and scratch files`,
|
|
272
320
|
...(h.preCommit ?? []).map((c) => `pre-commit ${c.id} — ${c.cmd.join(' ')}`),
|
package/src/repo-config.ts
CHANGED
|
@@ -22,6 +22,7 @@ import { validateSmokeConfig, type SmokeConfig } from './smoke/run-smoke.ts'
|
|
|
22
22
|
import { contractsFulfillness } from './rules/SK002-contracts-fulfillness.ts'
|
|
23
23
|
import { BUILTIN_RULE_IDS, validateRules } from './rules/rule-registry.ts'
|
|
24
24
|
import type { Rule } from './rules/rule.ts'
|
|
25
|
+
import { leakAllowProblems } from './hooks/leak-guard.ts'
|
|
25
26
|
import type { HooksConfig } from './hooks/run-hooks.ts'
|
|
26
27
|
import type { ReleaseConfig } from './release/run-release.ts'
|
|
27
28
|
import { UNIT_DECL_KEYS, type ReleaseUnitDecl } from './release/units.ts'
|
|
@@ -202,6 +203,12 @@ export function defineRepo(c: RepoConfig): RepoConfig {
|
|
|
202
203
|
if (c.rules != null) validateRules(c.rules, { label: 'defineRepo: rules', taken: BUILTIN_RULE_IDS })
|
|
203
204
|
// a smoke typo would land as a run that visits nothing and exits 0
|
|
204
205
|
if (c.smoke != null) validateSmokeConfig(c.smoke)
|
|
206
|
+
// a leak-guard waiver is the one config row that can make a gate quieter,
|
|
207
|
+
// so a malformed one dies here rather than being dropped at commit time
|
|
208
|
+
if (c.hooks?.leakAllow != null) {
|
|
209
|
+
const problems = leakAllowProblems(c.hooks.leakAllow, 'defineRepo: hooks.leakAllow', true)
|
|
210
|
+
if (problems.length > 0) throw new Error(problems[0])
|
|
211
|
+
}
|
|
205
212
|
if (c.credsReadCommand != null && !/\{name\}/.test(c.credsReadCommand))
|
|
206
213
|
throw new Error(`defineRepo: credsReadCommand must carry {name} (and {field} for get | account), e.g. 'my-secrets {field} {name}'`)
|
|
207
214
|
if (c.smoke?.login != null && (c.credsReadCommand ?? '') === '')
|