codebase-snapshot 0.9.9 → 0.9.14

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -21,6 +21,879 @@ Two versions exist on the registry that are not releases, and both are deprecate
21
21
  Deleting them requires account-level rights that the release pipeline does not have, so they are
22
22
  deprecated with an explanatory message instead. `latest` has pointed at 0.9.0 since it existed.
23
23
 
24
+ ---
25
+
26
+ ## 0.9.14
27
+
28
+ A fifth audit, of what the tool does to text it did not write. Three of the defects below were
29
+ editing the user's code: a redaction rule erased 36 lines of source, a truncation left a code fence
30
+ open, and a preview budget did not apply to a file with CR line endings. The rest are credentials
31
+ left in clear text, two records that disagreed with the files beside them, and the performance work
32
+ the previous audits kept deferring.
33
+
34
+ Every item has a test that fails without it, with the exception of two performance changes that have
35
+ no behavioural signal — both are named.
36
+
37
+ ### A redaction rule erased source code
38
+
39
+ `PEM_BLOCK_PATTERN` is lazy over `[\s\S]`, which places no restriction on what may lie between a
40
+ `BEGIN` and the next `END`. A file holding an *unterminated* header — a truncated key, a fixture, a
41
+ half-written file — and, later, a properly terminated block had **everything between the two
42
+ headers replaced with blank lines**, including the source code that happened to sit there.
43
+
44
+ This repository did it to its own test suite: the security fixture's "unterminated key" test was
45
+ followed by another test whose key was terminated, and the generator bridged the two — 36 lines of
46
+ assertions, braces and the second test's declaration, replaced with 36 empty newlines.
47
+
48
+ The body may no longer contain another `-----BEGIN`. A header with no `END` before the next `BEGIN`
49
+ does not match as a block, and `PEM_UNTERMINATED_PATTERN` handles its header and its own base64 body,
50
+ which is what that pattern is for. The newline after the header is deliberately still not required:
51
+ a PEM inside a JSON or `.env` value is written with escaped `\n`, two characters rather than a line
52
+ break, and a rule that insisted on one would mask the header alone and leave the base64 in clear.
53
+
54
+ ### A truncated section left its code fence open
55
+
56
+ When the fixed metadata of a section would not fit the part budget, the *rendered* section was cut at
57
+ a byte offset. The cut lands in the body or in the opening fence, and the closing fence it removes is
58
+ the one that ends the block: every section after it was rendered inside this file's code block, and
59
+ `parsePartSections` merged the next file into this one's body.
60
+
61
+ The body now goes and the notice takes its place inside the block, with the file marked `preview`
62
+ first so the re-rendered `- Meta:` line describes what the reader will actually find. A section that
63
+ is a closed, empty, labelled block is readable; an unclosed fence corrupts the files after it, and no
64
+ block at all is invisible to the part reader.
65
+
66
+ ### A file with CR line endings was never previewed
67
+
68
+ `countLines` and `truncatePreview` split on `\n` alone, so a CR-only file — an old Mac file, anything
69
+ written by a tool that predates LF on that platform — counted as **one line**. `largeFileBytes`
70
+ therefore never fired: a 5 MB CR file went into a part whole, with its carriage returns intact, and
71
+ `manifest.json` reported `lines: 1` for a 40 000-line file. The redaction pass counted terminators
72
+ with `/\r\n|\n|\r/g`, so `reports/audit.md` reported line numbers the manifest disagreed with.
73
+
74
+ All four places now normalise CR, LF and CRLF, and `countLines` counts by scanning rather than
75
+ splitting — it runs on every file body and every part, and normalising first allocated a second copy
76
+ of a multi-megabyte string to answer a question about its terminators.
77
+
78
+ ### Four credential shapes were left in clear text
79
+
80
+ **`${VAR:-fallback}`, unquoted.** `POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-MyProdPassword123!}`
81
+ defeated every rule, and each refusal was individually reasonable: the quoted rules need a quote
82
+ after the separator, and the unquoted ones need the value to begin with a word character — a
83
+ fallback begins with `$`. That idiom is the norm in `.env` files, docker-compose, GitHub Actions and
84
+ shell scripts. A rule for the shape now masks the fallback and keeps the variable name.
85
+
86
+ **`readonly` and `declare` in shell files.** Those words introduce a binding in TypeScript and an
87
+ assignment in POSIX sh and `.envrc` — the same word, opposite meaning. The guard that suppresses
88
+ redaction for declarations therefore suppressed it for exactly the files most likely to hold a
89
+ credential. Bare `declare` is gone from the keyword list (the form that can initialise a binding in
90
+ TS is `declare const`, matched through `const`), and the guard now knows which files are assignments.
91
+
92
+ **A credential inside a string literal.** `startsInsideStringLiteral` rejected any match whose name
93
+ began immediately after a quote, which is right for a list of key names and wrong for every
94
+ connection string: `const dsn = "password=Secret;host=db"` and `curl -d "password=Secret"` were
95
+ emitted in clear. The test now also asks whether the literal closes between the name and the value.
96
+
97
+ **The MCP line cap.** `createLineReader` checked `MAX_LINE_BYTES` only on the buffer it had just
98
+ appended to, so a chunk carrying a discarded line's terminator *and* the start of the next message
99
+ left that remainder unmeasured until the stream ended — and `onEnd` delivered it whole to
100
+ `JSON.parse`. It also compared `.length`, which is UTF-16 code units: a line of astral characters
101
+ could be twice the ceiling in bytes and still pass. The cap is applied at every path that can
102
+ deliver, and measured in bytes.
103
+
104
+ ### `--with-dependents` lost an importer, and three languages had no import graph
105
+
106
+ The closure was computed from `file.summary.imports`, and a previewed or oversized file has no
107
+ summary. An importer over `largeFileBytes` was therefore dropped as unchanged while `validate` and
108
+ `explain`, which read the same file from disk, called it a dependent — generate and validate
109
+ disagreed about the snapshot generate had just written. Import specifiers are cheap to extract from
110
+ text already in hand, so they are now collected for files whose summary was refused, and the size
111
+ ceiling lives in one place so all three commands refuse the same files.
112
+
113
+ `resolveSpecifier` only understood JavaScript's `./x` shape, so the Python, Ruby, C/C++ and Dart
114
+ imports the profiles have been recording for several releases resolved to nothing. Each language
115
+ spells "this directory" differently — Python with dots, Ruby with `require_relative`, C with a
116
+ quoted include, Dart with nothing at all — and each is now resolved on its own terms. Package-rooted
117
+ languages (Go modules, Rust `crate::`, PHP PSR-4, JVM packages, Swift modules) stay unresolved on
118
+ purpose: their specifiers go through a manifest, and guessing a path would fabricate edges that
119
+ `validate` then has to disagree about.
120
+
121
+ ### `setup --format json` could corrupt its own output
122
+
123
+ `setup` replaced `process.stdout.write` with a capture and restored the function it had captured on
124
+ entry. A nested generate with an `async` hook installed the hook diversion inside that window, so
125
+ `setup`'s restore ran first and removed it — the hook's `console.log`, after its first `await`, went
126
+ straight to the real stdout — and the hook's restore then put back `setup`'s *finished* capture.
127
+ Every later write in the process was appended to a dead string and silently discarded.
128
+
129
+ Both borrow stdout and neither can own it, so there is now one router, installed at most once and
130
+ released by reference counting, consulting two `AsyncLocalStorage` scopes.
131
+
132
+ The same function also merged a failed nested generate in as `manifest`: `{ setup, manifest: { error } }`
133
+ came out with exit code 1 and no top-level `error`. The failure is now at the top level.
134
+
135
+ ### A transient lock on the old snapshot could cost the new one
136
+
137
+ `fs.rmSync(backupDir)` ran inside the `try` whose `catch` restores the previous snapshot. `rmSync`
138
+ throws EPERM or EBUSY on Windows and on network filesystems when an indexer, an antivirus or an
139
+ editor holds a handle in the old snapshot — and by then `outDir` already holds the new one. The catch
140
+ deleted the new snapshot, restored the old one and rethrew; the retry then failed with ENOENT
141
+ because the staging directory no longer existed. Removing the backup is no longer part of the
142
+ transaction.
143
+
144
+ ### Two runs could take two different lock files
145
+
146
+ `realpathSync(stagingRoot)` was used when the root existed and the unresolved path when it did not —
147
+ and the run creates the root *after* taking the lock. On macOS, where `/var` is a symlink to
148
+ `/private/var`, process 1 hashed `/var/folders/…/snapshot.tmp` and process 2, starting once the
149
+ directory existed, hashed `/private/var/…`: two locks, no mutual exclusion, two concurrent writers.
150
+ The key now resolves the nearest existing ancestor. Reclaiming a stale lock also re-checks the file's
151
+ identity immediately before unlinking it, so two runs that both find the same dead lock cannot
152
+ delete each other's.
153
+
154
+ ### The MCP worker ran the hook the docs say it does not
155
+
156
+ `docs/mcp.md` tells the reader that `hooks.afterGenerate` does not run for a generation started from
157
+ MCP. It did: the worker resolved the config file with its hooks intact, so an after-generate script
158
+ ran from inside an MCP tool call, and a hook still running when the worker posted its result kept
159
+ the thread alive for as long as the server did. The worker now empties `hooks` before resolving, and
160
+ is terminated as soon as the result is settled.
161
+
162
+ ### A byte-order mark in `package.json` broke four readers
163
+
164
+ `findConfigFile` reads `package.json` through `stripJsonComments`, which drops a BOM, so `generate`
165
+ loaded a BOM'd project happily. Every other reader used bare `JSON.parse`, which throws on `\uFEFF`,
166
+ and every one of them caught the throw and reported the ordinary, wrong answer: an existing
167
+ `"snapshot"` block was invisible, `"type": "module"` was invisible, the scripts and devDependency
168
+ were skipped with a note, and `nextStepHint` told the user to run `npx codebase-snapshot generate` —
169
+ which resolves the *published* package instead of the one the run had just installed. The
170
+ `npm create` shim had the same bug in `hasSnapshotScript`.
171
+
172
+ A BOM is what a Windows editor writes when it saves UTF-8. All five readers now parse the same way
173
+ the config loader does, and `init` writes the mark back rather than silently removing it.
174
+
175
+ ### `validate` could not name seven of the settings it compared
176
+
177
+ `maxPartTokens`, `maxMonolithBytes`, `skeleton`, `sarif`, `changedSince`, `staged` and
178
+ `withDependents` were recorded in the manifest and folded into the configuration digest, but never
179
+ joined the field-by-field block. The digest can say *that* something changed; only the per-field
180
+ comparisons say *what*. Toggling `--skeleton` produced the generic "the settings differ" line and
181
+ nothing else.
182
+
183
+ ### A `.vue` or `.astro` script block was always parsed as `.ts`
184
+
185
+ `<Button onClick={() => go()}>` is a type assertion followed by six parse errors to the TypeScript
186
+ grammar, and no component at all, for a perfectly valid Vue `lang="tsx"` block. The script kind now
187
+ follows the component: Astro frontmatter is JSX-capable by construction, and a Vue or Svelte block
188
+ says so with `lang="tsx"`.
189
+
190
+ ### Performance
191
+
192
+ - **The split loop re-rendered every part on every pass.** `partitionFilesForBuckets` renders all
193
+ groups, finds the first one over budget, splits it, and goes round again — up to 64 times. Each
194
+ pass re-assembled, re-token-estimated and SHA-256 hashed *every* part. A group's rendered size
195
+ depends on its sections, its index and the *width* of the numbering, and the width only changes
196
+ when the part count crosses 1 000, so sizes are now measured instead of rendered and cached per
197
+ group. Each part is hashed exactly once.
198
+ - **The skeleton pass parsed with `setParentNodes: true`** and walked every node a second time to set
199
+ `.parent`, which nothing reads. It also expanded empty bodies from `{}` to `{ /* ... */ }`, making
200
+ small files larger, and rewrote the text once per span — 400 functions copied a 200 KB file 400
201
+ times.
202
+ - **`fullContents` retained every file body on runs that wrote no monolith.** The map held a
203
+ reference to every pre-partition body and was read only inside `if (config.includeMonolith)`,
204
+ pinning the multi-megabyte originals that the partitioner had just replaced with previews.
205
+ - **31 regexes were recompiled for every file.** `scanAndRedactSecrets` called `toGlobal` per rule
206
+ per call, and it is called once per file: a 20 000-file scan built 620 000 identical objects.
207
+ - **`--watch` re-derived the whole repository for one changed file.** The derivation — redaction,
208
+ findings, skeleton, preview, summary — is a pure function of (bytes, settings, path), so it is now
209
+ cached and reused when the content hash and a settings digest both match. Validity is the **content
210
+ hash**, not `mtime`: a formatter or a `git checkout` can rewrite a file within the same timestamp
211
+ tick at the same length. Measured on a synthetic 250-file TypeScript repository: cold 520 ms, warm
212
+ 196 ms of wall clock (2.65x) — the remainder is reading, hashing, partitioning and writing, which
213
+ a per-file cache cannot remove. The suite asserts the *work* instead, in CPU time on a denser
214
+ fixture: 1 046 ms against 48 ms, 21x, against a 3x floor.
215
+
216
+ ### What has no test, and why
217
+
218
+ The last two are structural. The regex cache and the `fullContents` change have no behavioural
219
+ signal — the output is byte-identical by construction — so they are covered by the existing suite
220
+ and by the reasoning above rather than by a new test. The `--watch` cache is different: four of its
221
+ five tests are correctness guards that pass on the previous build (that is the requirement — the
222
+ cache must be invisible), and the fifth measures the CPU time the cache saves and fails without it.
223
+
224
+ ---
225
+
226
+ ## 0.9.13
227
+
228
+ A fourth audit, this time of the seams: places where two code paths answer the same question with
229
+ different rules, and places where a command accepts something it does not apply. Every item below
230
+ has a test that fails without it, except one — the per-layer budget overflow, which I could not
231
+ reproduce and which is called out honestly at the end.
232
+
233
+ ### A scoped snapshot could expire and still validate
234
+
235
+ `validate` re-resolves the recorded changed-files scope and checks that every file it *skipped* as
236
+ `unchanged` is still outside that scope. It never asked the mirror question. So:
237
+
238
+ ```bash
239
+ git add src/a.ts
240
+ codebase-snapshot generate --staged # snapshot holds src/a.ts
241
+ git reset HEAD src/a.ts # same bytes on disk, nothing staged
242
+ codebase-snapshot validate --staged # said: "Snapshot is valid and up to date"
243
+ ```
244
+
245
+ The file on disk is byte-identical, so the hash comparison passed — and the scope the snapshot was
246
+ taken over had silently become empty. A file that is in the snapshot and would not be if it were
247
+ generated now is drift, and it is now reported, with `--with-dependents` still allowing for the
248
+ one-hop importer exception.
249
+
250
+ ### A part name in `manifest.json` could reach any file on the host
251
+
252
+ `validate` has confined part filenames since 0.9.9, but the **MCP server** resolved
253
+ `parts[].filename` with a bare `path.resolve`, in `snapshot_read` and in the `snapshot://parts/{n}`
254
+ resource. A repository under review could commit a manifest naming `../../../.env`, and a client
255
+ would read it — resource reads are auto-approved as read-only. A part name is bare by construction,
256
+ so `describeManifestShapeProblem` now requires one, which makes all five readers (`validate`,
257
+ `report`, `diff`, `snapshot_read`, the resource) refuse the same manifest for the same stated
258
+ reason.
259
+
260
+ `snapshot_explain` had the mirror problem. `toRelativePath` resolved *after* its "is it absolute?"
261
+ test and `normalizeTarget` returned anything starting with `..` unchanged, so `../other-repo/.env`
262
+ was joined to the root and passed to `existsSync`: one read-only, auto-approved call per probe, for
263
+ any path on the machine. Out-of-root targets now answer `outside-root` without touching the
264
+ filesystem. A file named `..foo.ts` is no longer mistaken for a traversal.
265
+
266
+ ### Four credentials that looked like code
267
+
268
+ - `password: "${DB_PASSWORD:-hunter2}"` — the shell and Compose idiom for a **default** — was
269
+ exempted as "an interpolation" because it starts with `${` and ends with `}`. So was
270
+ `${prefix}-real_secret-${suffix}` and any value with a `??` fallback. Only a value that is one
271
+ interpolation of a plain (optionally dotted) reference is exempt now; anything else in the braces
272
+ settles it the other way.
273
+ - `export api_key=hunter2` — bare `export` was in `DECLARATION_KEYWORDS`, so a shell assignment was
274
+ read as a JavaScript binding whose initializer is an expression. `export` can never be followed by
275
+ a bare assignment in JS or TS (`export foo = 1` does not parse), so the word only ever suppressed
276
+ redaction for the files most likely to hold a secret.
277
+ - An **indented, unterminated** private key — `privateKey: |` in YAML, or an indented template
278
+ literal — leaked every body line. The rule required base64 at column 0 and knew nothing about the
279
+ RFC 1421 header lines (`Proc-Type`, `DEK-Info`) that an encrypted key puts after the marker.
280
+ - A masked key block in a **CR-only** file swallowed its line breaks, because pass 1 counted `\n`
281
+ only, so every reported line number after it was wrong.
282
+
283
+ `.github/workflows/deprecate.yml` and `unpublish.yml` interpolated `${{ inputs.… }}` straight into
284
+ `run:` in jobs that carry `NPM_TOKEN`. GitHub expands an expression into the script *before* bash
285
+ parses it, so a dispatch input of `"; curl … $NODE_AUTH_TOKEN #` ran as a command. Inputs now arrive
286
+ through `env:` and are referenced quoted.
287
+
288
+ ### Two enumerations, two answers
289
+
290
+ The same repository produced different snapshots depending on `--discovery`, which is the one
291
+ property a reproducible snapshot tool cannot have:
292
+
293
+ - A tracked **file symlink** was dropped as `symlink` in `git` mode — the loop tested
294
+ `depth <= parts.length`, which includes the candidate itself — and emitted as a file by the walk.
295
+ - A **broken** symlink was `symlink` in `git` mode, because `targetReal` started at the link's own
296
+ path and so always compared as "inside the root". It is `read-error`, or `outside-root` when its
297
+ written target leaves the tree, in both modes now.
298
+
299
+ The output directory's crash leftovers had the same shape: the backup and partial-swap directories
300
+ were recognised by the hardcoded prefix `snapshot.backup-`. With `outDir: ".context"` an interrupted
301
+ run left `.context.backup-<pid>-<ts>` — a copy of the previous snapshot — to be scanned back in as
302
+ user source. The names now come from the resolved output and staging directories. `--watch` also
303
+ matched only `.backup-`, and only the directory itself, so an event for `part-001.md` *inside* a
304
+ `.partial-` copy was treated as a source edit: on a cross-device swap that is an infinite
305
+ regeneration loop.
306
+
307
+ ### The index described files that were not there
308
+
309
+ - `summarizeMarkdown` called a leading `---` a front-matter block if *any* later line was `---`,
310
+ so a document opening with a horizontal rule and using another as a section divider lost every
311
+ heading between them. The block must now also *be* YAML — a heading, a fence or prose means the
312
+ `---` was a rule. `...` closes a YAML document as well as `---`.
313
+ - The JSONC reader in `src/ast/structured.ts` ended a `//` comment only at `\n` and never removed a
314
+ trailing comma, so a `.jsonc` or an editor-reformatted `tsconfig.json` summarised as
315
+ `invalid-json` with no keys. Both are JSONC rather than extensions of it.
316
+ - `summarizeTypeScriptSource` looked at `ImportDeclaration` and `ExportDeclaration` and nothing
317
+ else, so `require("./x")`, `import("./x")` and `import fs = require("fs")` produced no import
318
+ edge. A CommonJS project had no import graph at all, and `--with-dependents` had nothing to walk.
319
+ `export { a } from "./x"` was not recorded as a re-export either.
320
+
321
+ ### Flags that were accepted and not applied
322
+
323
+ - `--only-artifacts audit` in a project whose config says `artifacts: { audit: false }` disabled the
324
+ other six and left audit off too: nothing written, exit 0. The flag was forwarded only as its
325
+ complement, and a suppression list cannot switch anything on.
326
+ - `findConfigFile` parsed `package.json` with bare `JSON.parse` while the loader used the JSONC
327
+ reader, so a BOM or a trailing comma made the two disagree about whether the project was
328
+ configured. And `"snapshot": []` counted as this tool's configuration, which skipped the setup
329
+ question entirely.
330
+ - `--skeleton` covered `.ts`, `.tsx`, `.mts` and `.cts` while `ts.createSourceFile` parses `.js`,
331
+ `.jsx`, `.mjs` and `.cjs` natively — so the files most repositories are made of were the ones it
332
+ could not compress, and `.d.ts` files, which have no bodies, were the ones it did parse.
333
+ - Over MCP, `WorkspaceOverrides` accepted only `maxPartBytes` and `outDir`. `snapshot_validate`
334
+ re-derives the configuration and compares digests, so a `--skeleton` snapshot could not be
335
+ validated at all. `maxPartTokens`, `skeleton`, `sarif`, `changedSince`, `staged` and
336
+ `withDependents` are now accepted by `snapshot_plan`, `snapshot_generate`, `snapshot_validate`
337
+ and `snapshot_explain` — schema and application coming from one definition, so a tool cannot
338
+ advertise a flag it would ignore.
339
+ - `npm create codebase-snapshot -- --preset=nextjs` ignored the preset: the shim parsed `argv` with
340
+ `indexOf`, which finds nothing for `--flag=value`, treats a missing value as "not given", and
341
+ ignores `--forse` entirely. `node:util.parseArgs` in strict mode now rejects all three by name.
342
+
343
+ ### Configuration answers that disagreed with each other
344
+
345
+ - `loadConfigFileDetailed` did not attach `CONFIG_DIR_SYMBOL` — only `loadConfigFile` did — so
346
+ `resolveConfig((await loadConfigFileDetailed(dir)).config)` scanned `process.cwd()`.
347
+ - `normalizeResolvedConfig` re-derived `artifacts` with `includeMonolith` as the default, so
348
+ `{ ...resolved, includeMonolith: true }` produced a config claiming `includeMonolith: true`
349
+ beside `artifacts.monolith: false`, and `generate` — which reads the map — wrote no monolith.
350
+ - `configDigest` hashed `budgets` with `JSON.stringify`, which preserves insertion order, so
351
+ `{ docs, api }` and `{ api, docs }` produced different digests and `validate` reported drift for
352
+ a snapshot nobody changed.
353
+
354
+ ### Cost
355
+
356
+ - `pathScanRules` returned `[...DEFAULT_SECRET_PATTERNS, ...extraRules]` although
357
+ `config.secretRules` already begins with those patterns: every file path, skip record and AST
358
+ summary string ran all 30 built-in rules twice. `disableDefaultPatterns` also had no effect on
359
+ paths while it worked on content.
360
+ - `globToRegExp` and `matchGitignorePattern` compiled a fresh `RegExp` per call — over a million per
361
+ run on a 20 000-file repository with 55 patterns. Both are cached by pattern. A malformed
362
+ character class no longer escapes as a raw `SyntaxError`: a `.gitignore` range like `[z-a]`
363
+ matches nothing (which is what git's own `fnmatch` does), while the same range in a configured
364
+ glob is reported as the configuration mistake it is, naming the pattern.
365
+ - `generateSnapshotAsync` resolved the configuration twice, and `validate` in the CLI re-read and
366
+ re-parsed `manifest.json` to re-run a shape check the validator had already run.
367
+ - `pendingHooks` was drained only by `generateSnapshotAsync`, so a synchronous caller left one
368
+ settled promise per output directory for the life of the process.
369
+
370
+ ### Lifecycle
371
+
372
+ - Each `hooks.afterGenerate` installed its own `process.stdout.write` wrapper and restored it only
373
+ if the current write was still its own. With two overlapping async hooks the first to finish never
374
+ restored, and the second put the first one's wrapper back — permanently. There is now one scope
375
+ and one wrapper, released by reference counting.
376
+ - The update check used `request.setTimeout`, an *idle socket* timeout that resets on every chunk:
377
+ a trickling proxy kept the CLI alive indefinitely. It has a wall-clock deadline now, `.unref()`d
378
+ so a pending check never holds the process open. And `isNewer` stripped prerelease tags before
379
+ comparing, so `0.9.13-alpha.1` was recommended to everyone on a stable release and a stable
380
+ release was never announced to anyone on a beta.
381
+
382
+ ### Smaller
383
+
384
+ - `wrapOneLine` measured `room` in terminal cells and cut with `String.prototype.slice`, which
385
+ counts UTF-16 code units. CJK text produced rows twice as wide as the frame believed, and a
386
+ surrogate pair could be cut in half. The cut advances by code point now.
387
+ - Part numbering is decided **before** rendering, in one pass over every budget bucket, instead of
388
+ partitioning each budget and re-numbering the merged set afterwards.
389
+
390
+ ### What I could not reproduce
391
+
392
+ The audit reported that per-layer budgets caused parts to overflow their budget by seven to twenty
393
+ bytes when `renumberParts` rewrote their headers, and were then hard-clipped into a broken document.
394
+ **I could not make that happen.** I swept file sizes against a two-budget configuration and again
395
+ with six layers and eight kinds whose names are 180 characters long, and no part ever came out over
396
+ its budget, with or without the clamp. The arithmetic explains why: `partitionFiles` reserves about
397
+ 2.2 kB of header room up front and shrinks sections to `budget - reserve`, so a rendered part sits
398
+ roughly 1.4 kB below its ceiling, and the header growth being reported is fourteen bytes.
399
+
400
+ The mechanism was removed anyway — the numbering pass now happens before rendering, so a fitted part
401
+ cannot be rewritten — but the claim was not verified, and this changelog will not pretend it was.
402
+ The four tests added with it pass on the previous build too; they pin the invariants the refactor
403
+ must not break rather than reproducing a defect.
404
+
405
+ ## 0.9.12
406
+
407
+ A third audit, this time of what the snapshot *costs* as well as what it gets wrong. Two of the
408
+ defects were circular — `validate` demanded a flag it refused to accept — and one of them meant a
409
+ config file could not turn a setting on at all. Every item below has a test that fails without it.
410
+
411
+ ### The parts were 32 000 tokens of duplicated metadata
412
+
413
+ Each file section carried nine metadata bullets and, for a TS/JSON/Markdown file, a
414
+ pretty-printed JSON summary of code that followed three lines later. On this repository — 234
415
+ files — that was **195 619 bytes, about 32 600 tokens, 12 % of the snapshot**, spent on a
416
+ `SHA-256:` line per file and on listing every variable and export immediately above the code that
417
+ declares it.
418
+
419
+ A section is now an anchor, a heading and one line:
420
+
421
+ ```text
422
+ - Meta: src/other | typescript | full | 3 lines (63 B)
423
+ ```
424
+
425
+ - The per-file `SHA-256` is **gone from the parts** and stays in `manifest.json`, which is the file
426
+ a machine reads. `snapshot_read` used to fall back to it when a path was too long for the
427
+ heading; it now matches the `<a id>` anchor, which is a digest of the exact `relativePath` and was
428
+ already there — exact even for the 300-character paths the heading caps.
429
+ - The structural summary is **not printed for a `full` file**: `manifest.json` keeps it, and
430
+ `preview` and `signatures` sections keep it, because there the body is not the file. A one-line
431
+ string summary survives either way.
432
+ - `- Content Fingerprint:` stays in the part header (~22 tokens per part), which anchors an answer
433
+ to a verifiable state of the repository.
434
+
435
+ Nothing else about the format moved, so `snapshot_read` and both structural parsers still read it.
436
+
437
+ ### `validate` refused the flag its own comparison needed
438
+
439
+ `manifest.config.digest` hashes `skeleton`, `sarif`, `maxPartTokens`, `changedSince`, `staged` and
440
+ `withDependents`, and `validate` compared that digest — but `COMMAND_OPTION_SCOPE` allowed those six
441
+ flags on `generate` only. So a snapshot generated with `--skeleton`:
442
+
443
+ 1. failed `validate` with "the settings differ from the ones this snapshot was generated with", and
444
+ 2. failed `validate --skeleton` with "`--skeleton` does not apply to validate".
445
+
446
+ There was no way to validate it. `validate` now accepts all six, `explain` accepts the three that
447
+ decide its answer, and `report` accepts none: it reads the manifest and prints what is there, so a
448
+ generation setting would change nothing.
449
+
450
+ ### A config file could not turn these four settings on
451
+
452
+ `parseArgs` returned `skeleton: values.skeleton === true`, so a run that did not pass the flag sent
453
+ **`false`** — and `resolveConfig` reads these as `cliOptions.x ?? config.x`, where `false` is an
454
+ explicit override. `skeleton: true` in `snapshot.config.ts` was silently turned off by every run
455
+ that did not repeat the flag on the command line. The same applied to `staged`, `withDependents`
456
+ and `sarif`. All four are now `undefined` when the flag is absent.
457
+
458
+ ### `explain` and `validate` ignored `--with-dependents`
459
+
460
+ The generator widened a changed-files scope by one hop of importers; both readers compared the raw
461
+ path set. A file that was **in** the snapshot because it imported something that changed was
462
+ explained as `included: false, reason: "unchanged"`, and `validate` called the snapshot current
463
+ while a file it should contain was missing from it. The import graph moved to
464
+ `src/scanner/dependents.ts` so all three apply one rule, and `validate` now re-parses a recorded
465
+ `unchanged` skip only when the manifest says the run used `--with-dependents`.
466
+
467
+ ### `setup` wrote a config its own snapshot did not match
468
+
469
+ `setup --discovery walk --large-file-bytes 100000` forwarded both flags to the generation it runs
470
+ and wrote the defaults into the config beside it, so a brand-new project's one-command setup
471
+ finished with a snapshot that `validate` immediately called drifted — three configuration
472
+ mismatches on the first run. `--max-part-tokens`, `--skeleton` and `--sarif` are now accepted by
473
+ `setup` too, and every one of them is written into the generated config as well as forwarded.
474
+
475
+ ### Redaction that corrupted what it protected
476
+
477
+ Three shapes were found in this repository's own snapshot, where the tool rewrote its own
478
+ documentation and tests:
479
+
480
+ - **Prose.** `credential-assignment-unquoted` masked any word after `secret: ` or `token: `, so
481
+ `indistinguishable from a secret: a` became `…from a secret: [REDACTED…]`. An unquoted value in a
482
+ line-oriented format never continues after a space (`PASSWORD: hunter2` ends the line; `secret: a
483
+ value` is a sentence), and a one-character bare value is not a credential. Quoted values keep
484
+ their one-character floor — quotes are evidence somebody meant to write a value.
485
+ - **Template literals.** `` `… an empty token: "${appliesTo}"` `` had the interpolation masked,
486
+ because `startsInsideStringLiteral` only looks at the character before the identifier. A value
487
+ that is entirely `${…}` is an expression, so it is not masked.
488
+ - **Doubly-escaped quotes.** `JSON.parse('{"password": "p@\\"ssw0rdINVENTED"}')` — how every
489
+ fixture that embeds JSON is written — emitted `"[REDACTED…]"ssw0rdINVENTED"`: a fragment of the
490
+ credential in clear and a line that no longer parses. A quote after `\\` that is followed by a
491
+ value character is an escaped quote in a deeper literal, and the value now runs to the real close.
492
+
493
+ ### An async `afterGenerate` hook's files were not recorded
494
+
495
+ The hook is documented as the way to write your own artifact into the snapshot directory, and that
496
+ directory is only replaceable while `.codebase-snapshot` accounts for every file in it. Only the
497
+ *rejection* path re-shipped the marker, so an async hook that succeeded left its file unrecorded
498
+ and the next `generate` refused the directory as foreign. The marker is now re-shipped when the
499
+ hook settles either way, and only while it still records this run — the lock is released when
500
+ `generateSnapshot` returns, so a slow hook can outlive the snapshot it belongs to.
501
+ `generateSnapshotAsync` now awaits the hook, so a caller of the async API gets control back when
502
+ the snapshot is actually finished.
503
+
504
+ ### Smaller fixes
505
+
506
+ - **`getGitInfo` used Node's 1 MB `maxBuffer`** where every sibling call passed 32 or 50 MB. On a
507
+ repository with enough dirty entries, `git status` overflowed, threw, and the `catch` returned
508
+ `isDirty: false` — reporting the dirtiest possible tree as clean. The limit is raised and shared,
509
+ and an overflow now reports `isDirty: true`, because a buffer that large only fills when there is
510
+ a great deal to report. All three git invocations use `execFileSync` instead of a shell string.
511
+ - **`.gitignore` character classes** were escaped as literals, so `*.[oa]`, `[Tt]humbs.db` and
512
+ `log[0-9].txt` matched nothing in `--discovery walk` while working in `git` mode.
513
+ - **A layer or kind rule** was glob-translated only when it contained `*` or `?`, so
514
+ `src/{api,web}/index.ts` fell through to a substring search and classified nothing; and a plain
515
+ pattern like `test` used `String.includes`, so it matched `src/contest/a.ts`. Both now go through
516
+ `matchesExclude`, which is the single implementation of "does this pattern select this path".
517
+ - **`export { handler as GET }`** and `export { GET } from "./handlers"` recorded the export but no
518
+ route method, so those App Router routes had no `methods` in `manifest.indexes.routes`.
519
+ `export default () => <div />` recorded no component for the same reason.
520
+ - **The last-resort clip** reserved room for its own truncation marker but not for the token count
521
+ rewritten afterwards, so a part clipped to exactly its budget came back over it. `rehashPart` also
522
+ estimated tokens from text that still carried the *old* number, unlike every other part.
523
+ - **Seven MCP tools ignored `outDir`**, so a snapshot generated into a custom directory could be
524
+ written by `snapshot_generate` and inspected by nothing.
525
+ - **`snapshot_read`** replaced the line range with the preview note instead of adding to it, so a
526
+ window into a preview came back with no `Lines X–Y of Z` at all; `signatures` was treated as a
527
+ full file.
528
+ - **Every published file ended with a `sourceMappingURL`** pointing at a `.map` that
529
+ `package.json#files` deliberately does not ship. The build no longer emits them.
530
+ - **`--watch` crashed on Node 18 on Linux**, where `fs.watch(cwd, { recursive: true })` does not
531
+ exist before Node 20.13. The package declares `engines: >=18`, and the run wrote a snapshot,
532
+ printed "Watching for changes", and *then* threw `ERR_FEATURE_UNAVAILABLE_ON_PLATFORM`. It is now
533
+ refused up front, with exit `2` and a message naming the version that works — and before anything
534
+ is written. A watcher that cannot start for any other reason (`ENOSPC`, the Linux inotify limit)
535
+ reports it as a message instead of a stack trace, and the snapshot it already wrote stays valid.
536
+ Found by the release's own Node 18 job: 0.9.10 and 0.9.11 never ran one.
537
+
538
+ ---
539
+
540
+ ## 0.9.11
541
+
542
+ A second audit, run against 0.9.10's own output, found that several defects the previous release
543
+ believed it had fixed were still reachable, that the changed-files scope was applied one layer too
544
+ late, and that this repository's documentation had been quietly corrupted. Every item below is
545
+ fixed with a test that fails without it.
546
+
547
+ ### The redactor was still editing the program
548
+
549
+ `credential-assignment-unquoted` exists for `.env`, YAML and compose files, where everything after
550
+ `NAME=` is a literal. In a source file the same shape is an **expression**. The bug is visible in
551
+ the 0.9.10 snapshot itself, in `parts/part-005.md`:
552
+
553
+ ```
554
+ const SINGLE_TOKEN = [REDACTED: potential secret-like value omitted from snapshot][^\sA-Za-z0-9_]/g;
555
+ ```
556
+
557
+ The regex literal's opening `/` was a legal match for `UNQUOTED_LITERAL`, so the pattern was
558
+ deleted and the line no longer parses — data loss in the artifact whose whole purpose is to
559
+ describe the code. Three changes close the class rather than the instance:
560
+
561
+ - an unquoted value must **begin** with a word character, so `/`, `+`, `-` and `%` can only
562
+ continue one;
563
+ - a value followed by `[`, `{` or `(` is refused, because `const accessToken = tokens[0]` is a
564
+ reference;
565
+ - a value that sits in a **declaration** is never a literal. `const`, `let`, `var`, `readonly`,
566
+ `private`, `protected`, `static`, `declare`, `export` and `using` are recognised, walking back to
567
+ the start of the identifier so the tail match (`TOKEN` inside `SINGLE_TOKEN`) is covered too.
568
+
569
+ The guard is bounded to a 40-character window: an unbounded version made a 236 KB line with 4,000
570
+ assignments cost 2.7 s, which the existing quadratic-scan test caught.
571
+
572
+ ### A changed-files scope was applied after the scan instead of before it
573
+
574
+ `--changed-since` and `--staged` read, decoded, secret-scanned and TypeScript-parsed every file in
575
+ the repository and then threw all but a handful away. Because every aggregate was bumped inside
576
+ that scan, the snapshot inherited the repository's numbers:
577
+
578
+ - **`validate` failed on every scoped snapshot.** Unchanged files are recorded in `skipped[]` with
579
+ reason `unchanged`, and `validate` then asked `wouldNowBeIncluded` about them — a question that
580
+ only answers "is this text?" — so every unchanged source file was reported as `new file:` and
581
+ the exit code was 1.
582
+ - **`manifest.layers`, `manifest.kinds`, `counts.largeFileCount`, `totals.redactedLines`** and the
583
+ encoding/preview/parse warnings all described files the snapshot did not contain. A two-file
584
+ snapshot reported the layer distribution and the secret count of the whole repository.
585
+ - **`explain` ignored the scope entirely** and answered `included: true` for exactly the files
586
+ `generate` was skipping.
587
+
588
+ The scope is now resolved once, before the scan, and applied to the candidate list whenever
589
+ `--with-dependents` is not in play. The dependents case still has to scan first — it needs every
590
+ file's imports — so it narrows afterwards and recomputes the counts, the findings and the warnings
591
+ from what survived.
592
+
593
+ ### `--with-dependents` walked the import graph transitively
594
+
595
+ The closure tested `keep.has(resolved)` while adding to `keep`, so `a.ts` changed pulled in
596
+ `b.ts` (which imports it) and then `c.ts` (which imports `b.ts`). Whether a two-hop file was
597
+ included depended on whether its name sorted after `b.ts`: the same repository produced different
598
+ snapshots depending on file names. The test is now made against the changed set, which makes the
599
+ closure exactly one hop.
600
+
601
+ ### Root-anchored `.gitignore` rules never matched in `walk` mode
602
+
603
+ `readGitignoreRules` keeps the leading `/`, and the matcher it was handed compares against a path
604
+ with no leading slash — so `/build` could not equal `build`, could not prefix `build/x`, and could
605
+ not appear between two slashes. `/secret-notes.txt` silently kept its file in the snapshot.
606
+
607
+ The same delegation made the opposite mistake one segment down: gitignore(5) anchors a pattern
608
+ containing a non-trailing `/` to the directory holding the `.gitignore`, so `doc/frotz` ignores the
609
+ top-level one only, while `matchesExclude` matched `a/doc/frotz` as well. Both are fixed by using
610
+ the gitignore matcher (which already implemented `anchored || body.includes("/")`) for every rule
611
+ rather than only for rules containing a backslash. `matchesExclude` is left to configuration
612
+ excludes, where "match at any depth" is the documented behaviour.
613
+
614
+ ### Spreading a resolved config with a new `maxPartBytes` changed nothing
615
+
616
+ `{ ...resolved, maxPartBytes: 64 * 1024 }` still satisfies `isResolvedConfig`, and
617
+ `normalizeResolvedConfig` kept the **old** `budgets` object. The partitioner reads
618
+ `budgets.default`, so the run produced 300 KB parts and the manifest recorded the 64 KB that was
619
+ ignored. `normalizeResolvedConfig` now normalises `budgets` against `maxPartBytes`, capping
620
+ per-layer budgets rather than replacing them.
621
+
622
+ A related inconsistency is fixed too: a config that sets only `budgets: { default: "100k" }`
623
+ recorded `maxPartBytes: 300k`, so one snapshot described two different budgets.
624
+
625
+ ### Smaller fixes
626
+
627
+ - **The MCP worker URL was built with string interpolation.** `file://${path}` turns a Windows path
628
+ into `file://C:/…` whose *host* is the drive letter, reads a `#` as a fragment and a `%` as an
629
+ escape. It uses `pathToFileURL` now, through an exported `workerEntry()`.
630
+ - **`--sarif` and the monolith disagreed.** The log was rendered in the write phase, so
631
+ `full-codebase.json` embedded a manifest with no `artifacts.sarif` while `manifest.json` had one.
632
+ The log is rendered before the monolith is serialised.
633
+ - **`validate` never checked `reports/audit.sarif`** against its recorded digest. It does now: a
634
+ security report uploaded to a third party is exactly the artifact that must be verified.
635
+ - **`--sarif` was missing from `configDigest`**, so two runs that write different directories
636
+ compared equal.
637
+ - **`--skip-artifacts … --sarif` claimed nothing was written** while writing `reports/audit.sarif`.
638
+ - **SARIF rule ids were positional.** They were handed out by position in the sorted set of rule
639
+ names a run happened to use, so adding one custom rule that sorts before the built-ins
640
+ (`acme-key` does) shifted every built-in id by one and re-pointed every open alert. Built-in
641
+ rules now take a fixed id from their declaration order and custom rules get an id hashed from
642
+ their own name. This renumbers the built-ins once, relative to 0.9.10, and never again.
643
+ - **`generate --watch` generated twice on startup** — `runCommand` generated, then the loop
644
+ generated again before any file changed.
645
+ - **`--watch` froze `outDir`/`tmpDir` at startup.** After the config moved them, the watcher saw
646
+ its own output as user content and regenerated without end.
647
+ - **The pre-commit hook hardcoded `snapshot/`**, so a project with `outDir: ".context"` got a hook
648
+ that printed "no snapshot found, skipping" and exited 0 on every commit — a gate that never ran.
649
+ The directory is baked in at install time, and the guard tests `manifest.json` rather than the
650
+ directory (a directory left by a failed run exists but holds no snapshot).
651
+ - **`--max-part-tokens 200` was silently raised** to the 1024-token floor. It is a usage error
652
+ now, and the message talks about tokens rather than bytes.
653
+ - **The MCP line cap was bypassed** by an oversized line that arrived complete with its terminator,
654
+ because the guard required the buffer to hold no `\n` at all. The cap now applies to every
655
+ complete line.
656
+ - **`snapshot_locate` searched only half the summary.** `variables` (where a `const`, a Python
657
+ constant, a Go `var`, a Rust `mod`, a C `#define` and a Terraform `variable` all land),
658
+ `directives`, `frontmatterKeys` and JSON `keys` are searched too.
659
+ - **`snapshot_read` printed `Lines 1–0 of 1` for an empty file** and returned the *last* line when
660
+ `fromLine` was past the end. Both are stated instead.
661
+ - **`.TSX` was parsed as `ScriptKind.TS`** — `isSkeletonizable` lowered the name and
662
+ `skeletonizeTypeScript` did not.
663
+ - **Markdown summaries:** an unclosed `---` at line 0 swallowed every heading, a shorter fence
664
+ inside a longer one closed it early, and a closing fence with an info string was accepted. All
665
+ three follow CommonMark now.
666
+ - **Destructured exports were invisible.** `export const { a, b } = obj` and
667
+ `export const [first] = pair` declare real exports and were dropped by an
668
+ `isIdentifier` check.
669
+ - **`rehashPart` left `estimatedTokens` at its pre-clip value** after `enforceBudget` cut a part,
670
+ and the part's own header kept stating the old number. Both are recomputed.
671
+ - **`setup --preset <unknown>` exited 1** where `init` exits 2, because the parse-time check was
672
+ written for `init` only.
673
+ - **`describe` omitted all seven 0.9.10 flags and six config keys.** `describe` is the contract an
674
+ agent reads instead of the docs, so this is the failure that matters most: the options simply did
675
+ not exist as far as a machine consumer was concerned. A test now compares `CLI_OPTIONS` and
676
+ `CONFIG_KEYS` against the contract in both directions.
677
+ - **`--changed-since` was passed to git as a positional revision.** A ref beginning with `-` is now
678
+ refused before git sees it, and `--` separates revisions from paths.
679
+ - **`--skeleton --include-monolith` labelled a body-stripped file `inclusionMode: "full"`.** The
680
+ monolith now keeps the full redacted body, so the label is true.
681
+
682
+ ### Documentation and repository hygiene
683
+
684
+ - **Three sentences had letters eaten by interpreted escapes**: `\v` in "Validate", `\r` in
685
+ "reports/", `\a` in "audit.md". A test now fails on any control byte in any tracked text file.
686
+ - **Four documents claimed the MCP server serves seven tools.** It serves eight. The count is now
687
+ compared against the live tool list.
688
+ - **`publish:create` could not run.** `npm publish packages/create` is parsed by npm as the GitHub
689
+ shorthand `ssh://git@github.com/packages/create.git` and fails with exit 128; it names
690
+ `./packages/create`.
691
+ - **`check-site-links` accepted a sibling directory.** `resolved.startsWith(SITE)` is true for
692
+ `../site-other`; the check requires the path separator.
693
+
694
+ ---
695
+
696
+ ## 0.9.10
697
+
698
+ An audit of this codebase, run against its own snapshot, found twenty defects, six performance
699
+ problems and seven missing capabilities. This release is all of them: every defect, every
700
+ performance problem and every capability named in that audit is in the tree, with a test that
701
+ fails without it.
702
+
703
+ ### Redaction was corrupting the code it exists to preserve
704
+
705
+ The worst defects were in the secret scanner, and the evidence was in this repository's own
706
+ snapshot. Four false-positive classes rewrote valid statements:
707
+
708
+ - `[:=]` matched the first `=` of `===`, and the unquoted-value rules then read the remaining `==`
709
+ as a credential. `clause.token === ts.SyntaxKind…` shipped as `clause.token =[REDACTED…]`, and so
710
+ did `if (token === "--")`.
711
+ - `env-secret-assignment` carried `/i`, so an indented reassignment matched: `key =
712
+ fs.realpathSync(stagingRoot)` was emitted with the call masked.
713
+ - Credential names are matched as a *tail* — that is what catches `dbPassword` — so a name inside a
714
+ string literal matched, and the string's closing quote was taken as the value's opening quote.
715
+ `["awsSecret:", "apiKey:"]` came out with two list elements welded together.
716
+ - `progressToken: "progressToken"` was masked as a credential. A camelCase value that spells its
717
+ own key is a symbol-table entry, not a secret.
718
+
719
+ Reserved words are no longer credential literals either: `token = await refreshToken(client)` was
720
+ emitted as `token = [REDACTED] refreshToken(client)`.
721
+
722
+ ### Scanning no longer depends on state outside the tree
723
+
724
+ - A `.gitignore` rule naming a nested directory (`public/build/`) never matched in `walk` mode: the
725
+ candidate was reduced to its first path segment, which cannot equal a two-segment rule. The same
726
+ bug dropped a regular *file* named `build` for a `build/` rule.
727
+ - How files are enumerated was held in a module-level variable written by `resolveConfig`, so two
728
+ scans in one process could disagree, and the second inherited the first one's mode.
729
+ - `validate` did not exclude the tool's own output directory, so it counted `snapshot/` as files
730
+ missing from the snapshot and reported a correct snapshot as drifting.
731
+ - `findSecretInPath` compiled thirty-plus `RegExp` objects per call, and it runs on every path and
732
+ every string in every summary.
733
+ - `getGitInfo` spawns one `git status` instead of three `git` processes.
734
+
735
+ ### The config digest now covers what decides a snapshot
736
+
737
+ Layer and kind *names* were hashed, not their matchers, so moving a layer from `src/api/**` to
738
+ `src/v2/**` produced an identical digest and `validate` said "valid and up to date" for a snapshot
739
+ whose files had all been reclassified. A rule's `flags` and `valueGroup` were missing too, and so
740
+ was `discovery`. `package.json`'s `"snapshot"` object now gets the unknown-key guard every other
741
+ config file already had, and `secretRules` is accepted in a config file as it already was in code.
742
+
743
+ ### `setup` stops dropping the flags it was given
744
+
745
+ `setup` builds the inner `generate` command line by hand, and five settings were never copied into
746
+ it, so the run reported one configuration and generated with another. It also refused
747
+ `--discovery`, `--large-file-bytes`, `--max-monolith-bytes` and `--include-monolith` outright.
748
+ `--only-artifacts monolith` enabled *nothing*: it turns the six default artifacts off and relied on
749
+ the monolith switch to turn the seventh on, and neither `init` nor `setup` passed that switch along.
750
+
751
+ ### Part budgets say what they enforce
752
+
753
+ - Every part records the budget it was built for. With per-layer budgets, verification compared
754
+ each part with the *largest* budget in the run, so a part twice over its own layer budget passed.
755
+ - `renumberParts` rewrites headers, so the byte bound is re-applied after renumbering.
756
+ - A body cut twice reported only the second cut: a 1,000-line file reduced to a preview and then to
757
+ ten lines said "11 lines omitted", because the count came from the preview instead of the file.
758
+ - Every part states its size in **tokens**, and `--max-part-tokens 32k` sets a ceiling a context
759
+ window can be measured against. The estimate's method is documented, not hidden.
760
+
761
+ ### Crash-proof locks, byte-accurate reads
762
+
763
+ - A run killed between creating its lock file and writing its pid leaves a zero-byte lock that
764
+ named nobody, so `lockOwnerIsGone` answered "not gone" forever and every later run refused to
765
+ start until the file was deleted by hand. Age is now the fallback evidence.
766
+ - `snapshot_read` truncated to its byte limit by slicing UTF-16 code units, returning up to three
767
+ times the limit for multibyte text and able to split a surrogate pair.
768
+ - `snapshot_locate` could not find a symbol a file does not export — most of the code in most files
769
+ — and could not find a Markdown heading at all. Layer and kind matched by exact equality while
770
+ every other category used a substring.
771
+ - `findSection` compared a heading with the manifest's raw path, which differ once the heading has
772
+ been capped or escaped; such a file became unreadable whenever another file shared its content.
773
+ - A watch loop had no `error` listener on `fs.watch`, so an `EPERM` or an inotify limit killed the
774
+ process instead of being reported.
775
+ - The update check skipped itself when any argument contained "json", so `explain package.json`
776
+ never asked.
777
+
778
+ ### Fewer syscalls
779
+
780
+ `generate` read every included file twice — a chunked NUL scan, then a full read and a second scan
781
+ of the same bytes. Below the 8 MB scan cap the decode is the complete check on bytes that had to be
782
+ read anyway, so the pre-scan is kept only above it. `explain` and `validate` had the same double
783
+ read.
784
+
785
+ ### New structure for six more languages
786
+
787
+ Swift, Dart, SQL, GraphQL, Protobuf and Terraform now have profiles. `src/scanner/language.ts`
788
+ already knew four of those extensions by name and no profile claimed them, so a repository in one of
789
+ those languages got a list of files and no idea what was inside them.
790
+
791
+ ### The site keeps its own links
792
+
793
+ `README.md` writes absolute GitHub URLs so the same text works on npmjs.com; the site builder only
794
+ rewrote *relative* links, so a reader on the docs site who clicked a guide link on the homepage was
795
+ handed back to GitHub. In-repo links to published pages are rewritten now; other repositories and
796
+ files the site does not publish are left alone.
797
+
798
+ ### Also
799
+
800
+ - `configDigest`, `classifyLayer`, `classifyKind`, `displayPath`, `lockPathFor`, `estimateTokens`,
801
+ `findSecretInPath` and `getGitInfo` are exported.
802
+ - `manifest.config` records `maxMonolithBytes` and `maxPartTokens`.
803
+ - `scripts/sync-docs.mjs` propagates `package.json#version` to `src/version.ts` and
804
+ `packages/create/package.json`, so a release cannot drift by forgetting a file. `--check` covers
805
+ them.
806
+ - `scripts/pack-check.mjs` requires the MCP build outputs, and the three wizard test files share one
807
+ fake-TTY harness.
808
+
809
+ ### `snapshot_dependents`: the import graph, in both directions
810
+
811
+ `manifest.indexes.imports` already held enough to answer "what uses this file" —
812
+ `byFile` records every specifier, `centralFiles` ranks the most-imported files — but nothing
813
+ an agent could ask. So the one question that decides what to read next, *what breaks if I
814
+ change this?*, was answered by reading parts until the picture appeared.
815
+
816
+ The eighth MCP tool takes a path and a direction and returns both edge sets with the part each
817
+ neighbour lives in, plus the specifiers that resolve to nothing inside the snapshot
818
+ (`node:fs`, a package name) rather than silently dropping them. Resolution runs through the
819
+ generator's own `buildImportResolver`, so the tool and `centralFiles` cannot describe different
820
+ graphs — a test asserts the counts agree.
821
+
822
+ ### `--install-hook` gates commits on `validate`
823
+
824
+ `docs/stale-context-is-your-bug.md` argues a snapshot is only trustworthy when something gates
825
+ on it, and `validate` is that gate — but wiring it into `.git/hooks` was thirty seconds of
826
+ manual work, which is the step a project skips. `init --install-hook` and `setup --install-hook`
827
+ write `.git/hooks/pre-commit`, which refuses a commit whose snapshot is stale and never installs
828
+ anything while doing it.
829
+
830
+ A `pre-commit` this tool did not write is left byte-for-byte alone, and the output names the line
831
+ to add instead: there is deliberately no flag to force the overwrite. A `.git` that is a *file*
832
+ (a worktree or submodule) is followed to the real git directory rather than failing with
833
+ `ENOTDIR`.
834
+
835
+ ### `--changed-since` and `--staged`: review packs the diff
836
+
837
+ Reviewing a pull request with a model is the case this exists for: the diff is small, the
838
+ repository is not, and packing all of it costs a context window to answer a question about four
839
+ files.
840
+
841
+ `--changed-since <ref>` is the working tree against that ref — committed, staged and unstaged
842
+ changes all count, because all three are what a reviewer is looking at — **plus untracked
843
+ files**, which `git diff` never lists and which are exactly what a new feature starts with.
844
+ `--staged` is the index against `HEAD`, and `--with-dependents` adds the one-hop importers,
845
+ which is the context needed to change a file safely rather than merely the file that moved.
846
+
847
+ Files outside the scope are recorded in `skipped[]` with a new reason, `unchanged` —
848
+ deliberately not `excluded`, which means the user's own glob rejected the path. Every failure is
849
+ loud: an unknown ref, a directory that is not a git repository, and `--with-dependents` without
850
+ a scope are all errors naming the way out, because a ref typo that produced an empty snapshot
851
+ would read as "this change touched nothing".
852
+
853
+ ### `--skeleton`: signatures instead of bodies
854
+
855
+ `docs/vs-repomix.md` concedes that a compression mode beats byte budgeting for a repository too
856
+ large to fit at all. This is that mode, built on the compiler that is already the one runtime
857
+ dependency: each function, method, constructor, accessor and arrow body becomes `{ /* ... */ }`
858
+ in place.
859
+
860
+ The rewrite is span-based rather than a `ts.createPrinter()` re-print, and that is the whole
861
+ feature — a printer normalises quotes, blank lines and comment positions, so a skeleton would
862
+ differ from its source everywhere instead of only where bodies were removed. Affected entries
863
+ carry `inclusionMode: "signatures"`, and the structural summary still describes the whole file.
864
+
865
+ ### `--sarif`: the findings a CI gate can read
866
+
867
+ `reports/audit.md` tells a human what was masked, and nothing could *gate* on it: a CI job that
868
+ wanted to fail a build over a leaked key had to scrape markdown. `--sarif` writes
869
+ `reports/audit.sarif` in SARIF 2.1.0, which is what GitHub Code Scanning reads.
870
+
871
+ The artefact is deliberately austere: path, line, level and rule — and **never the value**,
872
+ because it is meant to be uploaded to a third party and a second copy of a secret in someone
873
+ else's retention policy is a new problem rather than a report on the old one. Rule ids are
874
+ stable across runs so an alert is not orphaned, and each result carries a
875
+ `partialFingerprint` so Code Scanning keeps an alert open instead of closing and reopening it
876
+ on every run.
877
+
878
+ ### `snapshot_generate` runs off the JSON-RPC thread
879
+
880
+ The 0.9.4 measurement stands: a synchronous generate blocks the stdio server for about 7.5
881
+ seconds on a 3,000-file repository. In that window the process cannot answer `ping` or
882
+ acknowledge `notifications/cancelled`, and a client with a socket timeout reports a working tool
883
+ call as a dead server.
884
+
885
+ The work moves to a worker thread. A resolved config cannot cross a thread boundary —
886
+ `layers[].match` may be a function and `hooks.afterGenerate` always is — so the worker loads
887
+ the config file and applies the same CLI options the calling thread did, rather than receiving
888
+ values that would all look like explicit overrides. A worker that fails to start, exits without
889
+ posting, or throws is reported as a tool failure naming the reason; there is no quiet retry on the
890
+ calling thread, because that would hide exactly the condition this exists to avoid.
891
+
892
+ One consequence is documented rather than discovered: `hooks.afterGenerate` does not run for a
893
+ generation started from MCP, only for a CLI `generate`.
894
+
895
+ ---
896
+
24
897
  ## 0.9.9
25
898
 
26
899
  ### `snapshot -version` now answers the question it was asked