codebase-snapshot 0.9.9 → 0.9.14
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +873 -0
- package/README.md +3 -3
- package/dist/bin.js +6343 -3897
- package/dist/{chunk-KUVFTJHK.js → chunk-DUZK5VL2.js} +3036 -1049
- package/dist/{chunk-HQBEQ67K.js → chunk-J4W74N4Y.js} +496 -114
- package/dist/cli.cjs +6282 -3859
- package/dist/cli.d.cts +282 -3
- package/dist/cli.d.ts +282 -3
- package/dist/cli.js +198 -87
- package/dist/generate-worker-D_mUjgXw.d.cts +73 -0
- package/dist/generate-worker-D_mUjgXw.d.ts +73 -0
- package/dist/index.cjs +2981 -1057
- package/dist/index.d.cts +337 -25
- package/dist/index.d.ts +337 -25
- package/dist/index.js +1 -3
- package/dist/mcp.cjs +6426 -5065
- package/dist/mcp.d.cts +136 -68
- package/dist/mcp.d.ts +136 -68
- package/dist/mcp.js +2 -4
- package/dist/{types-Ca4PmkiQ.d.cts → types-DIX6zWSt.d.cts} +130 -2
- package/dist/{types-Ca4PmkiQ.d.ts → types-DIX6zWSt.d.ts} +130 -2
- package/docs/agent.md +1 -1
- package/docs/cli.md +26 -6
- package/docs/configuration.md +48 -9
- package/docs/how-to-feed-a-repo-to-claude.md +61 -0
- package/docs/mcp.md +36 -10
- package/docs/output.md +49 -4
- package/docs/stale-context-is-your-bug.md +13 -0
- package/docs/vs-repomix.md +1 -1
- package/package.json +5 -3
package/CHANGELOG.md
CHANGED
|
@@ -21,6 +21,879 @@ Two versions exist on the registry that are not releases, and both are deprecate
|
|
|
21
21
|
Deleting them requires account-level rights that the release pipeline does not have, so they are
|
|
22
22
|
deprecated with an explanatory message instead. `latest` has pointed at 0.9.0 since it existed.
|
|
23
23
|
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
## 0.9.14
|
|
27
|
+
|
|
28
|
+
A fifth audit, of what the tool does to text it did not write. Three of the defects below were
|
|
29
|
+
editing the user's code: a redaction rule erased 36 lines of source, a truncation left a code fence
|
|
30
|
+
open, and a preview budget did not apply to a file with CR line endings. The rest are credentials
|
|
31
|
+
left in clear text, two records that disagreed with the files beside them, and the performance work
|
|
32
|
+
the previous audits kept deferring.
|
|
33
|
+
|
|
34
|
+
Every item has a test that fails without it, with the exception of two performance changes that have
|
|
35
|
+
no behavioural signal — both are named.
|
|
36
|
+
|
|
37
|
+
### A redaction rule erased source code
|
|
38
|
+
|
|
39
|
+
`PEM_BLOCK_PATTERN` is lazy over `[\s\S]`, which places no restriction on what may lie between a
|
|
40
|
+
`BEGIN` and the next `END`. A file holding an *unterminated* header — a truncated key, a fixture, a
|
|
41
|
+
half-written file — and, later, a properly terminated block had **everything between the two
|
|
42
|
+
headers replaced with blank lines**, including the source code that happened to sit there.
|
|
43
|
+
|
|
44
|
+
This repository did it to its own test suite: the security fixture's "unterminated key" test was
|
|
45
|
+
followed by another test whose key was terminated, and the generator bridged the two — 36 lines of
|
|
46
|
+
assertions, braces and the second test's declaration, replaced with 36 empty newlines.
|
|
47
|
+
|
|
48
|
+
The body may no longer contain another `-----BEGIN`. A header with no `END` before the next `BEGIN`
|
|
49
|
+
does not match as a block, and `PEM_UNTERMINATED_PATTERN` handles its header and its own base64 body,
|
|
50
|
+
which is what that pattern is for. The newline after the header is deliberately still not required:
|
|
51
|
+
a PEM inside a JSON or `.env` value is written with escaped `\n`, two characters rather than a line
|
|
52
|
+
break, and a rule that insisted on one would mask the header alone and leave the base64 in clear.
|
|
53
|
+
|
|
54
|
+
### A truncated section left its code fence open
|
|
55
|
+
|
|
56
|
+
When the fixed metadata of a section would not fit the part budget, the *rendered* section was cut at
|
|
57
|
+
a byte offset. The cut lands in the body or in the opening fence, and the closing fence it removes is
|
|
58
|
+
the one that ends the block: every section after it was rendered inside this file's code block, and
|
|
59
|
+
`parsePartSections` merged the next file into this one's body.
|
|
60
|
+
|
|
61
|
+
The body now goes and the notice takes its place inside the block, with the file marked `preview`
|
|
62
|
+
first so the re-rendered `- Meta:` line describes what the reader will actually find. A section that
|
|
63
|
+
is a closed, empty, labelled block is readable; an unclosed fence corrupts the files after it, and no
|
|
64
|
+
block at all is invisible to the part reader.
|
|
65
|
+
|
|
66
|
+
### A file with CR line endings was never previewed
|
|
67
|
+
|
|
68
|
+
`countLines` and `truncatePreview` split on `\n` alone, so a CR-only file — an old Mac file, anything
|
|
69
|
+
written by a tool that predates LF on that platform — counted as **one line**. `largeFileBytes`
|
|
70
|
+
therefore never fired: a 5 MB CR file went into a part whole, with its carriage returns intact, and
|
|
71
|
+
`manifest.json` reported `lines: 1` for a 40 000-line file. The redaction pass counted terminators
|
|
72
|
+
with `/\r\n|\n|\r/g`, so `reports/audit.md` reported line numbers the manifest disagreed with.
|
|
73
|
+
|
|
74
|
+
All four places now normalise CR, LF and CRLF, and `countLines` counts by scanning rather than
|
|
75
|
+
splitting — it runs on every file body and every part, and normalising first allocated a second copy
|
|
76
|
+
of a multi-megabyte string to answer a question about its terminators.
|
|
77
|
+
|
|
78
|
+
### Four credential shapes were left in clear text
|
|
79
|
+
|
|
80
|
+
**`${VAR:-fallback}`, unquoted.** `POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-MyProdPassword123!}`
|
|
81
|
+
defeated every rule, and each refusal was individually reasonable: the quoted rules need a quote
|
|
82
|
+
after the separator, and the unquoted ones need the value to begin with a word character — a
|
|
83
|
+
fallback begins with `$`. That idiom is the norm in `.env` files, docker-compose, GitHub Actions and
|
|
84
|
+
shell scripts. A rule for the shape now masks the fallback and keeps the variable name.
|
|
85
|
+
|
|
86
|
+
**`readonly` and `declare` in shell files.** Those words introduce a binding in TypeScript and an
|
|
87
|
+
assignment in POSIX sh and `.envrc` — the same word, opposite meaning. The guard that suppresses
|
|
88
|
+
redaction for declarations therefore suppressed it for exactly the files most likely to hold a
|
|
89
|
+
credential. Bare `declare` is gone from the keyword list (the form that can initialise a binding in
|
|
90
|
+
TS is `declare const`, matched through `const`), and the guard now knows which files are assignments.
|
|
91
|
+
|
|
92
|
+
**A credential inside a string literal.** `startsInsideStringLiteral` rejected any match whose name
|
|
93
|
+
began immediately after a quote, which is right for a list of key names and wrong for every
|
|
94
|
+
connection string: `const dsn = "password=Secret;host=db"` and `curl -d "password=Secret"` were
|
|
95
|
+
emitted in clear. The test now also asks whether the literal closes between the name and the value.
|
|
96
|
+
|
|
97
|
+
**The MCP line cap.** `createLineReader` checked `MAX_LINE_BYTES` only on the buffer it had just
|
|
98
|
+
appended to, so a chunk carrying a discarded line's terminator *and* the start of the next message
|
|
99
|
+
left that remainder unmeasured until the stream ended — and `onEnd` delivered it whole to
|
|
100
|
+
`JSON.parse`. It also compared `.length`, which is UTF-16 code units: a line of astral characters
|
|
101
|
+
could be twice the ceiling in bytes and still pass. The cap is applied at every path that can
|
|
102
|
+
deliver, and measured in bytes.
|
|
103
|
+
|
|
104
|
+
### `--with-dependents` lost an importer, and three languages had no import graph
|
|
105
|
+
|
|
106
|
+
The closure was computed from `file.summary.imports`, and a previewed or oversized file has no
|
|
107
|
+
summary. An importer over `largeFileBytes` was therefore dropped as unchanged while `validate` and
|
|
108
|
+
`explain`, which read the same file from disk, called it a dependent — generate and validate
|
|
109
|
+
disagreed about the snapshot generate had just written. Import specifiers are cheap to extract from
|
|
110
|
+
text already in hand, so they are now collected for files whose summary was refused, and the size
|
|
111
|
+
ceiling lives in one place so all three commands refuse the same files.
|
|
112
|
+
|
|
113
|
+
`resolveSpecifier` only understood JavaScript's `./x` shape, so the Python, Ruby, C/C++ and Dart
|
|
114
|
+
imports the profiles have been recording for several releases resolved to nothing. Each language
|
|
115
|
+
spells "this directory" differently — Python with dots, Ruby with `require_relative`, C with a
|
|
116
|
+
quoted include, Dart with nothing at all — and each is now resolved on its own terms. Package-rooted
|
|
117
|
+
languages (Go modules, Rust `crate::`, PHP PSR-4, JVM packages, Swift modules) stay unresolved on
|
|
118
|
+
purpose: their specifiers go through a manifest, and guessing a path would fabricate edges that
|
|
119
|
+
`validate` then has to disagree about.
|
|
120
|
+
|
|
121
|
+
### `setup --format json` could corrupt its own output
|
|
122
|
+
|
|
123
|
+
`setup` replaced `process.stdout.write` with a capture and restored the function it had captured on
|
|
124
|
+
entry. A nested generate with an `async` hook installed the hook diversion inside that window, so
|
|
125
|
+
`setup`'s restore ran first and removed it — the hook's `console.log`, after its first `await`, went
|
|
126
|
+
straight to the real stdout — and the hook's restore then put back `setup`'s *finished* capture.
|
|
127
|
+
Every later write in the process was appended to a dead string and silently discarded.
|
|
128
|
+
|
|
129
|
+
Both borrow stdout and neither can own it, so there is now one router, installed at most once and
|
|
130
|
+
released by reference counting, consulting two `AsyncLocalStorage` scopes.
|
|
131
|
+
|
|
132
|
+
The same function also merged a failed nested generate in as `manifest`: `{ setup, manifest: { error } }`
|
|
133
|
+
came out with exit code 1 and no top-level `error`. The failure is now at the top level.
|
|
134
|
+
|
|
135
|
+
### A transient lock on the old snapshot could cost the new one
|
|
136
|
+
|
|
137
|
+
`fs.rmSync(backupDir)` ran inside the `try` whose `catch` restores the previous snapshot. `rmSync`
|
|
138
|
+
throws EPERM or EBUSY on Windows and on network filesystems when an indexer, an antivirus or an
|
|
139
|
+
editor holds a handle in the old snapshot — and by then `outDir` already holds the new one. The catch
|
|
140
|
+
deleted the new snapshot, restored the old one and rethrew; the retry then failed with ENOENT
|
|
141
|
+
because the staging directory no longer existed. Removing the backup is no longer part of the
|
|
142
|
+
transaction.
|
|
143
|
+
|
|
144
|
+
### Two runs could take two different lock files
|
|
145
|
+
|
|
146
|
+
`realpathSync(stagingRoot)` was used when the root existed and the unresolved path when it did not —
|
|
147
|
+
and the run creates the root *after* taking the lock. On macOS, where `/var` is a symlink to
|
|
148
|
+
`/private/var`, process 1 hashed `/var/folders/…/snapshot.tmp` and process 2, starting once the
|
|
149
|
+
directory existed, hashed `/private/var/…`: two locks, no mutual exclusion, two concurrent writers.
|
|
150
|
+
The key now resolves the nearest existing ancestor. Reclaiming a stale lock also re-checks the file's
|
|
151
|
+
identity immediately before unlinking it, so two runs that both find the same dead lock cannot
|
|
152
|
+
delete each other's.
|
|
153
|
+
|
|
154
|
+
### The MCP worker ran the hook the docs say it does not
|
|
155
|
+
|
|
156
|
+
`docs/mcp.md` tells the reader that `hooks.afterGenerate` does not run for a generation started from
|
|
157
|
+
MCP. It did: the worker resolved the config file with its hooks intact, so an after-generate script
|
|
158
|
+
ran from inside an MCP tool call, and a hook still running when the worker posted its result kept
|
|
159
|
+
the thread alive for as long as the server did. The worker now empties `hooks` before resolving, and
|
|
160
|
+
is terminated as soon as the result is settled.
|
|
161
|
+
|
|
162
|
+
### A byte-order mark in `package.json` broke four readers
|
|
163
|
+
|
|
164
|
+
`findConfigFile` reads `package.json` through `stripJsonComments`, which drops a BOM, so `generate`
|
|
165
|
+
loaded a BOM'd project happily. Every other reader used bare `JSON.parse`, which throws on `\uFEFF`,
|
|
166
|
+
and every one of them caught the throw and reported the ordinary, wrong answer: an existing
|
|
167
|
+
`"snapshot"` block was invisible, `"type": "module"` was invisible, the scripts and devDependency
|
|
168
|
+
were skipped with a note, and `nextStepHint` told the user to run `npx codebase-snapshot generate` —
|
|
169
|
+
which resolves the *published* package instead of the one the run had just installed. The
|
|
170
|
+
`npm create` shim had the same bug in `hasSnapshotScript`.
|
|
171
|
+
|
|
172
|
+
A BOM is what a Windows editor writes when it saves UTF-8. All five readers now parse the same way
|
|
173
|
+
the config loader does, and `init` writes the mark back rather than silently removing it.
|
|
174
|
+
|
|
175
|
+
### `validate` could not name seven of the settings it compared
|
|
176
|
+
|
|
177
|
+
`maxPartTokens`, `maxMonolithBytes`, `skeleton`, `sarif`, `changedSince`, `staged` and
|
|
178
|
+
`withDependents` were recorded in the manifest and folded into the configuration digest, but never
|
|
179
|
+
joined the field-by-field block. The digest can say *that* something changed; only the per-field
|
|
180
|
+
comparisons say *what*. Toggling `--skeleton` produced the generic "the settings differ" line and
|
|
181
|
+
nothing else.
|
|
182
|
+
|
|
183
|
+
### A `.vue` or `.astro` script block was always parsed as `.ts`
|
|
184
|
+
|
|
185
|
+
`<Button onClick={() => go()}>` is a type assertion followed by six parse errors to the TypeScript
|
|
186
|
+
grammar, and no component at all, for a perfectly valid Vue `lang="tsx"` block. The script kind now
|
|
187
|
+
follows the component: Astro frontmatter is JSX-capable by construction, and a Vue or Svelte block
|
|
188
|
+
says so with `lang="tsx"`.
|
|
189
|
+
|
|
190
|
+
### Performance
|
|
191
|
+
|
|
192
|
+
- **The split loop re-rendered every part on every pass.** `partitionFilesForBuckets` renders all
|
|
193
|
+
groups, finds the first one over budget, splits it, and goes round again — up to 64 times. Each
|
|
194
|
+
pass re-assembled, re-token-estimated and SHA-256 hashed *every* part. A group's rendered size
|
|
195
|
+
depends on its sections, its index and the *width* of the numbering, and the width only changes
|
|
196
|
+
when the part count crosses 1 000, so sizes are now measured instead of rendered and cached per
|
|
197
|
+
group. Each part is hashed exactly once.
|
|
198
|
+
- **The skeleton pass parsed with `setParentNodes: true`** and walked every node a second time to set
|
|
199
|
+
`.parent`, which nothing reads. It also expanded empty bodies from `{}` to `{ /* ... */ }`, making
|
|
200
|
+
small files larger, and rewrote the text once per span — 400 functions copied a 200 KB file 400
|
|
201
|
+
times.
|
|
202
|
+
- **`fullContents` retained every file body on runs that wrote no monolith.** The map held a
|
|
203
|
+
reference to every pre-partition body and was read only inside `if (config.includeMonolith)`,
|
|
204
|
+
pinning the multi-megabyte originals that the partitioner had just replaced with previews.
|
|
205
|
+
- **31 regexes were recompiled for every file.** `scanAndRedactSecrets` called `toGlobal` per rule
|
|
206
|
+
per call, and it is called once per file: a 20 000-file scan built 620 000 identical objects.
|
|
207
|
+
- **`--watch` re-derived the whole repository for one changed file.** The derivation — redaction,
|
|
208
|
+
findings, skeleton, preview, summary — is a pure function of (bytes, settings, path), so it is now
|
|
209
|
+
cached and reused when the content hash and a settings digest both match. Validity is the **content
|
|
210
|
+
hash**, not `mtime`: a formatter or a `git checkout` can rewrite a file within the same timestamp
|
|
211
|
+
tick at the same length. Measured on a synthetic 250-file TypeScript repository: cold 520 ms, warm
|
|
212
|
+
196 ms of wall clock (2.65x) — the remainder is reading, hashing, partitioning and writing, which
|
|
213
|
+
a per-file cache cannot remove. The suite asserts the *work* instead, in CPU time on a denser
|
|
214
|
+
fixture: 1 046 ms against 48 ms, 21x, against a 3x floor.
|
|
215
|
+
|
|
216
|
+
### What has no test, and why
|
|
217
|
+
|
|
218
|
+
The last two are structural. The regex cache and the `fullContents` change have no behavioural
|
|
219
|
+
signal — the output is byte-identical by construction — so they are covered by the existing suite
|
|
220
|
+
and by the reasoning above rather than by a new test. The `--watch` cache is different: four of its
|
|
221
|
+
five tests are correctness guards that pass on the previous build (that is the requirement — the
|
|
222
|
+
cache must be invisible), and the fifth measures the CPU time the cache saves and fails without it.
|
|
223
|
+
|
|
224
|
+
---
|
|
225
|
+
|
|
226
|
+
## 0.9.13
|
|
227
|
+
|
|
228
|
+
A fourth audit, this time of the seams: places where two code paths answer the same question with
|
|
229
|
+
different rules, and places where a command accepts something it does not apply. Every item below
|
|
230
|
+
has a test that fails without it, except one — the per-layer budget overflow, which I could not
|
|
231
|
+
reproduce and which is called out honestly at the end.
|
|
232
|
+
|
|
233
|
+
### A scoped snapshot could expire and still validate
|
|
234
|
+
|
|
235
|
+
`validate` re-resolves the recorded changed-files scope and checks that every file it *skipped* as
|
|
236
|
+
`unchanged` is still outside that scope. It never asked the mirror question. So:
|
|
237
|
+
|
|
238
|
+
```bash
|
|
239
|
+
git add src/a.ts
|
|
240
|
+
codebase-snapshot generate --staged # snapshot holds src/a.ts
|
|
241
|
+
git reset HEAD src/a.ts # same bytes on disk, nothing staged
|
|
242
|
+
codebase-snapshot validate --staged # said: "Snapshot is valid and up to date"
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
The file on disk is byte-identical, so the hash comparison passed — and the scope the snapshot was
|
|
246
|
+
taken over had silently become empty. A file that is in the snapshot and would not be if it were
|
|
247
|
+
generated now is drift, and it is now reported, with `--with-dependents` still allowing for the
|
|
248
|
+
one-hop importer exception.
|
|
249
|
+
|
|
250
|
+
### A part name in `manifest.json` could reach any file on the host
|
|
251
|
+
|
|
252
|
+
`validate` has confined part filenames since 0.9.9, but the **MCP server** resolved
|
|
253
|
+
`parts[].filename` with a bare `path.resolve`, in `snapshot_read` and in the `snapshot://parts/{n}`
|
|
254
|
+
resource. A repository under review could commit a manifest naming `../../../.env`, and a client
|
|
255
|
+
would read it — resource reads are auto-approved as read-only. A part name is bare by construction,
|
|
256
|
+
so `describeManifestShapeProblem` now requires one, which makes all five readers (`validate`,
|
|
257
|
+
`report`, `diff`, `snapshot_read`, the resource) refuse the same manifest for the same stated
|
|
258
|
+
reason.
|
|
259
|
+
|
|
260
|
+
`snapshot_explain` had the mirror problem. `toRelativePath` resolved *after* its "is it absolute?"
|
|
261
|
+
test and `normalizeTarget` returned anything starting with `..` unchanged, so `../other-repo/.env`
|
|
262
|
+
was joined to the root and passed to `existsSync`: one read-only, auto-approved call per probe, for
|
|
263
|
+
any path on the machine. Out-of-root targets now answer `outside-root` without touching the
|
|
264
|
+
filesystem. A file named `..foo.ts` is no longer mistaken for a traversal.
|
|
265
|
+
|
|
266
|
+
### Four credentials that looked like code
|
|
267
|
+
|
|
268
|
+
- `password: "${DB_PASSWORD:-hunter2}"` — the shell and Compose idiom for a **default** — was
|
|
269
|
+
exempted as "an interpolation" because it starts with `${` and ends with `}`. So was
|
|
270
|
+
`${prefix}-real_secret-${suffix}` and any value with a `??` fallback. Only a value that is one
|
|
271
|
+
interpolation of a plain (optionally dotted) reference is exempt now; anything else in the braces
|
|
272
|
+
settles it the other way.
|
|
273
|
+
- `export api_key=hunter2` — bare `export` was in `DECLARATION_KEYWORDS`, so a shell assignment was
|
|
274
|
+
read as a JavaScript binding whose initializer is an expression. `export` can never be followed by
|
|
275
|
+
a bare assignment in JS or TS (`export foo = 1` does not parse), so the word only ever suppressed
|
|
276
|
+
redaction for the files most likely to hold a secret.
|
|
277
|
+
- An **indented, unterminated** private key — `privateKey: |` in YAML, or an indented template
|
|
278
|
+
literal — leaked every body line. The rule required base64 at column 0 and knew nothing about the
|
|
279
|
+
RFC 1421 header lines (`Proc-Type`, `DEK-Info`) that an encrypted key puts after the marker.
|
|
280
|
+
- A masked key block in a **CR-only** file swallowed its line breaks, because pass 1 counted `\n`
|
|
281
|
+
only, so every reported line number after it was wrong.
|
|
282
|
+
|
|
283
|
+
`.github/workflows/deprecate.yml` and `unpublish.yml` interpolated `${{ inputs.… }}` straight into
|
|
284
|
+
`run:` in jobs that carry `NPM_TOKEN`. GitHub expands an expression into the script *before* bash
|
|
285
|
+
parses it, so a dispatch input of `"; curl … $NODE_AUTH_TOKEN #` ran as a command. Inputs now arrive
|
|
286
|
+
through `env:` and are referenced quoted.
|
|
287
|
+
|
|
288
|
+
### Two enumerations, two answers
|
|
289
|
+
|
|
290
|
+
The same repository produced different snapshots depending on `--discovery`, which is the one
|
|
291
|
+
property a reproducible snapshot tool cannot have:
|
|
292
|
+
|
|
293
|
+
- A tracked **file symlink** was dropped as `symlink` in `git` mode — the loop tested
|
|
294
|
+
`depth <= parts.length`, which includes the candidate itself — and emitted as a file by the walk.
|
|
295
|
+
- A **broken** symlink was `symlink` in `git` mode, because `targetReal` started at the link's own
|
|
296
|
+
path and so always compared as "inside the root". It is `read-error`, or `outside-root` when its
|
|
297
|
+
written target leaves the tree, in both modes now.
|
|
298
|
+
|
|
299
|
+
The output directory's crash leftovers had the same shape: the backup and partial-swap directories
|
|
300
|
+
were recognised by the hardcoded prefix `snapshot.backup-`. With `outDir: ".context"` an interrupted
|
|
301
|
+
run left `.context.backup-<pid>-<ts>` — a copy of the previous snapshot — to be scanned back in as
|
|
302
|
+
user source. The names now come from the resolved output and staging directories. `--watch` also
|
|
303
|
+
matched only `.backup-`, and only the directory itself, so an event for `part-001.md` *inside* a
|
|
304
|
+
`.partial-` copy was treated as a source edit: on a cross-device swap that is an infinite
|
|
305
|
+
regeneration loop.
|
|
306
|
+
|
|
307
|
+
### The index described files that were not there
|
|
308
|
+
|
|
309
|
+
- `summarizeMarkdown` called a leading `---` a front-matter block if *any* later line was `---`,
|
|
310
|
+
so a document opening with a horizontal rule and using another as a section divider lost every
|
|
311
|
+
heading between them. The block must now also *be* YAML — a heading, a fence or prose means the
|
|
312
|
+
`---` was a rule. `...` closes a YAML document as well as `---`.
|
|
313
|
+
- The JSONC reader in `src/ast/structured.ts` ended a `//` comment only at `\n` and never removed a
|
|
314
|
+
trailing comma, so a `.jsonc` or an editor-reformatted `tsconfig.json` summarised as
|
|
315
|
+
`invalid-json` with no keys. Both are JSONC rather than extensions of it.
|
|
316
|
+
- `summarizeTypeScriptSource` looked at `ImportDeclaration` and `ExportDeclaration` and nothing
|
|
317
|
+
else, so `require("./x")`, `import("./x")` and `import fs = require("fs")` produced no import
|
|
318
|
+
edge. A CommonJS project had no import graph at all, and `--with-dependents` had nothing to walk.
|
|
319
|
+
`export { a } from "./x"` was not recorded as a re-export either.
|
|
320
|
+
|
|
321
|
+
### Flags that were accepted and not applied
|
|
322
|
+
|
|
323
|
+
- `--only-artifacts audit` in a project whose config says `artifacts: { audit: false }` disabled the
|
|
324
|
+
other six and left audit off too: nothing written, exit 0. The flag was forwarded only as its
|
|
325
|
+
complement, and a suppression list cannot switch anything on.
|
|
326
|
+
- `findConfigFile` parsed `package.json` with bare `JSON.parse` while the loader used the JSONC
|
|
327
|
+
reader, so a BOM or a trailing comma made the two disagree about whether the project was
|
|
328
|
+
configured. And `"snapshot": []` counted as this tool's configuration, which skipped the setup
|
|
329
|
+
question entirely.
|
|
330
|
+
- `--skeleton` covered `.ts`, `.tsx`, `.mts` and `.cts` while `ts.createSourceFile` parses `.js`,
|
|
331
|
+
`.jsx`, `.mjs` and `.cjs` natively — so the files most repositories are made of were the ones it
|
|
332
|
+
could not compress, and `.d.ts` files, which have no bodies, were the ones it did parse.
|
|
333
|
+
- Over MCP, `WorkspaceOverrides` accepted only `maxPartBytes` and `outDir`. `snapshot_validate`
|
|
334
|
+
re-derives the configuration and compares digests, so a `--skeleton` snapshot could not be
|
|
335
|
+
validated at all. `maxPartTokens`, `skeleton`, `sarif`, `changedSince`, `staged` and
|
|
336
|
+
`withDependents` are now accepted by `snapshot_plan`, `snapshot_generate`, `snapshot_validate`
|
|
337
|
+
and `snapshot_explain` — schema and application coming from one definition, so a tool cannot
|
|
338
|
+
advertise a flag it would ignore.
|
|
339
|
+
- `npm create codebase-snapshot -- --preset=nextjs` ignored the preset: the shim parsed `argv` with
|
|
340
|
+
`indexOf`, which finds nothing for `--flag=value`, treats a missing value as "not given", and
|
|
341
|
+
ignores `--forse` entirely. `node:util.parseArgs` in strict mode now rejects all three by name.
|
|
342
|
+
|
|
343
|
+
### Configuration answers that disagreed with each other
|
|
344
|
+
|
|
345
|
+
- `loadConfigFileDetailed` did not attach `CONFIG_DIR_SYMBOL` — only `loadConfigFile` did — so
|
|
346
|
+
`resolveConfig((await loadConfigFileDetailed(dir)).config)` scanned `process.cwd()`.
|
|
347
|
+
- `normalizeResolvedConfig` re-derived `artifacts` with `includeMonolith` as the default, so
|
|
348
|
+
`{ ...resolved, includeMonolith: true }` produced a config claiming `includeMonolith: true`
|
|
349
|
+
beside `artifacts.monolith: false`, and `generate` — which reads the map — wrote no monolith.
|
|
350
|
+
- `configDigest` hashed `budgets` with `JSON.stringify`, which preserves insertion order, so
|
|
351
|
+
`{ docs, api }` and `{ api, docs }` produced different digests and `validate` reported drift for
|
|
352
|
+
a snapshot nobody changed.
|
|
353
|
+
|
|
354
|
+
### Cost
|
|
355
|
+
|
|
356
|
+
- `pathScanRules` returned `[...DEFAULT_SECRET_PATTERNS, ...extraRules]` although
|
|
357
|
+
`config.secretRules` already begins with those patterns: every file path, skip record and AST
|
|
358
|
+
summary string ran all 30 built-in rules twice. `disableDefaultPatterns` also had no effect on
|
|
359
|
+
paths while it worked on content.
|
|
360
|
+
- `globToRegExp` and `matchGitignorePattern` compiled a fresh `RegExp` per call — over a million per
|
|
361
|
+
run on a 20 000-file repository with 55 patterns. Both are cached by pattern. A malformed
|
|
362
|
+
character class no longer escapes as a raw `SyntaxError`: a `.gitignore` range like `[z-a]`
|
|
363
|
+
matches nothing (which is what git's own `fnmatch` does), while the same range in a configured
|
|
364
|
+
glob is reported as the configuration mistake it is, naming the pattern.
|
|
365
|
+
- `generateSnapshotAsync` resolved the configuration twice, and `validate` in the CLI re-read and
|
|
366
|
+
re-parsed `manifest.json` to re-run a shape check the validator had already run.
|
|
367
|
+
- `pendingHooks` was drained only by `generateSnapshotAsync`, so a synchronous caller left one
|
|
368
|
+
settled promise per output directory for the life of the process.
|
|
369
|
+
|
|
370
|
+
### Lifecycle
|
|
371
|
+
|
|
372
|
+
- Each `hooks.afterGenerate` installed its own `process.stdout.write` wrapper and restored it only
|
|
373
|
+
if the current write was still its own. With two overlapping async hooks the first to finish never
|
|
374
|
+
restored, and the second put the first one's wrapper back — permanently. There is now one scope
|
|
375
|
+
and one wrapper, released by reference counting.
|
|
376
|
+
- The update check used `request.setTimeout`, an *idle socket* timeout that resets on every chunk:
|
|
377
|
+
a trickling proxy kept the CLI alive indefinitely. It has a wall-clock deadline now, `.unref()`d
|
|
378
|
+
so a pending check never holds the process open. And `isNewer` stripped prerelease tags before
|
|
379
|
+
comparing, so `0.9.13-alpha.1` was recommended to everyone on a stable release and a stable
|
|
380
|
+
release was never announced to anyone on a beta.
|
|
381
|
+
|
|
382
|
+
### Smaller
|
|
383
|
+
|
|
384
|
+
- `wrapOneLine` measured `room` in terminal cells and cut with `String.prototype.slice`, which
|
|
385
|
+
counts UTF-16 code units. CJK text produced rows twice as wide as the frame believed, and a
|
|
386
|
+
surrogate pair could be cut in half. The cut advances by code point now.
|
|
387
|
+
- Part numbering is decided **before** rendering, in one pass over every budget bucket, instead of
|
|
388
|
+
partitioning each budget and re-numbering the merged set afterwards.
|
|
389
|
+
|
|
390
|
+
### What I could not reproduce
|
|
391
|
+
|
|
392
|
+
The audit reported that per-layer budgets caused parts to overflow their budget by seven to twenty
|
|
393
|
+
bytes when `renumberParts` rewrote their headers, and were then hard-clipped into a broken document.
|
|
394
|
+
**I could not make that happen.** I swept file sizes against a two-budget configuration and again
|
|
395
|
+
with six layers and eight kinds whose names are 180 characters long, and no part ever came out over
|
|
396
|
+
its budget, with or without the clamp. The arithmetic explains why: `partitionFiles` reserves about
|
|
397
|
+
2.2 kB of header room up front and shrinks sections to `budget - reserve`, so a rendered part sits
|
|
398
|
+
roughly 1.4 kB below its ceiling, and the header growth being reported is fourteen bytes.
|
|
399
|
+
|
|
400
|
+
The mechanism was removed anyway — the numbering pass now happens before rendering, so a fitted part
|
|
401
|
+
cannot be rewritten — but the claim was not verified, and this changelog will not pretend it was.
|
|
402
|
+
The four tests added with it pass on the previous build too; they pin the invariants the refactor
|
|
403
|
+
must not break rather than reproducing a defect.
|
|
404
|
+
|
|
405
|
+
## 0.9.12
|
|
406
|
+
|
|
407
|
+
A third audit, this time of what the snapshot *costs* as well as what it gets wrong. Two of the
|
|
408
|
+
defects were circular — `validate` demanded a flag it refused to accept — and one of them meant a
|
|
409
|
+
config file could not turn a setting on at all. Every item below has a test that fails without it.
|
|
410
|
+
|
|
411
|
+
### The parts were 32 000 tokens of duplicated metadata
|
|
412
|
+
|
|
413
|
+
Each file section carried nine metadata bullets and, for a TS/JSON/Markdown file, a
|
|
414
|
+
pretty-printed JSON summary of code that followed three lines later. On this repository — 234
|
|
415
|
+
files — that was **195 619 bytes, about 32 600 tokens, 12 % of the snapshot**, spent on a
|
|
416
|
+
`SHA-256:` line per file and on listing every variable and export immediately above the code that
|
|
417
|
+
declares it.
|
|
418
|
+
|
|
419
|
+
A section is now an anchor, a heading and one line:
|
|
420
|
+
|
|
421
|
+
```text
|
|
422
|
+
- Meta: src/other | typescript | full | 3 lines (63 B)
|
|
423
|
+
```
|
|
424
|
+
|
|
425
|
+
- The per-file `SHA-256` is **gone from the parts** and stays in `manifest.json`, which is the file
|
|
426
|
+
a machine reads. `snapshot_read` used to fall back to it when a path was too long for the
|
|
427
|
+
heading; it now matches the `<a id>` anchor, which is a digest of the exact `relativePath` and was
|
|
428
|
+
already there — exact even for the 300-character paths the heading caps.
|
|
429
|
+
- The structural summary is **not printed for a `full` file**: `manifest.json` keeps it, and
|
|
430
|
+
`preview` and `signatures` sections keep it, because there the body is not the file. A one-line
|
|
431
|
+
string summary survives either way.
|
|
432
|
+
- `- Content Fingerprint:` stays in the part header (~22 tokens per part), which anchors an answer
|
|
433
|
+
to a verifiable state of the repository.
|
|
434
|
+
|
|
435
|
+
Nothing else about the format moved, so `snapshot_read` and both structural parsers still read it.
|
|
436
|
+
|
|
437
|
+
### `validate` refused the flag its own comparison needed
|
|
438
|
+
|
|
439
|
+
`manifest.config.digest` hashes `skeleton`, `sarif`, `maxPartTokens`, `changedSince`, `staged` and
|
|
440
|
+
`withDependents`, and `validate` compared that digest — but `COMMAND_OPTION_SCOPE` allowed those six
|
|
441
|
+
flags on `generate` only. So a snapshot generated with `--skeleton`:
|
|
442
|
+
|
|
443
|
+
1. failed `validate` with "the settings differ from the ones this snapshot was generated with", and
|
|
444
|
+
2. failed `validate --skeleton` with "`--skeleton` does not apply to validate".
|
|
445
|
+
|
|
446
|
+
There was no way to validate it. `validate` now accepts all six, `explain` accepts the three that
|
|
447
|
+
decide its answer, and `report` accepts none: it reads the manifest and prints what is there, so a
|
|
448
|
+
generation setting would change nothing.
|
|
449
|
+
|
|
450
|
+
### A config file could not turn these four settings on
|
|
451
|
+
|
|
452
|
+
`parseArgs` returned `skeleton: values.skeleton === true`, so a run that did not pass the flag sent
|
|
453
|
+
**`false`** — and `resolveConfig` reads these as `cliOptions.x ?? config.x`, where `false` is an
|
|
454
|
+
explicit override. `skeleton: true` in `snapshot.config.ts` was silently turned off by every run
|
|
455
|
+
that did not repeat the flag on the command line. The same applied to `staged`, `withDependents`
|
|
456
|
+
and `sarif`. All four are now `undefined` when the flag is absent.
|
|
457
|
+
|
|
458
|
+
### `explain` and `validate` ignored `--with-dependents`
|
|
459
|
+
|
|
460
|
+
The generator widened a changed-files scope by one hop of importers; both readers compared the raw
|
|
461
|
+
path set. A file that was **in** the snapshot because it imported something that changed was
|
|
462
|
+
explained as `included: false, reason: "unchanged"`, and `validate` called the snapshot current
|
|
463
|
+
while a file it should contain was missing from it. The import graph moved to
|
|
464
|
+
`src/scanner/dependents.ts` so all three apply one rule, and `validate` now re-parses a recorded
|
|
465
|
+
`unchanged` skip only when the manifest says the run used `--with-dependents`.
|
|
466
|
+
|
|
467
|
+
### `setup` wrote a config its own snapshot did not match
|
|
468
|
+
|
|
469
|
+
`setup --discovery walk --large-file-bytes 100000` forwarded both flags to the generation it runs
|
|
470
|
+
and wrote the defaults into the config beside it, so a brand-new project's one-command setup
|
|
471
|
+
finished with a snapshot that `validate` immediately called drifted — three configuration
|
|
472
|
+
mismatches on the first run. `--max-part-tokens`, `--skeleton` and `--sarif` are now accepted by
|
|
473
|
+
`setup` too, and every one of them is written into the generated config as well as forwarded.
|
|
474
|
+
|
|
475
|
+
### Redaction that corrupted what it protected
|
|
476
|
+
|
|
477
|
+
Three shapes were found in this repository's own snapshot, where the tool rewrote its own
|
|
478
|
+
documentation and tests:
|
|
479
|
+
|
|
480
|
+
- **Prose.** `credential-assignment-unquoted` masked any word after `secret: ` or `token: `, so
|
|
481
|
+
`indistinguishable from a secret: a` became `…from a secret: [REDACTED…]`. An unquoted value in a
|
|
482
|
+
line-oriented format never continues after a space (`PASSWORD: hunter2` ends the line; `secret: a
|
|
483
|
+
value` is a sentence), and a one-character bare value is not a credential. Quoted values keep
|
|
484
|
+
their one-character floor — quotes are evidence somebody meant to write a value.
|
|
485
|
+
- **Template literals.** `` `… an empty token: "${appliesTo}"` `` had the interpolation masked,
|
|
486
|
+
because `startsInsideStringLiteral` only looks at the character before the identifier. A value
|
|
487
|
+
that is entirely `${…}` is an expression, so it is not masked.
|
|
488
|
+
- **Doubly-escaped quotes.** `JSON.parse('{"password": "p@\\"ssw0rdINVENTED"}')` — how every
|
|
489
|
+
fixture that embeds JSON is written — emitted `"[REDACTED…]"ssw0rdINVENTED"`: a fragment of the
|
|
490
|
+
credential in clear and a line that no longer parses. A quote after `\\` that is followed by a
|
|
491
|
+
value character is an escaped quote in a deeper literal, and the value now runs to the real close.
|
|
492
|
+
|
|
493
|
+
### An async `afterGenerate` hook's files were not recorded
|
|
494
|
+
|
|
495
|
+
The hook is documented as the way to write your own artifact into the snapshot directory, and that
|
|
496
|
+
directory is only replaceable while `.codebase-snapshot` accounts for every file in it. Only the
|
|
497
|
+
*rejection* path re-shipped the marker, so an async hook that succeeded left its file unrecorded
|
|
498
|
+
and the next `generate` refused the directory as foreign. The marker is now re-shipped when the
|
|
499
|
+
hook settles either way, and only while it still records this run — the lock is released when
|
|
500
|
+
`generateSnapshot` returns, so a slow hook can outlive the snapshot it belongs to.
|
|
501
|
+
`generateSnapshotAsync` now awaits the hook, so a caller of the async API gets control back when
|
|
502
|
+
the snapshot is actually finished.
|
|
503
|
+
|
|
504
|
+
### Smaller fixes
|
|
505
|
+
|
|
506
|
+
- **`getGitInfo` used Node's 1 MB `maxBuffer`** where every sibling call passed 32 or 50 MB. On a
|
|
507
|
+
repository with enough dirty entries, `git status` overflowed, threw, and the `catch` returned
|
|
508
|
+
`isDirty: false` — reporting the dirtiest possible tree as clean. The limit is raised and shared,
|
|
509
|
+
and an overflow now reports `isDirty: true`, because a buffer that large only fills when there is
|
|
510
|
+
a great deal to report. All three git invocations use `execFileSync` instead of a shell string.
|
|
511
|
+
- **`.gitignore` character classes** were escaped as literals, so `*.[oa]`, `[Tt]humbs.db` and
|
|
512
|
+
`log[0-9].txt` matched nothing in `--discovery walk` while working in `git` mode.
|
|
513
|
+
- **A layer or kind rule** was glob-translated only when it contained `*` or `?`, so
|
|
514
|
+
`src/{api,web}/index.ts` fell through to a substring search and classified nothing; and a plain
|
|
515
|
+
pattern like `test` used `String.includes`, so it matched `src/contest/a.ts`. Both now go through
|
|
516
|
+
`matchesExclude`, which is the single implementation of "does this pattern select this path".
|
|
517
|
+
- **`export { handler as GET }`** and `export { GET } from "./handlers"` recorded the export but no
|
|
518
|
+
route method, so those App Router routes had no `methods` in `manifest.indexes.routes`.
|
|
519
|
+
`export default () => <div />` recorded no component for the same reason.
|
|
520
|
+
- **The last-resort clip** reserved room for its own truncation marker but not for the token count
|
|
521
|
+
rewritten afterwards, so a part clipped to exactly its budget came back over it. `rehashPart` also
|
|
522
|
+
estimated tokens from text that still carried the *old* number, unlike every other part.
|
|
523
|
+
- **Seven MCP tools ignored `outDir`**, so a snapshot generated into a custom directory could be
|
|
524
|
+
written by `snapshot_generate` and inspected by nothing.
|
|
525
|
+
- **`snapshot_read`** replaced the line range with the preview note instead of adding to it, so a
|
|
526
|
+
window into a preview came back with no `Lines X–Y of Z` at all; `signatures` was treated as a
|
|
527
|
+
full file.
|
|
528
|
+
- **Every published file ended with a `sourceMappingURL`** pointing at a `.map` that
|
|
529
|
+
`package.json#files` deliberately does not ship. The build no longer emits them.
|
|
530
|
+
- **`--watch` crashed on Node 18 on Linux**, where `fs.watch(cwd, { recursive: true })` does not
|
|
531
|
+
exist before Node 20.13. The package declares `engines: >=18`, and the run wrote a snapshot,
|
|
532
|
+
printed "Watching for changes", and *then* threw `ERR_FEATURE_UNAVAILABLE_ON_PLATFORM`. It is now
|
|
533
|
+
refused up front, with exit `2` and a message naming the version that works — and before anything
|
|
534
|
+
is written. A watcher that cannot start for any other reason (`ENOSPC`, the Linux inotify limit)
|
|
535
|
+
reports it as a message instead of a stack trace, and the snapshot it already wrote stays valid.
|
|
536
|
+
Found by the release's own Node 18 job: 0.9.10 and 0.9.11 never ran one.
|
|
537
|
+
|
|
538
|
+
---
|
|
539
|
+
|
|
540
|
+
## 0.9.11
|
|
541
|
+
|
|
542
|
+
A second audit, run against 0.9.10's own output, found that several defects the previous release
|
|
543
|
+
believed it had fixed were still reachable, that the changed-files scope was applied one layer too
|
|
544
|
+
late, and that this repository's documentation had been quietly corrupted. Every item below is
|
|
545
|
+
fixed with a test that fails without it.
|
|
546
|
+
|
|
547
|
+
### The redactor was still editing the program
|
|
548
|
+
|
|
549
|
+
`credential-assignment-unquoted` exists for `.env`, YAML and compose files, where everything after
|
|
550
|
+
`NAME=` is a literal. In a source file the same shape is an **expression**. The bug is visible in
|
|
551
|
+
the 0.9.10 snapshot itself, in `parts/part-005.md`:
|
|
552
|
+
|
|
553
|
+
```
|
|
554
|
+
const SINGLE_TOKEN = [REDACTED: potential secret-like value omitted from snapshot][^\sA-Za-z0-9_]/g;
|
|
555
|
+
```
|
|
556
|
+
|
|
557
|
+
The regex literal's opening `/` was a legal match for `UNQUOTED_LITERAL`, so the pattern was
|
|
558
|
+
deleted and the line no longer parses — data loss in the artifact whose whole purpose is to
|
|
559
|
+
describe the code. Three changes close the class rather than the instance:
|
|
560
|
+
|
|
561
|
+
- an unquoted value must **begin** with a word character, so `/`, `+`, `-` and `%` can only
|
|
562
|
+
continue one;
|
|
563
|
+
- a value followed by `[`, `{` or `(` is refused, because `const accessToken = tokens[0]` is a
|
|
564
|
+
reference;
|
|
565
|
+
- a value that sits in a **declaration** is never a literal. `const`, `let`, `var`, `readonly`,
|
|
566
|
+
`private`, `protected`, `static`, `declare`, `export` and `using` are recognised, walking back to
|
|
567
|
+
the start of the identifier so the tail match (`TOKEN` inside `SINGLE_TOKEN`) is covered too.
|
|
568
|
+
|
|
569
|
+
The guard is bounded to a 40-character window: an unbounded version made a 236 KB line with 4,000
|
|
570
|
+
assignments cost 2.7 s, which the existing quadratic-scan test caught.
|
|
571
|
+
|
|
572
|
+
### A changed-files scope was applied after the scan instead of before it
|
|
573
|
+
|
|
574
|
+
`--changed-since` and `--staged` read, decoded, secret-scanned and TypeScript-parsed every file in
|
|
575
|
+
the repository and then threw all but a handful away. Because every aggregate was bumped inside
|
|
576
|
+
that scan, the snapshot inherited the repository's numbers:
|
|
577
|
+
|
|
578
|
+
- **`validate` failed on every scoped snapshot.** Unchanged files are recorded in `skipped[]` with
|
|
579
|
+
reason `unchanged`, and `validate` then asked `wouldNowBeIncluded` about them — a question that
|
|
580
|
+
only answers "is this text?" — so every unchanged source file was reported as `new file:` and
|
|
581
|
+
the exit code was 1.
|
|
582
|
+
- **`manifest.layers`, `manifest.kinds`, `counts.largeFileCount`, `totals.redactedLines`** and the
|
|
583
|
+
encoding/preview/parse warnings all described files the snapshot did not contain. A two-file
|
|
584
|
+
snapshot reported the layer distribution and the secret count of the whole repository.
|
|
585
|
+
- **`explain` ignored the scope entirely** and answered `included: true` for exactly the files
|
|
586
|
+
`generate` was skipping.
|
|
587
|
+
|
|
588
|
+
The scope is now resolved once, before the scan, and applied to the candidate list whenever
|
|
589
|
+
`--with-dependents` is not in play. The dependents case still has to scan first — it needs every
|
|
590
|
+
file's imports — so it narrows afterwards and recomputes the counts, the findings and the warnings
|
|
591
|
+
from what survived.
|
|
592
|
+
|
|
593
|
+
### `--with-dependents` walked the import graph transitively
|
|
594
|
+
|
|
595
|
+
The closure tested `keep.has(resolved)` while adding to `keep`, so `a.ts` changed pulled in
|
|
596
|
+
`b.ts` (which imports it) and then `c.ts` (which imports `b.ts`). Whether a two-hop file was
|
|
597
|
+
included depended on whether its name sorted after `b.ts`: the same repository produced different
|
|
598
|
+
snapshots depending on file names. The test is now made against the changed set, which makes the
|
|
599
|
+
closure exactly one hop.
|
|
600
|
+
|
|
601
|
+
### Root-anchored `.gitignore` rules never matched in `walk` mode
|
|
602
|
+
|
|
603
|
+
`readGitignoreRules` keeps the leading `/`, and the matcher it was handed compares against a path
|
|
604
|
+
with no leading slash — so `/build` could not equal `build`, could not prefix `build/x`, and could
|
|
605
|
+
not appear between two slashes. `/secret-notes.txt` silently kept its file in the snapshot.
|
|
606
|
+
|
|
607
|
+
The same delegation made the opposite mistake one segment down: gitignore(5) anchors a pattern
|
|
608
|
+
containing a non-trailing `/` to the directory holding the `.gitignore`, so `doc/frotz` ignores the
|
|
609
|
+
top-level one only, while `matchesExclude` matched `a/doc/frotz` as well. Both are fixed by using
|
|
610
|
+
the gitignore matcher (which already implemented `anchored || body.includes("/")`) for every rule
|
|
611
|
+
rather than only for rules containing a backslash. `matchesExclude` is left to configuration
|
|
612
|
+
excludes, where "match at any depth" is the documented behaviour.
|
|
613
|
+
|
|
614
|
+
### Spreading a resolved config with a new `maxPartBytes` changed nothing
|
|
615
|
+
|
|
616
|
+
`{ ...resolved, maxPartBytes: 64 * 1024 }` still satisfies `isResolvedConfig`, and
|
|
617
|
+
`normalizeResolvedConfig` kept the **old** `budgets` object. The partitioner reads
|
|
618
|
+
`budgets.default`, so the run produced 300 KB parts and the manifest recorded the 64 KB that was
|
|
619
|
+
ignored. `normalizeResolvedConfig` now normalises `budgets` against `maxPartBytes`, capping
|
|
620
|
+
per-layer budgets rather than replacing them.
|
|
621
|
+
|
|
622
|
+
A related inconsistency is fixed too: a config that sets only `budgets: { default: "100k" }`
|
|
623
|
+
recorded `maxPartBytes: 300k`, so one snapshot described two different budgets.
|
|
624
|
+
|
|
625
|
+
### Smaller fixes
|
|
626
|
+
|
|
627
|
+
- **The MCP worker URL was built with string interpolation.** `file://${path}` turns a Windows path
|
|
628
|
+
into `file://C:/…` whose *host* is the drive letter, reads a `#` as a fragment and a `%` as an
|
|
629
|
+
escape. It uses `pathToFileURL` now, through an exported `workerEntry()`.
|
|
630
|
+
- **`--sarif` and the monolith disagreed.** The log was rendered in the write phase, so
|
|
631
|
+
`full-codebase.json` embedded a manifest with no `artifacts.sarif` while `manifest.json` had one.
|
|
632
|
+
The log is rendered before the monolith is serialised.
|
|
633
|
+
- **`validate` never checked `reports/audit.sarif`** against its recorded digest. It does now: a
|
|
634
|
+
security report uploaded to a third party is exactly the artifact that must be verified.
|
|
635
|
+
- **`--sarif` was missing from `configDigest`**, so two runs that write different directories
|
|
636
|
+
compared equal.
|
|
637
|
+
- **`--skip-artifacts … --sarif` claimed nothing was written** while writing `reports/audit.sarif`.
|
|
638
|
+
- **SARIF rule ids were positional.** They were handed out by position in the sorted set of rule
|
|
639
|
+
names a run happened to use, so adding one custom rule that sorts before the built-ins
|
|
640
|
+
(`acme-key` does) shifted every built-in id by one and re-pointed every open alert. Built-in
|
|
641
|
+
rules now take a fixed id from their declaration order and custom rules get an id hashed from
|
|
642
|
+
their own name. This renumbers the built-ins once, relative to 0.9.10, and never again.
|
|
643
|
+
- **`generate --watch` generated twice on startup** — `runCommand` generated, then the loop
|
|
644
|
+
generated again before any file changed.
|
|
645
|
+
- **`--watch` froze `outDir`/`tmpDir` at startup.** After the config moved them, the watcher saw
|
|
646
|
+
its own output as user content and regenerated without end.
|
|
647
|
+
- **The pre-commit hook hardcoded `snapshot/`**, so a project with `outDir: ".context"` got a hook
|
|
648
|
+
that printed "no snapshot found, skipping" and exited 0 on every commit — a gate that never ran.
|
|
649
|
+
The directory is baked in at install time, and the guard tests `manifest.json` rather than the
|
|
650
|
+
directory (a directory left by a failed run exists but holds no snapshot).
|
|
651
|
+
- **`--max-part-tokens 200` was silently raised** to the 1024-token floor. It is a usage error
|
|
652
|
+
now, and the message talks about tokens rather than bytes.
|
|
653
|
+
- **The MCP line cap was bypassed** by an oversized line that arrived complete with its terminator,
|
|
654
|
+
because the guard required the buffer to hold no `\n` at all. The cap now applies to every
|
|
655
|
+
complete line.
|
|
656
|
+
- **`snapshot_locate` searched only half the summary.** `variables` (where a `const`, a Python
|
|
657
|
+
constant, a Go `var`, a Rust `mod`, a C `#define` and a Terraform `variable` all land),
|
|
658
|
+
`directives`, `frontmatterKeys` and JSON `keys` are searched too.
|
|
659
|
+
- **`snapshot_read` printed `Lines 1–0 of 1` for an empty file** and returned the *last* line when
|
|
660
|
+
`fromLine` was past the end. Both are stated instead.
|
|
661
|
+
- **`.TSX` was parsed as `ScriptKind.TS`** — `isSkeletonizable` lowered the name and
|
|
662
|
+
`skeletonizeTypeScript` did not.
|
|
663
|
+
- **Markdown summaries:** an unclosed `---` at line 0 swallowed every heading, a shorter fence
|
|
664
|
+
inside a longer one closed it early, and a closing fence with an info string was accepted. All
|
|
665
|
+
three follow CommonMark now.
|
|
666
|
+
- **Destructured exports were invisible.** `export const { a, b } = obj` and
|
|
667
|
+
`export const [first] = pair` declare real exports and were dropped by an
|
|
668
|
+
`isIdentifier` check.
|
|
669
|
+
- **`rehashPart` left `estimatedTokens` at its pre-clip value** after `enforceBudget` cut a part,
|
|
670
|
+
and the part's own header kept stating the old number. Both are recomputed.
|
|
671
|
+
- **`setup --preset <unknown>` exited 1** where `init` exits 2, because the parse-time check was
|
|
672
|
+
written for `init` only.
|
|
673
|
+
- **`describe` omitted all seven 0.9.10 flags and six config keys.** `describe` is the contract an
|
|
674
|
+
agent reads instead of the docs, so this is the failure that matters most: the options simply did
|
|
675
|
+
not exist as far as a machine consumer was concerned. A test now compares `CLI_OPTIONS` and
|
|
676
|
+
`CONFIG_KEYS` against the contract in both directions.
|
|
677
|
+
- **`--changed-since` was passed to git as a positional revision.** A ref beginning with `-` is now
|
|
678
|
+
refused before git sees it, and `--` separates revisions from paths.
|
|
679
|
+
- **`--skeleton --include-monolith` labelled a body-stripped file `inclusionMode: "full"`.** The
|
|
680
|
+
monolith now keeps the full redacted body, so the label is true.
|
|
681
|
+
|
|
682
|
+
### Documentation and repository hygiene
|
|
683
|
+
|
|
684
|
+
- **Three sentences had letters eaten by interpreted escapes**: `\v` in "Validate", `\r` in
|
|
685
|
+
"reports/", `\a` in "audit.md". A test now fails on any control byte in any tracked text file.
|
|
686
|
+
- **Four documents claimed the MCP server serves seven tools.** It serves eight. The count is now
|
|
687
|
+
compared against the live tool list.
|
|
688
|
+
- **`publish:create` could not run.** `npm publish packages/create` is parsed by npm as the GitHub
|
|
689
|
+
shorthand `ssh://git@github.com/packages/create.git` and fails with exit 128; it names
|
|
690
|
+
`./packages/create`.
|
|
691
|
+
- **`check-site-links` accepted a sibling directory.** `resolved.startsWith(SITE)` is true for
|
|
692
|
+
`../site-other`; the check requires the path separator.
|
|
693
|
+
|
|
694
|
+
---
|
|
695
|
+
|
|
696
|
+
## 0.9.10
|
|
697
|
+
|
|
698
|
+
An audit of this codebase, run against its own snapshot, found twenty defects, six performance
|
|
699
|
+
problems and seven missing capabilities. This release is all of them: every defect, every
|
|
700
|
+
performance problem and every capability named in that audit is in the tree, with a test that
|
|
701
|
+
fails without it.
|
|
702
|
+
|
|
703
|
+
### Redaction was corrupting the code it exists to preserve
|
|
704
|
+
|
|
705
|
+
The worst defects were in the secret scanner, and the evidence was in this repository's own
|
|
706
|
+
snapshot. Four false-positive classes rewrote valid statements:
|
|
707
|
+
|
|
708
|
+
- `[:=]` matched the first `=` of `===`, and the unquoted-value rules then read the remaining `==`
|
|
709
|
+
as a credential. `clause.token === ts.SyntaxKind…` shipped as `clause.token =[REDACTED…]`, and so
|
|
710
|
+
did `if (token === "--")`.
|
|
711
|
+
- `env-secret-assignment` carried `/i`, so an indented reassignment matched: `key =
|
|
712
|
+
fs.realpathSync(stagingRoot)` was emitted with the call masked.
|
|
713
|
+
- Credential names are matched as a *tail* — that is what catches `dbPassword` — so a name inside a
|
|
714
|
+
string literal matched, and the string's closing quote was taken as the value's opening quote.
|
|
715
|
+
`["awsSecret:", "apiKey:"]` came out with two list elements welded together.
|
|
716
|
+
- `progressToken: "progressToken"` was masked as a credential. A camelCase value that spells its
|
|
717
|
+
own key is a symbol-table entry, not a secret.
|
|
718
|
+
|
|
719
|
+
Reserved words are no longer credential literals either: `token = await refreshToken(client)` was
|
|
720
|
+
emitted as `token = [REDACTED] refreshToken(client)`.
|
|
721
|
+
|
|
722
|
+
### Scanning no longer depends on state outside the tree
|
|
723
|
+
|
|
724
|
+
- A `.gitignore` rule naming a nested directory (`public/build/`) never matched in `walk` mode: the
|
|
725
|
+
candidate was reduced to its first path segment, which cannot equal a two-segment rule. The same
|
|
726
|
+
bug dropped a regular *file* named `build` for a `build/` rule.
|
|
727
|
+
- How files are enumerated was held in a module-level variable written by `resolveConfig`, so two
|
|
728
|
+
scans in one process could disagree, and the second inherited the first one's mode.
|
|
729
|
+
- `validate` did not exclude the tool's own output directory, so it counted `snapshot/` as files
|
|
730
|
+
missing from the snapshot and reported a correct snapshot as drifting.
|
|
731
|
+
- `findSecretInPath` compiled thirty-plus `RegExp` objects per call, and it runs on every path and
|
|
732
|
+
every string in every summary.
|
|
733
|
+
- `getGitInfo` spawns one `git status` instead of three `git` processes.
|
|
734
|
+
|
|
735
|
+
### The config digest now covers what decides a snapshot
|
|
736
|
+
|
|
737
|
+
Layer and kind *names* were hashed, not their matchers, so moving a layer from `src/api/**` to
|
|
738
|
+
`src/v2/**` produced an identical digest and `validate` said "valid and up to date" for a snapshot
|
|
739
|
+
whose files had all been reclassified. A rule's `flags` and `valueGroup` were missing too, and so
|
|
740
|
+
was `discovery`. `package.json`'s `"snapshot"` object now gets the unknown-key guard every other
|
|
741
|
+
config file already had, and `secretRules` is accepted in a config file as it already was in code.
|
|
742
|
+
|
|
743
|
+
### `setup` stops dropping the flags it was given
|
|
744
|
+
|
|
745
|
+
`setup` builds the inner `generate` command line by hand, and five settings were never copied into
|
|
746
|
+
it, so the run reported one configuration and generated with another. It also refused
|
|
747
|
+
`--discovery`, `--large-file-bytes`, `--max-monolith-bytes` and `--include-monolith` outright.
|
|
748
|
+
`--only-artifacts monolith` enabled *nothing*: it turns the six default artifacts off and relied on
|
|
749
|
+
the monolith switch to turn the seventh on, and neither `init` nor `setup` passed that switch along.
|
|
750
|
+
|
|
751
|
+
### Part budgets say what they enforce
|
|
752
|
+
|
|
753
|
+
- Every part records the budget it was built for. With per-layer budgets, verification compared
|
|
754
|
+
each part with the *largest* budget in the run, so a part twice over its own layer budget passed.
|
|
755
|
+
- `renumberParts` rewrites headers, so the byte bound is re-applied after renumbering.
|
|
756
|
+
- A body cut twice reported only the second cut: a 1,000-line file reduced to a preview and then to
|
|
757
|
+
ten lines said "11 lines omitted", because the count came from the preview instead of the file.
|
|
758
|
+
- Every part states its size in **tokens**, and `--max-part-tokens 32k` sets a ceiling a context
|
|
759
|
+
window can be measured against. The estimate's method is documented, not hidden.
|
|
760
|
+
|
|
761
|
+
### Crash-proof locks, byte-accurate reads
|
|
762
|
+
|
|
763
|
+
- A run killed between creating its lock file and writing its pid leaves a zero-byte lock that
|
|
764
|
+
named nobody, so `lockOwnerIsGone` answered "not gone" forever and every later run refused to
|
|
765
|
+
start until the file was deleted by hand. Age is now the fallback evidence.
|
|
766
|
+
- `snapshot_read` truncated to its byte limit by slicing UTF-16 code units, returning up to three
|
|
767
|
+
times the limit for multibyte text and able to split a surrogate pair.
|
|
768
|
+
- `snapshot_locate` could not find a symbol a file does not export — most of the code in most files
|
|
769
|
+
— and could not find a Markdown heading at all. Layer and kind matched by exact equality while
|
|
770
|
+
every other category used a substring.
|
|
771
|
+
- `findSection` compared a heading with the manifest's raw path, which differ once the heading has
|
|
772
|
+
been capped or escaped; such a file became unreadable whenever another file shared its content.
|
|
773
|
+
- A watch loop had no `error` listener on `fs.watch`, so an `EPERM` or an inotify limit killed the
|
|
774
|
+
process instead of being reported.
|
|
775
|
+
- The update check skipped itself when any argument contained "json", so `explain package.json`
|
|
776
|
+
never asked.
|
|
777
|
+
|
|
778
|
+
### Fewer syscalls
|
|
779
|
+
|
|
780
|
+
`generate` read every included file twice — a chunked NUL scan, then a full read and a second scan
|
|
781
|
+
of the same bytes. Below the 8 MB scan cap the decode is the complete check on bytes that had to be
|
|
782
|
+
read anyway, so the pre-scan is kept only above it. `explain` and `validate` had the same double
|
|
783
|
+
read.
|
|
784
|
+
|
|
785
|
+
### New structure for six more languages
|
|
786
|
+
|
|
787
|
+
Swift, Dart, SQL, GraphQL, Protobuf and Terraform now have profiles. `src/scanner/language.ts`
|
|
788
|
+
already knew four of those extensions by name and no profile claimed them, so a repository in one of
|
|
789
|
+
those languages got a list of files and no idea what was inside them.
|
|
790
|
+
|
|
791
|
+
### The site keeps its own links
|
|
792
|
+
|
|
793
|
+
`README.md` writes absolute GitHub URLs so the same text works on npmjs.com; the site builder only
|
|
794
|
+
rewrote *relative* links, so a reader on the docs site who clicked a guide link on the homepage was
|
|
795
|
+
handed back to GitHub. In-repo links to published pages are rewritten now; other repositories and
|
|
796
|
+
files the site does not publish are left alone.
|
|
797
|
+
|
|
798
|
+
### Also
|
|
799
|
+
|
|
800
|
+
- `configDigest`, `classifyLayer`, `classifyKind`, `displayPath`, `lockPathFor`, `estimateTokens`,
|
|
801
|
+
`findSecretInPath` and `getGitInfo` are exported.
|
|
802
|
+
- `manifest.config` records `maxMonolithBytes` and `maxPartTokens`.
|
|
803
|
+
- `scripts/sync-docs.mjs` propagates `package.json#version` to `src/version.ts` and
|
|
804
|
+
`packages/create/package.json`, so a release cannot drift by forgetting a file. `--check` covers
|
|
805
|
+
them.
|
|
806
|
+
- `scripts/pack-check.mjs` requires the MCP build outputs, and the three wizard test files share one
|
|
807
|
+
fake-TTY harness.
|
|
808
|
+
|
|
809
|
+
### `snapshot_dependents`: the import graph, in both directions
|
|
810
|
+
|
|
811
|
+
`manifest.indexes.imports` already held enough to answer "what uses this file" —
|
|
812
|
+
`byFile` records every specifier, `centralFiles` ranks the most-imported files — but nothing
|
|
813
|
+
an agent could ask. So the one question that decides what to read next, *what breaks if I
|
|
814
|
+
change this?*, was answered by reading parts until the picture appeared.
|
|
815
|
+
|
|
816
|
+
The eighth MCP tool takes a path and a direction and returns both edge sets with the part each
|
|
817
|
+
neighbour lives in, plus the specifiers that resolve to nothing inside the snapshot
|
|
818
|
+
(`node:fs`, a package name) rather than silently dropping them. Resolution runs through the
|
|
819
|
+
generator's own `buildImportResolver`, so the tool and `centralFiles` cannot describe different
|
|
820
|
+
graphs — a test asserts the counts agree.
|
|
821
|
+
|
|
822
|
+
### `--install-hook` gates commits on `validate`
|
|
823
|
+
|
|
824
|
+
`docs/stale-context-is-your-bug.md` argues a snapshot is only trustworthy when something gates
|
|
825
|
+
on it, and `validate` is that gate — but wiring it into `.git/hooks` was thirty seconds of
|
|
826
|
+
manual work, which is the step a project skips. `init --install-hook` and `setup --install-hook`
|
|
827
|
+
write `.git/hooks/pre-commit`, which refuses a commit whose snapshot is stale and never installs
|
|
828
|
+
anything while doing it.
|
|
829
|
+
|
|
830
|
+
A `pre-commit` this tool did not write is left byte-for-byte alone, and the output names the line
|
|
831
|
+
to add instead: there is deliberately no flag to force the overwrite. A `.git` that is a *file*
|
|
832
|
+
(a worktree or submodule) is followed to the real git directory rather than failing with
|
|
833
|
+
`ENOTDIR`.
|
|
834
|
+
|
|
835
|
+
### `--changed-since` and `--staged`: review packs the diff
|
|
836
|
+
|
|
837
|
+
Reviewing a pull request with a model is the case this exists for: the diff is small, the
|
|
838
|
+
repository is not, and packing all of it costs a context window to answer a question about four
|
|
839
|
+
files.
|
|
840
|
+
|
|
841
|
+
`--changed-since <ref>` is the working tree against that ref — committed, staged and unstaged
|
|
842
|
+
changes all count, because all three are what a reviewer is looking at — **plus untracked
|
|
843
|
+
files**, which `git diff` never lists and which are exactly what a new feature starts with.
|
|
844
|
+
`--staged` is the index against `HEAD`, and `--with-dependents` adds the one-hop importers,
|
|
845
|
+
which is the context needed to change a file safely rather than merely the file that moved.
|
|
846
|
+
|
|
847
|
+
Files outside the scope are recorded in `skipped[]` with a new reason, `unchanged` —
|
|
848
|
+
deliberately not `excluded`, which means the user's own glob rejected the path. Every failure is
|
|
849
|
+
loud: an unknown ref, a directory that is not a git repository, and `--with-dependents` without
|
|
850
|
+
a scope are all errors naming the way out, because a ref typo that produced an empty snapshot
|
|
851
|
+
would read as "this change touched nothing".
|
|
852
|
+
|
|
853
|
+
### `--skeleton`: signatures instead of bodies
|
|
854
|
+
|
|
855
|
+
`docs/vs-repomix.md` concedes that a compression mode beats byte budgeting for a repository too
|
|
856
|
+
large to fit at all. This is that mode, built on the compiler that is already the one runtime
|
|
857
|
+
dependency: each function, method, constructor, accessor and arrow body becomes `{ /* ... */ }`
|
|
858
|
+
in place.
|
|
859
|
+
|
|
860
|
+
The rewrite is span-based rather than a `ts.createPrinter()` re-print, and that is the whole
|
|
861
|
+
feature — a printer normalises quotes, blank lines and comment positions, so a skeleton would
|
|
862
|
+
differ from its source everywhere instead of only where bodies were removed. Affected entries
|
|
863
|
+
carry `inclusionMode: "signatures"`, and the structural summary still describes the whole file.
|
|
864
|
+
|
|
865
|
+
### `--sarif`: the findings a CI gate can read
|
|
866
|
+
|
|
867
|
+
`reports/audit.md` tells a human what was masked, and nothing could *gate* on it: a CI job that
|
|
868
|
+
wanted to fail a build over a leaked key had to scrape markdown. `--sarif` writes
|
|
869
|
+
`reports/audit.sarif` in SARIF 2.1.0, which is what GitHub Code Scanning reads.
|
|
870
|
+
|
|
871
|
+
The artefact is deliberately austere: path, line, level and rule — and **never the value**,
|
|
872
|
+
because it is meant to be uploaded to a third party and a second copy of a secret in someone
|
|
873
|
+
else's retention policy is a new problem rather than a report on the old one. Rule ids are
|
|
874
|
+
stable across runs so an alert is not orphaned, and each result carries a
|
|
875
|
+
`partialFingerprint` so Code Scanning keeps an alert open instead of closing and reopening it
|
|
876
|
+
on every run.
|
|
877
|
+
|
|
878
|
+
### `snapshot_generate` runs off the JSON-RPC thread
|
|
879
|
+
|
|
880
|
+
The 0.9.4 measurement stands: a synchronous generate blocks the stdio server for about 7.5
|
|
881
|
+
seconds on a 3,000-file repository. In that window the process cannot answer `ping` or
|
|
882
|
+
acknowledge `notifications/cancelled`, and a client with a socket timeout reports a working tool
|
|
883
|
+
call as a dead server.
|
|
884
|
+
|
|
885
|
+
The work moves to a worker thread. A resolved config cannot cross a thread boundary —
|
|
886
|
+
`layers[].match` may be a function and `hooks.afterGenerate` always is — so the worker loads
|
|
887
|
+
the config file and applies the same CLI options the calling thread did, rather than receiving
|
|
888
|
+
values that would all look like explicit overrides. A worker that fails to start, exits without
|
|
889
|
+
posting, or throws is reported as a tool failure naming the reason; there is no quiet retry on the
|
|
890
|
+
calling thread, because that would hide exactly the condition this exists to avoid.
|
|
891
|
+
|
|
892
|
+
One consequence is documented rather than discovered: `hooks.afterGenerate` does not run for a
|
|
893
|
+
generation started from MCP, only for a CLI `generate`.
|
|
894
|
+
|
|
895
|
+
---
|
|
896
|
+
|
|
24
897
|
## 0.9.9
|
|
25
898
|
|
|
26
899
|
### `snapshot -version` now answers the question it was asked
|