@objectstack/metadata-fs 17.0.0-rc.5 → 17.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +323 -0
- package/README.md +16 -1
- package/dist/index.cjs +142 -19
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +115 -2
- package/dist/index.d.ts +115 -2
- package/dist/index.js +142 -19
- package/dist/index.js.map +1 -1
- package/package.json +3 -3
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,328 @@
|
|
|
1
1
|
# @objectstack/metadata-fs
|
|
2
2
|
|
|
3
|
+
## 17.0.0
|
|
4
|
+
|
|
5
|
+
### Patch Changes
|
|
6
|
+
|
|
7
|
+
- 2e836de: chore(packaging): CHANGELOG.md ships in every npm tarball (#4261)
|
|
8
|
+
|
|
9
|
+
The AGENTS.md post-task checklist requires breaking changesets to carry their
|
|
10
|
+
FROM → TO migration because "this text ships to consumers as `CHANGELOG.md`
|
|
11
|
+
inside the npm package and is what an upgrading agent greps after the tombstone
|
|
12
|
+
error." That delivery path was severed for 68 of the 69 publishable packages:
|
|
13
|
+
npm packs `package.json` / `README*` / `LICENSE*` unconditionally but — unlike
|
|
14
|
+
older npm versions — not `CHANGELOG.md`, and the canonical
|
|
15
|
+
`"files": ["dist", "README.md"]` whitelist never named it. Measured on npm
|
|
16
|
+
10.9.7: `npm pack --dry-run` on `@objectstack/types` shipped 3 files while its
|
|
17
|
+
70KB `CHANGELOG.md` stayed behind. Only `@objectstack/spec` listed it
|
|
18
|
+
explicitly.
|
|
19
|
+
|
|
20
|
+
The tombstone-error scenario is precisely the one where the repo is out of
|
|
21
|
+
reach — the upgrading agent has `node_modules` and nothing else — so the
|
|
22
|
+
migration text has to ride in the tarball. Every publishable package now
|
|
23
|
+
declares `CHANGELOG.md` in `files`, and the canonical whitelist is
|
|
24
|
+
`["dist", "README.md", "CHANGELOG.md"]`.
|
|
25
|
+
|
|
26
|
+
The other half is the gate: `check:published-files` gains a fifth invariant,
|
|
27
|
+
COMPLETE — a whitelist that fails to cover `CHANGELOG.md` fails the
|
|
28
|
+
always-required lint job, so the next package cannot silently sever the path
|
|
29
|
+
again. `@objectstack/spec`'s per-package EXTRA_ENTRIES exemption dissolves
|
|
30
|
+
into the canonical set.
|
|
31
|
+
|
|
32
|
+
Consumer-visible change: one more file per install (the package's changelog,
|
|
33
|
+
e.g. 70.8KB for `@objectstack/types`), and `grep -r "removed key"
|
|
34
|
+
node_modules/@objectstack/*/CHANGELOG.md` now finds the migration it was
|
|
35
|
+
promised.
|
|
36
|
+
|
|
37
|
+
- a1b66ef: `FileSystemRepository` no longer creates its root directory when it is attached — only when it first writes.
|
|
38
|
+
|
|
39
|
+
`start()` used to `mkdir` both `<root>` and `<root>/.objectstack/.log` unconditionally, so merely attaching a repository was a write. Because `MetadataPlugin` attaches one at `<project>/.objectstack/metadata` during every boot, a command that never writes metadata still brought a directory skeleton into existence. The loudest case is `os migrate plan`, a declared dry run: on a project that had never been started it left
|
|
40
|
+
|
|
41
|
+
```
|
|
42
|
+
.objectstack/metadata/.objectstack/.log
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
behind, which also destroyed the one signal — does `.objectstack/` exist? — by which the next command can tell a fresh project from a started one. This is the filesystem half of the same property the database half already covers: a dry run leaves nothing behind.
|
|
46
|
+
|
|
47
|
+
Attaching and reading a repository whose root does not exist is now explicitly supported and answers as an empty repository (`get`, `getByHash`, `list`, `history`, `watch`). The root, the type directories and the JSONL change log all appear on the first `put` / `delete`, and nothing about the boot's read-only character changes: no metadata is written that was not written before.
|
|
48
|
+
|
|
49
|
+
One behavioural note for direct users of the package: when the root is absent at `start()`, the chokidar watcher is armed by the first write instead, because chokidar cannot watch a path that does not yet exist. A root brought into existence by a third party while the process runs — with this repository never writing — is therefore not picked up until the next `start()`.
|
|
50
|
+
|
|
51
|
+
- c7e7900: fix(metadata-core,metadata-fs): hash the serialized form, so `put().version` identifies the bytes actually stored (#7856)
|
|
52
|
+
|
|
53
|
+
`hashSpec` canonicalised a `Date` to `{}`, because `canonicalize` walked a
|
|
54
|
+
value's own enumerable keys and a `Date` has none. `JSON.stringify` — what every
|
|
55
|
+
repository actually writes — turns the same `Date` into an ISO string. So the
|
|
56
|
+
hash of the in-memory spec and the hash of the bytes on disk were **different
|
|
57
|
+
hashes for the same item**, and the version handed back to a caller did not
|
|
58
|
+
identify what had been stored.
|
|
59
|
+
|
|
60
|
+
Measured on `main`, one spec carrying one `Date`:
|
|
61
|
+
|
|
62
|
+
```
|
|
63
|
+
canonicalize(in-memory) : {"createdAt":{},"label":"Home"}
|
|
64
|
+
JSON.stringify (bytes) : {"label":"Home","createdAt":"2024-01-01T00:00:00.000Z"}
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
`canonicalize` now honours `toJSON` exactly as `JSON.stringify` does —
|
|
68
|
+
consulted once per position, its result serialised as-is and never
|
|
69
|
+
re-consulted — which makes a new guarantee true by construction:
|
|
70
|
+
|
|
71
|
+
```
|
|
72
|
+
canonicalize(x) === canonicalize(JSON.parse(JSON.stringify(x)))
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
**Both repository implementations were wrong, in different places**, which is
|
|
76
|
+
why the fix is one function rather than two patches. `FileSystemRepository`
|
|
77
|
+
broke `put().version === get().hash`: it hashed the spec it was handed, wrote
|
|
78
|
+
`JSON.stringify` of it, and re-hashed the parse on the way back out.
|
|
79
|
+
`InMemoryRepository` broke the repository contract's invariant 4
|
|
80
|
+
(`item.hash === hashSpec(item.body)`): it stores `body` already serialised
|
|
81
|
+
(`clonePlain`) while hashing the in-memory spec, so the item it returns
|
|
82
|
+
disagreed with its own hash. `SysMetadataRepository` inherits the fix through
|
|
83
|
+
the same function.
|
|
84
|
+
|
|
85
|
+
Downstream, an incoherent version meant a repository could report an
|
|
86
|
+
`{op:'update', actor:'fs'}` for a file nothing outside the process had touched:
|
|
87
|
+
the head index held a hash the disk could never reproduce, so re-reading one's
|
|
88
|
+
own write looked like somebody else's edit. That surfaces without any watcher —
|
|
89
|
+
a restart rebuilds the index from disk and the version the caller was handed no
|
|
90
|
+
longer matches it.
|
|
91
|
+
|
|
92
|
+
**Ordinary specs hash exactly as before, and this is not a migration.** The new
|
|
93
|
+
path diverges only at a position carrying a callable `toJSON`; a graph without
|
|
94
|
+
one is byte-identical through `canonicalize`. Verified against this repository's
|
|
95
|
+
entire checked-in JSON corpus — 1973 files hashed under both the old and the new
|
|
96
|
+
implementation, **0 hashes changed** — and the `hashSpec({})` regression guard
|
|
97
|
+
in `metadata-core` is unmoved. Stored versions for ordinary specs keep their
|
|
98
|
+
meaning. Versions for `toJSON`-carrying specs do change, and those are exactly
|
|
99
|
+
the versions that never identified their stored bytes in the first place.
|
|
100
|
+
|
|
101
|
+
Also supported as a consequence: a class instance with a `toJSON` now hashes as
|
|
102
|
+
whatever it serialises to, rather than as its private fields. One without a
|
|
103
|
+
`toJSON` still hashes as its own enumerable keys — which is what
|
|
104
|
+
`JSON.stringify` writes for it.
|
|
105
|
+
|
|
106
|
+
The pin is table-driven and lives in the shared repository contract suite, so
|
|
107
|
+
every `MetadataRepository` implementation is held to it: `Date` at a key, `Date`
|
|
108
|
+
under an array index, a class whose `toJSON` yields a string, an object literal
|
|
109
|
+
carrying its own `toJSON`, a nested case, and a plain-JSON control row that
|
|
110
|
+
proves the fix did not simply change every hash.
|
|
111
|
+
|
|
112
|
+
- a1686f9: fix(metadata-fs): stop suppressing self-writes on a wall clock, so a poll tick can no longer swallow an external edit (#7335)
|
|
113
|
+
|
|
114
|
+
`FileSystemRepository` suppressed the watcher event its own `put()`/`delete()`
|
|
115
|
+
was about to produce by adding the path to a `selfWrites` Set and clearing it on
|
|
116
|
+
a fixed `setTimeout(…, 200)`. `handleFsChange` then dropped **any** event for a
|
|
117
|
+
path in that Set, without ever reading what the watcher had observed.
|
|
118
|
+
|
|
119
|
+
Under `usePolling: true, interval: 1000` chokidar compares state once per tick,
|
|
120
|
+
so our own write and an external edit landing between two ticks are delivered as
|
|
121
|
+
a **single** event carrying the _external_ content. Dropping that on a timer
|
|
122
|
+
destroyed the only notification the external edit would ever produce — the edit
|
|
123
|
+
was silently lost, and nothing later recovered it. The realistic trigger is a
|
|
124
|
+
`git checkout` or an editor save arriving while the process writes the same item:
|
|
125
|
+
the dev-mode authoring loop.
|
|
126
|
+
|
|
127
|
+
**Measured.** The filing recorded 0/360 instrumented iterations reaching the
|
|
128
|
+
window and called it derived rather than observed. That was a sampling artefact:
|
|
129
|
+
the delivery lag of a self-write event is
|
|
130
|
+
`(interval - (writeTime mod interval)) + awaitWriteFinish`, so a _fixed_
|
|
131
|
+
pre-edit sleep phase-locks the poll and pins the lag outside the window
|
|
132
|
+
(measured: 519–585 ms across 25 runs). Randomising the sleep so the lag samples
|
|
133
|
+
`[0, interval)` uniformly, 40 runs:
|
|
134
|
+
|
|
135
|
+
| delivery lag | runs | external edit |
|
|
136
|
+
| ------------ | ---- | ------------- |
|
|
137
|
+
| < 200 ms | 7 | **swallowed** |
|
|
138
|
+
| > 200 ms | 33 | delivered |
|
|
139
|
+
|
|
140
|
+
A perfect split on the wall-clock boundary — the mechanism, observed.
|
|
141
|
+
|
|
142
|
+
**The fix removes the pre-check rather than re-keying it**, because the
|
|
143
|
+
content-keyed suppression it was shadowing already existed one step further
|
|
144
|
+
down and needs no timer:
|
|
145
|
+
|
|
146
|
+
- `add`/`change` — `currentHead === hash` drops the event when the bytes on disk
|
|
147
|
+
are the bytes we last published. `put()` sets that head in the same
|
|
148
|
+
continuation as its `rename`, and `awaitWriteFinish` holds any event for a
|
|
149
|
+
further `stabilityThreshold`, so the index is never late.
|
|
150
|
+
- `unlink` — `!currentHead` drops the event when the index already agrees the
|
|
151
|
+
item is gone.
|
|
152
|
+
|
|
153
|
+
`delete()` additionally now retires the head **before** it unlinks rather than
|
|
154
|
+
after. `awaitWriteFinish` debounces only `add`/`change`, so that face gets no
|
|
155
|
+
stability cushion between the disk mutation and the event it produces; ordering
|
|
156
|
+
the index update first makes the downstream check a total suppression rather
|
|
157
|
+
than a race against the poll callback. A failed `unlink` restores the head
|
|
158
|
+
before rethrowing, so the error path is unchanged.
|
|
159
|
+
|
|
160
|
+
No API or configuration change; the repository publishes strictly more of the
|
|
161
|
+
external edits it was always meant to report.
|
|
162
|
+
|
|
163
|
+
One pre-existing limit is now documented rather than altered: identity is judged
|
|
164
|
+
on what round-trips through the file, so a spec whose in-memory form does not
|
|
165
|
+
(a `Date`, which canonicalises to `{}` in memory but to an ISO string once
|
|
166
|
+
written and re-read) is republished as an external `update`. Such a spec already
|
|
167
|
+
fails `put().version === get().hash` independently of the watcher, and the
|
|
168
|
+
200 ms window never covered it either — it expired some 360 ms before the event
|
|
169
|
+
it would have had to catch.
|
|
170
|
+
|
|
171
|
+
- ab07b53: fix(metadata-fs): register every written path with the watcher, so an item created while chokidar is still scanning is not invisible forever (#7282)
|
|
172
|
+
|
|
173
|
+
`FileSystemRepository`'s watcher could go **permanently blind to a single
|
|
174
|
+
item** — external edits to that file produced no `MetadataEvent` for the whole
|
|
175
|
+
life of the process, and nothing recovered short of a restart. The window is a
|
|
176
|
+
race between chokidar's asynchronous initial scan and the repository's own
|
|
177
|
+
first write, and both `start()` (which arms the watcher, after which the caller
|
|
178
|
+
may `put()` on the next tick) and `ensureRoot()` (which arms it in the middle
|
|
179
|
+
of the very first write, #7000) can open it.
|
|
180
|
+
|
|
181
|
+
Measured on chokidar 5 with this repository's options (`usePolling`,
|
|
182
|
+
`interval: 1000`):
|
|
183
|
+
|
|
184
|
+
1. chokidar reads `<root>/<type>/` and finds it EMPTY — the atomic `rename` in
|
|
185
|
+
`writeJsonAtomic` has not landed yet;
|
|
186
|
+
2. the rename lands, changing the directory's mtime;
|
|
187
|
+
3. chokidar calls `watchFile()` on that directory and libuv takes its polling
|
|
188
|
+
baseline stat, which already reflects step 2.
|
|
189
|
+
|
|
190
|
+
The directory's stat then never changes again, so no poll ever fires for it,
|
|
191
|
+
the directory is never re-read, the item file is never added to the watched
|
|
192
|
+
set, and no per-file watcher is created. `getWatched()` reports the type
|
|
193
|
+
directory as `[]` while the file sits in it, and neither `add` nor `change` is
|
|
194
|
+
ever emitted for that path.
|
|
195
|
+
|
|
196
|
+
The fix does not widen any timer. The only writer that can be inside that
|
|
197
|
+
window is the repository itself, so `put()` now tells the watcher explicitly
|
|
198
|
+
about the path it created instead of depending on a directory scan that may
|
|
199
|
+
never notice it. Registration is idempotent and emits nothing.
|
|
200
|
+
|
|
201
|
+
User-visible effect: `MetadataManager.subscribe()` (and every consumer of
|
|
202
|
+
`repo.watch()`) now reliably sees out-of-process edits — a hand edit, or a
|
|
203
|
+
`git checkout` bringing metadata JSON in — to items written earlier in the same
|
|
204
|
+
process. This was also the cause of four merge-queue ejections across three
|
|
205
|
+
PRs; the two time-based mitigations tried before it (a 20s/25s event deadline
|
|
206
|
+
and a wider pre-edit sleep) could not have worked, because the event was never
|
|
207
|
+
delivered rather than late.
|
|
208
|
+
|
|
209
|
+
- 684ab22: fix(metadata-fs): the `FileSystemRepository` watcher now sees external edits in the production layout
|
|
210
|
+
|
|
211
|
+
`MetadataPlugin` attaches the repository at `<project>/.objectstack/metadata`, and the
|
|
212
|
+
watcher's `ignored` matcher was a bare dotfile regex. chokidar applies that matcher to the
|
|
213
|
+
watched root path itself, not only to entries found underneath it, so the `.objectstack`
|
|
214
|
+
segment of the root matched and the entire watch was inert — `getWatched()` returned `{}`
|
|
215
|
+
and no event ever fired. Hand edits, a `git checkout` that brings metadata JSON in, and any
|
|
216
|
+
other out-of-process writer under `.objectstack/metadata/` were invisible until the next
|
|
217
|
+
`start()`, even though `MetadataManager.setRepository()` is wired to those events and uses
|
|
218
|
+
them to invalidate the registry and the `list()` cache.
|
|
219
|
+
|
|
220
|
+
The matcher is now evaluated against the path _relative_ to the watch root, so dot segments
|
|
221
|
+
belonging to the root itself are never considered while dotfiles under the root — including
|
|
222
|
+
the repository's own `.objectstack/` bookkeeping subtree — stay ignored as before.
|
|
223
|
+
|
|
224
|
+
- Updated dependencies [f5a4ef0]
|
|
225
|
+
- Updated dependencies [2e836de]
|
|
226
|
+
- Updated dependencies [121852d]
|
|
227
|
+
- Updated dependencies [db0d53c]
|
|
228
|
+
- Updated dependencies [c7e7900]
|
|
229
|
+
- Updated dependencies [72c3c86]
|
|
230
|
+
- Updated dependencies [3670cf9]
|
|
231
|
+
- Updated dependencies [2d8dba3]
|
|
232
|
+
- Updated dependencies [7372d46]
|
|
233
|
+
- Updated dependencies [5e247fd]
|
|
234
|
+
- Updated dependencies [1a53a02]
|
|
235
|
+
- Updated dependencies [a954634]
|
|
236
|
+
- Updated dependencies [fda61e4]
|
|
237
|
+
- Updated dependencies [db48ad5]
|
|
238
|
+
- Updated dependencies [65f184b]
|
|
239
|
+
- Updated dependencies [51a587d]
|
|
240
|
+
- Updated dependencies [c073b8c]
|
|
241
|
+
- Updated dependencies [946a131]
|
|
242
|
+
- Updated dependencies [ce92674]
|
|
243
|
+
- Updated dependencies [3d4c545]
|
|
244
|
+
- Updated dependencies [bb7cb41]
|
|
245
|
+
- @objectstack/metadata-core@17.0.0
|
|
246
|
+
|
|
247
|
+
## 17.0.0-rc.6
|
|
248
|
+
|
|
249
|
+
### Patch Changes
|
|
250
|
+
|
|
251
|
+
- a1b66ef: `FileSystemRepository` no longer creates its root directory when it is attached — only when it first writes.
|
|
252
|
+
|
|
253
|
+
`start()` used to `mkdir` both `<root>` and `<root>/.objectstack/.log` unconditionally, so merely attaching a repository was a write. Because `MetadataPlugin` attaches one at `<project>/.objectstack/metadata` during every boot, a command that never writes metadata still brought a directory skeleton into existence. The loudest case is `os migrate plan`, a declared dry run: on a project that had never been started it left
|
|
254
|
+
|
|
255
|
+
```
|
|
256
|
+
.objectstack/metadata/.objectstack/.log
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
behind, which also destroyed the one signal — does `.objectstack/` exist? — by which the next command can tell a fresh project from a started one. This is the filesystem half of the same property the database half already covers: a dry run leaves nothing behind.
|
|
260
|
+
|
|
261
|
+
Attaching and reading a repository whose root does not exist is now explicitly supported and answers as an empty repository (`get`, `getByHash`, `list`, `history`, `watch`). The root, the type directories and the JSONL change log all appear on the first `put` / `delete`, and nothing about the boot's read-only character changes: no metadata is written that was not written before.
|
|
262
|
+
|
|
263
|
+
One behavioural note for direct users of the package: when the root is absent at `start()`, the chokidar watcher is armed by the first write instead, because chokidar cannot watch a path that does not yet exist. A root brought into existence by a third party while the process runs — with this repository never writing — is therefore not picked up until the next `start()`.
|
|
264
|
+
|
|
265
|
+
- ab07b53: fix(metadata-fs): register every written path with the watcher, so an item created while chokidar is still scanning is not invisible forever (#7282)
|
|
266
|
+
|
|
267
|
+
`FileSystemRepository`'s watcher could go **permanently blind to a single
|
|
268
|
+
item** — external edits to that file produced no `MetadataEvent` for the whole
|
|
269
|
+
life of the process, and nothing recovered short of a restart. The window is a
|
|
270
|
+
race between chokidar's asynchronous initial scan and the repository's own
|
|
271
|
+
first write, and both `start()` (which arms the watcher, after which the caller
|
|
272
|
+
may `put()` on the next tick) and `ensureRoot()` (which arms it in the middle
|
|
273
|
+
of the very first write, #7000) can open it.
|
|
274
|
+
|
|
275
|
+
Measured on chokidar 5 with this repository's options (`usePolling`,
|
|
276
|
+
`interval: 1000`):
|
|
277
|
+
|
|
278
|
+
1. chokidar reads `<root>/<type>/` and finds it EMPTY — the atomic `rename` in
|
|
279
|
+
`writeJsonAtomic` has not landed yet;
|
|
280
|
+
2. the rename lands, changing the directory's mtime;
|
|
281
|
+
3. chokidar calls `watchFile()` on that directory and libuv takes its polling
|
|
282
|
+
baseline stat, which already reflects step 2.
|
|
283
|
+
|
|
284
|
+
The directory's stat then never changes again, so no poll ever fires for it,
|
|
285
|
+
the directory is never re-read, the item file is never added to the watched
|
|
286
|
+
set, and no per-file watcher is created. `getWatched()` reports the type
|
|
287
|
+
directory as `[]` while the file sits in it, and neither `add` nor `change` is
|
|
288
|
+
ever emitted for that path.
|
|
289
|
+
|
|
290
|
+
The fix does not widen any timer. The only writer that can be inside that
|
|
291
|
+
window is the repository itself, so `put()` now tells the watcher explicitly
|
|
292
|
+
about the path it created instead of depending on a directory scan that may
|
|
293
|
+
never notice it. Registration is idempotent and emits nothing.
|
|
294
|
+
|
|
295
|
+
User-visible effect: `MetadataManager.subscribe()` (and every consumer of
|
|
296
|
+
`repo.watch()`) now reliably sees out-of-process edits — a hand edit, or a
|
|
297
|
+
`git checkout` bringing metadata JSON in — to items written earlier in the same
|
|
298
|
+
process. This was also the cause of four merge-queue ejections across three
|
|
299
|
+
PRs; the two time-based mitigations tried before it (a 20s/25s event deadline
|
|
300
|
+
and a wider pre-edit sleep) could not have worked, because the event was never
|
|
301
|
+
delivered rather than late.
|
|
302
|
+
|
|
303
|
+
- 684ab22: fix(metadata-fs): the `FileSystemRepository` watcher now sees external edits in the production layout
|
|
304
|
+
|
|
305
|
+
`MetadataPlugin` attaches the repository at `<project>/.objectstack/metadata`, and the
|
|
306
|
+
watcher's `ignored` matcher was a bare dotfile regex. chokidar applies that matcher to the
|
|
307
|
+
watched root path itself, not only to entries found underneath it, so the `.objectstack`
|
|
308
|
+
segment of the root matched and the entire watch was inert — `getWatched()` returned `{}`
|
|
309
|
+
and no event ever fired. Hand edits, a `git checkout` that brings metadata JSON in, and any
|
|
310
|
+
other out-of-process writer under `.objectstack/metadata/` were invisible until the next
|
|
311
|
+
`start()`, even though `MetadataManager.setRepository()` is wired to those events and uses
|
|
312
|
+
them to invalidate the registry and the `list()` cache.
|
|
313
|
+
|
|
314
|
+
The matcher is now evaluated against the path _relative_ to the watch root, so dot segments
|
|
315
|
+
belonging to the root itself are never considered while dotfiles under the root — including
|
|
316
|
+
the repository's own `.objectstack/` bookkeeping subtree — stay ignored as before.
|
|
317
|
+
|
|
318
|
+
- Updated dependencies [121852d]
|
|
319
|
+
- Updated dependencies [5e247fd]
|
|
320
|
+
- Updated dependencies [1a53a02]
|
|
321
|
+
- Updated dependencies [a954634]
|
|
322
|
+
- Updated dependencies [3d4c545]
|
|
323
|
+
- Updated dependencies [bb7cb41]
|
|
324
|
+
- @objectstack/metadata-core@17.0.0-rc.6
|
|
325
|
+
|
|
3
326
|
## 17.0.0-rc.5
|
|
4
327
|
|
|
5
328
|
### Patch Changes
|
package/README.md
CHANGED
|
@@ -35,7 +35,7 @@ const repo = new FileSystemRepository({
|
|
|
35
35
|
root: './metadata',
|
|
36
36
|
org: 'system',
|
|
37
37
|
});
|
|
38
|
-
await repo.start(); // scan + open watcher
|
|
38
|
+
await repo.start(); // scan + open watcher — creates nothing on disk
|
|
39
39
|
|
|
40
40
|
const view = await repo.get({
|
|
41
41
|
org: 'system',
|
|
@@ -47,4 +47,19 @@ for await (const evt of repo.watch({})) {
|
|
|
47
47
|
}
|
|
48
48
|
```
|
|
49
49
|
|
|
50
|
+
## Root creation is a write, not an attach
|
|
51
|
+
|
|
52
|
+
`start()` never creates `<root>`. Attaching a repository whose root does not
|
|
53
|
+
exist is legal: reads answer as if the repository were empty, and the root —
|
|
54
|
+
together with `<root>/.objectstack/.log/` — appears on the **first write**
|
|
55
|
+
(`put` / `delete`). This is what keeps a read-only boot, such as the dry run
|
|
56
|
+
`os migrate plan` performs on a project that has never been started, from
|
|
57
|
+
leaving a directory skeleton behind (#7000, the filesystem half of #6743).
|
|
58
|
+
|
|
59
|
+
One consequence worth knowing: when the root is absent at `start()`, the
|
|
60
|
+
chokidar watcher is armed by the first write instead, because chokidar cannot
|
|
61
|
+
watch a path that does not exist yet. A root brought into existence by a third
|
|
62
|
+
party while the process runs, without this repository ever writing, is
|
|
63
|
+
therefore not picked up until the next `start()`.
|
|
64
|
+
|
|
50
65
|
See ADR-0008 (incl. §0 amendment) and the `metadata-branch-removal` changeset.
|
package/dist/index.cjs
CHANGED
|
@@ -243,8 +243,6 @@ var FileSystemRepository = class {
|
|
|
243
243
|
this.heads = /* @__PURE__ */ new Map();
|
|
244
244
|
/** Next seq counter, hydrated from the log on `start()`. */
|
|
245
245
|
this.nextSeq = 1;
|
|
246
|
-
/** Paths we wrote ourselves; suppress the resulting chokidar event. */
|
|
247
|
-
this.selfWrites = /* @__PURE__ */ new Set();
|
|
248
246
|
this.watcher = null;
|
|
249
247
|
this.started = false;
|
|
250
248
|
this.org = opts.org;
|
|
@@ -255,15 +253,42 @@ var FileSystemRepository = class {
|
|
|
255
253
|
this.log = new JsonlLog(logFile(this.layout));
|
|
256
254
|
}
|
|
257
255
|
// ── Lifecycle ───────────────────────────────────────────────────────
|
|
256
|
+
/**
|
|
257
|
+
* Attach the repository. **Creates nothing on disk** (#7000).
|
|
258
|
+
*
|
|
259
|
+
* Attaching is not a write. `start()` used to `mkdir` both the root and
|
|
260
|
+
* `<root>/.objectstack/.log` unconditionally, which meant every read-only
|
|
261
|
+
* boot that merely attaches a repository left a skeleton behind — most
|
|
262
|
+
* visibly `os migrate plan`, a declared dry run, on a project that has
|
|
263
|
+
* never been started. That is the same property #6743 ruled on for
|
|
264
|
+
* `.objectstack/data/`: a dry run leaves nothing behind, and the existence
|
|
265
|
+
* of `.objectstack/` has to stay a usable "this project has been started"
|
|
266
|
+
* signal.
|
|
267
|
+
*
|
|
268
|
+
* Every read path below already treats a missing root as an empty
|
|
269
|
+
* repository (`scanHeads` swallows ENOENT, `JsonlLog` guards on
|
|
270
|
+
* `existsSync`, `get` guards on `existsSync`), so the root is materialized
|
|
271
|
+
* by `ensureRoot()` on the first write instead.
|
|
272
|
+
*/
|
|
258
273
|
async start() {
|
|
259
274
|
if (this.started) return;
|
|
260
275
|
this.started = true;
|
|
261
|
-
await import_promises2.default.mkdir(this.layout.root, { recursive: true });
|
|
262
|
-
await import_promises2.default.mkdir(logDir(this.layout), { recursive: true });
|
|
263
276
|
await this.scanHeads();
|
|
264
277
|
const highest = await this.log.highestSeq();
|
|
265
278
|
this.nextSeq = highest + 1;
|
|
266
|
-
if (!this.disableWatch) this.startWatcher();
|
|
279
|
+
if (!this.disableWatch && (0, import_node_fs2.existsSync)(this.layout.root)) this.startWatcher();
|
|
280
|
+
}
|
|
281
|
+
/**
|
|
282
|
+
* Bring the repository root into existence. Called by every write path
|
|
283
|
+
* immediately before it touches the disk — `start()` deliberately does not
|
|
284
|
+
* create it (#7000), so this is the single seam where the root appears.
|
|
285
|
+
*
|
|
286
|
+
* It is also where a watcher that `start()` could not arm (missing root)
|
|
287
|
+
* gets armed, so "external edits are detected" survives the change.
|
|
288
|
+
*/
|
|
289
|
+
async ensureRoot() {
|
|
290
|
+
await import_promises2.default.mkdir(this.layout.root, { recursive: true });
|
|
291
|
+
if (this.started && !this.disableWatch && !this.watcher) this.startWatcher();
|
|
267
292
|
}
|
|
268
293
|
async close() {
|
|
269
294
|
if (this.watcher) {
|
|
@@ -381,13 +406,10 @@ var FileSystemRepository = class {
|
|
|
381
406
|
const seq = this.nextSeq++;
|
|
382
407
|
const ts = this.now().toISOString();
|
|
383
408
|
const file = itemPath(this.layout, ref.type, ref.name);
|
|
409
|
+
await this.ensureRoot();
|
|
384
410
|
await import_promises2.default.mkdir(typeDir(this.layout, ref.type), { recursive: true });
|
|
385
|
-
|
|
386
|
-
|
|
387
|
-
await writeJsonAtomic(file, spec);
|
|
388
|
-
} finally {
|
|
389
|
-
setTimeout(() => this.selfWrites.delete(file), 200);
|
|
390
|
-
}
|
|
411
|
+
await writeJsonAtomic(file, spec);
|
|
412
|
+
this.trackWrittenPath(file);
|
|
391
413
|
this.heads.set(key, hash);
|
|
392
414
|
const evt = {
|
|
393
415
|
seq,
|
|
@@ -427,13 +449,14 @@ var FileSystemRepository = class {
|
|
|
427
449
|
throw new import_metadata_core.ConflictError(ref, opts.parentVersion, currentHead);
|
|
428
450
|
}
|
|
429
451
|
const file = itemPath(this.layout, ref.type, ref.name);
|
|
430
|
-
this.
|
|
452
|
+
await this.ensureRoot();
|
|
453
|
+
this.heads.delete(key);
|
|
431
454
|
try {
|
|
432
455
|
if ((0, import_node_fs2.existsSync)(file)) await import_promises2.default.unlink(file);
|
|
433
|
-
}
|
|
434
|
-
|
|
456
|
+
} catch (err) {
|
|
457
|
+
if (currentHead !== null) this.heads.set(key, currentHead);
|
|
458
|
+
throw err;
|
|
435
459
|
}
|
|
436
|
-
this.heads.delete(key);
|
|
437
460
|
const seq = this.nextSeq++;
|
|
438
461
|
const ts = this.now().toISOString();
|
|
439
462
|
const evt = {
|
|
@@ -502,10 +525,58 @@ var FileSystemRepository = class {
|
|
|
502
525
|
}
|
|
503
526
|
return last;
|
|
504
527
|
}
|
|
528
|
+
/**
|
|
529
|
+
* Register a path this repository just wrote with the watcher (#7282).
|
|
530
|
+
*
|
|
531
|
+
* chokidar's initial scan is asynchronous, and every write path here can be
|
|
532
|
+
* running **while it is still walking the tree** — `start()` arms the watcher
|
|
533
|
+
* and the caller may `put()` on the next tick, and `ensureRoot()` arms it in
|
|
534
|
+
* the middle of the very first write. With `usePolling` that combination has
|
|
535
|
+
* a permanently-blinding interleaving, measured on chokidar 5 with this
|
|
536
|
+
* repository's own options:
|
|
537
|
+
*
|
|
538
|
+
* 1. chokidar reads `<root>/<type>/` and finds it EMPTY — the atomic
|
|
539
|
+
* `rename` in `writeJsonAtomic` has not landed yet.
|
|
540
|
+
* 2. the rename lands; the directory's mtime changes.
|
|
541
|
+
* 3. chokidar calls `watchFile()` on that directory, and libuv takes its
|
|
542
|
+
* polling baseline stat — which already reflects step 2.
|
|
543
|
+
*
|
|
544
|
+
* From then on the directory's stat never changes again, so no poll ever
|
|
545
|
+
* fires for it, `_handleRead` never re-runs, the item file is never added to
|
|
546
|
+
* the watched set, and no per-file watcher is ever created. chokidar emits
|
|
547
|
+
* neither `add` nor `change` for that path **for the life of the process** —
|
|
548
|
+
* `getWatched()` reports the type directory as `[]` forever while the file
|
|
549
|
+
* sits in it. That is the whole of #7282: the four merge-queue ejections all
|
|
550
|
+
* waited out their deadlines (20s, then 25541ms against 25s) on an event that
|
|
551
|
+
* was never going to be delivered, which is why widening the deadline and
|
|
552
|
+
* widening the pre-edit sleep both changed nothing, and why lowering
|
|
553
|
+
* `interval` would change nothing either — a shorter poll re-compares against
|
|
554
|
+
* the same unchanged directory stat.
|
|
555
|
+
*
|
|
556
|
+
* The window is exactly "files that exist at baseline time but were absent
|
|
557
|
+
* from the snapshot read a moment earlier", and the only writer that can be
|
|
558
|
+
* inside it is us. So we close it at the source: tell the watcher explicitly
|
|
559
|
+
* about every path we create, instead of hoping its scan happened to see it.
|
|
560
|
+
*
|
|
561
|
+
* `add()` is idempotent here — `_handleFile` returns early when the parent
|
|
562
|
+
* directory already tracks the basename — and it emits nothing, because
|
|
563
|
+
* chokidar treats an explicit `add()` as an initial add and `ignoreInitial`
|
|
564
|
+
* is set. Its effect is the one we need: `_watchWithNodeFs` registers the
|
|
565
|
+
* basename with the parent directory (without which chokidar drops `change`
|
|
566
|
+
* events for the file) and starts the per-file poll.
|
|
567
|
+
*/
|
|
568
|
+
trackWrittenPath(file) {
|
|
569
|
+
const w = this.watcher;
|
|
570
|
+
if (!w || w.closed) return;
|
|
571
|
+
w.add(file);
|
|
572
|
+
}
|
|
505
573
|
startWatcher() {
|
|
506
|
-
const
|
|
507
|
-
|
|
508
|
-
//
|
|
574
|
+
const root = this.layout.root;
|
|
575
|
+
const w = import_chokidar.default.watch(root, {
|
|
576
|
+
// Skip dotfiles under the root — including the repository's own
|
|
577
|
+
// `.objectstack/` bookkeeping subtree — matched on the path RELATIVE
|
|
578
|
+
// to the watch root (#7150). See `isIgnoredWatchPath`.
|
|
579
|
+
ignored: (p) => isIgnoredWatchPath(root, p),
|
|
509
580
|
ignoreInitial: true,
|
|
510
581
|
depth: 2,
|
|
511
582
|
awaitWriteFinish: { stabilityThreshold: 50, pollInterval: 20 },
|
|
@@ -521,8 +592,55 @@ var FileSystemRepository = class {
|
|
|
521
592
|
w.on("unlink", (p) => void this.handleFsChange(p, "unlink"));
|
|
522
593
|
this.watcher = w;
|
|
523
594
|
}
|
|
595
|
+
/**
|
|
596
|
+
* Translate a watcher event into a `MetadataEvent`, or drop it.
|
|
597
|
+
*
|
|
598
|
+
* ## Self-writes are suppressed by content identity, never by a clock (#7335)
|
|
599
|
+
*
|
|
600
|
+
* This used to open with `if (this.selfWrites.has(absPath)) return;` — a
|
|
601
|
+
* `Set` that `put()`/`delete()` added the path to and a `setTimeout(…, 200)`
|
|
602
|
+
* cleared. That check discarded **every** event for a recently-written path
|
|
603
|
+
* without ever looking at what the watcher had actually observed, which is
|
|
604
|
+
* the whole defect: with `usePolling`, chokidar compares state once per
|
|
605
|
+
* `interval`, so our write and an external edit landing between two ticks
|
|
606
|
+
* are delivered as **one** event carrying the *external* content. Dropping
|
|
607
|
+
* it on a wall clock destroyed the only notification that edit would ever
|
|
608
|
+
* produce.
|
|
609
|
+
*
|
|
610
|
+
* Measured on `origin/main` @ `69fde55`, 40 iterations, poll phase
|
|
611
|
+
* randomised so the delivery lag samples `[0, interval)` uniformly:
|
|
612
|
+
*
|
|
613
|
+
* delivery lag < 200ms → 7 runs → external edit SWALLOWED, every time
|
|
614
|
+
* delivery lag > 200ms → 33 runs → external edit delivered, every time
|
|
615
|
+
*
|
|
616
|
+
* A perfect split on the wall-clock boundary, and the reason earlier
|
|
617
|
+
* instrumentation saw 0/360: a *fixed* pre-edit sleep phase-locks the poll,
|
|
618
|
+
* pinning the lag (measured: 519–585ms across 25 runs) safely outside the
|
|
619
|
+
* window. Nothing about the window was rare — it was unsampled.
|
|
620
|
+
*
|
|
621
|
+
* What remains is the check that was already doing the real work one step
|
|
622
|
+
* down, and it needs no timer because it compares the content the watcher
|
|
623
|
+
* **read** against the index:
|
|
624
|
+
*
|
|
625
|
+
* - `add`/`change` — `currentHead === hash` drops the event when the bytes
|
|
626
|
+
* on disk are the bytes we last published. `put()` sets that head in the
|
|
627
|
+
* same continuation as its `rename`, and `awaitWriteFinish` holds the
|
|
628
|
+
* event for a further `stabilityThreshold`, so it is never late.
|
|
629
|
+
* - `unlink` — `!currentHead` drops the event when the index already
|
|
630
|
+
* agrees the item is gone. `delete()` retires the head *before* it
|
|
631
|
+
* unlinks, precisely because this face gets no `awaitWriteFinish` delay.
|
|
632
|
+
*
|
|
633
|
+
* Both faces are pinned together in `test/self-write-suppression.test.ts`.
|
|
634
|
+
*
|
|
635
|
+
* Note the deliberate limit: identity is judged on what round-trips through
|
|
636
|
+
* the file, so a spec whose in-memory form does not (a `Date`, which
|
|
637
|
+
* canonicalises to `{}` in memory but to an ISO string once written and
|
|
638
|
+
* re-read) is republished as an external `update`. That predates this change
|
|
639
|
+
* and is independent of it — such a spec already fails `put().version ===
|
|
640
|
+
* get().hash`, and the 200ms window never covered it either, expiring some
|
|
641
|
+
* 360ms before the event it would have had to catch.
|
|
642
|
+
*/
|
|
524
643
|
async handleFsChange(absPath, kind) {
|
|
525
|
-
if (this.selfWrites.has(absPath)) return;
|
|
526
644
|
const parsed = parseItemPath(this.layout, absPath);
|
|
527
645
|
if (!parsed) return;
|
|
528
646
|
const ref = {
|
|
@@ -573,6 +691,11 @@ var FileSystemRepository = class {
|
|
|
573
691
|
});
|
|
574
692
|
}
|
|
575
693
|
};
|
|
694
|
+
function isIgnoredWatchPath(root, absPath) {
|
|
695
|
+
const rel = import_node_path3.default.relative(root, absPath);
|
|
696
|
+
if (rel === "" || rel.startsWith("..")) return false;
|
|
697
|
+
return rel.split(/[\\/]/).some((segment) => segment.startsWith("."));
|
|
698
|
+
}
|
|
576
699
|
async function readJson(file) {
|
|
577
700
|
try {
|
|
578
701
|
const text = await import_promises2.default.readFile(file, "utf8");
|