@objectstack/metadata-fs 17.0.0 → 17.2.0

package/CHANGELOG.md

@@ -1,5 +1,116 @@
 # @objectstack/metadata-fs
 
+## 17.2.0
+
+### Patch Changes
+
+- 46644e2: `FileSystemRepository.close()` now terminates every live `watch()` iterator
+  instead of leaving it parked (#11127). A consumer holding a `for await` over
+  `watch()` at shutdown never saw its loop end — on a repository that was already
+  gone.
+  
+  `close()` retired the chokidar watcher and the resync sweep and stopped there.
+  It never reached the event broker, and the broker had no teardown of its own:
+  `subscribe`/`unsubscribe` add to and delete from a plain `Set`, and nothing else
+  emptied it. Each iterator parks its pending `next()` on a `waiter` that only two
+  things can settle — a broker `push`, or the iterator's own terminator, which ran
+  from `iterator.return()`/`throw()` and from nowhere else. After `close()` the
+  chokidar source was gone so no `push` could arrive, and the subscriber was still
+  registered with nothing left to run its terminator.
+  
+  Unlike the sibling defect in `SysMetadataRepository` (#11021) this was not
+  filter-dependent: there was no drain attempt at all, so every subscription shape
+  hung, `watch({})` included. Measured before the fix: nine cases —
+  `watch({org}, seq)`, `watch({org})`, `watch({})`, a ref-exact filter, a watcher
+  over the real chokidar watcher, a watcher with no pull outstanding, four
+  concurrent watchers, and the `return()`-symmetry comparison — were all still
+  unsettled 2s after `close()`. `MetadataManager.startRepositoryWatch()`, which
+  awaits `iter.next()` in a loop, is exactly the shape that hung.
+  
+  The broker now holds each subscription's terminator next to its event sink, and
+  `close()` runs every terminator — the same routine the consumer's own
+  `iterator.return()` runs, so a parked `next()` settles with `{ done: true }` and
+  no value, and so does every later one. Shutdown is deliberately **not** delivered
+  as an event: a synthetic drain event is subject to the very filters `watch()`
+  applies to real ones, and delivering an event has never ended an iterator
+  (invariant 8, `@objectstack/metadata-core`'s `repository.ts`).
+  
+  One narrower path is closed with it. `watch()` returns a deferred iterable whose
+  subscriber registers only once the eager log read resolves, so a `close()`
+  landing inside that window swept a broker the subscription had not yet joined —
+  the same forever-parked shape by a different route. `watch()` now carries the
+  close generation it was opened under, and a subscription that arrives after a
+  shutdown terminates on arrival.
+  
+  Invariant 8 named `FileSystemRepository` as its one known non-conformance. With
+  this change the invariant has no declared exceptions, and its text says so.
+- Updated dependencies [8cc8401]
+- Updated dependencies [26f3588]
+- Updated dependencies [05bc692]
+- Updated dependencies [f334d66]
+- Updated dependencies [2810695]
+  - @objectstack/metadata-core@17.2.0
+
+## 17.1.0
+
+### Patch Changes
+
+- bd2fc8b: fix(metadata-fs): an external write reaches subscribers even when the watcher's single delivery attempt is lost — content-keyed reconciliation behind the poll (#9339)
+  
+  `FileSystemRepository`'s watcher gave an externally-written file **exactly one**
+  chance to be noticed, and losing it was permanent and silent. Under
+  `usePolling`, chokidar re-reads a directory only when its stat *strictly*
+  advances; an external write advances the type directory's mtime once, so poll
+  #2..#N compare an unchanged stat and can never rediscover the file. Measured on
+  #9339 with a fault-injection harness: with that single read suppressed, fifteen
+  further poll ticks never find the file — a 20s deadline and a 200s deadline buy
+  the same one attempt. That is the structural reason behind #7282's empirical
+  finding that the event is *"never delivered, not slow"*, and why widening the
+  deadline (#7208) and lowering `interval` were both spent before they were tried.
+  
+  **At least six independent one-shot gates sit on that attempt**, spanning three
+  layers — the kernel timestamp (the directory mtime does not strictly advance),
+  chokidar's readdir throttle and readdir snapshot, and chokidar's emit gates
+  (`_throttle('add')`, a stale `_pendingWrites` entry, the `awaitWriteFinish`
+  ENOENT early return). Each produces a byte-identical observable: no event, ever,
+  for that path. They are indistinguishable at the point of failure, which is why
+  #7282's close — picked from that family — covered one member and reopened.
+  
+  **The fix does not name a member.** A bounded, content-keyed reconciliation
+  sweep runs alongside the watcher and compares what is on disk against `heads`,
+  the index that already defines what the repository believes it holds, publishing
+  any divergence through the *same* handler the watcher feeds. Its only premise is
+  that the bytes on disk stopped matching the index, so it is robust across all six
+  by construction — and equally across a seventh nobody has found.
+  
+  - **Cadence** — one pass over `<root>/<type>/*.json` every 2s (twice the poll
+    interval), the same walk `start()` already performs once. Sweeps are chained
+    rather than intervalled, so they can never overlap or stack behind a slow
+    disk; the timer is `unref`ed and is retired by `close()`; and it is armed only
+    alongside the watcher, so a `disableWatch` repository pays nothing.
+  - **Exactly-once is preserved.** Suppression stays content-keyed (`#7335`): the
+    sweep republishes nothing the watcher already delivered, and recognises this
+    repository's own `put()` by content rather than by a clock.
+  - **Events are indistinguishable from the fast path** — same `op`,
+    `parentHash`, `source: 'fs'` and actor, because they are produced by the same
+    code. A subscriber cannot be made to care which path noticed.
+  - **A recovered path is re-armed** with the watcher through the seam `put()`
+    already uses, so a loss upstream of chokidar's `_handleFile` does not leave
+    the file dependent on the sweep forever.
+  - `put()`'s existing direct registration (#7336) is unchanged, as are
+    `usePolling`, `interval`, and `awaitWriteFinish`.
+  
+  ⚠️ **Bound on the claim.** The six gates are *forced fault injections*, not the
+  CI mechanism, which was never identified and may be a seventh. What is measured
+  is that the fix converts **six of six** forced one-shot gates from permanent
+  loss to delivery (3/3 runs each), where all six returned an empty event list
+  before it. That is not the same statement as "the flake is fixed".
+- Updated dependencies [b6c7690]
+- Updated dependencies [845e164]
+- Updated dependencies [1a7f907]
+- Updated dependencies [7fc01db]
+  - @objectstack/metadata-core@17.1.0
+
 ## 17.0.0
 
 ### Patch Changes

package/dist/index.cjs

@@ -144,6 +144,16 @@ function createBroker(matches) {
         if (!matches(evt, s.filter)) continue;
         s.push(evt);
       }
+    },
+    terminateAll: () => {
+      const snapshot = Array.from(subs);
+      subs.clear();
+      for (const s of snapshot) {
+        try {
+          s.terminate();
+        } catch {
+        }
+      }
     }
   };
 }
@@ -158,6 +168,10 @@ function createWatchIterable(args) {
   const subscriber = {
     filter: args.filter,
     closed: false,
+    // Assigned below, once `close` exists. Termination and the consumer's own
+    // `return()` are ONE routine, deliberately: invariant 8 requires shutdown
+    // to be indistinguishable from `iterator.return()`.
+    terminate: () => void 0,
     push: (evt) => {
       if (subscriber.closed) return;
       const k = evtKey(evt);
@@ -206,6 +220,8 @@ function createWatchIterable(args) {
     }
     return { value: void 0, done: true };
   };
+  subscriber.terminate = close;
+  if (args.arrivesClosed?.()) close();
   const iterator = {
     next: () => {
       if (closed) return Promise.resolve({ value: void 0, done: true });
@@ -235,6 +251,8 @@ var matchRefFilter = (ref, filter) => {
   return true;
 };
 var matchEvent = (evt, filter) => matchRefFilter(evt.ref, filter);
+var RESYNC_INTERVAL_MS = 2e3;
+var isEnoent = (err) => err?.code === "ENOENT";
 var FileSystemRepository = class {
   constructor(opts) {
     this.mutex = new KeyedMutex();
@@ -245,6 +263,25 @@ var FileSystemRepository = class {
     this.nextSeq = 1;
     this.watcher = null;
     this.started = false;
+    /** Pending reconciliation sweep (#9339). Chained, never overlapping. */
+    this.resyncTimer = null;
+    /** False before the watcher is armed and from `close()` onwards. */
+    this.resyncEnabled = false;
+    /**
+     * Sweep read faults already reported, keyed `CODE @ path`, so a standing
+     * fault is announced once rather than every 2s (AGENTS.md: say it once, at
+     * the first degradation). An entry is cleared when that path reads again.
+     */
+    this.resyncFaults = /* @__PURE__ */ new Set();
+    /**
+     * Bumped by every `close()`. `watch()` reads it before its deferred log
+     * replay starts and hands the comparison to `createWatchIterable`, so a
+     * subscription that registers AFTER the shutdown sweep terminates on
+     * arrival instead of parking forever (#11127). A counter rather than a
+     * boolean because `start()` may follow `close()`: a repository restart must
+     * not poison the watchers opened after it.
+     */
+    this.closeGeneration = 0;
     this.org = opts.org;
     this.fsActor = opts.fsActor ?? "fs";
     this.disableWatch = opts.disableWatch ?? false;
@@ -290,7 +327,35 @@ var FileSystemRepository = class {
     await import_promises2.default.mkdir(this.layout.root, { recursive: true });
     if (this.started && !this.disableWatch && !this.watcher) this.startWatcher();
   }
+  /**
+   * Shut the repository down, ending every live `watch()` iterator.
+   *
+   * **Shutdown terminates; it does not emit** — invariant 8 in
+   * `@objectstack/metadata-core`'s `repository.ts`, and the reason this method
+   * reaches the broker at all. It used to retire the chokidar watcher and the
+   * resync sweep and stop there. The broker has no teardown of its own
+   * (`subscribe`/`unsubscribe` add to and delete from a plain `Set`), and each
+   * iterator parks its pending `next()` on a `waiter` that only a broker
+   * `push` or the iterator's own terminator can settle. After `close()` the
+   * chokidar source was gone, so no `push` could arrive; the subscriber was
+   * still registered, and nothing ran its terminator. A consumer holding a
+   * `for await` at shutdown — `MetadataManager.startRepositoryWatch()` is
+   * exactly that shape — therefore never saw its loop end, for EVERY
+   * subscription shape including `watch({})`.
+   *
+   * Termination is expressed as termination: each subscription's
+   * `terminate()`, which is the same routine the consumer's own
+   * `iterator.return()` runs, so no consumer has to tell "the repository shut
+   * down under me" apart from "I broke my own loop". A synthetic drain event
+   * would be the wrong shape and was measured to be so (#11021): the
+   * subscriptions most in need of draining are exactly the ones whose filter
+   * or numeric `since` drops it, and delivering an event has never ended an
+   * iterator.
+   */
   async close() {
+    this.stopResync();
+    this.closeGeneration++;
+    this.broker.terminateAll();
     if (this.watcher) {
       await this.watcher.close();
       this.watcher = null;
@@ -365,6 +430,7 @@ var FileSystemRepository = class {
         if (matchEvent(evt, filter)) replay.push(evt);
       }
     })();
+    const generation = this.closeGeneration;
     return deferredIterable(promise.then(
       () => createWatchIterable({
         filter,
@@ -372,7 +438,8 @@ var FileSystemRepository = class {
         replay,
         broker: this.broker,
         matches: matchEvent,
-        branchKeyOf: (e) => e.ref.org
+        branchKeyOf: (e) => e.ref.org,
+        arrivesClosed: () => this.closeGeneration !== generation
       })
     ));
   }
@@ -591,6 +658,228 @@ var FileSystemRepository = class {
     w.on("change", (p) => void this.handleFsChange(p, "change"));
     w.on("unlink", (p) => void this.handleFsChange(p, "unlink"));
     this.watcher = w;
+    this.startResync();
+  }
+  /**
+   * Publish the `delete` face of an externally-observed removal.
+   *
+   * Extracted from `handleFsChange` unchanged so the reconciliation sweep
+   * (#9339) can reuse it **verbatim** rather than growing a second copy of the
+   * event shape. The one-line invariant: the caller already holds the per-key
+   * mutex, and `!currentHead` is the content-keyed suppression that makes our
+   * own `delete()` a no-op here.
+   */
+  async publishExternalDelete(ref, key) {
+    const currentHead = this.heads.get(key) ?? null;
+    if (!currentHead) return;
+    this.heads.delete(key);
+    const seq = this.nextSeq++;
+    const evt = {
+      seq,
+      op: "delete",
+      ref: { ...ref, version: void 0 },
+      hash: null,
+      parentHash: currentHead,
+      actor: this.fsActor,
+      ts: this.now().toISOString(),
+      source: "fs"
+    };
+    await this.log.append(evt);
+    this.broker.publish(evt);
+  }
+  startResync() {
+    this.resyncEnabled = true;
+    this.scheduleResync();
+  }
+  stopResync() {
+    this.resyncEnabled = false;
+    if (this.resyncTimer) {
+      clearTimeout(this.resyncTimer);
+      this.resyncTimer = null;
+    }
+  }
+  /**
+   * Schedule the next sweep — chained, never `setInterval` (#9339).
+   *
+   * A chained timeout cannot stack: the next sweep is armed only once the
+   * previous one has finished, so a saturated runner degrades to *fewer*
+   * sweeps instead of a growing backlog of overlapping tree walks. The timer
+   * is `unref`ed because a backstop must never be the reason a process stays
+   * alive.
+   */
+  scheduleResync() {
+    if (!this.resyncEnabled || this.resyncTimer) return;
+    const timer = setTimeout(() => {
+      this.resyncTimer = null;
+      void this.resync().finally(() => this.scheduleResync());
+    }, RESYNC_INTERVAL_MS);
+    timer.unref?.();
+    this.resyncTimer = timer;
+  }
+  /**
+   * Announce a sweep read that could not run — the non-silence half of #8895's
+   * "discriminate or propagate".
+   *
+   * ## Why `error` and not `warn`
+   *
+   * AGENTS.md decides the level with one question: *after the degradation, does
+   * the system still look "normal" from the outside while something it claims
+   * is persisted has not actually landed?* Here it does. Nothing throws, the
+   * watcher stays armed, `getWatched()` stays populated, `start()` succeeded —
+   * and the repository's index quietly stops tracking what is on disk. That is
+   * the rule's second limb verbatim ("persisted state and runtime state
+   * disagree"), not the functional-degradation limb: no capability is visibly
+   * smaller, so nobody finds out by using the missing thing.
+   *
+   * The counter-argument — *this is only a backstop, the watcher is still the
+   * fast path* — is why the level is arguable, and it does not survive the
+   * failing errno. The sharp case is fd exhaustion: EMFILE/ENFILE break this
+   * `readdir` and chokidar's `fs.watchFile` polling **at the same time and for
+   * the same reason**, so the fast path is not an independent fallback under
+   * precisely the load that produces this fault. A backstop that is silently
+   * absent whenever it is most needed is a durability-shaped degradation.
+   *
+   * ⚠️ AGENTS.md also warns against over-applying `error`, and the discipline
+   * that answers it is the ledger, not a quieter level: an `error` owes the
+   * consequence and the fix, said **once** at the first degradation rather than
+   * once per failed read. A sweep runs every 2s forever, so an unlatched
+   * `console.error` here would be the mirror-image failure the same rule names.
+   *
+   * ⛔ It deliberately does NOT throw. This runs on a background timer; taking
+   * a process down on a transient EACCES would be worse than the bug. The bar
+   * met here is non-silence, not propagation.
+   *
+   * The channel is `console.error` because this class has no logger: nothing is
+   * injected through `FileSystemRepositoryOptions`, and widening that public
+   * surface to carry one is out of scope for this fix.
+   */
+  reportResyncFault(target, err) {
+    const code = err?.code ?? "UNKNOWN";
+    const key = `${code} @ ${target}`;
+    if (this.resyncFaults.has(key)) return;
+    this.resyncFaults.add(key);
+    console.error(
+      `[FileSystemRepository] metadata reconciliation sweep could not read ${target} (${code}). CONSEQUENCE: external edits under this path are no longer reconciled, so this repository's index and its watch() subscribers can drift from what is on disk while everything keeps reporting healthy. The chokidar watcher is not an independent fallback here \u2014 fd exhaustion degrades both. FIX: restore read access to the path; the sweep recovers by itself on the first successful read. Reported once per path and error code.`
+    );
+  }
+  /** Re-arm reporting for a path that reads again, so a recurrence is heard. */
+  clearResyncFault(target) {
+    if (this.resyncFaults.size === 0) return;
+    const suffix = ` @ ${target}`;
+    for (const key of this.resyncFaults) {
+      if (key.endsWith(suffix)) this.resyncFaults.delete(key);
+    }
+  }
+  /**
+   * Content-keyed reconciliation sweep — the backstop that makes external-edit
+   * detection a guarantee rather than a single chance (#9339, #7282).
+   *
+   * ## Why the watcher alone cannot be the guarantee
+   *
+   * An external write to `<root>/<type>/<name>.json` reaches a subscriber only
+   * if chokidar notices it, and under `usePolling` it gets **exactly one**
+   * opportunity to do so: the write advances the type directory's mtime once,
+   * and chokidar re-reads a directory only when its stat *strictly advances*,
+   * so every later poll compares an unchanged stat and can never rediscover
+   * the file. Measured on #9339 with a fault-injection harness: with the one
+   * read suppressed, fifteen further poll ticks never find the new file, and a
+   * 20s deadline and a 200s deadline buy the same single attempt. That is the
+   * structural reason behind #7282's empirical finding that the event is
+   * "never delivered, not slow", and why widening the deadline (#7208) and
+   * lowering `interval` were both spent before they were tried.
+   *
+   * At least six independent one-shot gates sit on that single attempt,
+   * spanning three layers — the kernel timestamp (the directory mtime does not
+   * strictly advance), chokidar's readdir throttle and readdir snapshot, and
+   * chokidar's emit gates (`_throttle('add')`, a stale `_pendingWrites` entry,
+   * the `awaitWriteFinish` ENOENT early return). Each one produces a
+   * byte-identical observable: no event, ever, for that path.
+   *
+   * ## Why this shape, and not a narrower one
+   *
+   * ⚠️ The six are indistinguishable at the point of failure, so **any fix
+   * that has to name which gate fired is a fix for one member of a family** —
+   * which is exactly how #7282 was closed and exactly why it reopened. This
+   * sweep never asks. It compares what is on disk against `heads`, the index
+   * that already defines what this repository believes it holds, and publishes
+   * the divergence through the same `handleFsChange` the watcher feeds. It is
+   * therefore robust across all six *by construction*, and equally across a
+   * seventh nobody has found: the only property it relies on is that the bytes
+   * on disk stopped matching the index.
+   *
+   * `put()` is unaffected and keeps its direct registration (`trackWrittenPath`
+   * calls `watcher.add` and bypasses the whole chain, which is why the `put()`
+   * half of this family was already closed by #7336 and the external-write half
+   * was not).
+   *
+   * ## Cost, and why it is bounded
+   *
+   * One pass over `<root>/<type>/*.json` per sweep — the same walk `start()`
+   * already performs once — with no retry loop inside it and no work at all
+   * when nothing diverged. Sweeps are chained, so they cannot overlap; the
+   * timer is `unref`ed and dies with `close()`; and it is armed only alongside
+   * the watcher, so a `disableWatch` repository pays nothing.
+   *
+   * Discovery is by content, never by stat: a stat pre-filter would reintroduce
+   * a time key of exactly the kind this replaces.
+   */
+  async resync() {
+    const root = this.layout.root;
+    let entries = [];
+    try {
+      entries = await import_promises2.default.readdir(root, { withFileTypes: true });
+      this.clearResyncFault(root);
+    } catch (err) {
+      if (!isEnoent(err)) this.reportResyncFault(root, err);
+      return;
+    }
+    const onDisk = /* @__PURE__ */ new Set();
+    const unreadableTypes = /* @__PURE__ */ new Set();
+    for (const entry of entries) {
+      if (!entry.isDirectory()) continue;
+      if (entry.name.startsWith(".")) continue;
+      const dir = import_node_path3.default.join(root, entry.name);
+      let files = [];
+      try {
+        files = await import_promises2.default.readdir(dir);
+        this.clearResyncFault(dir);
+      } catch (err) {
+        if (!isEnoent(err)) {
+          this.reportResyncFault(dir, err);
+          unreadableTypes.add(entry.name);
+        }
+        continue;
+      }
+      for (const file of files) {
+        if (!file.endsWith(".json") || file.startsWith(".")) continue;
+        const abs = import_node_path3.default.join(dir, file);
+        const parsed = parseItemPath(this.layout, abs);
+        if (!parsed) continue;
+        const ref = {
+          org: this.org,
+          type: parsed.type,
+          name: parsed.name
+        };
+        const key = (0, import_metadata_core.refKey)(ref);
+        onDisk.add(key);
+        const before = this.heads.get(key);
+        await this.handleFsChange(abs, "add");
+        if (this.heads.get(key) !== before) {
+          this.trackWrittenPath(abs);
+        }
+      }
+    }
+    for (const key of [...this.heads.keys()]) {
+      if (onDisk.has(key)) continue;
+      const ref = parseRefKey(key);
+      if (!ref) continue;
+      if (unreadableTypes.has(ref.type)) continue;
+      const file = itemPath(this.layout, ref.type, ref.name);
+      await this.mutex.run(key, async () => {
+        if ((0, import_node_fs2.existsSync)(file)) return;
+        await this.publishExternalDelete(ref, key);
+      });
+    }
   }
   /**
    * Translate a watcher event into a `MetadataEvent`, or drop it.
@@ -651,22 +940,7 @@ var FileSystemRepository = class {
     const key = (0, import_metadata_core.refKey)(ref);
     await this.mutex.run(key, async () => {
       if (kind === "unlink") {
-        const currentHead2 = this.heads.get(key) ?? null;
-        if (!currentHead2) return;
-        this.heads.delete(key);
-        const seq2 = this.nextSeq++;
-        const evt2 = {
-          seq: seq2,
-          op: "delete",
-          ref: { ...ref, version: void 0 },
-          hash: null,
-          parentHash: currentHead2,
-          actor: this.fsActor,
-          ts: this.now().toISOString(),
-          source: "fs"
-        };
-        await this.log.append(evt2);
-        this.broker.publish(evt2);
+        await this.publishExternalDelete(ref, key);
         return;
       }
       const body = await readJson(absPath);