@objectstack/metadata-fs 17.0.0 → 17.1.0

package/CHANGELOG.md

@@ -1,5 +1,65 @@
 # @objectstack/metadata-fs
 
+## 17.1.0
+
+### Patch Changes
+
+- bd2fc8b: fix(metadata-fs): an external write reaches subscribers even when the watcher's single delivery attempt is lost — content-keyed reconciliation behind the poll (#9339)
+  
+  `FileSystemRepository`'s watcher gave an externally-written file **exactly one**
+  chance to be noticed, and losing it was permanent and silent. Under
+  `usePolling`, chokidar re-reads a directory only when its stat *strictly*
+  advances; an external write advances the type directory's mtime once, so poll
+  #2..#N compare an unchanged stat and can never rediscover the file. Measured on
+  #9339 with a fault-injection harness: with that single read suppressed, fifteen
+  further poll ticks never find the file — a 20s deadline and a 200s deadline buy
+  the same one attempt. That is the structural reason behind #7282's empirical
+  finding that the event is *"never delivered, not slow"*, and why widening the
+  deadline (#7208) and lowering `interval` were both spent before they were tried.
+  
+  **At least six independent one-shot gates sit on that attempt**, spanning three
+  layers — the kernel timestamp (the directory mtime does not strictly advance),
+  chokidar's readdir throttle and readdir snapshot, and chokidar's emit gates
+  (`_throttle('add')`, a stale `_pendingWrites` entry, the `awaitWriteFinish`
+  ENOENT early return). Each produces a byte-identical observable: no event, ever,
+  for that path. They are indistinguishable at the point of failure, which is why
+  #7282's close — picked from that family — covered one member and reopened.
+  
+  **The fix does not name a member.** A bounded, content-keyed reconciliation
+  sweep runs alongside the watcher and compares what is on disk against `heads`,
+  the index that already defines what the repository believes it holds, publishing
+  any divergence through the *same* handler the watcher feeds. Its only premise is
+  that the bytes on disk stopped matching the index, so it is robust across all six
+  by construction — and equally across a seventh nobody has found.
+  
+  - **Cadence** — one pass over `<root>/<type>/*.json` every 2s (twice the poll
+    interval), the same walk `start()` already performs once. Sweeps are chained
+    rather than intervalled, so they can never overlap or stack behind a slow
+    disk; the timer is `unref`ed and is retired by `close()`; and it is armed only
+    alongside the watcher, so a `disableWatch` repository pays nothing.
+  - **Exactly-once is preserved.** Suppression stays content-keyed (`#7335`): the
+    sweep republishes nothing the watcher already delivered, and recognises this
+    repository's own `put()` by content rather than by a clock.
+  - **Events are indistinguishable from the fast path** — same `op`,
+    `parentHash`, `source: 'fs'` and actor, because they are produced by the same
+    code. A subscriber cannot be made to care which path noticed.
+  - **A recovered path is re-armed** with the watcher through the seam `put()`
+    already uses, so a loss upstream of chokidar's `_handleFile` does not leave
+    the file dependent on the sweep forever.
+  - `put()`'s existing direct registration (#7336) is unchanged, as are
+    `usePolling`, `interval`, and `awaitWriteFinish`.
+  
+  ⚠️ **Bound on the claim.** The six gates are *forced fault injections*, not the
+  CI mechanism, which was never identified and may be a seventh. What is measured
+  is that the fix converts **six of six** forced one-shot gates from permanent
+  loss to delivery (3/3 runs each), where all six returned an empty event list
+  before it. That is not the same statement as "the flake is fixed".
+- Updated dependencies [b6c7690]
+- Updated dependencies [845e164]
+- Updated dependencies [1a7f907]
+- Updated dependencies [7fc01db]
+  - @objectstack/metadata-core@17.1.0
+
 ## 17.0.0
 
 ### Patch Changes

package/dist/index.cjs

@@ -235,6 +235,8 @@ var matchRefFilter = (ref, filter) => {
   return true;
 };
 var matchEvent = (evt, filter) => matchRefFilter(evt.ref, filter);
+var RESYNC_INTERVAL_MS = 2e3;
+var isEnoent = (err) => err?.code === "ENOENT";
 var FileSystemRepository = class {
   constructor(opts) {
     this.mutex = new KeyedMutex();
@@ -245,6 +247,16 @@ var FileSystemRepository = class {
     this.nextSeq = 1;
     this.watcher = null;
     this.started = false;
+    /** Pending reconciliation sweep (#9339). Chained, never overlapping. */
+    this.resyncTimer = null;
+    /** False before the watcher is armed and from `close()` onwards. */
+    this.resyncEnabled = false;
+    /**
+     * Sweep read faults already reported, keyed `CODE @ path`, so a standing
+     * fault is announced once rather than every 2s (AGENTS.md: say it once, at
+     * the first degradation). An entry is cleared when that path reads again.
+     */
+    this.resyncFaults = /* @__PURE__ */ new Set();
     this.org = opts.org;
     this.fsActor = opts.fsActor ?? "fs";
     this.disableWatch = opts.disableWatch ?? false;
@@ -291,6 +303,7 @@ var FileSystemRepository = class {
     if (this.started && !this.disableWatch && !this.watcher) this.startWatcher();
   }
   async close() {
+    this.stopResync();
     if (this.watcher) {
       await this.watcher.close();
       this.watcher = null;
@@ -591,6 +604,228 @@ var FileSystemRepository = class {
     w.on("change", (p) => void this.handleFsChange(p, "change"));
     w.on("unlink", (p) => void this.handleFsChange(p, "unlink"));
     this.watcher = w;
+    this.startResync();
+  }
+  /**
+   * Publish the `delete` face of an externally-observed removal.
+   *
+   * Extracted from `handleFsChange` unchanged so the reconciliation sweep
+   * (#9339) can reuse it **verbatim** rather than growing a second copy of the
+   * event shape. The one-line invariant: the caller already holds the per-key
+   * mutex, and `!currentHead` is the content-keyed suppression that makes our
+   * own `delete()` a no-op here.
+   */
+  async publishExternalDelete(ref, key) {
+    const currentHead = this.heads.get(key) ?? null;
+    if (!currentHead) return;
+    this.heads.delete(key);
+    const seq = this.nextSeq++;
+    const evt = {
+      seq,
+      op: "delete",
+      ref: { ...ref, version: void 0 },
+      hash: null,
+      parentHash: currentHead,
+      actor: this.fsActor,
+      ts: this.now().toISOString(),
+      source: "fs"
+    };
+    await this.log.append(evt);
+    this.broker.publish(evt);
+  }
+  startResync() {
+    this.resyncEnabled = true;
+    this.scheduleResync();
+  }
+  stopResync() {
+    this.resyncEnabled = false;
+    if (this.resyncTimer) {
+      clearTimeout(this.resyncTimer);
+      this.resyncTimer = null;
+    }
+  }
+  /**
+   * Schedule the next sweep — chained, never `setInterval` (#9339).
+   *
+   * A chained timeout cannot stack: the next sweep is armed only once the
+   * previous one has finished, so a saturated runner degrades to *fewer*
+   * sweeps instead of a growing backlog of overlapping tree walks. The timer
+   * is `unref`ed because a backstop must never be the reason a process stays
+   * alive.
+   */
+  scheduleResync() {
+    if (!this.resyncEnabled || this.resyncTimer) return;
+    const timer = setTimeout(() => {
+      this.resyncTimer = null;
+      void this.resync().finally(() => this.scheduleResync());
+    }, RESYNC_INTERVAL_MS);
+    timer.unref?.();
+    this.resyncTimer = timer;
+  }
+  /**
+   * Announce a sweep read that could not run — the non-silence half of #8895's
+   * "discriminate or propagate".
+   *
+   * ## Why `error` and not `warn`
+   *
+   * AGENTS.md decides the level with one question: *after the degradation, does
+   * the system still look "normal" from the outside while something it claims
+   * is persisted has not actually landed?* Here it does. Nothing throws, the
+   * watcher stays armed, `getWatched()` stays populated, `start()` succeeded —
+   * and the repository's index quietly stops tracking what is on disk. That is
+   * the rule's second limb verbatim ("persisted state and runtime state
+   * disagree"), not the functional-degradation limb: no capability is visibly
+   * smaller, so nobody finds out by using the missing thing.
+   *
+   * The counter-argument — *this is only a backstop, the watcher is still the
+   * fast path* — is why the level is arguable, and it does not survive the
+   * failing errno. The sharp case is fd exhaustion: EMFILE/ENFILE break this
+   * `readdir` and chokidar's `fs.watchFile` polling **at the same time and for
+   * the same reason**, so the fast path is not an independent fallback under
+   * precisely the load that produces this fault. A backstop that is silently
+   * absent whenever it is most needed is a durability-shaped degradation.
+   *
+   * ⚠️ AGENTS.md also warns against over-applying `error`, and the discipline
+   * that answers it is the ledger, not a quieter level: an `error` owes the
+   * consequence and the fix, said **once** at the first degradation rather than
+   * once per failed read. A sweep runs every 2s forever, so an unlatched
+   * `console.error` here would be the mirror-image failure the same rule names.
+   *
+   * ⛔ It deliberately does NOT throw. This runs on a background timer; taking
+   * a process down on a transient EACCES would be worse than the bug. The bar
+   * met here is non-silence, not propagation.
+   *
+   * The channel is `console.error` because this class has no logger: nothing is
+   * injected through `FileSystemRepositoryOptions`, and widening that public
+   * surface to carry one is out of scope for this fix.
+   */
+  reportResyncFault(target, err) {
+    const code = err?.code ?? "UNKNOWN";
+    const key = `${code} @ ${target}`;
+    if (this.resyncFaults.has(key)) return;
+    this.resyncFaults.add(key);
+    console.error(
+      `[FileSystemRepository] metadata reconciliation sweep could not read ${target} (${code}). CONSEQUENCE: external edits under this path are no longer reconciled, so this repository's index and its watch() subscribers can drift from what is on disk while everything keeps reporting healthy. The chokidar watcher is not an independent fallback here \u2014 fd exhaustion degrades both. FIX: restore read access to the path; the sweep recovers by itself on the first successful read. Reported once per path and error code.`
+    );
+  }
+  /** Re-arm reporting for a path that reads again, so a recurrence is heard. */
+  clearResyncFault(target) {
+    if (this.resyncFaults.size === 0) return;
+    const suffix = ` @ ${target}`;
+    for (const key of this.resyncFaults) {
+      if (key.endsWith(suffix)) this.resyncFaults.delete(key);
+    }
+  }
+  /**
+   * Content-keyed reconciliation sweep — the backstop that makes external-edit
+   * detection a guarantee rather than a single chance (#9339, #7282).
+   *
+   * ## Why the watcher alone cannot be the guarantee
+   *
+   * An external write to `<root>/<type>/<name>.json` reaches a subscriber only
+   * if chokidar notices it, and under `usePolling` it gets **exactly one**
+   * opportunity to do so: the write advances the type directory's mtime once,
+   * and chokidar re-reads a directory only when its stat *strictly advances*,
+   * so every later poll compares an unchanged stat and can never rediscover
+   * the file. Measured on #9339 with a fault-injection harness: with the one
+   * read suppressed, fifteen further poll ticks never find the new file, and a
+   * 20s deadline and a 200s deadline buy the same single attempt. That is the
+   * structural reason behind #7282's empirical finding that the event is
+   * "never delivered, not slow", and why widening the deadline (#7208) and
+   * lowering `interval` were both spent before they were tried.
+   *
+   * At least six independent one-shot gates sit on that single attempt,
+   * spanning three layers — the kernel timestamp (the directory mtime does not
+   * strictly advance), chokidar's readdir throttle and readdir snapshot, and
+   * chokidar's emit gates (`_throttle('add')`, a stale `_pendingWrites` entry,
+   * the `awaitWriteFinish` ENOENT early return). Each one produces a
+   * byte-identical observable: no event, ever, for that path.
+   *
+   * ## Why this shape, and not a narrower one
+   *
+   * ⚠️ The six are indistinguishable at the point of failure, so **any fix
+   * that has to name which gate fired is a fix for one member of a family** —
+   * which is exactly how #7282 was closed and exactly why it reopened. This
+   * sweep never asks. It compares what is on disk against `heads`, the index
+   * that already defines what this repository believes it holds, and publishes
+   * the divergence through the same `handleFsChange` the watcher feeds. It is
+   * therefore robust across all six *by construction*, and equally across a
+   * seventh nobody has found: the only property it relies on is that the bytes
+   * on disk stopped matching the index.
+   *
+   * `put()` is unaffected and keeps its direct registration (`trackWrittenPath`
+   * calls `watcher.add` and bypasses the whole chain, which is why the `put()`
+   * half of this family was already closed by #7336 and the external-write half
+   * was not).
+   *
+   * ## Cost, and why it is bounded
+   *
+   * One pass over `<root>/<type>/*.json` per sweep — the same walk `start()`
+   * already performs once — with no retry loop inside it and no work at all
+   * when nothing diverged. Sweeps are chained, so they cannot overlap; the
+   * timer is `unref`ed and dies with `close()`; and it is armed only alongside
+   * the watcher, so a `disableWatch` repository pays nothing.
+   *
+   * Discovery is by content, never by stat: a stat pre-filter would reintroduce
+   * a time key of exactly the kind this replaces.
+   */
+  async resync() {
+    const root = this.layout.root;
+    let entries = [];
+    try {
+      entries = await import_promises2.default.readdir(root, { withFileTypes: true });
+      this.clearResyncFault(root);
+    } catch (err) {
+      if (!isEnoent(err)) this.reportResyncFault(root, err);
+      return;
+    }
+    const onDisk = /* @__PURE__ */ new Set();
+    const unreadableTypes = /* @__PURE__ */ new Set();
+    for (const entry of entries) {
+      if (!entry.isDirectory()) continue;
+      if (entry.name.startsWith(".")) continue;
+      const dir = import_node_path3.default.join(root, entry.name);
+      let files = [];
+      try {
+        files = await import_promises2.default.readdir(dir);
+        this.clearResyncFault(dir);
+      } catch (err) {
+        if (!isEnoent(err)) {
+          this.reportResyncFault(dir, err);
+          unreadableTypes.add(entry.name);
+        }
+        continue;
+      }
+      for (const file of files) {
+        if (!file.endsWith(".json") || file.startsWith(".")) continue;
+        const abs = import_node_path3.default.join(dir, file);
+        const parsed = parseItemPath(this.layout, abs);
+        if (!parsed) continue;
+        const ref = {
+          org: this.org,
+          type: parsed.type,
+          name: parsed.name
+        };
+        const key = (0, import_metadata_core.refKey)(ref);
+        onDisk.add(key);
+        const before = this.heads.get(key);
+        await this.handleFsChange(abs, "add");
+        if (this.heads.get(key) !== before) {
+          this.trackWrittenPath(abs);
+        }
+      }
+    }
+    for (const key of [...this.heads.keys()]) {
+      if (onDisk.has(key)) continue;
+      const ref = parseRefKey(key);
+      if (!ref) continue;
+      if (unreadableTypes.has(ref.type)) continue;
+      const file = itemPath(this.layout, ref.type, ref.name);
+      await this.mutex.run(key, async () => {
+        if ((0, import_node_fs2.existsSync)(file)) return;
+        await this.publishExternalDelete(ref, key);
+      });
+    }
   }
   /**
    * Translate a watcher event into a `MetadataEvent`, or drop it.
@@ -651,22 +886,7 @@ var FileSystemRepository = class {
     const key = (0, import_metadata_core.refKey)(ref);
     await this.mutex.run(key, async () => {
       if (kind === "unlink") {
-        const currentHead2 = this.heads.get(key) ?? null;
-        if (!currentHead2) return;
-        this.heads.delete(key);
-        const seq2 = this.nextSeq++;
-        const evt2 = {
-          seq: seq2,
-          op: "delete",
-          ref: { ...ref, version: void 0 },
-          hash: null,
-          parentHash: currentHead2,
-          actor: this.fsActor,
-          ts: this.now().toISOString(),
-          source: "fs"
-        };
-        await this.log.append(evt2);
-        this.broker.publish(evt2);
+        await this.publishExternalDelete(ref, key);
         return;
       }
       const body = await readJson(absPath);