zen-fs-config 0.5.3 → 0.5.5

package/DESIGN.md

@@ -704,31 +704,208 @@ createConfigRepo('my-app', options?)
   │
   ├─ 5. If options.backendInfo provided:
   │     ├─ Generate replica ID (options.primaryBackendId or auto)
-  │     └─ Write descriptor to .meta/backends/{replicaId}.json (if not exists)
+  │     ├─ Dedup check: same type + options (stable key) already registered?
+  │     └─ Write descriptor to .meta/backends/{replicaId}.json (if not duplicate)
   │
   ├─ 6. Read all backend descriptors from .meta/backends/
+  │     └─ Dedup: remove duplicates (same type + options, different ID)
+  │        ├─ Delete duplicate files on ALL replicas directly
+  │        └─ Create tombstone + delete local file
   │
-  ├─ 7. Determine nodeId (explicit > localStorage > auto-generated)
+  ├─ 7. Determine nodeId (explicit parameter > auto-generated)
   │
-  ├─ 8. Create ConfigRepo with primary = 'local-idb'
+  ├─ 8. Create final ConfigRepo instance (primary = 'local-idb')
   │
-  ├─ 9. setupSync: for each backend (except local-idb):
-  │     ├─ Create backend instance
-  │     ├─ Create SyncPair(IndexedDB, replica, bi-directional)
-  │     └─ syncEngine.watch(pairId)
+  ├─ 9. setupSync: for each replica backend:
+  │     ├─ Create backend instance (e.g., Gitee, RemoteStorage)
+  │     ├─ Create SyncPair(IndexedDB ↔ replica, bi-directional)
+  │     ├─ Register conflict handler
+  │     └─ NOTE: Does NOT call watch() yet (see §11.4 for why)
   │
-  ├─ 10. Load app data group references from .meta/app-data-groups/{appId}/
-  │      For each referenced data-sync group:
-  │      ├─ Create data-sync backends (merge account fields if accountBackendId set)
-  │      ├─ Create SyncPair for each data backend
-  │      └─ syncEngine.watch(pairId)
+  ├─ 10. Load config cache from IndexedDB (fast, local-only)
   │
-  ├─ 11. syncMetaToReplicas: push .meta/ changes to all replicas (background)
+  ├─ 11. initialSyncAndDedup() — only if replicas exist:
+  │     ├─ unwatchAll()      — safety: clear any stale snapshots
+  │     ├─ syncAll()         — full bidirectional sync (no cached snapshot
+  │     │                      → every file is compared, remote-only files
+  │     │                      are pulled to local)
+  │     ├─ readAllBackendDescriptors() — dedup duplicates pulled from remote
+  │     │   ├─ Delete dup files on ALL replicas directly
+  │     │   └─ Create tombstones for deduped descriptors
+  │     ├─ processTombstones() — delete deduped files on all replicas
+  │     └─ watchAll()       — start monitoring for future changes
+  │         (snapshots now reflect the fully synced state)
   │
-  └─ 12. Load config cache from IndexedDB
+  ├─ 12. syncMetaToReplicas() — background push of .meta/ changes
+  │     (watchers already running, this just speeds up initial propagation)
+  │
+  └─ 13. Return ConfigRepo instance
+```
+
+### 11.2 Why "Sync Before Watch" (Critical Design Decision)
+
+The sync engine (`zen-fs-sync`) uses **snapshot-based change detection**. When `watch()` is called on a SyncPair, it triggers `buildInitialSnapshots()` which:
+
+1. Builds a snapshot of the source (IndexedDB) — walks all files, records `path`, `size`, `mtimeMs`
+2. Builds a snapshot of the target (remote backend) — same process
+3. Caches **separate** snapshots: `prevSrcSnap` (source) and `prevTgtSnap` (target)
+
+On the next `syncAll()`, `syncBidirectional()` compares each side's current snapshot with its own previous snapshot independently. If both sides are unchanged → **"unchanged" → skip sync entirely**.
+
+**The problem**: `buildInitialSnapshots()` only *reads* file metadata — it does NOT copy any files. So if the remote has files that the local doesn't (e.g., duplicate backend descriptors written by another node), the cached snapshots reflect the un-synced state. The subsequent sync sees "snapshots already match this state" and skips — the file is never actually copied to local, and local-only dedup logic never runs.
+
+**The fix**: Always perform a full `syncAll()` **before** `watch()`. With no cached snapshot, `syncBidirectional()` does a complete comparison and copies all missing files. After sync completes, `watch()` builds snapshots from the now-consistent state.
+
+This pattern is applied in three places:
+- `createConfigRepo()` → `initialSyncAndDedup()` (sync → dedup → watch)
+- `addBackend()` → `syncMetaToReplicas()` then `watch()` (sync → watch)
+- `AppDataGroupImpl.connect()` → `syncAll()` then `watchAll()` (sync → watch)
+
+### 11.3 `flush()` — Manual Sync Trigger
+
+```
+flush()
+  │
+  ├─ 1. processTombstones()
+  │     For each tombstone in /.meta/.deleted/:
+  │     ├─ Delete the actual file on primary (in case re-created)
+  │     ├─ Delete the actual file on ALL replicas
+  │     └─ Delete version sidecars on all replicas
+  │
+  ├─ 2. syncAll()
+  │     For each SyncPair (IndexedDB ↔ replica):
+  │     ├─ Build current snapshots of both sides
+  │     ├─ Compare with cached snapshot (if any)
+  │     ├─ Detect changes: Created / Modified / Deleted
+  │     ├─ Resolve conflicts (source-wins strategy)
+  │     └─ Copy files in both directions as needed
+  │
+  ├─ 3. readAllBackendDescriptors() — post-sync dedup
+  │     Sync may have pulled duplicate backend descriptors from remote.
+  │     Re-run dedup to catch and remove them.
+  │     ├─ Delete dup files on ALL replicas directly
+  │     └─ Create tombstones for deduped descriptors
+  │
+  ├─ 4. processTombstones() — process any new tombstones from step 3
+  │
+  ├─ 5. updateTombstoneConfirmations()
+  │     Mark each tombstone as confirmed by all replica backends
+  │
+  ├─ 6. gcTombstones()
+  │     Remove tombstones confirmed by ALL backends in the topology
+  │
+  └─ Return SyncResult[] (one per sync pair)
+```
+
+### 11.4 Tombstone-Based Deletion Propagation
+
+When a file is deleted via `deleteFile(path)`:
+
+```
+deleteFile('/.meta/backends/old-backend.json')
+  │
+  ├─ 1. Write tombstone: /.meta/.deleted/++meta__backends__old-backend++json.json
+  │     { path, deletedAt, deletedBy, confirmedBy: [primaryBackendId] }
+  │
+  ├─ 2. Delete the actual file on primary (IndexedDB)
+  │
+  └─ 3. Delete version sidecar (.old-backend.json.version) on primary
+```
+
+On the next `processTombstones()` (called by `flush()` or `initialSyncAndDedup()`):
+
+```
+For each tombstone:
+  ├─ Delete file on primary (in case sync re-created it)
+  ├─ Delete file on ALL replicas
+  ├─ Delete version sidecar on ALL replicas
+  └─ Tombstone file itself is synced to replicas via syncAll()
+     → Late-joining replicas see the tombstone and delete the file
+```
+
+**Why tombstones?** Without them, bi-directional sync treats a deleted local file as "missing → needs to be copied from remote". The tombstone explicitly signals "this file was intentionally deleted" so all replicas honor the deletion. Tombstones are garbage-collected after all backends confirm receipt.
+
+### 11.5 Backend Deduplication
+
+When `readAllBackendDescriptors()` detects two backends with the same `type` + `options` (using stable key ordering) but different IDs:
+
+```
+Detected: rs-1 and rs-2 have identical type + options
+  │
+  ├─ 1. Keep the one with the earliest mtime (created first)
+  │
+  ├─ 2. For each duplicate:
+  │     ├─ Delete descriptor file on ALL replicas directly
+  │     │   (prevents sync from pulling it back)
+  │     ├─ Delete version sidecar on ALL replicas
+  │     └─ Create tombstone + delete local file
+  │
+  └─ 3. Return deduplicated list (duplicates removed)
+```
+
+The stable key function (`backendDedupKey`) sorts object keys recursively, so `{ token: 'a', owner: 'b' }` and `{ owner: 'b', token: 'a' }` produce the same key and are correctly detected as duplicates.
+
+### 11.6 Dynamic Backend Management
+
+**`addBackend(id, type, options)`**:
+
+```
+  ├─ 1. Dedup check: reject if same type+options already registered
+  ├─ 2. Create backend instance
+  ├─ 3. Write descriptor to .meta/backends/{id}.json
+  ├─ 4. Create SyncPair (IndexedDB ↔ new replica, bi-directional)
+  ├─ 5. syncMetaToReplicas() — full sync FIRST (pull + push)
+  └─ 6. watch(pairId) — start monitoring AFTER sync completes
+```
+
+**`removeBackend(id)`**:
+
+```
+  ├─ 1. Delete descriptor file on the remote backend DIRECTLY
+  │     (must happen before removing sync pair — otherwise can't reach remote)
+  ├─ 2. Delete version sidecar on remote
+  ├─ 3. Create tombstone + delete local descriptor file
+  ├─ 4. Remove sync pair (stops watching + disposes)
+  ├─ 5. Remove from replicaBackends map
+  ├─ 6. Dispose backend instance
+  ├─ 7. processTombstones() — propagate deletion to remaining replicas
+  └─ 8. flush() — sync + GC tombstones
 ```
 
-### 11.2 Standalone Data-Sync Group (`createDataSyncGroup`)
+### 11.7 Watch Mode (Auto-Sync)
+
+After initialization, each SyncPair runs in **watch mode** with hybrid change detection:
+
+```
+watch() triggers:
+  │
+  ├─ 1. Register onChange callbacks (if backend supports it)
+  │     Local backends (IndexedDB) push change notifications
+  │     → triggers debounced sync (default 300ms)
+  │
+  ├─ 2. buildInitialSnapshots()
+  │     ├─ BiDirectional: cache separate source and target snapshots
+  │     └─ OneWay: cache source snapshot only
+  │
+  └─ 3. Start poll timers (if backend supports shouldSync)
+       ├─ Remote backends poll shouldSync() every pollIntervalMs (default 30min)
+       └─ Fallback: if no onChange and no shouldSync, poll every interval
+```
+
+**State guard**: If `unwatch()` is called during `buildInitialSnapshots()` (which is async), the snapshots are discarded — they won't be cached. This prevents stale snapshots from causing sync skips.
+
+**Snapshot comparison** in `syncBidirectional()`:
+1. Build current snapshots of both sides (via `getSnapshot()`)
+2. Compare each side independently against its own previous snapshot:
+   - `srcChanged = !snapshotsEqual(prevSrcSnap, currentSrcSnap)`
+   - `tgtChanged = !snapshotsEqual(prevTgtSnap, currentTgtSnap)`
+   - If neither changed and both previous snapshots exist → skip sync entirely
+3. Cache current snapshots as `prevSrcSnap` and `prevTgtSnap` for next comparison
+4. If either side changed, proceed with full diff and file operations
+
+**Key difference from previous design**: The old approach merged source and target snapshots into a single map (`source ∪ target`), which lost information about which filesystem a file belonged to. The new approach keeps them separate, enabling precise per-side change detection and bidirectional deletion propagation (see §11.10).
+
+### 11.8 Standalone Data-Sync Group (`createDataSyncGroup`)
 
 ```
 createDataSyncGroup('my-app', options?)
@@ -743,11 +920,16 @@ createDataSyncGroup('my-app', options?)
   ├─ 3. Create IndexedDB as local primary (for offline access)
   │
   ├─ 4. Setup sync: IndexedDB ↔ each data backend (bi-directional)
+  │     NOTE: Does NOT watch yet — sync first
+  │
+  ├─ 5. syncAll() — pull data from remote backends
+  │
+  ├─ 6. watchAll() — start monitoring AFTER sync completes
   │
-  └─ 5. Return DataSyncGroup handle with direct fs access
+  └─ 7. Return DataSyncGroup handle with direct fs access
 ```
 
-### 11.3 Unified Entry Point (`connect`)
+### 11.9 Unified Entry Point (`connect`)
 
 `createConfigRepo` and `createDataSyncGroup` are lower-level factory functions. The recommended entry point is `connect`, which auto-detects the group type and dispatches to the appropriate factory:
 
@@ -815,6 +997,110 @@ interface ConnectResult {
 
 **Offline / zero-parameter mode**: When no `backendInfo` is provided, `connect` defaults to `config-sync` and creates an IndexedDB-only repo (same as `createConfigRepo` with no options).
 
+### 11.10 Snapshot Optimization Design
+
+This section describes three interrelated optimizations to the sync engine's snapshot mechanism.
+
+#### 11.10.1 FS-Provided `createSnapshot()`
+
+The `SyncableFS` interface now includes an optional `createSnapshot()` method:
+
+```typescript
+interface SyncableFS {
+  // ... existing methods ...
+
+  /**
+   * Optional: Build a filesystem snapshot.
+   * Returns a map of relative path → {size, mtimeMs} for all files under root.
+   * Returns null if the filesystem is unreachable.
+   *
+   * Backends that can provide a more efficient snapshot than the generic
+   * walkFiles+stat approach should implement this method.
+   */
+  createSnapshot?(root: string, filter?: SyncFilter): Promise<Map<string, FileSnapshot> | null>;
+}
+```
+
+The sync engine's `getSnapshot()` helper dispatches to the FS-provided method when available, falling back to the generic `buildSnapshot()` (walkFiles + stat) otherwise:
+
+```typescript
+private async getSnapshot(fs: SyncableFS): Promise<Map<string, FileSnapshot> | null> {
+  if (fs.createSnapshot) {
+    return fs.createSnapshot(this.root, this.options.filter);
+  }
+  return buildSnapshot(fs, this.root, this.options.filter);
+}
+```
+
+**Optimization examples**:
+- **Gitee/GitHub**: Use Git tree API to fetch all file metadata in a single request instead of walking files one by one
+- **IndexedDB**: Use `getAll()` for batch querying instead of individual `stat()` calls
+- **InMemory**: Directly iterate the internal Map (no async I/O overhead)
+
+Backends that do not implement `createSnapshot()` are fully supported — the generic fallback produces identical results.
+
+#### 11.10.2 Separate Source and Target Snapshots
+
+**Previous design** (merged snapshots):
+- `buildInitialSnapshots()` merged source and target into a single map: `sourceSnapshots = new Map([...srcSnap, ...tgtSnap])`
+- `syncBidirectional()` compared `currentMerged` with the cached merged snapshot
+- **Problem**: The merged map lost which filesystem a file belonged to. A file present on target but not source could be "new on target" or "deleted from source" — the merged snapshot couldn't distinguish.
+
+**New design** (separate snapshots):
+- `buildInitialSnapshots()` caches two independent maps: `prevSrcSnap` and `prevTgtSnap`
+- `syncBidirectional()` compares each side independently:
+  ```
+  srcChanged = !snapshotsEqual(prevSrcSnap, currentSrcSnap)
+  tgtChanged = !snapshotsEqual(prevTgtSnap, currentTgtSnap)
+  if (!srcChanged && !tgtChanged && prevSrcSnap && prevTgtSnap) → skip sync
+  ```
+- Each side's change is detected independently, preserving file-location information
+
+#### 11.10.3 Bidirectional Deletion Propagation
+
+When a file exists on one side but not the other, the sync engine uses previous snapshots to distinguish "created" from "deleted":
+
+```
+File on target, not on source:
+  ├─ Was it on source in the previous snapshot (prevSrcSnap)?
+  │   ├─ Yes → file was deleted from source → delete from target too (propagate deletion)
+  │   └─ No  → file was created on target → copy to source
+
+File on source, not on target:
+  ├─ Was it on target in the previous snapshot (prevTgtSnap)?
+  │   ├─ Yes → file was deleted from target → delete from source too (propagate deletion)
+  │   └─ No  → file was created on source → copy to target
+```
+
+**Without this mechanism**, deleting a file on one side would cause the sync engine to see "the other side still has it → copy it back", effectively undoing the deletion.
+
+**Relationship with tombstones**: The tombstone mechanism (§11.4) and bidirectional deletion propagation operate at different layers and complement each other:
+
+| Mechanism | Layer | Trigger | How It Works |
+|-----------|-------|---------|--------------|
+| Tombstone | `zen-fs-config` (application) | Application calls `deleteFile()` | Writes a `.meta/.deleted/` marker, physically deletes file on all replicas **before** sync runs |
+| Deletion propagation | `zen-fs-sync` (engine) | Sync detects one-side-only file | Compares with previous snapshot to determine if file was created or deleted |
+
+Tombstones handle application-initiated deletions (the common case). Deletion propagation handles deletions that bypass the tombstone flow — e.g., external modifications on the remote backend, or files removed by other sync mechanisms.
+
+#### 11.10.4 `shouldSync()` vs Snapshot Comparison
+
+These two mechanisms are complementary, not interchangeable:
+
+| Mechanism | Purpose | Cost | When Used |
+|-----------|---------|------|-----------|
+| `shouldSync()` | Fast "has anything changed?" boolean | O(1) for remote (ETag/commit check) | `onRemotePoll()` — decide whether to trigger sync at all |
+| Snapshot comparison | "What exactly changed?" detail | O(n) filesystem traversal | `syncBidirectional()` — decide what to copy/delete |
+
+`shouldSync()` is **not** part of the snapshot comparison because:
+1. `shouldSync()` updates its internal baseline after each call — calling it again during sync would return stale results
+2. `shouldSync()` returning false doesn't guarantee the snapshot is unchanged — it means the FS's own change detection says nothing changed, which could miss edge cases
+3. `shouldSync()` returning true doesn't tell us **which** files changed — snapshots are still needed for that
+
+The two-level optimization works as follows:
+1. **Level 1**: `shouldSync()` in remote poll → if false, skip sync trigger entirely (saves the O(n) snapshot build)
+2. **Level 2**: Separate snapshot comparison in sync → if both sides unchanged, skip file operations (saves I/O)
+
 ## 12. Data Flow
 
 ### Read Path

package/dist/index.js

@@ -274,6 +274,12 @@ function backendToSyncableFS(backend, name) {
   if (typeof backend.shouldSync === "function") {
     syncable.shouldSync = () => backend.shouldSync();
   }
+  if (typeof backend.createSnapshot === "function") {
+    syncable.createSnapshot = (root, filter) => backend.createSnapshot(root, filter);
+  }
+  if (typeof backend.writeFileWithMtime === "function") {
+    syncable.writeFileWithMtime = (path, data, mtimeMs) => backend.writeFileWithMtime(path, data, mtimeMs);
+  }
   if (typeof backend.checkForUpdates === "function") {
     syncable.checkForUpdates = () => backend.checkForUpdates();
   }
@@ -404,6 +410,12 @@ async function wrapZenFSFileSystem(config) {
   backend.onChange = (callback) => {
     changeCallback = callback;
   };
+  if (typeof isolatedFS.createSnapshot === "function") {
+    backend.createSnapshot = (root, filter) => isolatedFS.createSnapshot(root, filter);
+  }
+  if (typeof isolatedFS.writeFileWithMtime === "function") {
+    backend.writeFileWithMtime = (path, data, mtimeMs) => isolatedFS.writeFileWithMtime(path, data, mtimeMs);
+  }
   return backend;
 }
 var inMemoryCounter = 0;
@@ -1179,6 +1191,7 @@ var ConfigRepo = class {
     try {
       const entries = await this.cachedFS.readdir(BACKENDS_DIR);
       const items = [];
+      const corruptFiles = [];
       for (const entry of entries) {
         if (!entry.endsWith(".json")) continue;
         const filePath = `${BACKENDS_DIR}/${entry}`;
@@ -1193,8 +1206,37 @@ var ConfigRepo = class {
             } catch {
             }
             items.push({ desc, mtime });
+          } else {
+            console.warn(`[ConfigRepo] Backend descriptor ${entry} is missing id/type fields, marking for cleanup`);
+            corruptFiles.push(filePath);
           }
+        } catch (parseErr) {
+          console.warn(`[ConfigRepo] Backend descriptor ${entry} has corrupted JSON: ${parseErr}. Marking for cleanup.`);
+          corruptFiles.push(filePath);
+        }
+      }
+      for (const corruptPath of corruptFiles) {
+        for (const [, replica] of this.replicaBackends) {
+          try {
+            await replica.instance.unlink(corruptPath);
+          } catch {
+          }
+          try {
+            await replica.instance.unlink(versionPathFor(corruptPath));
+          } catch {
+          }
+        }
+        try {
+          await this.deleteFile(corruptPath);
         } catch {
+          try {
+            await this.cachedFS.unlink(corruptPath);
+          } catch {
+          }
+          try {
+            await this.cachedFS.unlink(versionPathFor(corruptPath));
+          } catch {
+          }
         }
       }
       const seen = /* @__PURE__ */ new Map();

package/dist/index.mjs

@@ -217,6 +217,12 @@ function backendToSyncableFS(backend, name) {
   if (typeof backend.shouldSync === "function") {
     syncable.shouldSync = () => backend.shouldSync();
   }
+  if (typeof backend.createSnapshot === "function") {
+    syncable.createSnapshot = (root, filter) => backend.createSnapshot(root, filter);
+  }
+  if (typeof backend.writeFileWithMtime === "function") {
+    syncable.writeFileWithMtime = (path, data, mtimeMs) => backend.writeFileWithMtime(path, data, mtimeMs);
+  }
   if (typeof backend.checkForUpdates === "function") {
     syncable.checkForUpdates = () => backend.checkForUpdates();
   }
@@ -347,6 +353,12 @@ async function wrapZenFSFileSystem(config) {
   backend.onChange = (callback) => {
     changeCallback = callback;
   };
+  if (typeof isolatedFS.createSnapshot === "function") {
+    backend.createSnapshot = (root, filter) => isolatedFS.createSnapshot(root, filter);
+  }
+  if (typeof isolatedFS.writeFileWithMtime === "function") {
+    backend.writeFileWithMtime = (path, data, mtimeMs) => isolatedFS.writeFileWithMtime(path, data, mtimeMs);
+  }
   return backend;
 }
 var inMemoryCounter = 0;
@@ -1122,6 +1134,7 @@ var ConfigRepo = class {
     try {
       const entries = await this.cachedFS.readdir(BACKENDS_DIR);
       const items = [];
+      const corruptFiles = [];
       for (const entry of entries) {
         if (!entry.endsWith(".json")) continue;
         const filePath = `${BACKENDS_DIR}/${entry}`;
@@ -1136,8 +1149,37 @@ var ConfigRepo = class {
             } catch {
             }
             items.push({ desc, mtime });
+          } else {
+            console.warn(`[ConfigRepo] Backend descriptor ${entry} is missing id/type fields, marking for cleanup`);
+            corruptFiles.push(filePath);
           }
+        } catch (parseErr) {
+          console.warn(`[ConfigRepo] Backend descriptor ${entry} has corrupted JSON: ${parseErr}. Marking for cleanup.`);
+          corruptFiles.push(filePath);
+        }
+      }
+      for (const corruptPath of corruptFiles) {
+        for (const [, replica] of this.replicaBackends) {
+          try {
+            await replica.instance.unlink(corruptPath);
+          } catch {
+          }
+          try {
+            await replica.instance.unlink(versionPathFor(corruptPath));
+          } catch {
+          }
+        }
+        try {
+          await this.deleteFile(corruptPath);
         } catch {
+          try {
+            await this.cachedFS.unlink(corruptPath);
+          } catch {
+          }
+          try {
+            await this.cachedFS.unlink(versionPathFor(corruptPath));
+          } catch {
+          }
         }
       }
       const seen = /* @__PURE__ */ new Map();

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "zen-fs-config",
-  "version": "0.5.3",
+  "version": "0.5.5",
   "description": "Distributed config management library built on ZenFS, zen-fs-cache, and zen-fs-sync",
   "main": "dist/index.js",
   "module": "dist/index.mjs",