@statewalker/webrun-files-sync 0.2.2 → 0.3.0

package/README.md

@@ -1,20 +1,50 @@
 # @statewalker/webrun-files-sync
 
-rclone-like file-synchronisation engine (copy / sync / bisync / move / check) over two `FilesApi` endpoints.
+## What it is
 
-## Overview
+An rclone-like synchronisation engine for two `FilesApi` endpoints from
+`@statewalker/webrun-files`. It supports five operations (`copy`, `sync`, `bisync`, `move`,
+`check`) and splits each into a pure planning step and an execution step that verifies and can
+resume.
 
-`files-sync` is Axis A of the two-axis VCS architecture — the git-independent "sync" engine. It **plans, then executes**: `plan(a, b, op, opts)` is pure (mutates neither endpoint) and returns a serializable `SyncPlan`; `execute(plan, a, b, opts)` applies it, emitting events and resuming from an optional checkpoint after interruption. Bidirectional `bisync` is a genuine **three-way merge over a sync-owned `SyncAnchor`** (delegated to `@statewalker/webrun-merge`), never two chained one-way syncs.
+## Why it exists
 
-It imports **only** `@statewalker/webrun-files` and `@statewalker/webrun-merge` and knows nothing of git or versioning — that is the hard Axis A ✗↔ Axis B ban. Content identity (`hashContent`), the per-copy `Transfer` strategy, and conflict resolution are all injected.
+Keeping two file trees in step (a browser store and a server, a local folder and a cloud bucket)
+is a different job from versioning: it needs cheap change detection, safe deletes and two-way
+reconciliation, not commits. Because every storage backend in the ecosystem is a `FilesApi`, one
+engine covers all pairs of backends. Two-way sync is a real three-way merge against the state
+recorded after the previous sync, done by `@statewalker/webrun-merge`, so a delete on one side is
+propagated instead of being undone by a copy from the other side.
 
-## Installation
+## How to use
 
 ```bash
 pnpm add @statewalker/webrun-files-sync
 ```
 
-## Quick Start
+| Import | Contents |
+| --- | --- |
+| `@statewalker/webrun-files-sync` | `plan`, `execute`, `buildAnchor`, `snapshot`, `isChanged`, `createStreamingTransfer`, and the types `SyncOptions`, `SyncPlan`, `SyncAction`, `SyncEvent`, `SyncOp`, `SyncConflict`, `SyncAnchor`, `AnchorStore`, `CheckpointStore`, `Transfer`, `PathFilter`, `VerificationMode`, `Resolution`, `FilesApi`, `ByteStream`. ESM, runs in browsers, Node and workers. |
+
+| Operation | Effect on `b` (and `a`) |
+| --- | --- |
+| `copy` | New and changed files from `a` are written to `b`. Nothing is deleted. |
+| `sync` | Like `copy`, and files in `b` that are not in `a` are deleted. |
+| `move` | Like `copy`, then each source file in `a` is deleted once its copy verifies. |
+| `check` | Compares only. `execute` reports warnings and conflicts and changes nothing. |
+| `bisync` | Changes from both sides are applied to both sides; conflicts are reported. |
+
+`plan(a, b, op, opts)` returns a JSON-serializable `SyncPlan` (`actions`, `conflicts`,
+`warnings`). `execute(plan, a, b, opts)` yields `warning`, `conflict`, `start`, `done`, `skipped`
+and `failed` events.
+
+`SyncOptions`: `hashContent` (required); `filter?(path)`; `transfer?` (default
+`createStreamingTransfer()`); `verify?` (default `"size"`); `anchorStore?` and `pairKey?`
+(default `"default"`) for bisync; `resolve?(conflict)`; `quickFingerprint?`; `checkpoint?`.
+
+## Examples
+
+One-way copy:
 
 ```typescript
 import { createHash } from "node:crypto";
@@ -27,33 +57,91 @@ async function sha256(input: AsyncIterable<Uint8Array>): Promise<string> {
   return `sha256:${h.digest("hex")}`;
 }
 
-const a = new MemFilesApi({ initialFiles: { "/new.txt": "new", "/changed.txt": "v2" } });
+const a = new MemFilesApi({ initialFiles: { "/new.txt": "new", "/changed.txt": "version 2" } });
 const b = new MemFilesApi({ initialFiles: { "/changed.txt": "v1", "/extra.txt": "keep" } });
 
-// Pure plan — dry-runnable, serializable, mutates nothing.
 const p = await plan(a, b, "copy", { hashContent: sha256 });
-
-// Execute, streaming per-action events.
 for await (const event of execute(p, a, b, { hashContent: sha256 })) {
-  if (event.type === "done") console.log("synced", event.action);
+  if (event.type === "done") console.log(event.action);
+}
+// b: new.txt, changed.txt ("version 2"), extra.txt (kept)
+// Had both versions of changed.txt the same size and mtime, it would count as unchanged.
+```
+
+Two-way sync with a stored anchor:
+
+```typescript
+import type { AnchorStore, SyncAnchor } from "@statewalker/webrun-files-sync";
+
+const anchors = new Map<string, SyncAnchor>();
+const anchorStore: AnchorStore = {
+  read: async (key) => anchors.get(key),
+  write: async (key, anchor) => void anchors.set(key, anchor),
+};
+const opts = { hashContent: sha256, anchorStore, pairKey: "laptop<->server" };
+
+for await (const e of execute(await plan(a, b, "bisync", opts), a, b, opts)) {
+  if (e.type === "conflict") console.log("conflict", e.conflict.kind, e.conflict.path);
 }
-// b now has new.txt + updated changed.txt; extra.txt is kept (copy never deletes).
 ```
 
-## API
+Resumable execution: pass a `checkpoint` (`{ read(): Promise<Set<number>>, add(index) }`) backed by
+durable storage; a second `execute` with the same plan skips the recorded action indices.
+
+## Internals
+
+### Plan first, then execute
+
+`plan` reads both endpoints and writes nothing, so a plan can be shown to a user, stored, or
+tested. `execute` applies the actions in order, verifies each copy, records each finished index in
+the checkpoint, and keeps going after a failed action (it emits `failed` and continues).
+
+### Change detection stops at the first decisive check
+
+```
+path/kind ─> size ─> equal mtime ─> head+tail sample ─> full hashContent
+ differ?     differ?   equal?        (quickFingerprint)   differ?
+ changed     changed   unchanged     differ? changed      changed
+```
+
+Most files are decided by size or modification time without reading their bytes. With
+`quickFingerprint`, the first and last 64 bytes are compared before hashing the whole file.
+
+### How bisync works
+
+```
+anchor (state after last sync) ─┐
+endpoint a ─────────────────────┼─> webrun-merge ─> actions for both sides + conflicts
+endpoint b ─────────────────────┘
+```
+
+The anchor stores each path's content id, size and mtime. After a bisync with no failures and no
+unresolved conflicts, `execute` writes a new anchor built from `a`. A file changed on both sides
+is always a conflict: bisync never line-merges file contents. Pass `resolve` to settle conflicts
+programmatically.
+
+### What will surprise you
+
+- **First bisync without an anchor only adds.** With no stored anchor there is no common base, so
+  bisync copies files missing on either side, deletes nothing, reports files that differ as
+  conflicts, and warns: `no sync anchor; performing conservative union (additions only, no
+  deletions)`. Without an `anchorStore`, every bisync runs this way.
+- **Equal size and mtime means unchanged.** A file edited without changing its size or
+  modification time is not detected (the same rule rclone uses by default).
+- **Verification modes collapse to two checks.** `size` and `mtime` both compare byte counts
+  (modification times are not comparable across endpoints); `content-hash`, `backend-hash` and
+  `read-after-write` all hash both files with `hashContent`.
+- **A failed verification is an event, not an exception.** Look for
+  `{ type: "failed", reason: "destination verification failed" }`. In a `move`, an unverified
+  copy keeps the source: `{ type: "skipped", reason: "destination not verified; source retained" }`.
+- **`check` changes nothing**, even if you pass its plan to `execute`.
+- The default transfer handles only copies: `streaming transfer only handles copy/update, got <kind>`.
 
-- **`plan(a, b, op, opts): Promise<SyncPlan>`** — pure planner. `op` is `"copy" | "sync" | "bisync" | "move" | "check"`. `copy` never deletes destination files; `sync` deletes extraneous ones; `move` is copy+verify+delete; `check` compares only; `bisync` runs the 3-way merge.
-- **`execute(plan, a, b, opts): AsyncIterable<SyncEvent>`** — applies a plan, verifying each action and skipping already-completed indices from `opts.checkpoint`.
-- **`buildAnchor(...)`** — build a `SyncAnchor` (bisync base) from an endpoint.
-- **`snapshot(files, filter?)` / `isChanged(...)`** — the change-detection ladder primitives.
-- **`createStreamingTransfer(): Transfer`** — the default whole-file streaming copy executor.
+### Dependencies
 
-`SyncOptions` carries the required `hashContent`, plus optional `filter`, `transfer`, `verify` (`VerificationMode`), `anchorStore` / `pairKey` (bisync), `resolve`, `quickFingerprint`, and `checkpoint`. Types: `SyncPlan`, `SyncAction`, `SyncConflict`, `SyncEvent`, `SyncAnchor`, `AnchorStore`, `CheckpointStore`, `Transfer`.
+`@statewalker/webrun-merge` (runtime: bisync) and `@statewalker/webrun-files` (the `FilesApi`
+type). The package does not depend on any VCS code.
 
-## Notes
+## License
 
-- **Plan-then-execute + resumable.** Plans are serializable and auditable; `execute` records each completed action index, so an interrupted run resumes with no duplicate work.
-- **Cheap-first change detection.** A short-circuiting ladder (path/type → size → stable mtime → optional quick fingerprint → full `hashContent`) avoids hashing byte-identical files.
-- **Injected `Transfer` seam.** The default streams whole files; a chunk-dedup transfer (from `@statewalker/webrun-content-transfer`) plugs into the *same* seam with no rewrite.
-- **bisync = 3-way over an anchor** via `@statewalker/webrun-merge`, using the same injected content id — not two one-way syncs.
-- Built red/green TDD.
+MIT

package/package.json

@@ -1,11 +1,15 @@
 {
   "name": "@statewalker/webrun-files-sync",
-  "version": "0.2.2",
+  "version": "0.3.0",
   "description": "rclone-like file-synchronisation engine (copy/sync/bisync/move/check) over the webrun-files FilesApi",
+  "bugs": {
+    "url": "https://github.com/statewalker/webrun-sync/issues"
+  },
+  "homepage": "https://github.com/statewalker/webrun-sync/tree/main/packages/webrun-files-sync#readme",
   "license": "MIT",
   "repository": {
     "type": "git",
-    "url": "https://github.com/statewalker/webrun-vcs.git",
+    "url": "git+https://github.com/statewalker/webrun-sync.git",
     "directory": "packages/webrun-files-sync"
   },
   "private": false,
@@ -27,11 +31,11 @@
     "src"
   ],
   "dependencies": {
-    "@statewalker/webrun-files": "^0.10.0",
-    "@statewalker/webrun-merge": "^0.2.1"
+    "@statewalker/webrun-files": "^0.12.0",
+    "@statewalker/webrun-merge": "^0.3.0"
   },
   "devDependencies": {
-    "@statewalker/webrun-files-mem": "^0.10.0",
+    "@statewalker/webrun-files-mem": "^0.12.0",
     "rolldown": "^1.2.12",
     "vitest": "^5.0.3",
     "typescript": "^7.0.2",
@@ -46,6 +50,7 @@
     "format": "biome format --write .",
     "format:check": "biome format .",
     "format:fix": "biome check --write --unsafe .",
-    "typecheck": "tsc --noEmit"
+    "typecheck": "tsc --noEmit",
+    "typecheck:tests": "tsc -p tsconfig.tests.json"
   }
 }