@statewalker/webrun-files-sync 0.2.2 → 0.2.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +113 -25
  2. package/package.json +7 -3
package/README.md CHANGED
@@ -1,20 +1,50 @@
1
1
  # @statewalker/webrun-files-sync
2
2
 
3
- rclone-like file-synchronisation engine (copy / sync / bisync / move / check) over two `FilesApi` endpoints.
3
+ ## What it is
4
4
 
5
- ## Overview
5
+ An rclone-like synchronisation engine for two `FilesApi` endpoints from
6
+ `@statewalker/webrun-files`. It supports five operations (`copy`, `sync`, `bisync`, `move`,
7
+ `check`) and splits each into a pure planning step and an execution step that verifies and can
8
+ resume.
6
9
 
7
- `files-sync` is Axis A of the two-axis VCS architecture — the git-independent "sync" engine. It **plans, then executes**: `plan(a, b, op, opts)` is pure (mutates neither endpoint) and returns a serializable `SyncPlan`; `execute(plan, a, b, opts)` applies it, emitting events and resuming from an optional checkpoint after interruption. Bidirectional `bisync` is a genuine **three-way merge over a sync-owned `SyncAnchor`** (delegated to `@statewalker/webrun-merge`), never two chained one-way syncs.
10
+ ## Why it exists
8
11
 
9
- It imports **only** `@statewalker/webrun-files` and `@statewalker/webrun-merge` and knows nothing of git or versioning — that is the hard Axis A ✗↔ Axis B ban. Content identity (`hashContent`), the per-copy `Transfer` strategy, and conflict resolution are all injected.
12
+ Keeping two file trees in step (a browser store and a server, a local folder and a cloud bucket)
13
+ is a different job from versioning: it needs cheap change detection, safe deletes and two-way
14
+ reconciliation, not commits. Because every storage backend in the ecosystem is a `FilesApi`, one
15
+ engine covers all pairs of backends. Two-way sync is a real three-way merge against the state
16
+ recorded after the previous sync, done by `@statewalker/webrun-merge`, so a delete on one side is
17
+ propagated instead of being undone by a copy from the other side.
10
18
 
11
- ## Installation
19
+ ## How to use
12
20
 
13
21
  ```bash
14
22
  pnpm add @statewalker/webrun-files-sync
15
23
  ```
16
24
 
17
- ## Quick Start
25
+ | Import | Contents |
26
+ | --- | --- |
27
+ | `@statewalker/webrun-files-sync` | `plan`, `execute`, `buildAnchor`, `snapshot`, `isChanged`, `createStreamingTransfer`, and the types `SyncOptions`, `SyncPlan`, `SyncAction`, `SyncEvent`, `SyncOp`, `SyncConflict`, `SyncAnchor`, `AnchorStore`, `CheckpointStore`, `Transfer`, `PathFilter`, `VerificationMode`, `Resolution`, `FilesApi`, `ByteStream`. ESM, runs in browsers, Node and workers. |
28
+
29
+ | Operation | Effect on `b` (and `a`) |
30
+ | --- | --- |
31
+ | `copy` | New and changed files from `a` are written to `b`. Nothing is deleted. |
32
+ | `sync` | Like `copy`, and files in `b` that are not in `a` are deleted. |
33
+ | `move` | Like `copy`, then each source file in `a` is deleted once its copy verifies. |
34
+ | `check` | Compares only. `execute` reports warnings and conflicts and changes nothing. |
35
+ | `bisync` | Changes from both sides are applied to both sides; conflicts are reported. |
36
+
37
+ `plan(a, b, op, opts)` returns a JSON-serializable `SyncPlan` (`actions`, `conflicts`,
38
+ `warnings`). `execute(plan, a, b, opts)` yields `warning`, `conflict`, `start`, `done`, `skipped`
39
+ and `failed` events.
40
+
41
+ `SyncOptions`: `hashContent` (required); `filter?(path)`; `transfer?` (default
42
+ `createStreamingTransfer()`); `verify?` (default `"size"`); `anchorStore?` and `pairKey?`
43
+ (default `"default"`) for bisync; `resolve?(conflict)`; `quickFingerprint?`; `checkpoint?`.
44
+
45
+ ## Examples
46
+
47
+ One-way copy:
18
48
 
19
49
  ```typescript
20
50
  import { createHash } from "node:crypto";
@@ -27,33 +57,91 @@ async function sha256(input: AsyncIterable<Uint8Array>): Promise<string> {
27
57
  return `sha256:${h.digest("hex")}`;
28
58
  }
29
59
 
30
- const a = new MemFilesApi({ initialFiles: { "/new.txt": "new", "/changed.txt": "v2" } });
60
+ const a = new MemFilesApi({ initialFiles: { "/new.txt": "new", "/changed.txt": "version 2" } });
31
61
  const b = new MemFilesApi({ initialFiles: { "/changed.txt": "v1", "/extra.txt": "keep" } });
32
62
 
33
- // Pure plan — dry-runnable, serializable, mutates nothing.
34
63
  const p = await plan(a, b, "copy", { hashContent: sha256 });
35
-
36
- // Execute, streaming per-action events.
37
64
  for await (const event of execute(p, a, b, { hashContent: sha256 })) {
38
- if (event.type === "done") console.log("synced", event.action);
65
+ if (event.type === "done") console.log(event.action);
66
+ }
67
+ // b: new.txt, changed.txt ("version 2"), extra.txt (kept)
68
+ // Had both versions of changed.txt the same size and mtime, it would count as unchanged.
69
+ ```
70
+
71
+ Two-way sync with a stored anchor:
72
+
73
+ ```typescript
74
+ import type { AnchorStore, SyncAnchor } from "@statewalker/webrun-files-sync";
75
+
76
+ const anchors = new Map<string, SyncAnchor>();
77
+ const anchorStore: AnchorStore = {
78
+ read: async (key) => anchors.get(key),
79
+ write: async (key, anchor) => void anchors.set(key, anchor),
80
+ };
81
+ const opts = { hashContent: sha256, anchorStore, pairKey: "laptop<->server" };
82
+
83
+ for await (const e of execute(await plan(a, b, "bisync", opts), a, b, opts)) {
84
+ if (e.type === "conflict") console.log("conflict", e.conflict.kind, e.conflict.path);
39
85
  }
40
- // b now has new.txt + updated changed.txt; extra.txt is kept (copy never deletes).
41
86
  ```
42
87
 
43
- ## API
88
+ Resumable execution: pass a `checkpoint` (`{ read(): Promise<Set<number>>, add(index) }`) backed by
89
+ durable storage; a second `execute` with the same plan skips the recorded action indices.
90
+
91
+ ## Internals
92
+
93
+ ### Plan first, then execute
94
+
95
+ `plan` reads both endpoints and writes nothing, so a plan can be shown to a user, stored, or
96
+ tested. `execute` applies the actions in order, verifies each copy, records each finished index in
97
+ the checkpoint, and keeps going after a failed action (it emits `failed` and continues).
98
+
99
+ ### Change detection stops at the first decisive check
100
+
101
+ ```
102
+ path/kind ─> size ─> equal mtime ─> head+tail sample ─> full hashContent
103
+ differ? differ? equal? (quickFingerprint) differ?
104
+ changed changed unchanged differ? changed changed
105
+ ```
106
+
107
+ Most files are decided by size or modification time without reading their bytes. With
108
+ `quickFingerprint`, the first and last 64 bytes are compared before hashing the whole file.
109
+
110
+ ### How bisync works
111
+
112
+ ```
113
+ anchor (state after last sync) ─┐
114
+ endpoint a ─────────────────────┼─> webrun-merge ─> actions for both sides + conflicts
115
+ endpoint b ─────────────────────┘
116
+ ```
117
+
118
+ The anchor stores each path's content id, size and mtime. After a bisync with no failures and no
119
+ unresolved conflicts, `execute` writes a new anchor built from `a`. A file changed on both sides
120
+ is always a conflict: bisync never line-merges file contents. Pass `resolve` to settle conflicts
121
+ programmatically.
122
+
123
+ ### What will surprise you
124
+
125
+ - **First bisync without an anchor only adds.** With no stored anchor there is no common base, so
126
+ bisync copies files missing on either side, deletes nothing, reports files that differ as
127
+ conflicts, and warns: `no sync anchor; performing conservative union (additions only, no
128
+ deletions)`. Without an `anchorStore`, every bisync runs this way.
129
+ - **Equal size and mtime means unchanged.** A file edited without changing its size or
130
+ modification time is not detected (the same rule rclone uses by default).
131
+ - **Verification modes collapse to two checks.** `size` and `mtime` both compare byte counts
132
+ (modification times are not comparable across endpoints); `content-hash`, `backend-hash` and
133
+ `read-after-write` all hash both files with `hashContent`.
134
+ - **A failed verification is an event, not an exception.** Look for
135
+ `{ type: "failed", reason: "destination verification failed" }`. In a `move`, an unverified
136
+ copy keeps the source: `{ type: "skipped", reason: "destination not verified; source retained" }`.
137
+ - **`check` changes nothing**, even if you pass its plan to `execute`.
138
+ - The default transfer handles only copies: `streaming transfer only handles copy/update, got <kind>`.
44
139
 
45
- - **`plan(a, b, op, opts): Promise<SyncPlan>`** — pure planner. `op` is `"copy" | "sync" | "bisync" | "move" | "check"`. `copy` never deletes destination files; `sync` deletes extraneous ones; `move` is copy+verify+delete; `check` compares only; `bisync` runs the 3-way merge.
46
- - **`execute(plan, a, b, opts): AsyncIterable<SyncEvent>`** — applies a plan, verifying each action and skipping already-completed indices from `opts.checkpoint`.
47
- - **`buildAnchor(...)`** — build a `SyncAnchor` (bisync base) from an endpoint.
48
- - **`snapshot(files, filter?)` / `isChanged(...)`** — the change-detection ladder primitives.
49
- - **`createStreamingTransfer(): Transfer`** — the default whole-file streaming copy executor.
140
+ ### Dependencies
50
141
 
51
- `SyncOptions` carries the required `hashContent`, plus optional `filter`, `transfer`, `verify` (`VerificationMode`), `anchorStore` / `pairKey` (bisync), `resolve`, `quickFingerprint`, and `checkpoint`. Types: `SyncPlan`, `SyncAction`, `SyncConflict`, `SyncEvent`, `SyncAnchor`, `AnchorStore`, `CheckpointStore`, `Transfer`.
142
+ `@statewalker/webrun-merge` (runtime: bisync) and `@statewalker/webrun-files` (the `FilesApi`
143
+ type). The package does not depend on any VCS code.
52
144
 
53
- ## Notes
145
+ ## License
54
146
 
55
- - **Plan-then-execute + resumable.** Plans are serializable and auditable; `execute` records each completed action index, so an interrupted run resumes with no duplicate work.
56
- - **Cheap-first change detection.** A short-circuiting ladder (path/type → size → stable mtime → optional quick fingerprint → full `hashContent`) avoids hashing byte-identical files.
57
- - **Injected `Transfer` seam.** The default streams whole files; a chunk-dedup transfer (from `@statewalker/webrun-content-transfer`) plugs into the *same* seam with no rewrite.
58
- - **bisync = 3-way over an anchor** via `@statewalker/webrun-merge`, using the same injected content id — not two one-way syncs.
59
- - Built red/green TDD.
147
+ MIT
package/package.json CHANGED
@@ -1,11 +1,15 @@
1
1
  {
2
2
  "name": "@statewalker/webrun-files-sync",
3
- "version": "0.2.2",
3
+ "version": "0.2.3",
4
4
  "description": "rclone-like file-synchronisation engine (copy/sync/bisync/move/check) over the webrun-files FilesApi",
5
+ "bugs": {
6
+ "url": "https://github.com/statewalker/webrun-sync/issues"
7
+ },
8
+ "homepage": "https://github.com/statewalker/webrun-sync/tree/main/packages/webrun-files-sync#readme",
5
9
  "license": "MIT",
6
10
  "repository": {
7
11
  "type": "git",
8
- "url": "https://github.com/statewalker/webrun-vcs.git",
12
+ "url": "git+https://github.com/statewalker/webrun-sync.git",
9
13
  "directory": "packages/webrun-files-sync"
10
14
  },
11
15
  "private": false,
@@ -28,7 +32,7 @@
28
32
  ],
29
33
  "dependencies": {
30
34
  "@statewalker/webrun-files": "^0.10.0",
31
- "@statewalker/webrun-merge": "^0.2.1"
35
+ "@statewalker/webrun-merge": "^0.2.2"
32
36
  },
33
37
  "devDependencies": {
34
38
  "@statewalker/webrun-files-mem": "^0.10.0",