@fizzyflow/wdoublesync_cli 1.0.2

package/README.md

@@ -0,0 +1,150 @@
+# wdoublesync cli
+
+CLI tool to sync local folders to [Walrus](https://walrus.xyz) decentralised storage on the Sui network.
+
+Built on top of the [`wdoublesync`](https://github.com/fizzyFlow/wdoublesync) library.
+
+Each `push` stores a versioned, gzip-compressed snapshot (or diff) inside an [EndlessVector](https://github.com/fizzyFlow/endless_vector) on-chain object. Any past version can be restored at any time with `pull`. Folders can optionally be encrypted with [Seal](https://github.com/MystenLabs/seal).
+
+
+## Installation
+
+```bash
+pnpm install
+# make the binary globally available (optional)
+pnpm link --global
+```
+
+## Usage
+
+```
+wdoublesync push [vector-id] [options]   Sync current folder to a vector (creates one if no id given)
+wdoublesync pull <vector-id> [options]   Restore vector contents to the current folder
+wdoublesync info <vector-id> [options]   Show vector metadata and local sync status
+wdoublesync watch <vector-id> [options]  Watch folder and auto-push on changes, auto-pull on remote updates
+```
+
+### Options
+
+| Flag | Description |
+|---|---|
+| `--chain <name>` | Chain: `mainnet`, `testnet`, `devnet`, `localnet` (default: `testnet`) |
+| `--key <suiprivkey>` | Sui private key (or set `WDOUBLESYNC_KEY` env var) |
+| `--phrase <mnemonic>` | Mnemonic phrase instead of a raw key |
+| `--version <n>` | Version to restore (`pull` only, default: latest) |
+| `--exclude <p1,p2>` | Extra exclude patterns (comma-separated) |
+| `--no-compress` | Disable gzip compression |
+| `--manifest` | Write/use `.wdoublesync` manifest for faster change detection |
+| `--force-snapshot` | Push a full snapshot regardless of prior history (repairs a corrupt vector) |
+| `--poll-interval <s>` | `watch`: seconds between remote version checks (default: `2`) |
+| `--debounce <ms>` | `watch`: quiet period in ms before pushing after a local change (default: `1000`) |
+| `--push-only` | `watch`: disable auto-pull |
+| `--pull-only` | `watch`: disable auto-push |
+| `--help` | Show help |
+
+### Authentication
+
+Supply a signing key in one of three ways (checked in order):
+
+1. `--key suiprivkey1...`
+2. `--phrase "word1 word2 ..."`
+3. `WDOUBLESYNC_KEY` environment variable
+
+`pull` and `info` work without a key for public (unencrypted) vectors. A key is required for Seal-encrypted vectors and for any `push`.
+
+## Examples
+
+### Push current folder (first time)
+
+```bash
+cd my-project
+wdoublesync push --chain testnet --key suiprivkey1...
+# prints the new vector id, e.g.:
+#   created: 0xabc123...
+#   version 1 pushed (full snapshot, gzip compressed)
+```
+
+### Push an update
+
+```bash
+wdoublesync push 0xabc123... --chain testnet --key suiprivkey1...
+# version 2 pushed (diff, gzip compressed)
+```
+
+### Pull the latest version into the current folder
+
+```bash
+mkdir restored && cd restored
+wdoublesync pull 0xabc123... --chain testnet
+```
+
+### Pull a specific version
+
+```bash
+wdoublesync pull 0xabc123... --chain testnet --version 1
+```
+
+### Inspect a vector without touching the local folder
+
+```bash
+wdoublesync info 0xabc123... --chain testnet
+```
+
+### Fast incremental pushes with a manifest
+
+```bash
+wdoublesync push 0xabc123... --chain testnet --key suiprivkey1... --manifest
+# subsequent pushes are skipped when nothing has changed locally
+```
+
+### Watch a folder (auto push + pull)
+
+```bash
+wdoublesync watch 0xabc123... --chain testnet --key suiprivkey1...
+# pushes local changes 1 s after the last edit
+# pulls remote updates every 2 s
+# Ctrl-C to stop
+```
+
+Tune the timing:
+
+```bash
+wdoublesync watch 0xabc123... --debounce 3000 --poll-interval 5
+```
+
+Watch in push-only or pull-only mode:
+
+```bash
+wdoublesync watch 0xabc123... --push-only   # no auto-pull
+wdoublesync watch 0xabc123... --pull-only   # no auto-push (read-only mirror)
+```
+
+### Repair a corrupt vector with a force snapshot
+
+If a diff patch was pushed against a stale base (e.g. a race condition in `watch`), subsequent pulls will fail. Fix it by pushing a new full snapshot:
+
+```bash
+wdoublesync push 0xabc123... --chain testnet --key suiprivkey1... --force-snapshot
+# skips chain replay and pushes the current folder as a self-contained full snapshot
+# restore() will recover from this snapshot, skipping any corrupt diffs before it
+```
+
+## How it works
+
+1. **push** — scans the current directory, computes a tree hash, and compares it against the last stored version. If changes are detected, a compressed diff (or full snapshot on the first push) is uploaded to Walrus and appended to the EndlessVector on-chain.
+2. **pull** — reads the requested version from the EndlessVector, decrypts it if Seal-encrypted, and writes only changed files to disk. Files absent from the stored version are deleted.
+3. **info** — reads EndlessVector metadata from the chain (version count, binary size, history) and checks whether the local folder matches any stored version.
+4. **watch** — combines push and pull in a loop. A filesystem watcher triggers a debounced push on local changes. A poll interval checks the remote vector for new versions and pulls them if found. Push and pull never run concurrently.
+
+### Default excludes
+
+The following are always excluded from snapshots: `node_modules`, `.git`, `.env`, `.DS_Store`, `.wdoublesync`, `pnpm-lock.yaml`, `package-lock.json`. Add more with `--exclude`.
+
+## Dependencies
+
+| Package | Role |
+|---|---|
+| `@fizzyflow/wdoublesync` | Folder-diff / snapshot layer |
+| `@fizzyflow/doublesync` | Core CDC store and snapshot primitives |
+| `@fizzyflow/endless-vector` | On-chain EndlessVector (Sui + Walrus + Seal) |
+| `suidouble` | Sui client / key management |

package/bin/wdoublesync.js

@@ -0,0 +1,121 @@
+#!/usr/bin/env node
+
+import { push, pull, info, watch, rebate } from '../lib/commands.js';
+
+const USAGE = `wdoublesync — sync local folders to EndlessVector on Sui+Walrus
+
+Usage:
+  wdoublesync push [vector-id] [options]   Sync current folder to vector (creates if no id)
+  wdoublesync pull <vector-id> [options]   Restore vector contents to current folder
+  wdoublesync info <vector-id> [options]   Show vector metadata
+  wdoublesync watch <vector-id> [options]  Watch folder and auto-push/pull
+  wdoublesync rebate <vector-id> [options] Archive history, burn archives, push fresh snapshot
+
+Options:
+  --chain <name>            Chain: mainnet, testnet, devnet, localnet (default: testnet)
+  --key <suiprivkey>        Sui private key (or set WDOUBLESYNC_KEY env var)
+  --phrase <mnemonic>       Mnemonic phrase
+  --version <n>             Version to restore (pull only, default: latest)
+  --exclude <p1,p2>         Extra exclude patterns (comma-separated)
+  --no-compress             Disable gzip compression
+  --manifest                Write/use .wdoublesync manifest for faster change detection
+  --force-snapshot          Push a full snapshot regardless of prior history (repairs corrupt vectors)
+  --poll-interval <s>       Watch: seconds between remote checks (default: 2)
+  --debounce <ms>           Watch: ms quiet period before pushing after a change (default: 1000)
+  --push-only               Watch: disable auto-pull
+  --pull-only               Watch: disable auto-push
+  --help                    Show this help
+`;
+
+function parseArgs(argv) {
+    const args = {
+        command: null,
+        vectorId: null,
+        chain: 'testnet',
+        key: null,
+        phrase: null,
+        version: null,
+        exclude: null,
+        compress: 'gzip',
+        manifest: false,
+        forceSnapshot: false,
+        pollInterval: 2,
+        debounce: 1000,
+        pushOnly: false,
+        pullOnly: false,
+    };
+
+    const raw = argv.slice(2);
+
+    if (raw.length === 0 || raw.includes('--help')) {
+        console.log(USAGE);
+        process.exit(0);
+    }
+
+    args.command = raw[0];
+
+    let i = 1;
+
+    // Next positional arg is vector-id if it looks like one (starts with 0x)
+    if (i < raw.length && !raw[i].startsWith('--')) {
+        args.vectorId = raw[i];
+        i++;
+    }
+
+    while (i < raw.length) {
+        const flag = raw[i];
+        if (flag === '--chain' && i + 1 < raw.length) {
+            args.chain = raw[++i];
+        } else if (flag === '--key' && i + 1 < raw.length) {
+            args.key = raw[++i];
+        } else if (flag === '--phrase' && i + 1 < raw.length) {
+            args.phrase = raw[++i];
+        } else if (flag === '--version' && i + 1 < raw.length) {
+            args.version = raw[++i];
+        } else if (flag === '--exclude' && i + 1 < raw.length) {
+            args.exclude = raw[++i];
+        } else if (flag === '--no-compress') {
+            args.compress = false;
+        } else if (flag === '--manifest') {
+            args.manifest = true;
+        } else if (flag === '--force-snapshot') {
+            args.forceSnapshot = true;
+        } else if (flag === '--poll-interval' && i + 1 < raw.length) {
+            args.pollInterval = Number(raw[++i]);
+        } else if (flag === '--debounce' && i + 1 < raw.length) {
+            args.debounce = Number(raw[++i]);
+        } else if (flag === '--push-only') {
+            args.pushOnly = true;
+        } else if (flag === '--pull-only') {
+            args.pullOnly = true;
+        }
+        i++;
+    }
+
+    return args;
+}
+
+const args = parseArgs(process.argv);
+
+try {
+    if (args.command === 'push') {
+        await push(args);
+    } else if (args.command === 'pull') {
+        await pull(args);
+    } else if (args.command === 'info') {
+        await info(args);
+    } else if (args.command === 'watch') {
+        await watch(args);
+    } else if (args.command === 'rebate') {
+        await rebate(args);
+    } else {
+        console.error('Unknown command:', args.command);
+        console.log(USAGE);
+        process.exit(1);
+    }
+    process.exit(0);
+} catch (err) {
+    console.error('Error:', err.message);
+    if (err.stack) console.error(err.stack);
+    process.exit(1);
+}

package/lib/DoubleSyncFSFile.js

@@ -0,0 +1,23 @@
+import { readFile, stat } from 'node:fs/promises';
+import { basename } from 'node:path';
+import { DoubleSyncFile } from '@fizzyflow/doublesync';
+
+export class DoubleSyncFSFile extends DoubleSyncFile {
+    constructor(filePath) {
+        super();
+        this._path = filePath;
+        this._name = basename(filePath);
+    }
+
+    get name() { return this._name; }
+
+    async getContent() {
+        const buf = await readFile(this._path);
+        return new Uint8Array(buf.buffer, buf.byteOffset, buf.byteLength);
+    }
+
+    async getSize() {
+        const s = await stat(this._path);
+        return s.size;
+    }
+}

package/lib/DoubleSyncFSFolder.js

@@ -0,0 +1,37 @@
+import { readdir, stat } from 'node:fs/promises';
+import { join, basename } from 'node:path';
+import { DoubleSyncFolder } from '@fizzyflow/doublesync';
+import { DoubleSyncFSFile } from './DoubleSyncFSFile.js';
+
+const DEFAULT_EXCLUDES = ['node_modules', '.git', '.env', '.DS_Store', '.wdoublesync', 'pnpm-lock.yaml', 'package-lock.json'];
+
+export class DoubleSyncFSFolder extends DoubleSyncFolder {
+    constructor(dirPath, excludes = DEFAULT_EXCLUDES) {
+        super();
+        this._path = dirPath;
+        this._name = basename(dirPath);
+        this._excludes = excludes;
+    }
+
+    get name() { return this._name; }
+
+    async list() {
+        const entries = await readdir(this._path, { withFileTypes: true });
+        const children = [];
+
+        for (const entry of entries) {
+            if (this._excludes.includes(entry.name)) continue;
+            if (entry.name.startsWith('.')) continue;
+
+            const fullPath = join(this._path, entry.name);
+
+            if (entry.isDirectory()) {
+                children.push(new DoubleSyncFSFolder(fullPath, this._excludes));
+            } else if (entry.isFile()) {
+                children.push(new DoubleSyncFSFile(fullPath));
+            }
+        }
+
+        return children;
+    }
+}