@fizzyflow/wdoublesync_cli 1.0.2 → 1.0.5

package/README.md

@@ -1,11 +1,14 @@
 # wdoublesync cli
 
-CLI tool to sync local folders to [Walrus](https://walrus.xyz) decentralised storage on the Sui network.
+CLI tool to sync local folders to Sui and [Walrus](https://walrus.xyz) decentralised storage on the Sui network.
 
 Built on top of the [`wdoublesync`](https://github.com/fizzyFlow/wdoublesync) library.
 
-Each `push` stores a versioned, gzip-compressed snapshot (or diff) inside an [EndlessVector](https://github.com/fizzyFlow/endless_vector) on-chain object. Any past version can be restored at any time with `pull`. Folders can optionally be encrypted with [Seal](https://github.com/MystenLabs/seal).
+Each `push` stores a versioned, gzip-compressed diff or shapshot inside an [EndlessVector](https://github.com/fizzyFlow/endless_vector) on-chain object. Any past version can be restored at any time with `pull`. Folders can optionally be encrypted with [Seal](https://github.com/MystenLabs/seal).
 
+## AI Agent Skill
+
+wdoublesync ships with a [SKILL.md](https://github.com/FizzyFlow/wdoublesync_cli/blob/main/SKILL.md) that teaches AI coding agents how to use it autonomously. Once loaded, an agent can push folders to on-chain storage, pull specific versions for inspection or rollback, watch a directory for bi-directional sync, and run rebate to compact a vector's archive — all without hand-holding on the commands. This is useful when you want an agent to persist its own working state, archive project snapshots, or coordinate versioned data across sessions. The skill also covers how to combine wdoublesync with [MemWal](https://github.com/MystenLabs/MemWal) for workflows.
 
 ## Installation
 
@@ -18,12 +21,15 @@ pnpm link --global
 ## Usage
 
 ```
-wdoublesync push [vector-id] [options]   Sync current folder to a vector (creates one if no id given)
-wdoublesync pull <vector-id> [options]   Restore vector contents to the current folder
-wdoublesync info <vector-id> [options]   Show vector metadata and local sync status
-wdoublesync watch <vector-id> [options]  Watch folder and auto-push on changes, auto-pull on remote updates
+wdoublesync push [vector-id] [path] [options]   Sync folder to a vector (creates one if no id given)
+wdoublesync pull <vector-id> [path] [options]   Restore vector contents to a folder
+wdoublesync info [vector-id] [path] [options]   Show chain/wallet info, or full vector metadata
+wdoublesync watch <vector-id> [path] [options]  Watch folder: auto-push on changes, auto-pull on remote updates
+wdoublesync rebate <vector-id> [path] [options] Burn archive patches and push folder as a fresh snapshot
 ```
 
+The `[path]` argument is optional — if omitted, the current working directory is used.
+
 ### Options
 
 | Flag | Description |
@@ -54,11 +60,15 @@ Supply a signing key in one of three ways (checked in order):
 
 ## Examples
 
-### Push current folder (first time)
+### Push a folder (first time)
 
 ```bash
-cd my-project
+# from current directory
 wdoublesync push --chain testnet --key suiprivkey1...
+
+# or pass path explicitly
+wdoublesync push ~/my-project --chain testnet --key suiprivkey1...
+
 # prints the new vector id, e.g.:
 #   created: 0xabc123...
 #   version 1 pushed (full snapshot, gzip compressed)
@@ -68,26 +78,32 @@ wdoublesync push --chain testnet --key suiprivkey1...
 
 ```bash
 wdoublesync push 0xabc123... --chain testnet --key suiprivkey1...
-# version 2 pushed (diff, gzip compressed)
+
+# or with explicit path
+wdoublesync push 0xabc123... ~/my-project --chain testnet --key suiprivkey1...
 ```
 
-### Pull the latest version into the current folder
+### Pull the latest version
 
 ```bash
-mkdir restored && cd restored
-wdoublesync pull 0xabc123... --chain testnet
+wdoublesync pull 0xabc123... ~/restored --chain testnet
 ```
 
 ### Pull a specific version
 
 ```bash
-wdoublesync pull 0xabc123... --chain testnet --version 1
+wdoublesync pull 0xabc123... ~/restored --chain testnet --version 1
 ```
 
-### Inspect a vector without touching the local folder
+### Inspect a vector
 
 ```bash
+# check your wallet + chain (no vector ID needed)
+wdoublesync info
+
+# full vector metadata + local sync status
 wdoublesync info 0xabc123... --chain testnet
+wdoublesync info 0xabc123... ~/my-project --chain testnet
 ```
 
 ### Fast incremental pushes with a manifest
@@ -101,6 +117,9 @@ wdoublesync push 0xabc123... --chain testnet --key suiprivkey1... --manifest
 
 ```bash
 wdoublesync watch 0xabc123... --chain testnet --key suiprivkey1...
+
+# or with explicit path
+wdoublesync watch 0xabc123... ~/my-project --chain testnet --key suiprivkey1...
 # pushes local changes 1 s after the last edit
 # pulls remote updates every 2 s
 # Ctrl-C to stop
@@ -119,6 +138,17 @@ wdoublesync watch 0xabc123... --push-only   # no auto-pull
 wdoublesync watch 0xabc123... --pull-only   # no auto-push (read-only mirror)
 ```
 
+### Rebate (compact archive)
+
+After many incremental syncs, burn old archive patches and push the current folder as a fresh single snapshot. Reduces storage cost.
+
+```bash
+wdoublesync rebate 0xabc123... --chain testnet --key suiprivkey1...
+
+# or with explicit path
+wdoublesync rebate 0xabc123... ~/my-project --chain testnet --key suiprivkey1...
+```
+
 ### Repair a corrupt vector with a force snapshot
 
 If a diff patch was pushed against a stale base (e.g. a race condition in `watch`), subsequent pulls will fail. Fix it by pushing a new full snapshot:
@@ -135,6 +165,7 @@ wdoublesync push 0xabc123... --chain testnet --key suiprivkey1... --force-snapsh
 2. **pull** — reads the requested version from the EndlessVector, decrypts it if Seal-encrypted, and writes only changed files to disk. Files absent from the stored version are deleted.
 3. **info** — reads EndlessVector metadata from the chain (version count, binary size, history) and checks whether the local folder matches any stored version.
 4. **watch** — combines push and pull in a loop. A filesystem watcher triggers a debounced push on local changes. A poll interval checks the remote vector for new versions and pulls them if found. Push and pull never run concurrently.
+5. **rebate** — burns all existing archive patches on-chain and pushes the current folder as a single fresh full snapshot, reducing Walrus storage costs after many incremental syncs.
 
 ### Default excludes
 

package/SKILL.md

@@ -0,0 +1,297 @@
+---
+name: wdoublesync
+version: 0.1.7
+description: Claude agent skill for decentralized file storage, versioning, and sync on Sui + Walrus + Seal. Enables agents to push local folders as on-chain vectors, pull versions, manage Walrus storage, and watch for changes.
+keywords: [walrus, sui, storage, versioning, sync, seal-encryption, blockchain, defi]
+---
+
+# wdoublesync
+
+Decentralized file storage, versioning, and sync on **Sui + Walrus + Seal**.
+
+An abstract virtual filesystem that synchronizes folder state on-chain via Sui + Walrus using Content-Defined Chunking (CDC) to compute minimal binary diffs. Files can optionally be encrypted with Seal so only authorized addresses can decrypt them.
+
+## When to Use
+
+- **Archive and restore folder snapshots** on-chain with full version history
+- **Efficient incremental sync** — only changed chunks uploaded, not entire files
+- **End-to-end encrypted storage** — data encrypted with Seal before Walrus upload
+- **Verify data integrity** — pull any version and reconstruct exact folder state
+- **Extend storage** before Walrus epochs expire without re-uploading data
+- **Clean archive** with rebate workflow (burn old archive, push as fresh snapshot)
+
+## When NOT to Use
+
+- For **semantic/vector search over text** — if you need to index and search text by meaning, use [MemWal](https://raw.githubusercontent.com/MystenLabs/MemWal/refs/heads/dev/SKILL.md) instead (vector database with embeddings)
+
+## MemWal + WDoubleSync
+
+Feel free to combine MemWal and WDoubleSync in your agent workflows. For example, you can use MemWal to store and search text embeddings, and WDoubleSync to store the original files or folders that correspond to those embeddings. This allows you to have both semantic search capabilities and versioned file storage in one system.
+
+## Installation
+
+Check if `wdoublesync` is already installed:
+
+```bash
+wdoublesync --help
+```
+
+if not, install via npm:
+
+**Global npm install**
+```bash
+npm install -g @fizzyflow/wdoublesync_cli
+# Now use: wdoublesync <command> (installed from npm)
+```
+
+After you have access to `wdoublesync`, you can run `wdoublesync --help` to see the available commands and options. Use `wdoublesync info` to check your currently connected wallet (`your wallet` field), if there's no - wallet, you can set one with `--key` or `--phrase` flags, or by exporting `WDOUBLESYNC_KEY` environment variable. Feel free to ask user to set up a wallet if they haven't done so yet, as it's required for push operations and pull encrypted vectors.
+
+### Set up your key (required for push and pull of encrypted vectors)
+
+```bash
+export WDOUBLESYNC_KEY=suiprivkey1...
+```
+
+For **read-only public vectors**, the key is not required.
+
+### Push a folder (creates new vector if no ID provided)
+
+Be sure to run this command from the folder you want to sync:
+
+```bash
+cd ~/my-folder
+wdoublesync push
+# Creates new vector + syncs current folder
+# Output: created: 0x1234... / syncing ./ → 0x1234...
+```
+
+This creates a new EndlessVector on-chain and pushes the current folder as a snapshot encrypted by Seal. The output will show the vector ID (e.g., `0x1234...`) which you can use for future pulls or pushes.
+
+### Pull a vector to restore it
+
+```bash
+mkdir ~/restored-folder && cd ~/restored-folder
+wdoublesync pull 0x1234...
+# Restores vector to current directory
+```
+
+### View vector metadata
+
+```bash
+wdoublesync info 0x1234...
+# Output: ID, owner, version count, total size, encryption status
+```
+
+### Watch for changes (bi-directional sync)
+
+```bash
+cd ~/my-data
+wdoublesync watch 0x1234... --poll-interval 5
+# Monitors local folder, auto-pushes changes
+# Checks remote every 5 seconds, auto-pulls updates
+```
+
+## API Surface
+
+All commands operate on the **current working directory** — `cd` into your folder first.
+
+| Command | Arguments | Use Case |
+|---------|-----------|----------|
+| **push** | `[vectorId]` `[options]` | Sync current folder to vector; creates new vector if no ID given |
+| **pull** | `vectorId` `[options]` | Restore vector (default: latest version) into current directory |
+| **info** | `[vectorId]` | No ID: show chain + wallet. With ID: full vector details |
+| **watch** | `vectorId` `[options]` | Bi-directional sync: auto-push local changes, auto-pull remote updates |
+| **rebate** | `vectorId` | Burn archive patches, push current folder as fresh single snapshot |
+
+### Command Examples
+
+#### push (create new vector)
+
+```bash
+cd ~/my-data
+wdoublesync push
+# Syncs current folder as new vector
+# Output: creating new EndlessVector on testnet...
+#         created: 0xabc123...
+#         syncing ./ → 0xabc123...
+```
+
+#### push (with manifest for faster change detection)
+
+```bash
+cd ~/my-data
+wdoublesync push 0x1234... --manifest
+# Writes .wdoublesync manifest; faster change detection on future pushes
+```
+
+#### push (full snapshot, not incremental)
+
+```bash
+cd ~/my-data
+wdoublesync push 0x1234... --force-snapshot
+# Uploads entire snapshot, useful if history is corrupted
+```
+
+#### pull (latest version)
+
+```bash
+mkdir ~/restored && cd ~/restored
+wdoublesync pull 0x1234...
+# Restores latest version to current directory
+```
+
+#### pull (specific version)
+
+```bash
+cd ~/restored
+wdoublesync pull 0x1234... --version 10
+# Restores version 10 instead of latest
+```
+
+#### watch (bi-directional)
+
+```bash
+cd ~/my-data
+wdoublesync watch 0x1234... --poll-interval 5
+# Monitors current folder, auto-pushes on change (debounce: 1000ms default)
+# Checks remote every 5 seconds, auto-pulls new versions
+# Runs indefinitely; stop with Ctrl+C
+```
+
+#### watch (push-only)
+
+```bash
+cd ~/my-data
+wdoublesync watch 0x1234... --push-only
+# Only auto-push on local changes, don't pull remote updates
+```
+
+#### watch (pull-only)
+
+```bash
+cd ~/my-data
+wdoublesync watch 0x1234... --pull-only
+# Only pull remote updates, don't auto-push local changes
+```
+
+#### rebate
+
+When user asks to rebate, you can explain that this command is useful after many incremental syncs to reclaim on-chain storage. It burns the archive history and pushes the current folder as a fresh single snapshot. Make sure user understands that after rebate, only the latest snapshot is available, and old patches will be burned.
+
+```bash
+cd ~/my-data
+wdoublesync rebate 0x1234...
+# Burns archive history, pushes current folder as fresh single snapshot
+# Useful after many incremental syncs to reclaim on-chain storage
+```
+
+
+## Configuration
+
+### Command-line flags
+
+| Flag | Arguments | Default | Use |
+|------|-----------|---------|-----|
+| `--chain` | `mainnet` \| `testnet` \| `devnet` \| `localnet` | `testnet` | Which Sui network to use |
+| `--key` | Sui private key (`suiprivkey1...`) | — | Sign transactions (or use env vars) |
+| `--phrase` | Mnemonic phrase | — | Alternative: derive key from mnemonic |
+| `--no-compress` | — | — | Push: disable gzip compression |
+| `--manifest` | — | — | Push: write/use `.wdoublesync` file for faster change detection |
+| `--force-snapshot` | — | — | Push: full snapshot (repairs corrupt vectors) |
+| `--version` | Number | `latest` | Pull: restore specific version instead of latest |
+| `--exclude` | Patterns (comma-separated) | — | Extra file patterns to exclude from sync |
+| `--poll-interval` | Seconds | `2` | Watch: check for remote updates every N seconds |
+| `--debounce` | Milliseconds | `1000` | Watch: quiet period before pushing after local change |
+| `--push-only` | — | — | Watch: disable auto-pull |
+| `--pull-only` | — | — | Watch: disable auto-push |
+
+### Environment variables
+
+| Variable | Type | Use |
+|----------|------|-----|
+| `WDOUBLESYNC_KEY` | Sui private key (`suiprivkey1...`) | Sign transactions (alternative to `--key` flag) |
+
+## Example Workflows
+
+### Workflow 1: Create and verify
+
+```bash
+# 1. Push a folder (creates new vector on first push)
+cd ~/important-data
+wdoublesync push
+# Output: creating new EndlessVector on testnet...
+#         created: 0xabc123...
+
+# 2. Verify by pulling to a fresh folder
+mkdir ~/verify-tmp && cd ~/verify-tmp
+wdoublesync pull 0xabc123...
+
+# 3. Check file counts match
+ls ~/important-data | wc -l
+ls ~/verify-tmp | wc -l
+```
+
+### Workflow 2: Multi-version history and rollback
+
+```bash
+cd ~/data
+# Push version 1
+wdoublesync push 0xabc123...
+
+# Make changes, push version 2
+echo "new file" > new.txt
+wdoublesync push 0xabc123...
+
+# List all versions
+wdoublesync info 0xabc123...
+
+# Rollback to version 1
+mkdir ~/data-v1 && cd ~/data-v1
+wdoublesync pull 0xabc123... --version 1
+# Folder now matches state from version 1
+```
+
+### Workflow 3: Rebate to compact archive
+
+After many incremental syncs, archive grows with lots of patches. Rebate burns old data and rebuilds as single snapshot:
+
+```bash
+# Before rebate: many small patches
+wdoublesync info 0xabc123...
+# Output: version 87, total size 45 MB (many patches)
+
+# Run rebate from current directory
+cd ~/data
+wdoublesync rebate 0xabc123...
+# Output: Burned 86 items, pushed version 88 (3.2 MB)
+
+# After rebate: single snapshot instead of 87 patches
+# Much more efficient for Walrus storage
+```
+
+### Workflow 4: Watch for external changes
+
+```bash
+# Another user pushes to the same vector
+# You want to auto-sync your local copy
+
+cd ~/data
+wdoublesync watch 0xabc123... --poll-interval 5
+# Monitors local folder for changes, auto-pushes them
+# Also checks remote every 5 seconds for updates, auto-pulls
+# Press Ctrl+C to stop
+```
+
+## Troubleshooting
+
+| Issue | Cause | Solution |
+|-------|-------|----------|
+| **"No signing key"** | Trying to push/pull without credentials | Export `WDOUBLESYNC_KEY=suiprivkey1...` or use `--key` / `--phrase` flag |
+| **"vector-id is required"** | Command needs a vector ID but none provided | Pass the vector ID as the first argument: `wdoublesync pull 0x...` |
+| **"This vector is Seal-encrypted"** | Vector is encrypted but no key provided | Provide `--key` or `--phrase`, or export `WDOUBLESYNC_KEY` |
+| **"at() is out of range, this part of archive has been burned"** | Trying to restore a version whose archive was burned by rebate | Only the latest snapshot is available after rebate; pull without `--version` |
+| **Connection errors** | RPC endpoint unreachable or wrong chain | You may need to try again little later |
+
+## Links
+
+- **Repository**: https://github.com/FizzyFlow/wdoublesync_cli

package/bin/wdoublesync.js

@@ -1,6 +1,17 @@
 #!/usr/bin/env node
 
-import { push, pull, info, watch, rebate } from '../lib/commands.js';
+// Intercept process.exit() calls from dependencies so they don't interrupt
+// the top-level await before it settles. Converts them to thrown errors that
+// our try/catch handles, then sets process.exitCode without calling exit().
+const _realExit = process.exit.bind(process);
+process.exit = (code) => {
+    const err = new Error(`process.exit(${code ?? 0})`);
+    err._isProcessExit = true;
+    err._exitCode = code ?? 0;
+    throw err;
+};
+
+import { push, pull, info, watch, rebate } from '../lib/commands/index.js';
 
 const USAGE = `wdoublesync — sync local folders to EndlessVector on Sui+Walrus
 
@@ -31,6 +42,7 @@ function parseArgs(argv) {
     const args = {
         command: null,
         vectorId: null,
+        path: process.cwd(),
         chain: 'testnet',
         key: null,
         phrase: null,
@@ -49,18 +61,30 @@ function parseArgs(argv) {
 
     if (raw.length === 0 || raw.includes('--help')) {
         console.log(USAGE);
-        process.exit(0);
+        _realExit(0);
     }
 
     args.command = raw[0];
 
     let i = 1;
 
-    // Next positional arg is vector-id if it looks like one (starts with 0x)
-    if (i < raw.length && !raw[i].startsWith('--')) {
-        args.vectorId = raw[i];
+    // Collect up to two positional args: [vectorId] [path]
+    // vectorId starts with 0x, path starts with /, ~, or .
+    const positionals = [];
+    while (i < raw.length && !raw[i].startsWith('--')) {
+        positionals.push(raw[i]);
         i++;
     }
+    if (positionals.length === 2) {
+        args.vectorId = positionals[0];
+        args.path = positionals[1];
+    } else if (positionals.length === 1) {
+        if (positionals[0].startsWith('0x')) {
+            args.vectorId = positionals[0];
+        } else {
+            args.path = positionals[0];
+        }
+    }
 
     while (i < raw.length) {
         const flag = raw[i];
@@ -97,6 +121,12 @@ function parseArgs(argv) {
 
 const args = parseArgs(process.argv);
 
+// The Sui/Walrus SDKs use undici (Node.js fetch) which unref's its HTTP connections.
+// With only unref'd I/O in flight, the event loop drains and Node.js exits with code 13
+// ("Detected unsettled top-level await") before network responses arrive.
+// A ref'd interval keeps the event loop alive for the duration of the command.
+const keepAlive = setInterval(() => {}, 60_000);
+
 try {
     if (args.command === 'push') {
         await push(args);
@@ -111,11 +141,16 @@ try {
     } else {
         console.error('Unknown command:', args.command);
         console.log(USAGE);
-        process.exit(1);
+        process.exitCode = 1;
     }
-    process.exit(0);
 } catch (err) {
-    console.error('Error:', err.message);
-    if (err.stack) console.error(err.stack);
-    process.exit(1);
+    if (err._isProcessExit) {
+        process.exitCode = err._exitCode;
+    } else {
+        console.error('Error:', err.message);
+        if (err.stack) console.error(err.stack);
+        process.exitCode = 1;
+    }
+} finally {
+    clearInterval(keepAlive);
 }