@augmentd-labs/canvas-edge 0.7.0 → 0.7.2

package/node_modules/canvas-stored/README.md

@@ -317,7 +317,7 @@ await mirror.stop();            // drains running jobs
 
 - **View**: Stored's index holds both sides — local keys through the file watcher/scan, hub keys through the `canvas` driver's feed poller. **Base ledger** (`Ledger`, LMDB sub-db `mirror`): `base/<mirror>/<key>` → `{ sha256, size, mtime, remoteSeq }`, `cursor/<mirror>`, `skip/<mirror>/<key>`, `state/<mirror>`. Keys + digests are the identity; hub document ids are never stored.
 - **Decision**: per dirty key, the protocol's three-way table (`decide(L, B, R)`, exported from `src/sync/Mirror.js`); R is confirmed with `HEAD` before a push or a conflict.
-- **Jobs** (`JobQueue`, LMDB sub-dbs `jobs` + `jobs.dedupe`): `push`, `pull`, `delete-remote`, `trash-local`, `rename-remote`, `rename-remote-dir`, `rename-local`, `conflict`; dedupe per `<kind>|<key>`, backoff `min(60s·2^n, 1h)`, running→pending recovery on open, one job per key in flight, concurrency 4 (a brand-new key skips the pre-push HEAD; the PUT's If-None-Match:* guards it). Hub requests abort after `requestTimeoutMs` (60 s) plus an upload allowance (`minUploadBytesPerSec`, 50 kB/s) so a dropped connection never pins a worker. Every byte move goes through `stored.copy`/`move`/`renameObject`/`removeObject`, so locations, succession and echo suppression are the ordinary ones. Base is written only after the operation succeeded; the feed cursor after a whole batch reconciled.
+- **Jobs** (`JobQueue`, LMDB sub-dbs `jobs` + `jobs.dedupe`): `push`, `pull`, `delete-remote`, `trash-local`, `rename-remote`, `rename-remote-dir`, `rename-local`, `conflict`; dedupe per `<kind>|<key>`, backoff `min(60s·2^n, 1h)`, running→pending recovery on open, one job per key in flight, concurrency 4 (a brand-new key skips the pre-push HEAD; the PUT's If-None-Match:* guards it). Hub requests abort after `requestTimeoutMs` (60 s) plus an upload allowance (`minUploadBytesPerSec`, 50 kB/s) so a dropped connection never pins a worker. A directory rename the hub answers `202` (still running) is requested again with the same `operationId` until the hub reports the result. Requests to the hub run outside the storage mutation queue (an upload streams unqueued, a download is staged and only its commit is queued, a delete or a single-file rename is one request and one index transaction), so the four workers overlap on the network. Every byte move goes through `stored.copy`/`move`/`renameObject`/`removeObject`, so locations, succession and echo suppression are the ordinary ones. Base is written only after the operation succeeded; the feed cursor after a whole batch reconciled.
 - **Conflicts** (`prompt`): the device version is copied to the `conflicts` backend as `<stem> (conflict from <device> <YYYY-MM-DD HHmm>).<ext>`, uploaded with `X-Canvas-Conflict-Of` (inbox), the hub version is pulled into place, base := R. `rename`: the upload is an ordinary object at the conflict-copy key (the hub tags it); the copy then arrives on every device like any other object.
 - **Renames**: a local `mv` (paired by Stored's inode window, or by content after downtime) → `POST objects/rename`. Renames are collected for `moveSettleMs` (1.5 s); when a whole folder went (the source folder is gone from the device, the target is new to the hub) they are sent as one request with `directory: true`, whatever the number of files. An inferred local deletion waits `deleteDelayMs` (5 s) before it is sent, so the second half of a move can still claim it. A hub rename → local `rename(2)` when the local copy is clean. **Hub delete** → local copy to the `trash` backend. **Offline** is a state: jobs stay queued, the engine retries with backoff and catches up (`poll`, or a full listing after `410`).
 - **Keys**: `src/sync/keys.js` — `normalizeKey` (NFC, `/`), `validateKey` (traversal, NUL, Windows reserved names, `<>:"|?*`, trailing dot/space, length) → skipped visibly (`skip` event, `status().skips`), `isIgnored(key, patterns)` (picomatch, `dot:true`), `matchesPrefixes`, `conflictKey`, `MIRROR_IGNORE_DEFAULTS`.

package/node_modules/canvas-stored/package.json

@@ -1,6 +1,6 @@
 {
   "name": "canvas-stored",
-  "version": "1.18.0",
+  "version": "1.20.0",
   "description": "",
   "main": "./src/index.js",
   "type": "module",

package/node_modules/canvas-stored/src/backends/canvas/index.js

@@ -42,6 +42,8 @@ const RETRY_DELAYS_MS = [500, 1500, 4000];
 // large file on a slow uplink is never cut short.
 const REQUEST_TIMEOUT_MS = 60_000;
 const MIN_UPLOAD_BYTES_PER_SEC = 50_000;
+// Pause between two requests for a directory rename the hub is still running.
+const DIRECTORY_RENAME_POLL_MS = [500, 1000, 2000, 5000];
 const JSON_HEADERS = { Accept: 'application/json' };
 
 const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
@@ -568,8 +570,10 @@ export default class CanvasBackend extends StorageBackend {
 
     /**
      * `POST objects/rename` with `directory: true`: the hub moves the folder
-     * and everything under it in one operation. `operationId` makes a retry
-     * after a lost response return the first result.
+     * and everything under it in one operation. A large folder is answered
+     * `202` while the hub works; the same request is repeated until it
+     * reports the result. `operationId` makes every repeat, and a retry
+     * after a lost response, return the first result.
      */
     async renameDirectory(from, to, { operationId, origin = null } = {}) {
         const fromKey = normalizeKey(from);
@@ -577,13 +581,19 @@ export default class CanvasBackend extends StorageBackend {
         if (!fromKey || !toKey || !operationId) throw new CanvasHubError('canvas directory rename: from, to and operationId are required', { code: 'REFUSED' });
         const body = { from: fromKey, to: toKey, directory: true, operationId };
         if (origin || this.#deviceId) body.origin = origin || this.#deviceId;
-        const { payload } = await this.#json(`${this.#backendUrl()}/objects/rename`, {
-            method: 'POST',
-            headers: { 'Content-Type': 'application/json' },
-            body: JSON.stringify(body),
-        }, { key: fromKey });
-        if (payload?.directory !== true) throw new CanvasHubError('the hub did not acknowledge a directory rename', { code: 'REFUSED', hubCode: 'NOT_IMPLEMENTED', key: fromKey });
         const prefix = `${fromKey}/`;
+        let payload;
+        for (let attempt = 0; ; attempt += 1) {
+            const answer = await this.#json(`${this.#backendUrl()}/objects/rename`, {
+                method: 'POST',
+                headers: { 'Content-Type': 'application/json' },
+                body: JSON.stringify(body),
+            }, { key: fromKey });
+            payload = answer.payload;
+            if (answer.status !== 202) break;
+            await sleep(DIRECTORY_RENAME_POLL_MS[Math.min(attempt, DIRECTORY_RENAME_POLL_MS.length - 1)]);
+        }
+        if (payload?.directory !== true) throw new CanvasHubError('the hub did not acknowledge a directory rename', { code: 'REFUSED', hubCode: 'NOT_IMPLEMENTED', key: fromKey });
         for (const [key, row] of [...this.#files]) {
             if (!key.startsWith(prefix)) continue;
             const next = `${toKey}/${key.slice(prefix.length)}`;

package/node_modules/canvas-stored/src/index.js

@@ -27,6 +27,8 @@ import { detectMimeType, detectMimeFromHead } from './utils/mime.js';
 const debug = Debug('stored');
 // Source retirement belongs to the final move event, never a source-loss unlink.
 const MOVE_SOURCE = Symbol('move-source');
+// A remote source already downloaded for this copy: { path, checksums, size, backend, key }.
+const STAGED = Symbol('staged-source');
 
 const HEAD_BYTES = 4096;
 
@@ -101,6 +103,7 @@ export default class Stored extends EventEmitter2 {
     #sweepTimer = null;
     #directoryMovesDb = null;
     #directoryMoveTasks = new Map();
+    #moveUnqueued;
     #replicas;
 
     constructor(config = {}) {
@@ -112,7 +115,15 @@ export default class Stored extends EventEmitter2 {
         for (const name of ['put', 'copy', 'move', 'writeObject', 'removeObject', 'removeToTrash', 'renameObject', 'renameDirectory', 'trashDirectory', 'removeDirectory',
             'ensureDirectory', 'removeEmptyDirectory', 'restoreTrash', 'discardTrash', 'restoreRetained', 'deleteByUrl', 'delete']) {
             const method = this[name].bind(this);
-            this[name] = (...args) => this.withStorageMutation(() => method(...args), name);
+            if (name === 'move') this.#moveUnqueued = method;
+            if (name === 'copy') this[name] = (idOrKey, options = {}) => this.#routeCopy(method, idOrKey, options);
+            // A delete or rename on a streaming remote is one request and one
+            // index transaction, so several run side by side.
+            else if (name === 'removeObject' || name === 'renameObject') {
+                this[name] = (backendName, ...rest) => (this.#streamsRemote(backendName)
+                    ? method(backendName, ...rest)
+                    : this.withStorageMutation(() => method(backendName, ...rest), name));
+            } else this[name] = (...args) => this.withStorageMutation(() => method(...args), name);
         }
         this.#paths = resolveStoredPaths(config);
         this.#config = {
@@ -983,7 +994,8 @@ export default class Stored extends EventEmitter2 {
             return { ok: false, reason: 'target-exists', key: toKey };
         }
 
-        const result = await this.move(`${backendName}:${fromKey}`, {
+        const move = this.#streamsRemote(backendName) ? this.#moveUnqueued : this.move;
+        const result = await move(`${backendName}:${fromKey}`, {
             to: backendName,
             key: toKey,
             from: `stored://${backendName}/${fromKey}`,
@@ -1196,6 +1208,58 @@ export default class Stored extends EventEmitter2 {
      *   pick the next free `name-1.ext`, or replace it.
      * @returns {Promise<{ ok:true, id, added:string[], locations } | { ok:false, reason:string }>}
      */
+    /**
+     * Network time stays out of the mutation executor, so transfers to and
+     * from a hub run side by side. An upload to a streaming remote touches no
+     * local bytes and records its result in one index transaction, so it runs
+     * unqueued. A download is staged to a temp file first; only the commit
+     * onto the local target is queued.
+     */
+    async #routeCopy(method, idOrKey, options) {
+        const names = (Array.isArray(options.to) ? options.to : [options.to]).filter(Boolean).map(String);
+        const targets = names.map((name) => this.#backends.get(name));
+        if (names.length === 1 && this.#streamsRemote(names[0])) {
+            return method(idOrKey, options);
+        }
+        const staged = this.#closing ? null : await this.#stageRemoteSource(idOrKey, options, targets);
+        if (!staged) return this.withStorageMutation(() => method(idOrKey, options), 'copy');
+        try {
+            return await this.withStorageMutation(() => method(idOrKey, { ...options, [STAGED]: staged }), 'copy');
+        } finally {
+            await fsp.rm(staged.path, { force: true }).catch(() => {});
+        }
+    }
+
+    // A remote that takes writes directly (the canvas driver): it holds no
+    // local bytes, so its requests need no place in the mutation queue.
+    #streamsRemote(backendName) {
+        const backend = this.#backends.get(String(backendName));
+        return backend?.type === 'remote' && typeof backend.putStream === 'function';
+    }
+
+    // Download a remote source next to its first local target. Null when the
+    // source is not remote or cannot be read; the queued copy then reports it.
+    async #stageRemoteSource(idOrKey, options, targets) {
+        const meta = this.#index.get(idOrKey);
+        const location = meta ? this.#resolveSourceLocation(meta, options.from) : null;
+        const backend = location ? this.#backends.get(location.backend) : null;
+        if (backend?.type !== 'remote') return null;
+        const firstLocal = targets.find((t) => t?.type === 'local');
+        const tempDir = firstLocal ? firstLocal.tempDir : path.join(this.#paths.cache, '.tmp');
+        const tempPath = path.join(tempDir, `${Date.now()}-${crypto.randomUUID()}`);
+        try {
+            const source = await backend.get(location.key, { stream: true });
+            if (!source) return null;
+            await fsp.mkdir(tempDir, { recursive: true });
+            const { checksums, size } = await this.#hashToFile(source, tempPath);
+            return { path: tempPath, checksums, size, backend: location.backend, key: location.key };
+        } catch (err) {
+            debug(`Staging ${location.backend}:${location.key} failed: ${err.message}`);
+            await fsp.rm(tempPath, { force: true }).catch(() => {});
+            return null;
+        }
+    }
+
     async copy(idOrKey, options = {}) {
         const plan = await this.#planTransfer(idOrKey, options);
         if (!plan.ok) return plan;
@@ -1207,7 +1271,7 @@ export default class Stored extends EventEmitter2 {
 
         const release = this.#holdKeys(targets.map(t => `${t.name}:${t.key || destKey}`));
         try {
-            const transferred = await this.#streamToTargets(meta, sourceLocation, targets, destKey, this.#transferOptions(options));
+            const transferred = await this.#streamToTargets(meta, sourceLocation, targets, destKey, this.#transferOptions(options), options[STAGED]);
             if (!transferred.ok) return transferred;
             // A conflict-inbox upload placed nothing at the key: no location.
             if (transferred.conflict && !transferred.locations.length) {
@@ -1632,7 +1696,13 @@ export default class Stored extends EventEmitter2 {
             // an unmounted share, a failed sync) must not turn a copy into a
             // silent no-op that reports success.
             const ours = (meta.locations || []).some(l => l.backend === name && l.key === destKey);
-            const present = await backend.stat(destKey).catch(() => null);
+            // A streaming remote enforces If-Match / If-None-Match itself, so
+            // the indexed view of the key stands in for a round trip.
+            const preconditioned = backend.type === 'remote' && typeof backend.putStream === 'function'
+                && (options.ifMatch != null || options.ifNoneMatch != null);
+            const present = preconditioned
+                ? (this.#index.get(`${name}:${destKey}`) ? { key: destKey } : null)
+                : await backend.stat(destKey).catch(() => null);
             if (ours && present) continue;
             if (ours) debug(`Stale location ${name}:${destKey} (indexed, no bytes) — re-transferring`);
 
@@ -1724,7 +1794,7 @@ export default class Stored extends EventEmitter2 {
      * content identity before committing. The temp file is staged on the first
      * local target's filesystem so `#commit` can hardlink instead of copying.
      */
-    async #streamToTargets(meta, sourceLocation, targets, destKey, options = {}) {
+    async #streamToTargets(meta, sourceLocation, targets, destKey, options = {}, staged = null) {
         const sourceBackend = this.#backends.get(sourceLocation.backend);
         // One remote target that speaks `putStream`: stream straight to it —
         // no temp file, no cache entry, no SyncQueue. The hub verifies the
@@ -1732,16 +1802,17 @@ export default class Stored extends EventEmitter2 {
         if (targets.length === 1 && targets[0].backend.type === 'remote' && typeof targets[0].backend.putStream === 'function') {
             return this.#streamToRemote(meta, sourceBackend, sourceLocation, targets[0], destKey, options);
         }
-        const source = await sourceBackend.get(sourceLocation.key, { stream: true });
-        if (!source) return { ok: false, reason: 'source-unreadable', backend: sourceLocation.backend, key: sourceLocation.key };
+        const prefetched = staged && staged.backend === sourceLocation.backend && staged.key === sourceLocation.key ? staged : null;
+        const source = prefetched ? null : await sourceBackend.get(sourceLocation.key, { stream: true });
+        if (!prefetched && !source) return { ok: false, reason: 'source-unreadable', backend: sourceLocation.backend, key: sourceLocation.key };
 
         const firstLocal = targets.find(t => t.backend.type === 'local')?.backend;
         const tempDir = firstLocal ? firstLocal.tempDir : path.join(this.#paths.cache, '.tmp');
-        const tempPath = path.join(tempDir, `${Date.now()}-${crypto.randomUUID()}`);
-        await fsp.mkdir(tempDir, { recursive: true });
+        const tempPath = prefetched ? prefetched.path : path.join(tempDir, `${Date.now()}-${crypto.randomUUID()}`);
+        if (!prefetched) await fsp.mkdir(tempDir, { recursive: true });
 
         try {
-            const { checksums, size } = await this.#hashToFile(source, tempPath);
+            const { checksums, size } = prefetched || await this.#hashToFile(source, tempPath);
             const id = formatId(checksums, this.#config.primaryChecksum);
             if (id !== meta.id) {
                 debug(`Transfer aborted: ${sourceLocation.backend}:${sourceLocation.key} hashes to ${id}, indexed as ${meta.id}`);

package/node_modules/canvas-stored/src/sync/Mirror.js

@@ -926,6 +926,7 @@ export default class Mirror extends EventEmitter {
         const key = job.key;
         const { L, B, Lloc, Lmeta } = this.#local3(key);
         if (!L || !Lmeta) { this.#markDirty(key); return; }
+        if (L === B) return;                                    // an earlier job already pushed these bytes
         // A brand-new key (no base, nothing in the hub view) skips the HEAD:
         // the PUT carries If-None-Match:* and a 412 re-decides below, so the
         // extra round trip bought nothing — and on a slow link it was half the

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@augmentd-labs/canvas-edge",
-  "version": "0.7.0",
+  "version": "0.7.2",
   "description": "Canvas device runtime: folder mirrors (canvas-stored Mirror engine) + edge tunnel client (dist artifact)",
   "license": "AGPL-3.0-or-later",
   "type": "module",
@@ -21,7 +21,7 @@
   },
   "dependencies": {
     "@augmentd-labs/canvas-protocol": "^0.5.0",
-    "canvas-stored": "1.18.0",
+    "canvas-stored": "1.20.0",
     "@aws-sdk/client-s3": "^3.1149.0",
     "cacache": "^20.0.4",
     "chokidar": "^5.0.0",
@@ -48,6 +48,6 @@
   "bundleDependencies": [
     "canvas-stored"
   ],
-  "canvasRev": "9cc56fa",
+  "canvasRev": "8758ca3",
   "canvasSource": "runtimes/edge"
 }