@shieldfive/cli 0.4.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +32 -0
- package/README.md +79 -22
- package/package.json +1 -1
- package/src/cli.mjs +71 -21
- package/src/migrate.mjs +510 -0
- package/src/upload.mjs +76 -26
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,38 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## Unreleased
|
|
9
9
|
|
|
10
|
+
## 0.5.0 — 2026-11-16
|
|
11
|
+
|
|
12
|
+
### Behaviour changes
|
|
13
|
+
|
|
14
|
+
- **`sf push` copies the whole folder tree.** It creates one vault folder named
|
|
15
|
+
after the source and recreates every subfolder inside it (folder keys and
|
|
16
|
+
names made as the web app's Upload folder makes them). It used to upload only
|
|
17
|
+
the files at the top of the folder, to the vault root. Through the agent it
|
|
18
|
+
still does that, and refuses a folder with subfolders.
|
|
19
|
+
- **`sf push` is resumable.** A 0600 state file in `~/.shieldfive/push/` records
|
|
20
|
+
finished folders and files; rerunning skips them. A file whose finalize was
|
|
21
|
+
cut off is settled by asking the server, not uploaded twice.
|
|
22
|
+
|
|
23
|
+
### Added
|
|
24
|
+
|
|
25
|
+
- Retries with backoff for network errors and 5xx/408/429 answers on the
|
|
26
|
+
session, the direct PUT, each part and the folder calls; 3 files at a time
|
|
27
|
+
(`SF_PUSH_CONCURRENCY`); per-file output and a progress line every 10 s;
|
|
28
|
+
first Ctrl-C finishes uploads in progress, second quits.
|
|
29
|
+
- The access token is refreshed during a long push, including in the middle of
|
|
30
|
+
a large file.
|
|
31
|
+
- File and folder names are sent in Unicode NFC.
|
|
32
|
+
|
|
33
|
+
### Verified
|
|
34
|
+
|
|
35
|
+
- Run against production on 2026-11-16 with the canary account: 248 files /
|
|
36
|
+
534 MB in 10 folders (520 MB and 12 MB files, Unicode/long/duplicate names),
|
|
37
|
+
killed and interrupted midway, resumed; all 247 non-empty files `ready` with
|
|
38
|
+
the recorded ids, sizes and folders; three of them (including the 520 MB
|
|
39
|
+
file) downloaded byte-identical in the web app; the 0-byte file skipped
|
|
40
|
+
(production does not accept empty files yet).
|
|
41
|
+
|
|
10
42
|
## 0.4.0 — 2026-10-12
|
|
11
43
|
|
|
12
44
|
### Behaviour changes
|
package/README.md
CHANGED
|
@@ -39,18 +39,27 @@ size, uploading nothing.
|
|
|
39
39
|
|
|
40
40
|
- The underlying cryptography ([`@shieldfive/crypto`](https://github.com/shieldfive/crypto))
|
|
41
41
|
has **not** undergone an external audit yet. Treat this as early software.
|
|
42
|
-
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
42
|
+
- **`sf push` was run against production on 2026-11-16** (0.5.0, with the
|
|
43
|
+
dedicated canary account) on a tree of 248 files (534 MB) in 10 folders: a
|
|
44
|
+
520 MB file, a 12 MB file, 240 small files, an empty folder, a 0-byte file,
|
|
45
|
+
Unicode, 200-character and duplicate names, two `.DS_Store` files and a
|
|
46
|
+
symlink. The first run was killed (SIGKILL) after 81 files, the second was
|
|
47
|
+
stopped with Ctrl-C after 60 more, the third resumed and finished, and a
|
|
48
|
+
fourth uploaded nothing. In the production database all 247 non-empty files
|
|
49
|
+
are `ready`, each with its id, plaintext size and folder exactly as the client
|
|
50
|
+
recorded them, and the 10 folders form the source tree. The hidden files and
|
|
51
|
+
the symlink were skipped. Three of the files (a small one at the top, the
|
|
52
|
+
Unicode-named one and the 520 MB one) were then downloaded in the web app and
|
|
53
|
+
decrypted byte-identical. Uploading left the account's download allowance
|
|
54
|
+
counter unchanged. One `sf sync` pass of a flat folder uploaded both of its
|
|
55
|
+
files to the vault root. Production does not accept 0-byte files yet:
|
|
56
|
+
`sf push` skips them, says so, and exits 2.
|
|
57
|
+
- These `sf push` behaviours are covered by offline tests only and have **not**
|
|
58
|
+
been exercised against production: retries after network errors and 5xx/429
|
|
59
|
+
answers, stopping on a full vault, refreshing the session during a run longer
|
|
60
|
+
than the access token's lifetime, settling a file whose finalize was cut off,
|
|
61
|
+
a file changing while it uploads, a vault folder deleted between runs, and a
|
|
62
|
+
vault password different from the account password.
|
|
54
63
|
- `sf sync` is **append-only**: it uploads new and changed files; it does not
|
|
55
64
|
yet mirror local deletions or renames into the vault.
|
|
56
65
|
- **`sf login` and the agent have not been run against production.** They are
|
|
@@ -83,7 +92,7 @@ trust it):
|
|
|
83
92
|
git clone https://github.com/shieldfive/cli.git
|
|
84
93
|
cd cli
|
|
85
94
|
npm install # @shieldfive/crypto (+ libsodium), @supabase/supabase-js, @noble/*
|
|
86
|
-
npm test
|
|
95
|
+
npm test
|
|
87
96
|
```
|
|
88
97
|
|
|
89
98
|
## Commands
|
|
@@ -92,7 +101,7 @@ npm test # 107 tests
|
|
|
92
101
|
sf login [--idle=<hours>] sign in once; an agent holds the session in memory (locks after 8 h idle)
|
|
93
102
|
sf status is an agent running, and for which account
|
|
94
103
|
sf logout [--everywhere] revoke the session and stop the agent; --everywhere signs out every device
|
|
95
|
-
sf push <folder> [--include-dotfiles]
|
|
104
|
+
sf push <folder> [--include-dotfiles] copy a whole folder tree into a new vault folder; rerun to resume
|
|
96
105
|
sf sync <folder> [--watch] [--interval=N] upload new/changed files; --watch keeps a poll loop (N seconds, default 5)
|
|
97
106
|
sf verify <file>... report whether each file is safely stored in your vault
|
|
98
107
|
sf encrypt <folder> encrypt each file locally, upload nothing (no account needed)
|
|
@@ -100,14 +109,36 @@ sf inspect <folder> read-only recursive migration inven
|
|
|
100
109
|
```
|
|
101
110
|
|
|
102
111
|
`sf push` and `sf sync` pick the direct or multipart upload path automatically
|
|
103
|
-
from the file size.
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
112
|
+
from the file size. Symlinks are never followed (files are opened with
|
|
113
|
+
`O_NOFOLLOW`), and names starting with `.` (`.DS_Store`, `.git`, `.env`) are
|
|
114
|
+
skipped unless you pass `sf push --include-dotfiles`. An empty (0-byte) file is
|
|
115
|
+
uploaded only when the server says it accepts empty files; otherwise it is
|
|
116
|
+
reported as skipped and the exit code is 2. If an upload fails after its session
|
|
117
|
+
was created, the pending session is cancelled so it does not hold quota.
|
|
118
|
+
|
|
119
|
+
`sf push` copies the whole tree: it creates one vault folder named after
|
|
120
|
+
`<folder>` and recreates every subfolder (empty ones too) inside it, the way the
|
|
121
|
+
web app's Upload folder does. Three files upload at a time
|
|
122
|
+
(`SF_PUSH_CONCURRENCY`, 1-8); large files also send three parts at a time
|
|
123
|
+
(`SF_UPLOAD_CONCURRENCY`). Network errors, 5xx, 408 and 429 answers are retried
|
|
124
|
+
with backoff. It prints each file as it finishes and a progress line every 10
|
|
125
|
+
seconds. A full vault or a refused session stops the run; any other file that
|
|
126
|
+
fails is listed, the rest continue, and the exit code is 2.
|
|
127
|
+
|
|
128
|
+
It is resumable. A state file in `~/.shieldfive/push/` (mode 0600; file names
|
|
129
|
+
and vault ids, never keys) records each folder made and each file once its
|
|
130
|
+
upload is final, so running the same command again skips those and uploads the
|
|
131
|
+
rest; a file interrupted halfway starts again from its beginning. The first
|
|
132
|
+
Ctrl-C lets the uploads in progress finish and exits 130; a second quits at once.
|
|
133
|
+
A run killed during a file's last step is settled on the next run by asking the
|
|
134
|
+
server whether that file arrived, so it is not uploaded twice. A file that
|
|
135
|
+
changes while it uploads is reported, and uploaded again on the next run. Files
|
|
136
|
+
you delete from the vault after a run are not re-uploaded by a rerun.
|
|
137
|
+
|
|
138
|
+
Through the agent (`sf login`), `sf push` uploads only the files at the top of
|
|
139
|
+
the folder, to the vault root, and refuses a folder with subfolders; use the
|
|
140
|
+
environment variables below for a tree. `sf sync` is also top-level only and
|
|
141
|
+
uploads to the vault root.
|
|
111
142
|
|
|
112
143
|
`sf inspect` reports JSON counts of files, directories, bytes and skipped entries.
|
|
113
144
|
It includes nested/hidden files and empty folders, reads metadata only, and
|
|
@@ -182,6 +213,32 @@ With `sf login` you are asked once. The agent refreshes the stepped-up session
|
|
|
182
213
|
on its own, so `sf sync --watch` keeps working on a two-factor account until the
|
|
183
214
|
agent locks or you log out.
|
|
184
215
|
|
|
216
|
+
## Migrating from Dropbox or Google Drive with rclone
|
|
217
|
+
|
|
218
|
+
For most people the web app is simpler: sync Dropbox to your computer, then
|
|
219
|
+
**Upload > Upload folder** in the browser
|
|
220
|
+
(see [Move from Dropbox to ShieldFive](https://shieldfive.com/help/move-from-dropbox)).
|
|
221
|
+
For a large or scripted move:
|
|
222
|
+
|
|
223
|
+
```sh
|
|
224
|
+
# 1. Copy your cloud files to a local folder (rclone.org; `rclone config` first)
|
|
225
|
+
rclone copy dropbox: ~/dropbox-export --progress
|
|
226
|
+
# Google Drive: rclone copy gdrive: ~/drive-export --progress
|
|
227
|
+
|
|
228
|
+
# 2. Encrypt and upload it, keeping the folder structure
|
|
229
|
+
export SF_EMAIL=you@example.com
|
|
230
|
+
read -rs SF_PASSWORD; echo; export SF_PASSWORD
|
|
231
|
+
sf push ~/dropbox-export
|
|
232
|
+
|
|
233
|
+
# 3. If it stops for any reason, run the same command again: it resumes.
|
|
234
|
+
sf push ~/dropbox-export
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
Your vault gets a folder named `dropbox-export` with the same tree inside. The
|
|
238
|
+
command exits 0 only when every file is in the vault. Uploads do not count
|
|
239
|
+
toward your monthly download allowance. `sf push` copies once; it does not keep
|
|
240
|
+
the folder in sync afterwards.
|
|
241
|
+
|
|
185
242
|
## What the agent can and cannot do
|
|
186
243
|
|
|
187
244
|
The agent keeps two things in memory: your signed-in session and your unlocked
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@shieldfive/cli",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.5.0",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"description": "Open-source command-line client for ShieldFive: encrypts files on your machine before upload (contents: AES-256-GCM under an ML-KEM-1024 + X25519 hybrid key; names: AES-256-GCM under your vault key) and pushes or syncs them to your vault. Read the code.",
|
|
6
6
|
"license": "Apache-2.0",
|
package/src/cli.mjs
CHANGED
|
@@ -401,6 +401,17 @@ async function cmdLogout(flags) {
|
|
|
401
401
|
async function cmdPush(folder, flags) {
|
|
402
402
|
const { agent } = await chooseAuth()
|
|
403
403
|
if (agent) {
|
|
404
|
+
// The agent uploads single files to the vault root; it cannot make vault
|
|
405
|
+
// folders yet. Refuse a tree rather than flatten it.
|
|
406
|
+
const { readdir: readDir } = await import('node:fs/promises')
|
|
407
|
+
const entries = await readDir(resolve(folder), { withFileTypes: true })
|
|
408
|
+
if (entries.some((e) => e.isDirectory() && !e.name.startsWith('.'))) {
|
|
409
|
+
throw new Error(
|
|
410
|
+
'this folder has subfolders, and pushing through the agent (sf login) only ' +
|
|
411
|
+
'uploads the files at its top level. To copy the whole tree, run sf logout, ' +
|
|
412
|
+
'then set SF_EMAIL and SF_PASSWORD and run sf push again.',
|
|
413
|
+
)
|
|
414
|
+
}
|
|
404
415
|
const { request } = await import('./agent/client.mjs')
|
|
405
416
|
process.stderr.write(`pushing to account ${agent.account} through the agent\n`)
|
|
406
417
|
let differs = 0
|
|
@@ -431,29 +442,68 @@ async function cmdPush(folder, flags) {
|
|
|
431
442
|
}
|
|
432
443
|
|
|
433
444
|
const cfg = readLiveConfig(flags)
|
|
434
|
-
const {
|
|
435
|
-
const {
|
|
436
|
-
|
|
437
|
-
|
|
438
|
-
|
|
439
|
-
|
|
440
|
-
|
|
441
|
-
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
process.
|
|
451
|
-
skipped++
|
|
452
|
-
continue
|
|
445
|
+
const { runPush } = await import('./migrate.mjs')
|
|
446
|
+
const { createSessionTokens } = await import('./sync.mjs')
|
|
447
|
+
const session = await authAndUnlock(cfg)
|
|
448
|
+
const accessToken = createSessionTokens({
|
|
449
|
+
accessToken: session.accessToken,
|
|
450
|
+
refreshToken: session.refreshToken,
|
|
451
|
+
expiresAt: session.expiresAt,
|
|
452
|
+
refresh: session.refreshSession,
|
|
453
|
+
})
|
|
454
|
+
|
|
455
|
+
// First Ctrl-C: start nothing new, let uploads in progress finish and be
|
|
456
|
+
// recorded. Second: quit now. Either way a rerun picks up where this left off.
|
|
457
|
+
const controller = new AbortController()
|
|
458
|
+
const onSigint = () => {
|
|
459
|
+
if (controller.signal.aborted) {
|
|
460
|
+
process.stderr.write('\nquitting now; run the same command again to resume\n')
|
|
461
|
+
process.exit(130)
|
|
453
462
|
}
|
|
454
|
-
|
|
463
|
+
controller.abort()
|
|
464
|
+
process.stderr.write(
|
|
465
|
+
'\nstopping: finishing the uploads in progress (Ctrl-C again to quit now). ' +
|
|
466
|
+
'Run the same command again to resume.\n',
|
|
467
|
+
)
|
|
468
|
+
}
|
|
469
|
+
process.on('SIGINT', onSigint)
|
|
470
|
+
let summary
|
|
471
|
+
try {
|
|
472
|
+
summary = await runPush({
|
|
473
|
+
source: folder,
|
|
474
|
+
apiBaseUrl: cfg.apiBaseUrl,
|
|
475
|
+
accessToken,
|
|
476
|
+
rootKey: session.rootKey,
|
|
477
|
+
account: cfg.email.toLowerCase(),
|
|
478
|
+
includeDotfiles: pushListOptions(flags).includeDotfiles,
|
|
479
|
+
signal: controller.signal,
|
|
480
|
+
})
|
|
481
|
+
} finally {
|
|
482
|
+
process.off('SIGINT', onSigint)
|
|
483
|
+
}
|
|
484
|
+
const fmt = (n) => `${(n / 1e9).toFixed(2)} GB`
|
|
485
|
+
process.stderr.write(
|
|
486
|
+
`done: ${summary.uploaded} uploaded (${fmt(summary.bytesUploaded)}), ` +
|
|
487
|
+
`${summary.alreadyThere} already there, ${summary.foldersCreated} folders created` +
|
|
488
|
+
(summary.failed.length ? `, ${summary.failed.length} FAILED` : '') +
|
|
489
|
+
(summary.changed.length ? `, ${summary.changed.length} changed while uploading` : '') +
|
|
490
|
+
(summary.skippedEmpty.length ? `, ${summary.skippedEmpty.length} empty skipped` : '') +
|
|
491
|
+
'\n',
|
|
492
|
+
)
|
|
493
|
+
if (summary.stoppedBy) {
|
|
494
|
+
process.stderr.write(`stopped: ${summary.stoppedBy.message}\n`)
|
|
495
|
+
process.exitCode = 1
|
|
496
|
+
return
|
|
497
|
+
}
|
|
498
|
+
if (summary.interrupted) {
|
|
499
|
+
process.stderr.write(`interrupted with ${summary.notStarted} files not started; run the same command again to resume\n`)
|
|
500
|
+
process.exitCode = 130
|
|
501
|
+
return
|
|
502
|
+
}
|
|
503
|
+
if (summary.failed.length || summary.changed.length || summary.skippedEmpty.length) {
|
|
504
|
+
process.stderr.write('some files are not in the vault; run the same command again to retry them\n')
|
|
505
|
+
process.exitCode = 2
|
|
455
506
|
}
|
|
456
|
-
if (skipped) process.exitCode = 2
|
|
457
507
|
}
|
|
458
508
|
|
|
459
509
|
function pushListOptions(flags) {
|
package/src/migrate.mjs
ADDED
|
@@ -0,0 +1,510 @@
|
|
|
1
|
+
// ShieldFive CLI — `sf push`: copy a whole folder tree into the vault, the way a
|
|
2
|
+
// migration from Dropbox or Google Drive needs it.
|
|
3
|
+
//
|
|
4
|
+
// - The tree is recreated in the vault under one new top-level folder named
|
|
5
|
+
// after the source folder. Folder names and keys are made exactly as the
|
|
6
|
+
// web app's "Upload folder" makes them (a fresh folder key wrapped by the
|
|
7
|
+
// parent's key, a v6 name sealed to the folder's row id).
|
|
8
|
+
// - Rerunnable. A state file (outside the source folder, so a read-only
|
|
9
|
+
// export works) records the vault folder made for each directory and each
|
|
10
|
+
// file once its upload is final. A rerun skips those and uploads the rest.
|
|
11
|
+
// Before a file starts, its row id is recorded as pending, so a run killed
|
|
12
|
+
// during finalize is settled on the next run by asking the server whether
|
|
13
|
+
// that row is ready, rather than uploading the file twice.
|
|
14
|
+
// - Transient failures (network, 5xx, 408, 429) are retried with backoff. A
|
|
15
|
+
// full vault or a refused session stops the run; other per-file failures
|
|
16
|
+
// are reported and the run goes on.
|
|
17
|
+
//
|
|
18
|
+
// No key is ever written to disk: folder keys are unwrapped from the server's
|
|
19
|
+
// folder tree with the vault root key on every run.
|
|
20
|
+
|
|
21
|
+
import { createHash } from 'node:crypto'
|
|
22
|
+
import { lstat, mkdir, open, readFile, readdir, rename } from 'node:fs/promises'
|
|
23
|
+
import { homedir } from 'node:os'
|
|
24
|
+
import { basename, join, relative, resolve, sep } from 'node:path'
|
|
25
|
+
|
|
26
|
+
import { encryptNameV6 } from '@shieldfive/crypto/vault'
|
|
27
|
+
|
|
28
|
+
import {
|
|
29
|
+
EmptyFileUnsupportedError,
|
|
30
|
+
httpError,
|
|
31
|
+
newRowBoundId,
|
|
32
|
+
resolveToken,
|
|
33
|
+
uploadFile,
|
|
34
|
+
} from './upload.mjs'
|
|
35
|
+
import {
|
|
36
|
+
base64ToBytes,
|
|
37
|
+
bytesToBase64,
|
|
38
|
+
generateRandomKeyB64,
|
|
39
|
+
hashMetadataV4,
|
|
40
|
+
unwrapKeyB64,
|
|
41
|
+
wrapKeyB64,
|
|
42
|
+
} from './uploadCrypto.mjs'
|
|
43
|
+
|
|
44
|
+
const STATE_VERSION = 1
|
|
45
|
+
const DEFAULT_CONCURRENCY = 3
|
|
46
|
+
const MAX_CONCURRENCY = 8
|
|
47
|
+
const MAX_FILE_ATTEMPTS = 8
|
|
48
|
+
const RETRY_BASE_MS = 2000
|
|
49
|
+
const RETRY_MAX_MS = 60_000
|
|
50
|
+
const VERIFY_BATCH = 100
|
|
51
|
+
|
|
52
|
+
const sleep = (ms) => new Promise((r) => setTimeout(r, ms))
|
|
53
|
+
|
|
54
|
+
export function resolvePushConcurrency(env = process.env) {
|
|
55
|
+
const raw = Number.parseInt(env.SF_PUSH_CONCURRENCY ?? '', 10)
|
|
56
|
+
if (Number.isInteger(raw) && raw >= 1) return Math.min(raw, MAX_CONCURRENCY)
|
|
57
|
+
return DEFAULT_CONCURRENCY
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
export function defaultStatePath(source, home = homedir()) {
|
|
61
|
+
const id = createHash('sha256').update(resolve(source)).digest('hex').slice(0, 32)
|
|
62
|
+
return join(home, '.shieldfive', 'push', `${id}.json`)
|
|
63
|
+
}
|
|
64
|
+
|
|
65
|
+
// Names go to the vault in NFC, the form browsers and the other apps use, so a
|
|
66
|
+
// name read from a macOS disk in decomposed form looks the same in the vault.
|
|
67
|
+
const vaultName = (name) => name.normalize('NFC')
|
|
68
|
+
|
|
69
|
+
// Walk the source. lstat, never stat: a symlink is listed as skipped, never
|
|
70
|
+
// followed (it could point outside the folder). Dotfiles (.DS_Store, .git,
|
|
71
|
+
// .env, ...) are skipped unless includeDotfiles. Directories are returned
|
|
72
|
+
// parents first; relative paths use '/'.
|
|
73
|
+
export async function scanTree(source, { includeDotfiles = false } = {}) {
|
|
74
|
+
const dirs = []
|
|
75
|
+
const files = []
|
|
76
|
+
const skipped = { symlinks: [], dotfiles: [], other: [] }
|
|
77
|
+
async function walk(dir, rel) {
|
|
78
|
+
const names = (await readdir(dir)).sort()
|
|
79
|
+
for (const name of names) {
|
|
80
|
+
const path = join(dir, name)
|
|
81
|
+
const relPath = rel ? `${rel}/${name}` : name
|
|
82
|
+
if (name.startsWith('.') && !includeDotfiles) {
|
|
83
|
+
skipped.dotfiles.push(relPath)
|
|
84
|
+
continue
|
|
85
|
+
}
|
|
86
|
+
const info = await lstat(path)
|
|
87
|
+
if (info.isSymbolicLink()) skipped.symlinks.push(relPath)
|
|
88
|
+
else if (info.isDirectory()) {
|
|
89
|
+
dirs.push(relPath)
|
|
90
|
+
await walk(path, relPath)
|
|
91
|
+
} else if (info.isFile()) {
|
|
92
|
+
files.push({ rel: relPath, dir: rel, name, path, size: info.size, mtimeMs: info.mtimeMs })
|
|
93
|
+
} else skipped.other.push(relPath)
|
|
94
|
+
}
|
|
95
|
+
}
|
|
96
|
+
await walk(source, '')
|
|
97
|
+
dirs.sort((a, b) => a.split('/').length - b.split('/').length || (a < b ? -1 : 1))
|
|
98
|
+
return { dirs, files, skipped }
|
|
99
|
+
}
|
|
100
|
+
|
|
101
|
+
const signatureOf = ({ size, mtimeMs }) => `${size}:${mtimeMs}`
|
|
102
|
+
|
|
103
|
+
export async function loadState(statePath, { account, source }) {
|
|
104
|
+
const fresh = { version: STATE_VERSION, account, source, rootFolderId: null, folders: {}, files: {}, pending: {} }
|
|
105
|
+
let raw
|
|
106
|
+
try {
|
|
107
|
+
raw = await readFile(statePath, 'utf8')
|
|
108
|
+
} catch (err) {
|
|
109
|
+
if (err.code === 'ENOENT') return fresh
|
|
110
|
+
throw err
|
|
111
|
+
}
|
|
112
|
+
const parsed = JSON.parse(raw)
|
|
113
|
+
// Another account's ids mean nothing here: start over for this one.
|
|
114
|
+
if (!parsed || parsed.version !== STATE_VERSION || parsed.account !== account) return fresh
|
|
115
|
+
return {
|
|
116
|
+
...fresh,
|
|
117
|
+
rootFolderId: parsed.rootFolderId ?? null,
|
|
118
|
+
folders: parsed.folders ?? {},
|
|
119
|
+
files: parsed.files ?? {},
|
|
120
|
+
pending: parsed.pending ?? {},
|
|
121
|
+
}
|
|
122
|
+
}
|
|
123
|
+
|
|
124
|
+
// Write to a temp sibling, fsync, rename: a crash never leaves half a file.
|
|
125
|
+
// 0600: it lists file names and vault ids.
|
|
126
|
+
export async function saveState(statePath, state) {
|
|
127
|
+
await mkdir(join(statePath, '..'), { recursive: true, mode: 0o700 })
|
|
128
|
+
const tmp = `${statePath}.tmp`
|
|
129
|
+
const handle = await open(tmp, 'w', 0o600)
|
|
130
|
+
try {
|
|
131
|
+
await handle.chmod(0o600)
|
|
132
|
+
await handle.writeFile(JSON.stringify(state))
|
|
133
|
+
await handle.sync()
|
|
134
|
+
} finally {
|
|
135
|
+
await handle.close()
|
|
136
|
+
}
|
|
137
|
+
await rename(tmp, statePath)
|
|
138
|
+
}
|
|
139
|
+
|
|
140
|
+
async function apiJson(apiBaseUrl, accessToken, path, { method = 'GET', body } = {}) {
|
|
141
|
+
const res = await fetch(new URL(path, apiBaseUrl), {
|
|
142
|
+
method,
|
|
143
|
+
headers: {
|
|
144
|
+
Authorization: `Bearer ${await resolveToken(accessToken)}`,
|
|
145
|
+
...(body ? { 'Content-Type': 'application/json' } : {}),
|
|
146
|
+
},
|
|
147
|
+
body: body ? JSON.stringify(body) : undefined,
|
|
148
|
+
})
|
|
149
|
+
const json = await res.json().catch(() => ({}))
|
|
150
|
+
if (!res.ok) {
|
|
151
|
+
const err = httpError(`${method} ${path} failed (HTTP ${res.status}): ${JSON.stringify(json)}`, res.status, json)
|
|
152
|
+
err.retryAfterMs = Number(res.headers.get('retry-after')) * 1000 || null
|
|
153
|
+
throw err
|
|
154
|
+
}
|
|
155
|
+
return json
|
|
156
|
+
}
|
|
157
|
+
|
|
158
|
+
export function isTransient(err) {
|
|
159
|
+
if (err?.status === undefined) {
|
|
160
|
+
// Network / transport failures only (fetch throws TypeError 'fetch failed').
|
|
161
|
+
return (
|
|
162
|
+
err?.name === 'TypeError' ||
|
|
163
|
+
err?.finalizeOutcomeUnknown === true ||
|
|
164
|
+
/fetch failed|ECONNRESET|ETIMEDOUT|EPIPE|ENOTFOUND|EAI_AGAIN|socket|network/i.test(err?.message ?? '')
|
|
165
|
+
)
|
|
166
|
+
}
|
|
167
|
+
return err.status >= 500 || err.status === 408 || err.status === 429
|
|
168
|
+
}
|
|
169
|
+
|
|
170
|
+
// A refusal that every remaining file would meet too: stop the run.
|
|
171
|
+
export function isRunFatal(err) {
|
|
172
|
+
return (
|
|
173
|
+
err?.status === 401 ||
|
|
174
|
+
err?.status === 402 ||
|
|
175
|
+
err?.status === 412 ||
|
|
176
|
+
(err?.status === 413 && err?.serverCode !== 'too_many_parts') ||
|
|
177
|
+
(err?.status === 403 && /mfa|step/i.test(err?.message ?? ''))
|
|
178
|
+
)
|
|
179
|
+
}
|
|
180
|
+
|
|
181
|
+
async function withRetry(fn, { onRetry = () => {}, attempts = MAX_FILE_ATTEMPTS } = {}) {
|
|
182
|
+
for (let attempt = 1; ; attempt += 1) {
|
|
183
|
+
try {
|
|
184
|
+
return await fn(attempt)
|
|
185
|
+
} catch (err) {
|
|
186
|
+
if (attempt >= attempts || !isTransient(err) || isRunFatal(err)) throw err
|
|
187
|
+
const delay = Math.min(err.retryAfterMs ?? RETRY_BASE_MS * 2 ** (attempt - 1), RETRY_MAX_MS)
|
|
188
|
+
onRetry(err, attempt, delay)
|
|
189
|
+
await sleep(delay)
|
|
190
|
+
}
|
|
191
|
+
}
|
|
192
|
+
}
|
|
193
|
+
|
|
194
|
+
function flattenTree(nodes, out = new Map()) {
|
|
195
|
+
for (const n of nodes ?? []) {
|
|
196
|
+
out.set(n.id, n)
|
|
197
|
+
flattenTree(n.children, out)
|
|
198
|
+
}
|
|
199
|
+
return out
|
|
200
|
+
}
|
|
201
|
+
|
|
202
|
+
// The folder's key, unwrapped down the chain from the root key; null when the
|
|
203
|
+
// folder is gone or sits in the Bin (a binned folder is not a place to resume
|
|
204
|
+
// into: the user threw it away).
|
|
205
|
+
function folderKeyFor(id, byId, rootKeyB64, cache) {
|
|
206
|
+
if (cache.has(id)) return cache.get(id)
|
|
207
|
+
const node = byId.get(id)
|
|
208
|
+
let key = null
|
|
209
|
+
if (node && !node.is_bin && node.fk_wrapped && node.fk_iv) {
|
|
210
|
+
const parentKey = node.parent_id ? folderKeyFor(node.parent_id, byId, rootKeyB64, cache) : rootKeyB64
|
|
211
|
+
if (parentKey) {
|
|
212
|
+
try {
|
|
213
|
+
key = unwrapKeyB64({ wrappingKeyB64: parentKey, wrappedKeyB64: node.fk_wrapped, ivB64: node.fk_iv })
|
|
214
|
+
} catch {
|
|
215
|
+
key = null
|
|
216
|
+
}
|
|
217
|
+
}
|
|
218
|
+
}
|
|
219
|
+
cache.set(id, key)
|
|
220
|
+
return key
|
|
221
|
+
}
|
|
222
|
+
|
|
223
|
+
// Build the create request once; retries resend the same id and body, and the
|
|
224
|
+
// server answers a repeated create whose first answer was lost with the folder
|
|
225
|
+
// it already made, so a retry never makes a second folder.
|
|
226
|
+
async function prepareFolder({ rootKeyB64, name, parentId, parentKeyB64 }) {
|
|
227
|
+
const id = newRowBoundId()
|
|
228
|
+
const envelope = await encryptNameV6({ name, folderKey: base64ToBytes(parentKeyB64), rowId: id })
|
|
229
|
+
const folderKey = generateRandomKeyB64()
|
|
230
|
+
const wrapped = wrapKeyB64({ wrappingKeyB64: parentKeyB64, keyToWrapB64: folderKey })
|
|
231
|
+
const body = {
|
|
232
|
+
id,
|
|
233
|
+
name_encrypted: JSON.stringify(envelope),
|
|
234
|
+
nameHash: hashMetadataV4(name.toLowerCase(), rootKeyB64),
|
|
235
|
+
parentFolderId: parentId,
|
|
236
|
+
fk_wrapped: wrapped.wrapped,
|
|
237
|
+
fk_iv: wrapped.iv,
|
|
238
|
+
}
|
|
239
|
+
return { id, key: folderKey, body }
|
|
240
|
+
}
|
|
241
|
+
|
|
242
|
+
async function sendFolder({ apiBaseUrl, accessToken, prepared }) {
|
|
243
|
+
const res = await apiJson(apiBaseUrl, accessToken, '/api/folders/create', { method: 'POST', body: prepared.body })
|
|
244
|
+
if (res?.folder?.id !== prepared.id) throw new Error(`folders/create returned ${res?.folder?.id} for ${prepared.id}`)
|
|
245
|
+
return { id: prepared.id, key: prepared.key }
|
|
246
|
+
}
|
|
247
|
+
|
|
248
|
+
async function verifyReady({ apiBaseUrl, accessToken, ids }) {
|
|
249
|
+
const ready = new Set()
|
|
250
|
+
for (let i = 0; i < ids.length; i += VERIFY_BATCH) {
|
|
251
|
+
const batch = ids.slice(i, i + VERIFY_BATCH)
|
|
252
|
+
const res = await withRetry(() =>
|
|
253
|
+
apiJson(apiBaseUrl, accessToken, '/api/mobile/media/verify', { method: 'POST', body: { fileIds: batch } }),
|
|
254
|
+
)
|
|
255
|
+
for (const id of res.verified ?? []) ready.add(id)
|
|
256
|
+
}
|
|
257
|
+
return ready
|
|
258
|
+
}
|
|
259
|
+
|
|
260
|
+
const fmtBytes = (n) => {
|
|
261
|
+
const u = ['B', 'KB', 'MB', 'GB', 'TB']
|
|
262
|
+
let i = 0
|
|
263
|
+
while (n >= 1000 && i < u.length - 1) {
|
|
264
|
+
n /= 1000
|
|
265
|
+
i += 1
|
|
266
|
+
}
|
|
267
|
+
return `${i ? n.toFixed(1) : n} ${u[i]}`
|
|
268
|
+
}
|
|
269
|
+
|
|
270
|
+
/**
|
|
271
|
+
* Push `source` into the vault. Returns a summary; never throws for a single
|
|
272
|
+
* file's failure (those are in summary.failed), only for run-level problems.
|
|
273
|
+
*/
|
|
274
|
+
export async function runPush({
|
|
275
|
+
source,
|
|
276
|
+
apiBaseUrl,
|
|
277
|
+
accessToken,
|
|
278
|
+
rootKey,
|
|
279
|
+
account,
|
|
280
|
+
statePath = defaultStatePath(source),
|
|
281
|
+
includeDotfiles = false,
|
|
282
|
+
concurrency = resolvePushConcurrency(),
|
|
283
|
+
uploadFn = uploadFile,
|
|
284
|
+
log = (m) => process.stdout.write(`${m}\n`),
|
|
285
|
+
status = (m) => process.stderr.write(`${m}\n`),
|
|
286
|
+
signal,
|
|
287
|
+
progressEveryMs = 10_000,
|
|
288
|
+
}) {
|
|
289
|
+
source = resolve(source)
|
|
290
|
+
const rootKeyB64 = bytesToBase64(rootKey)
|
|
291
|
+
const scan = await scanTree(source, { includeDotfiles })
|
|
292
|
+
const totalBytes = scan.files.reduce((s, f) => s + f.size, 0)
|
|
293
|
+
status(
|
|
294
|
+
`found ${scan.files.length} files (${fmtBytes(totalBytes)}) in ${scan.dirs.length} folders` +
|
|
295
|
+
(scan.skipped.dotfiles.length ? `; skipping ${scan.skipped.dotfiles.length} hidden (use --include-dotfiles to send them)` : '') +
|
|
296
|
+
(scan.skipped.symlinks.length ? `; skipping ${scan.skipped.symlinks.length} symlinks` : '') +
|
|
297
|
+
(scan.skipped.other.length ? `; skipping ${scan.skipped.other.length} special files` : ''),
|
|
298
|
+
)
|
|
299
|
+
for (const p of scan.skipped.symlinks) log(`skipped ${p}: symbolic link (not followed)`)
|
|
300
|
+
for (const p of scan.skipped.other) log(`skipped ${p}: not a regular file`)
|
|
301
|
+
|
|
302
|
+
const state = await loadState(statePath, { account, source })
|
|
303
|
+
const save = (() => {
|
|
304
|
+
let chain = Promise.resolve()
|
|
305
|
+
// Serialised; one failed write (disk full) fails its caller but does not
|
|
306
|
+
// poison every later save.
|
|
307
|
+
return () => (chain = chain.catch(() => {}).then(() => saveState(statePath, state)))
|
|
308
|
+
})()
|
|
309
|
+
|
|
310
|
+
// Folders: resume into the ones recorded, create the rest.
|
|
311
|
+
const tree = await withRetry(() => apiJson(apiBaseUrl, accessToken, '/api/folders/tree'))
|
|
312
|
+
const byId = flattenTree(tree.tree)
|
|
313
|
+
const keyCache = new Map()
|
|
314
|
+
const folderIds = new Map() // rel dir ('' = top) -> { id, key }
|
|
315
|
+
|
|
316
|
+
const ensure = async (rel, name, parent) => {
|
|
317
|
+
const recorded = rel === '' ? state.rootFolderId : state.folders[rel]
|
|
318
|
+
const node = recorded ? byId.get(recorded) : null
|
|
319
|
+
if (node && (node.parent_id ?? null) === (parent?.id ?? null)) {
|
|
320
|
+
const key = folderKeyFor(recorded, byId, rootKeyB64, keyCache)
|
|
321
|
+
if (key) {
|
|
322
|
+
folderIds.set(rel, { id: recorded, key })
|
|
323
|
+
return false
|
|
324
|
+
}
|
|
325
|
+
}
|
|
326
|
+
const prepared = await prepareFolder({
|
|
327
|
+
rootKeyB64,
|
|
328
|
+
name: vaultName(name),
|
|
329
|
+
parentId: parent?.id ?? null,
|
|
330
|
+
parentKeyB64: parent?.key ?? rootKeyB64,
|
|
331
|
+
})
|
|
332
|
+
const made = await withRetry(
|
|
333
|
+
() => sendFolder({ apiBaseUrl, accessToken, prepared }),
|
|
334
|
+
{ onRetry: (e, n, d) => status(`creating folder ${rel || name}: ${e.message}; retrying in ${Math.round(d / 1000)} s`) },
|
|
335
|
+
)
|
|
336
|
+
folderIds.set(rel, made)
|
|
337
|
+
if (rel === '') {
|
|
338
|
+
state.rootFolderId = made.id
|
|
339
|
+
// A new top folder means nothing recorded under the old one is there.
|
|
340
|
+
state.folders = {}
|
|
341
|
+
state.files = {}
|
|
342
|
+
state.pending = {}
|
|
343
|
+
} else {
|
|
344
|
+
state.folders[rel] = made.id
|
|
345
|
+
// The folder was (re)made, so files recorded in the old one are not in
|
|
346
|
+
// this one: forget them so they upload again.
|
|
347
|
+
for (const key of Object.keys(state.files)) {
|
|
348
|
+
const cut = key.lastIndexOf('/')
|
|
349
|
+
if ((cut === -1 ? '' : key.slice(0, cut)) === rel) delete state.files[key]
|
|
350
|
+
}
|
|
351
|
+
}
|
|
352
|
+
await save()
|
|
353
|
+
return true
|
|
354
|
+
}
|
|
355
|
+
|
|
356
|
+
const topCreated = await ensure('', basename(source), null)
|
|
357
|
+
status(`${topCreated ? 'created' : 'resuming into'} vault folder "${vaultName(basename(source))}"`)
|
|
358
|
+
let foldersCreated = 0
|
|
359
|
+
for (const rel of scan.dirs) {
|
|
360
|
+
const cut = rel.lastIndexOf('/')
|
|
361
|
+
const parent = folderIds.get(cut === -1 ? '' : rel.slice(0, cut))
|
|
362
|
+
if (await ensure(rel, cut === -1 ? rel : rel.slice(cut + 1), parent)) foldersCreated += 1
|
|
363
|
+
}
|
|
364
|
+
|
|
365
|
+
// Settle uploads a previous run started but may not have seen finish.
|
|
366
|
+
const pendingEntries = Object.entries(state.pending)
|
|
367
|
+
if (pendingEntries.length) {
|
|
368
|
+
const ready = await verifyReady({ apiBaseUrl, accessToken, ids: pendingEntries.map(([, p]) => p.fileId) })
|
|
369
|
+
for (const [rel, p] of pendingEntries) {
|
|
370
|
+
if (ready.has(p.fileId)) state.files[rel] = { signature: p.signature, fileId: p.fileId }
|
|
371
|
+
delete state.pending[rel]
|
|
372
|
+
}
|
|
373
|
+
await save()
|
|
374
|
+
}
|
|
375
|
+
|
|
376
|
+
const summary = { uploaded: 0, alreadyThere: 0, failed: [], changed: [], skippedEmpty: [], bytesUploaded: 0, foldersCreated, stoppedBy: null, interrupted: false }
|
|
377
|
+
const todo = []
|
|
378
|
+
for (const f of scan.files) {
|
|
379
|
+
const done = state.files[f.rel]
|
|
380
|
+
if (done && done.signature === signatureOf(f)) summary.alreadyThere += 1
|
|
381
|
+
else todo.push(f)
|
|
382
|
+
}
|
|
383
|
+
if (summary.alreadyThere) status(`${summary.alreadyThere} files already uploaded by an earlier run; skipping them`)
|
|
384
|
+
const todoBytes = todo.reduce((s, f) => s + f.size, 0)
|
|
385
|
+
status(`uploading ${todo.length} files (${fmtBytes(todoBytes)}), ${concurrency} at a time`)
|
|
386
|
+
|
|
387
|
+
let inFlightBytes = 0
|
|
388
|
+
let completed = 0
|
|
389
|
+
const started = Date.now()
|
|
390
|
+
const timer = progressEveryMs
|
|
391
|
+
? setInterval(() => {
|
|
392
|
+
const secs = (Date.now() - started) / 1000
|
|
393
|
+
const sent = summary.bytesUploaded + inFlightBytes
|
|
394
|
+
status(
|
|
395
|
+
`progress: ${completed}/${todo.length} files, ${fmtBytes(sent)} of ${fmtBytes(todoBytes)}` +
|
|
396
|
+
(secs > 0 ? `, ${fmtBytes(sent / secs)}/s` : ''),
|
|
397
|
+
)
|
|
398
|
+
}, progressEveryMs)
|
|
399
|
+
: null
|
|
400
|
+
timer?.unref?.()
|
|
401
|
+
|
|
402
|
+
let next = 0
|
|
403
|
+
const stopping = () => summary.stoppedBy || signal?.aborted
|
|
404
|
+
|
|
405
|
+
const uploadOne = async (f) => {
|
|
406
|
+
const before = await lstat(f.path).catch(() => null)
|
|
407
|
+
if (!before?.isFile()) {
|
|
408
|
+
summary.failed.push({ rel: f.rel, message: 'file disappeared or is no longer a regular file' })
|
|
409
|
+
log(`failed ${f.rel}: file disappeared or is no longer a regular file`)
|
|
410
|
+
return
|
|
411
|
+
}
|
|
412
|
+
const entry = { ...f, size: before.size, mtimeMs: before.mtimeMs }
|
|
413
|
+
const folder = folderIds.get(f.dir)
|
|
414
|
+
let progressed = 0
|
|
415
|
+
try {
|
|
416
|
+
await withRetry(
|
|
417
|
+
async () => {
|
|
418
|
+
const fileId = newRowBoundId()
|
|
419
|
+
state.pending[f.rel] = { fileId, signature: signatureOf(entry) }
|
|
420
|
+
await save()
|
|
421
|
+
progressed = 0
|
|
422
|
+
try {
|
|
423
|
+
await uploadFn({
|
|
424
|
+
apiBaseUrl,
|
|
425
|
+
accessToken,
|
|
426
|
+
rootKey,
|
|
427
|
+
name: vaultName(f.name),
|
|
428
|
+
path: f.path,
|
|
429
|
+
size: entry.size,
|
|
430
|
+
rowId: fileId,
|
|
431
|
+
folderId: folder.id,
|
|
432
|
+
folderKey: base64ToBytes(folder.key),
|
|
433
|
+
onPlaintextChunk: (c) => {
|
|
434
|
+
progressed += c.length
|
|
435
|
+
inFlightBytes += c.length
|
|
436
|
+
},
|
|
437
|
+
})
|
|
438
|
+
} catch (err) {
|
|
439
|
+
inFlightBytes -= progressed
|
|
440
|
+
progressed = 0
|
|
441
|
+
// Finalize may have landed even though its answer was lost.
|
|
442
|
+
const landed =
|
|
443
|
+
err?.finalizeOutcomeUnknown &&
|
|
444
|
+
(await verifyReady({ apiBaseUrl, accessToken, ids: [fileId] }).catch(() => new Set())).has(fileId)
|
|
445
|
+
if (!landed) {
|
|
446
|
+
// Keep the pending record when we cannot tell: the next run asks again.
|
|
447
|
+
if (!err?.finalizeOutcomeUnknown) {
|
|
448
|
+
delete state.pending[f.rel]
|
|
449
|
+
await save()
|
|
450
|
+
}
|
|
451
|
+
throw err
|
|
452
|
+
}
|
|
453
|
+
}
|
|
454
|
+
inFlightBytes -= progressed
|
|
455
|
+
state.files[f.rel] = { signature: signatureOf(entry), fileId }
|
|
456
|
+
delete state.pending[f.rel]
|
|
457
|
+
await save()
|
|
458
|
+
},
|
|
459
|
+
{
|
|
460
|
+
onRetry: (e, n, d) =>
|
|
461
|
+
status(`${f.rel}: ${e.message.slice(0, 200)}; retry ${n} in ${Math.round(d / 1000)} s`),
|
|
462
|
+
},
|
|
463
|
+
)
|
|
464
|
+
} catch (err) {
|
|
465
|
+
if (err instanceof EmptyFileUnsupportedError) {
|
|
466
|
+
summary.skippedEmpty.push(f.rel)
|
|
467
|
+
log(`skipped ${f.rel}: empty file, and this server does not accept empty files`)
|
|
468
|
+
return
|
|
469
|
+
}
|
|
470
|
+
if (isRunFatal(err)) {
|
|
471
|
+
summary.stoppedBy ??= err
|
|
472
|
+
return
|
|
473
|
+
}
|
|
474
|
+
summary.failed.push({ rel: f.rel, message: err.message })
|
|
475
|
+
log(`failed ${f.rel}: ${err.message}`)
|
|
476
|
+
return
|
|
477
|
+
}
|
|
478
|
+
summary.uploaded += 1
|
|
479
|
+
summary.bytesUploaded += entry.size
|
|
480
|
+
completed += 1
|
|
481
|
+
const after = await lstat(f.path).catch(() => null)
|
|
482
|
+
if (!after || signatureOf(after) !== signatureOf(entry)) {
|
|
483
|
+
// What was read is in the vault; the file on disk is different now. Do
|
|
484
|
+
// not record it as done, so the next run uploads the new version.
|
|
485
|
+
delete state.files[f.rel]
|
|
486
|
+
await save()
|
|
487
|
+
summary.changed.push(f.rel)
|
|
488
|
+
log(`uploaded ${f.rel}, but it changed while uploading; run again to upload the new version`)
|
|
489
|
+
} else {
|
|
490
|
+
log(`[${completed}/${todo.length}] uploaded ${f.rel} (${fmtBytes(entry.size)})`)
|
|
491
|
+
}
|
|
492
|
+
}
|
|
493
|
+
|
|
494
|
+
const worker = async () => {
|
|
495
|
+
while (!stopping() && next < todo.length) {
|
|
496
|
+
const f = todo[next++]
|
|
497
|
+
await uploadOne(f)
|
|
498
|
+
}
|
|
499
|
+
}
|
|
500
|
+
try {
|
|
501
|
+
await Promise.all(Array.from({ length: Math.min(concurrency, todo.length || 1) }, worker))
|
|
502
|
+
} finally {
|
|
503
|
+
if (timer) clearInterval(timer)
|
|
504
|
+
}
|
|
505
|
+
summary.interrupted = Boolean(signal?.aborted)
|
|
506
|
+
summary.notStarted = todo.length - next
|
|
507
|
+
return summary
|
|
508
|
+
}
|
|
509
|
+
|
|
510
|
+
export const _internals = { flattenTree, folderKeyFor, signatureOf, relative, sep }
|
package/src/upload.mjs
CHANGED
|
@@ -50,12 +50,29 @@ const CAPABILITIES_PATH = '/api/files/upload-capabilities'
|
|
|
50
50
|
// part upload is idempotent by (part number, SHA-1) — retrying overwrites the
|
|
51
51
|
// same part. Bound the retries and back off so one transient blip does not
|
|
52
52
|
// abort a whole multi-gigabyte upload.
|
|
53
|
-
const MAX_PART_ATTEMPTS =
|
|
54
|
-
const PART_RETRY_BASE_DELAY_MS =
|
|
53
|
+
const MAX_PART_ATTEMPTS = 5
|
|
54
|
+
const PART_RETRY_BASE_DELAY_MS = 1000
|
|
55
55
|
|
|
56
56
|
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms))
|
|
57
57
|
const partRetryDelayMs = (attempt) =>
|
|
58
|
-
PART_RETRY_BASE_DELAY_MS * 2 ** (attempt - 1) //
|
|
58
|
+
PART_RETRY_BASE_DELAY_MS * 2 ** (attempt - 1) // 1s, 2s, 4s, 8s
|
|
59
|
+
|
|
60
|
+
// `accessToken` is a string, or a function returning the current one. A
|
|
61
|
+
// function is resolved at every request, so an upload that outlives the access
|
|
62
|
+
// token (about an hour; a multi-gigabyte file can) picks up a refreshed session
|
|
63
|
+
// for its later part URLs and its finalize.
|
|
64
|
+
export async function resolveToken(accessToken) {
|
|
65
|
+
return typeof accessToken === 'function' ? accessToken() : accessToken
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
// An Error that keeps the HTTP status and the server's error code, so callers
|
|
69
|
+
// can tell a transient failure (5xx, 429) from a terminal one (quota, 4xx).
|
|
70
|
+
export function httpError(message, status, body) {
|
|
71
|
+
const err = new Error(message)
|
|
72
|
+
err.status = status
|
|
73
|
+
if (body && typeof body === 'object' && typeof body.code === 'string') err.serverCode = body.code
|
|
74
|
+
return err
|
|
75
|
+
}
|
|
59
76
|
|
|
60
77
|
// Read a file in fixed-size chunks without holding it all in memory. Each yield
|
|
61
78
|
// is an independent copy (the read buffer is reused), so the caller may retain
|
|
@@ -120,22 +137,30 @@ export async function createUploadSession({
|
|
|
120
137
|
name,
|
|
121
138
|
size,
|
|
122
139
|
rowId,
|
|
140
|
+
folderId = null,
|
|
141
|
+
folderKey = null,
|
|
123
142
|
}) {
|
|
124
143
|
const rootKeyB64 = bytesToBase64(rootKey)
|
|
144
|
+
// The parent's key: the root key at the vault root, else the folder's own
|
|
145
|
+
// key. The name is sealed under it and the envelope key wrapped by it,
|
|
146
|
+
// exactly as the web uploader does for a file inside a folder. The name hash
|
|
147
|
+
// is always keyed by the root key (web hashMetadataClient(name, rootKey)).
|
|
148
|
+
const parentKey = folderKey ?? rootKey
|
|
149
|
+
const parentKeyB64 = bytesToBase64(parentKey)
|
|
125
150
|
// The client picks the row id (UUIDv7, sent as clientUploadId), so the very
|
|
126
151
|
// first stored name is already the v6 envelope bound to that row. No v4
|
|
127
152
|
// (row-unbound) name is ever written, even for an upload that fails before
|
|
128
153
|
// finalize.
|
|
129
|
-
const nameEnvelopeV6 = await encryptNameV6({ name, folderKey:
|
|
154
|
+
const nameEnvelopeV6 = await encryptNameV6({ name, folderKey: parentKey, rowId })
|
|
130
155
|
// For suite 0x03 this is the classical envelope key; csk_wrapped holds it
|
|
131
|
-
// under the parent key
|
|
156
|
+
// under the parent key.
|
|
132
157
|
const csk = generateRandomKeyB64()
|
|
133
|
-
const cskEnvelope = wrapKeyB64({ wrappingKeyB64:
|
|
158
|
+
const cskEnvelope = wrapKeyB64({ wrappingKeyB64: parentKeyB64, keyToWrapB64: csk })
|
|
134
159
|
|
|
135
160
|
const res = await fetch(new URL(CREATE_SESSION_PATH, apiBaseUrl), {
|
|
136
161
|
method: 'POST',
|
|
137
162
|
headers: {
|
|
138
|
-
Authorization: `Bearer ${accessToken}`,
|
|
163
|
+
Authorization: `Bearer ${await resolveToken(accessToken)}`,
|
|
139
164
|
'Content-Type': 'application/json',
|
|
140
165
|
},
|
|
141
166
|
// v1 wire format: no cipher_nonce_prefix and no chunkSize (the server
|
|
@@ -148,7 +173,7 @@ export async function createUploadSession({
|
|
|
148
173
|
csk_iv: cskEnvelope.iv,
|
|
149
174
|
cipherVersion: CIPHER_VERSION_PQ,
|
|
150
175
|
sizeBytes: size,
|
|
151
|
-
folderId
|
|
176
|
+
folderId,
|
|
152
177
|
contentType: 'application/octet-stream',
|
|
153
178
|
thumbnail_ciphertext_b64: null,
|
|
154
179
|
thumbnail_nonce_prefix: null,
|
|
@@ -156,8 +181,10 @@ export async function createUploadSession({
|
|
|
156
181
|
})
|
|
157
182
|
const body = await res.json().catch(() => ({}))
|
|
158
183
|
if (!res.ok) {
|
|
159
|
-
throw
|
|
184
|
+
throw httpError(
|
|
160
185
|
`create-upload-session failed (HTTP ${res.status}): ${JSON.stringify(body)}`,
|
|
186
|
+
res.status,
|
|
187
|
+
body,
|
|
161
188
|
)
|
|
162
189
|
}
|
|
163
190
|
return { session: body, csk, nameEncryptedV6: JSON.stringify(nameEnvelopeV6) }
|
|
@@ -178,18 +205,34 @@ export async function createUploadSession({
|
|
|
178
205
|
// Returns nothing. A presigned PUT answers with an ETag, not a Backblaze file
|
|
179
206
|
// id, so there is no storage id to hand to finalize.
|
|
180
207
|
async function putDirectPresigned({ session, ciphertext }) {
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
208
|
+
// Retried on network errors and 5xx / 408 / 429: the presigned URL names one
|
|
209
|
+
// object key, so a repeated PUT overwrites the same object with the same bytes.
|
|
210
|
+
let lastError
|
|
211
|
+
for (let tries = 1; tries <= MAX_PART_ATTEMPTS; tries += 1) {
|
|
212
|
+
let res
|
|
213
|
+
try {
|
|
214
|
+
res = await fetch(session.uploadUrl, {
|
|
215
|
+
method: 'PUT',
|
|
216
|
+
headers: {
|
|
217
|
+
'Content-Type': session.contentType || 'application/octet-stream',
|
|
218
|
+
},
|
|
219
|
+
body: ciphertext,
|
|
220
|
+
})
|
|
221
|
+
} catch (err) {
|
|
222
|
+
lastError = err
|
|
223
|
+
if (tries < MAX_PART_ATTEMPTS) await sleep(partRetryDelayMs(tries))
|
|
224
|
+
continue
|
|
225
|
+
}
|
|
226
|
+
if (res.ok) return
|
|
227
|
+
lastError = httpError(
|
|
190
228
|
`Direct upload failed (HTTP ${res.status}): ${await res.text().catch(() => '')}`,
|
|
229
|
+
res.status,
|
|
191
230
|
)
|
|
231
|
+
const retryable = res.status >= 500 || res.status === 429 || res.status === 408
|
|
232
|
+
if (!retryable) throw lastError
|
|
233
|
+
if (tries < MAX_PART_ATTEMPTS) await sleep(partRetryDelayMs(tries))
|
|
192
234
|
}
|
|
235
|
+
throw lastError
|
|
193
236
|
}
|
|
194
237
|
|
|
195
238
|
// Ask the server for a fresh b2_upload_part URL + token when the current one
|
|
@@ -198,15 +241,17 @@ async function refreshUploadPartUrl({ apiBaseUrl, accessToken, fileId }) {
|
|
|
198
241
|
const res = await fetch(new URL(PART_URL_PATH, apiBaseUrl), {
|
|
199
242
|
method: 'POST',
|
|
200
243
|
headers: {
|
|
201
|
-
Authorization: `Bearer ${accessToken}`,
|
|
244
|
+
Authorization: `Bearer ${await resolveToken(accessToken)}`,
|
|
202
245
|
'Content-Type': 'application/json',
|
|
203
246
|
},
|
|
204
247
|
body: JSON.stringify({ fileId }),
|
|
205
248
|
})
|
|
206
249
|
const body = await res.json().catch(() => ({}))
|
|
207
250
|
if (!res.ok || !body.uploadUrl || !body.authToken) {
|
|
208
|
-
throw
|
|
251
|
+
throw httpError(
|
|
209
252
|
`upload-part-url failed (HTTP ${res.status}): ${JSON.stringify(body)}`,
|
|
253
|
+
res.status,
|
|
254
|
+
body,
|
|
210
255
|
)
|
|
211
256
|
}
|
|
212
257
|
return { uploadUrl: body.uploadUrl, authToken: body.authToken }
|
|
@@ -273,9 +318,10 @@ async function putPartToBackblaze({
|
|
|
273
318
|
|
|
274
319
|
if (res.ok) return { uploadUrl: url, authToken: token }
|
|
275
320
|
|
|
276
|
-
lastError =
|
|
321
|
+
lastError = httpError(
|
|
277
322
|
`Backblaze part ${partNumber} upload failed (HTTP ${res.status}): ` +
|
|
278
323
|
`${await res.text().catch(() => '')}`,
|
|
324
|
+
res.status,
|
|
279
325
|
)
|
|
280
326
|
|
|
281
327
|
// 5xx / 408 / 429 are worth retrying; other 4xx (after the auth refresh
|
|
@@ -310,7 +356,7 @@ async function finalizeUpload({
|
|
|
310
356
|
res = await fetch(new URL(COMPLETE_PATH, apiBaseUrl), {
|
|
311
357
|
method: 'POST',
|
|
312
358
|
headers: {
|
|
313
|
-
Authorization: `Bearer ${accessToken}`,
|
|
359
|
+
Authorization: `Bearer ${await resolveToken(accessToken)}`,
|
|
314
360
|
'Content-Type': 'application/json',
|
|
315
361
|
},
|
|
316
362
|
body: JSON.stringify({ fileId, b2FileId, partSha1Array, proof, ciphertextHash, nameEncryptedV6 }),
|
|
@@ -323,8 +369,10 @@ async function finalizeUpload({
|
|
|
323
369
|
}
|
|
324
370
|
const body = await res.json().catch(() => ({}))
|
|
325
371
|
if (!res.ok) {
|
|
326
|
-
throw
|
|
372
|
+
throw httpError(
|
|
327
373
|
`complete-upload failed (HTTP ${res.status}): ${JSON.stringify(body)}`,
|
|
374
|
+
res.status,
|
|
375
|
+
body,
|
|
328
376
|
)
|
|
329
377
|
}
|
|
330
378
|
return body
|
|
@@ -498,7 +546,7 @@ export async function cancelUploadSession({ apiBaseUrl, accessToken, fileId }) {
|
|
|
498
546
|
const res = await fetch(new URL(DELETE_PATH, apiBaseUrl), {
|
|
499
547
|
method: 'POST',
|
|
500
548
|
headers: {
|
|
501
|
-
Authorization: `Bearer ${accessToken}`,
|
|
549
|
+
Authorization: `Bearer ${await resolveToken(accessToken)}`,
|
|
502
550
|
'Content-Type': 'application/json',
|
|
503
551
|
},
|
|
504
552
|
body: JSON.stringify({ ids: [fileId] }),
|
|
@@ -526,7 +574,7 @@ export class EmptyFileUnsupportedError extends Error {
|
|
|
526
574
|
async function serverAcceptsEmptyFiles({ apiBaseUrl, accessToken }) {
|
|
527
575
|
try {
|
|
528
576
|
const res = await fetch(new URL(CAPABILITIES_PATH, apiBaseUrl), {
|
|
529
|
-
headers: { Authorization: `Bearer ${accessToken}` },
|
|
577
|
+
headers: { Authorization: `Bearer ${await resolveToken(accessToken)}` },
|
|
530
578
|
})
|
|
531
579
|
if (!res.ok) return false
|
|
532
580
|
const body = await res.json().catch(() => ({}))
|
|
@@ -541,7 +589,7 @@ async function serverAcceptsEmptyFiles({ apiBaseUrl, accessToken }) {
|
|
|
541
589
|
// chunks to record what was actually uploaded; hashing the file separately,
|
|
542
590
|
// before or after, cannot see a change made and undone while it was being read.
|
|
543
591
|
// `rowId` is for tests only; callers let it default to a fresh UUIDv7.
|
|
544
|
-
export async function uploadFile({ apiBaseUrl, accessToken, rootKey, name, path, size, onPlaintextChunk, rowId = newRowBoundId() }) {
|
|
592
|
+
export async function uploadFile({ apiBaseUrl, accessToken, rootKey, name, path, size, onPlaintextChunk, rowId = newRowBoundId(), folderId = null, folderKey = null }) {
|
|
545
593
|
// Derived before anything is sent, so a vault whose identity cannot be
|
|
546
594
|
// derived fails before a session (and a pending row) exists.
|
|
547
595
|
const recipientPublicKey = await vaultMlKemPublicKey(rootKey)
|
|
@@ -555,6 +603,8 @@ export async function uploadFile({ apiBaseUrl, accessToken, rootKey, name, path,
|
|
|
555
603
|
name,
|
|
556
604
|
size,
|
|
557
605
|
rowId,
|
|
606
|
+
folderId,
|
|
607
|
+
folderKey,
|
|
558
608
|
})
|
|
559
609
|
// From here on a pending row exists. Any failure before finalize answers
|
|
560
610
|
// cancels it (see cancelUploadSession).
|