@shieldfive/cli 0.1.0 → 0.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md ADDED
@@ -0,0 +1,103 @@
1
+ # Changelog
2
+
3
+ All notable changes to `@shieldfive/cli` will be documented in this file.
4
+
5
+ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
6
+ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
+
8
+ ## 0.2.1 - 2026-09-14
9
+
10
+ ### Fixed
11
+
12
+ - **Files of 5 MiB or less could not be uploaded at all, since 2026-09-03.** The
13
+ server moved the direct-upload path to a presigned S3 PUT and stopped
14
+ returning `authToken` (web `b52b4c8`, PR #726); the client hard-required that
15
+ token and threw `Direct upload session is missing credentials.` before sending
16
+ a byte. The direct path now `PUT`s the ciphertext straight at the presigned
17
+ URL with no `Authorization` and no `X-Bz-*` headers, and sends no `b2FileId`
18
+ at finalize — a presigned PUT returns an ETag, not a Backblaze file id, and
19
+ forwarding it would have been persisted verbatim as the storage id. Files
20
+ larger than 5 MiB were never affected; the multipart path is unchanged and
21
+ keeps its token, its `X-Bz-*` headers and its session-derived `b2FileId`.
22
+
23
+ ### Changed
24
+
25
+ - **The direct path no longer sends a wire checksum.** It previously sent
26
+ `X-Bz-Content-Sha1`, which Backblaze's native upload verified server-side and
27
+ rejected on mismatch. A presigned PUT signed for `host` only carries no
28
+ equivalent, and the server's finalize check compares the client's
29
+ `ciphertextHash` against the client's own `partSha1Array[0]` — both
30
+ client-supplied, so always equal. That protection is removed and not replaced:
31
+ integrity on this path now rests on the upload proof and on AES-GCM's tag at
32
+ download. Stated rather than left to be discovered.
33
+ - The direct path also no longer checks the upload response shape. It used to
34
+ require a `fileId` in the body; a presigned PUT returns none, so any 2xx is
35
+ now accepted and a PUT that answered 200 without durably storing the object
36
+ surfaces later as finalize's 409 rather than immediately.
37
+
38
+ ### Notes
39
+
40
+ - **The rewritten direct path has not been run against production.** It is
41
+ covered by the offline mock test only, which is the same class of evidence
42
+ that let 0.2.0 ship broken — the old fixture supplied an `authToken` the
43
+ server had stopped sending. The mock now matches the live response shape, and
44
+ four mutations of the client (re-adding `Authorization`, re-sending
45
+ `b2FileId`, reverting to `POST`, restoring the token requirement) each fail
46
+ the suite. That is a contract test, not a live one. One `sf push` of a file
47
+ under 5 MiB and one over it, confirmed visible and decryptable in the web app,
48
+ is still required.
49
+ - The direct path remains single-attempt: one `fetch`, no retry and no
50
+ signature refresh, where the multipart path retries. A transient 5xx or a
51
+ signature that expires mid-upload fails the file. Pre-existing, out of scope
52
+ here, and worth fixing before `sf sync --watch` is relied on.
53
+
54
+ ## 0.2.0 - 2026-07-01
55
+
56
+ ### Added
57
+
58
+ - **Two-factor authentication (TOTP).** Accounts with an authenticator app
59
+ enrolled now complete the Supabase AAL2 step-up during sign-in, which the
60
+ server's upload gate requires. After the password, the CLI prompts
61
+ `Two-factor code (6 digits):` (or reads `SF_TOTP_CODE` in non-interactive
62
+ contexts), runs the standard challenge/verify against the verified TOTP
63
+ factor, and uses the elevated token for uploads. Only TOTP is supported; SMS
64
+ and other factors are not. Accounts without 2FA are unaffected and are never
65
+ prompted.
66
+
67
+ ### Notes
68
+
69
+ - The step-up flow is covered by unit tests over a mocked Supabase client
70
+ (no-MFA, success, missing factor, malformed/wrong code, lookup failure). The
71
+ live challenge/verify round-trip against a real 2FA account has not yet been
72
+ exercised.
73
+
74
+ ## 0.1.0 - 2026-07-01
75
+
76
+ Initial public release.
77
+
78
+ ### Added
79
+
80
+ - `sf encrypt <folder>` - encrypt each file locally (post-quantum hybrid demo);
81
+ no account and no upload.
82
+ - `sf push <folder>` - sign in, unlock the vault, encrypt, and upload a folder.
83
+ Picks the direct or multipart path automatically from file size.
84
+ - `sf sync <folder> [--watch] [--interval=<seconds>]` - upload new and changed
85
+ files, once or on a poll loop, tracked by a local `.shieldfive-sync.json`
86
+ manifest (size + mtime) so unchanged files are skipped.
87
+ - Chunked AES-GCM file encryption with a server-verified HMAC upload proof.
88
+ - Streaming chunk reader so large files never load fully into memory; multipart
89
+ upload for files larger than 5 MiB.
90
+ - Baked-in ShieldFive public backend configuration, so only `SF_EMAIL` and
91
+ `SF_PASSWORD` are required (endpoints override via env for a dev backend).
92
+
93
+ ### Notes
94
+
95
+ - The cryptographic core (`@shieldfive/crypto`) has not had an external audit.
96
+ Early software.
97
+ - The direct (single-chunk) upload path was validated against production as the
98
+ backend stood on 2026-07-01. **That no longer holds:** the server moved the
99
+ direct path to a presigned PUT on 2026-09-03, and this claim was false from
100
+ that date until the fix in 0.2.1. The multipart path is verified
101
+ byte-for-byte against the server contract and tested end to end with mocks,
102
+ but has not yet been run live with a real large file.
103
+ - `sf sync` is append-only: it does not yet mirror local deletions or renames.
package/README.md CHANGED
@@ -27,9 +27,14 @@ size, uploading nothing.
27
27
 
28
28
  - The underlying cryptography ([`@shieldfive/crypto`](https://github.com/shieldfive/crypto))
29
29
  has **not** undergone an external audit yet. Treat this as early software.
30
- - The direct upload path (files up to one 5 MiB chunk) has been run end to end
31
- against the production backend: files uploaded by this CLI appear in the web
32
- app with their real names and decrypt correctly.
30
+ - **Neither upload path has been exercised against production since 2026-09-03.**
31
+ The direct (≤ 5 MiB) path was validated end to end against the backend as it
32
+ stood at 0.1.0, but the server then moved that path to a presigned S3 PUT and
33
+ stopped issuing an upload token. 0.2.0 hard-required that token, so it could
34
+ not upload any file of 5 MiB or less at all. This release rewrites the direct
35
+ path to the presigned PUT. It is covered by the offline mock test in
36
+ [`test/upload.flow.test.mjs`](test/upload.flow.test.mjs) and sends byte-for-byte
37
+ the request the web app sends, but it has not been run live.
33
38
  - The **multipart** path (files larger than 5 MiB) is covered by a byte-for-byte
34
39
  review against the server contract and a mock end-to-end test that decrypts
35
40
  every part, but has not yet been exercised against production with a real
@@ -42,13 +47,26 @@ size, uploading nothing.
42
47
  - Node.js >= 20
43
48
  - A ShieldFive account with a vault (create one in the web app first)
44
49
 
45
- Not yet published to npm. For now, clone and run:
50
+ ## Install
51
+
52
+ ```sh
53
+ npm install -g @shieldfive/cli # provides the `sf` command
54
+ ```
55
+
56
+ Or run it without installing:
57
+
58
+ ```sh
59
+ npx @shieldfive/cli --help
60
+ ```
61
+
62
+ To read and build from source (the whole point — verify what it does before you
63
+ trust it):
46
64
 
47
65
  ```sh
48
66
  git clone https://github.com/shieldfive/cli.git
49
67
  cd cli
50
68
  npm install # @shieldfive/crypto (+ libsodium), @supabase/supabase-js, @noble/*
51
- npm test # 27/27
69
+ npm test # 40/40
52
70
  ```
53
71
 
54
72
  ## Commands
@@ -81,11 +99,22 @@ sf sync ./my-folder --watch
81
99
 
82
100
  The backend endpoints default to ShieldFive, so there is nothing else to
83
101
  configure. `SF_API_BASE_URL`, `SF_SUPABASE_URL`, and `SF_SUPABASE_ANON_KEY` are
84
- baked in with ShieldFive's public values — the Supabase anon key is a `role=anon`
85
- JWT gated by row-level security, the same value any browser receives — and only
86
- need overriding if you point the CLI at a development backend. If two-factor
87
- authentication is enabled on your account, the MFA step-up is not yet handled by
88
- the CLI.
102
+ baked in with ShieldFive's public values (the Supabase anon key is a `role=anon`
103
+ JWT gated by row-level security, the same value any browser receives) and only
104
+ need overriding if you point the CLI at a development backend.
105
+
106
+ ### Two-factor authentication
107
+
108
+ If your account has an authenticator app (TOTP) enabled, the CLI performs the
109
+ AAL2 step-up automatically: after your password it prompts
110
+
111
+ ```
112
+ Two-factor code (6 digits):
113
+ ```
114
+
115
+ Enter the current code from your authenticator. In a non-interactive context
116
+ (scripts, CI) set `SF_TOTP_CODE` to the current code instead of being prompted.
117
+ Only TOTP is supported; SMS and other factors are not.
89
118
 
90
119
  ## How the upload works (for auditors)
91
120
 
@@ -107,6 +136,15 @@ protocol and server-side proof expect.)
107
136
  The server issues `proofKey` when it creates the session and verifies the proof
108
137
  when finalizing. It ties the stored ciphertext to the session without the
109
138
  server ever seeing plaintext.
139
+ - **Direct** (≤ 5 MiB) — the single encrypted chunk is `PUT` at a presigned S3
140
+ URL the server issues with the session. The URL's SigV4 signature is the only
141
+ credential and it authorises exactly one object key, so the request carries no
142
+ `Authorization` header and no `X-Bz-*` headers; the client sends no storage id
143
+ at finalize, because the server reads the real one off the stored object.
144
+ Because the signature covers `host` only, nothing on this path binds the body:
145
+ the server compares the client's `ciphertextHash` against the client's own
146
+ `partSha1Array[0]`, so they always agree. Integrity here rests on the upload
147
+ proof and on AES-GCM's tag at download, not on a wire checksum.
110
148
  - **Multipart** (> 5 MiB) — the file is streamed one 5 MiB chunk at a time (never
111
149
  loaded whole into memory), each chunk uploaded as a Backblaze part. The
112
150
  ciphertext hash the server checks is SHA-1 over the concatenated raw part
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@shieldfive/cli",
3
- "version": "0.1.0",
3
+ "version": "0.2.1",
4
4
  "type": "module",
5
5
  "description": "Open-source command-line client for ShieldFive: encrypt a folder on your machine and sync it to your EU-hosted, post-quantum vault. Read the code; it encrypts before upload.",
6
6
  "license": "Apache-2.0",
@@ -34,6 +34,7 @@
34
34
  "src",
35
35
  "README.md",
36
36
  "SECURITY.md",
37
+ "CHANGELOG.md",
37
38
  "LICENSE"
38
39
  ],
39
40
  "engines": {
package/src/auth.mjs CHANGED
@@ -1,16 +1,88 @@
1
1
  // ShieldFive CLI — authentication.
2
2
  //
3
- // Signs in via Supabase to obtain a Bearer access token. The web upload routes
4
- // now accept this token (Track A: cookie -> Bearer), so it is what `sf push`
5
- // will use. Same token shape the mobile app uses.
3
+ // Signs in via Supabase to obtain a Bearer access token, elevating to AAL2 when
4
+ // the account has two-factor authentication (TOTP) enabled. The upload routes
5
+ // accept this token, and their server-side MFA gate requires an AAL2 token for
6
+ // enrolled accounts — so the step-up here is what makes `sf push` / `sf sync`
7
+ // work with 2FA on.
6
8
  //
7
- // NEEDS LIVE TEST: requires your real Supabase URL + anon key + account. If you
8
- // have 2FA enabled, an AAL2 step-up is required and is NOT handled here yet
9
- // (mirror the mobile MFA flow when you wire it).
9
+ // This uses the standard Supabase MFA API (challenge + verify) directly on the
10
+ // signed-in client. The resulting AAL2 session token is what the server checks
11
+ // via getAuthenticatorAssuranceLevel.
10
12
 
11
13
  import { createClient } from '@supabase/supabase-js'
12
14
 
13
- export async function signIn({ supabaseUrl, anonKey, email, password }) {
15
+ // If the account requires an AAL2 step-up, run the TOTP challenge/verify flow
16
+ // and return the elevated access token. Returns null when no step-up is needed
17
+ // (the caller keeps the AAL1 token). `getTotpCode` is an async function that
18
+ // yields the 6-digit code; it is only invoked when a step-up is actually needed.
19
+ export async function stepUpMfaIfRequired({ supabase, getTotpCode }) {
20
+ const { data: aal, error: aalError } =
21
+ await supabase.auth.mfa.getAuthenticatorAssuranceLevel()
22
+ if (aalError) {
23
+ throw new Error(`Could not check two-factor status: ${aalError.message}`)
24
+ }
25
+ const needsStepUp = aal?.nextLevel === 'aal2' && aal?.currentLevel !== 'aal2'
26
+ if (!needsStepUp) return null
27
+
28
+ if (typeof getTotpCode !== 'function') {
29
+ throw new Error(
30
+ 'This account has two-factor authentication enabled. Run in an ' +
31
+ 'interactive terminal, or set SF_TOTP_CODE to your current 6-digit code.',
32
+ )
33
+ }
34
+
35
+ const { data: factors, error: factorsError } =
36
+ await supabase.auth.mfa.listFactors()
37
+ if (factorsError) {
38
+ throw new Error(`Could not list two-factor methods: ${factorsError.message}`)
39
+ }
40
+ const totp = (factors?.totp ?? []).filter(
41
+ (f) => f?.id && f?.status === 'verified',
42
+ )
43
+ if (totp.length === 0) {
44
+ throw new Error(
45
+ 'Two-factor is required but no verified authenticator app (TOTP) is ' +
46
+ 'enrolled on this account. The CLI supports TOTP only.',
47
+ )
48
+ }
49
+ const factorId = totp[0].id
50
+
51
+ const { data: challenge, error: challengeError } =
52
+ await supabase.auth.mfa.challenge({ factorId })
53
+ if (challengeError) {
54
+ throw new Error(
55
+ `Could not start two-factor challenge: ${challengeError.message}`,
56
+ )
57
+ }
58
+
59
+ const code = String(await getTotpCode()).trim()
60
+ if (!/^\d{6}$/.test(code)) {
61
+ throw new Error('Expected a 6-digit authenticator code.')
62
+ }
63
+
64
+ const { data: verified, error: verifyError } = await supabase.auth.mfa.verify({
65
+ factorId,
66
+ challengeId: challenge.id,
67
+ code,
68
+ })
69
+ if (verifyError) {
70
+ throw new Error(`Two-factor verification failed: ${verifyError.message}`)
71
+ }
72
+ const token = verified?.access_token
73
+ if (!token) {
74
+ throw new Error('Two-factor verification did not return a session token.')
75
+ }
76
+ return token
77
+ }
78
+
79
+ export async function signIn({
80
+ supabaseUrl,
81
+ anonKey,
82
+ email,
83
+ password,
84
+ getTotpCode,
85
+ }) {
14
86
  const supabase = createClient(supabaseUrl, anonKey, {
15
87
  auth: {
16
88
  persistSession: false,
@@ -24,12 +96,11 @@ export async function signIn({ supabaseUrl, anonKey, email, password }) {
24
96
  })
25
97
  if (error) throw new Error(`Sign-in failed: ${error.message}`)
26
98
 
27
- const accessToken = data.session?.access_token
28
- if (!accessToken) {
29
- throw new Error(
30
- 'No access token returned. If 2FA is enabled, the MFA step-up is not yet ' +
31
- 'implemented in the CLI.',
32
- )
99
+ const aal1Token = data.session?.access_token
100
+ if (!aal1Token) {
101
+ throw new Error('Sign-in returned no access token.')
33
102
  }
34
- return { accessToken }
103
+
104
+ const aal2Token = await stepUpMfaIfRequired({ supabase, getTotpCode })
105
+ return { accessToken: aal2Token ?? aal1Token }
35
106
  }
package/src/cli.mjs CHANGED
@@ -13,6 +13,7 @@
13
13
  import { webcrypto } from 'node:crypto'
14
14
  import { readFile, readdir, stat } from 'node:fs/promises'
15
15
  import { join } from 'node:path'
16
+ import { createInterface } from 'node:readline'
16
17
 
17
18
  import { deriveVaultIdentity, encryptFileForVault } from './sfCrypto.mjs'
18
19
 
@@ -29,8 +30,9 @@ function usage() {
29
30
  '',
30
31
  'push / sync need only your ShieldFive account:',
31
32
  ' SF_EMAIL, SF_PASSWORD [SF_VAULT_PASSWORD if your vault password differs]',
32
- 'SF_API_BASE_URL, SF_SUPABASE_URL, SF_SUPABASE_ANON_KEY default to ShieldFive',
33
- '(override only for a development backend).',
33
+ 'If two-factor (TOTP) is enabled, you are prompted for a 6-digit code',
34
+ '(or set SF_TOTP_CODE). SF_API_BASE_URL, SF_SUPABASE_URL, SF_SUPABASE_ANON_KEY',
35
+ 'default to ShieldFive (override only for a development backend).',
34
36
  '',
35
37
  ].join('\n') + '\n',
36
38
  )
@@ -98,9 +100,36 @@ function readLiveConfig() {
98
100
  }
99
101
  }
100
102
 
101
- // Sign in (Supabase -> Bearer) then fetch + unlock the vault root key. Shared by
102
- // `push` and `sync`. Network deps are imported lazily so `sf encrypt` runs
103
- // without @supabase/supabase-js.
103
+ // Prompt for a single line on the terminal. A TOTP code is short-lived, not a
104
+ // long-term secret, so echoing it is fine (unlike the password).
105
+ function promptLine(question) {
106
+ return new Promise((resolve, reject) => {
107
+ const rl = createInterface({ input: process.stdin, output: process.stderr })
108
+ rl.question(question, (answer) => {
109
+ rl.close()
110
+ resolve(answer)
111
+ })
112
+ rl.on('error', reject)
113
+ })
114
+ }
115
+
116
+ // Source a TOTP code for the 2FA step-up: SF_TOTP_CODE if set (handy for
117
+ // scripting), otherwise prompt interactively. Only invoked when the account
118
+ // actually requires a step-up.
119
+ async function readTotpCode() {
120
+ if (process.env.SF_TOTP_CODE) return process.env.SF_TOTP_CODE
121
+ if (!process.stdin.isTTY) {
122
+ throw new Error(
123
+ 'Two-factor code required but no terminal is attached. Set SF_TOTP_CODE ' +
124
+ 'to your current 6-digit authenticator code.',
125
+ )
126
+ }
127
+ return promptLine('Two-factor code (6 digits): ')
128
+ }
129
+
130
+ // Sign in (Supabase -> Bearer, with 2FA step-up if enabled) then fetch + unlock
131
+ // the vault root key. Shared by `push` and `sync`. Network deps are imported
132
+ // lazily so `sf encrypt` runs without @supabase/supabase-js.
104
133
  async function authAndUnlock(cfg) {
105
134
  const { signIn } = await import('./auth.mjs')
106
135
  const { fetchAndUnlockVault } = await import('./vault.mjs')
@@ -111,6 +140,7 @@ async function authAndUnlock(cfg) {
111
140
  anonKey: cfg.anonKey,
112
141
  email: cfg.email,
113
142
  password: cfg.password,
143
+ getTotpCode: readTotpCode,
114
144
  })
115
145
 
116
146
  process.stderr.write('unlocking vault…\n')
package/src/upload.mjs CHANGED
@@ -1,10 +1,13 @@
1
- // ShieldFive CLI — upload: create-session -> chunk-encrypt -> Backblaze PUT ->
2
- // finalize. Ported from the apps (mobile cloudApi.ts + syncService.ts).
1
+ // ShieldFive CLI — upload: create-session -> chunk-encrypt -> upload -> finalize.
2
+ // Ported from the apps (mobile cloudApi.ts + syncService.ts).
3
3
  //
4
- // Two paths, chosen by the server:
5
- // direct — a single-chunk file (<= chunkSize). One b2_upload_file POST.
6
- // large — a multipart file (> chunkSize). b2_upload_part per chunk, then
7
- // b2_finish_large_file (server-side) at finalize.
4
+ // Two paths, chosen by the server, and they no longer share a wire protocol:
5
+ // direct — a single-chunk file (<= chunkSize). One presigned S3 PUT, no
6
+ // Authorization header and no storage id at finalize. The server
7
+ // moved this path off b2_upload_file in web b52b4c8 (PR #726).
8
+ // large — a multipart file (> chunkSize). b2_upload_part per chunk, with a
9
+ // Backblaze token and X-Bz-* headers, then b2_finish_large_file
10
+ // (server-side) at finalize. Unchanged.
8
11
  //
9
12
  // Chunks are read from disk one at a time (never the whole file into memory),
10
13
  // so a multi-gigabyte upload holds at most one chunk (~5 MiB) resident.
@@ -27,6 +30,17 @@ const COMPLETE_PATH = '/api/files/complete-upload'
27
30
  const PART_URL_PATH = '/api/files/upload-part-url'
28
31
  const CHUNK_SIZE = 5 * 1024 * 1024
29
32
 
33
+ // Backblaze documents retrying b2_upload_part on 5xx / network failures, and a
34
+ // part upload is idempotent by (part number, SHA-1) — retrying overwrites the
35
+ // same part. Bound the retries and back off so one transient blip does not
36
+ // abort a whole multi-gigabyte upload.
37
+ const MAX_PART_ATTEMPTS = 3
38
+ const PART_RETRY_BASE_DELAY_MS = 250
39
+
40
+ const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms))
41
+ const partRetryDelayMs = (attempt) =>
42
+ PART_RETRY_BASE_DELAY_MS * 2 ** (attempt - 1) // 250ms, 500ms, ...
43
+
30
44
  // Read a file in fixed-size chunks without holding it all in memory. Each yield
31
45
  // is an independent copy (the read buffer is reused), so the caller may retain
32
46
  // or transform a chunk across the next read safely.
@@ -67,7 +81,7 @@ export async function createUploadSession({
67
81
  size,
68
82
  }) {
69
83
  const rootKeyB64 = bytesToBase64(rootKey)
70
- const nameEnvelope = encryptMetadataV4(name, rootKeyB64)
84
+ const nameEnvelope = await encryptMetadataV4(name, rootKeyB64)
71
85
  const csk = generateRandomKeyB64()
72
86
  const cskEnvelope = wrapKeyB64({ wrappingKeyB64: rootKeyB64, keyToWrapB64: csk })
73
87
  const noncePrefix = generateNoncePrefixB64()
@@ -101,29 +115,33 @@ export async function createUploadSession({
101
115
  return { session: body, csk, noncePrefix }
102
116
  }
103
117
 
104
- // Direct upload: b2_upload_file. Returns the storage file id the server patches
105
- // into the row before verifying the proof.
106
- async function putDirectToBackblaze({ session, ciphertext, sha1 }) {
118
+ // Direct upload: a presigned S3 PUT (web b52b4c8 / PR #726), not b2_upload_file.
119
+ // The SigV4 signature in the URL IS the credential and it authorises exactly one
120
+ // object key, so Authorization and X-Bz-* are neither required nor signed --
121
+ // sending them would be noise, and an Authorization header would mislead the
122
+ // next reader into thinking the token still matters here.
123
+ //
124
+ // Content-Type is sent so the stored object carries a sensible type. It is
125
+ // currently unsigned, but only because the vault call site presigns without
126
+ // ContentLength; if that ever changes, this header becomes load-bearing and a
127
+ // mismatch 403s every direct upload. See the web repo's
128
+ // tests/storage/presignedPutContract.test.ts, which pins the signed-header set.
129
+ //
130
+ // Returns nothing. A presigned PUT answers with an ETag, not a Backblaze file
131
+ // id, so there is no storage id to hand to finalize.
132
+ async function putDirectPresigned({ session, ciphertext }) {
107
133
  const res = await fetch(session.uploadUrl, {
108
- method: 'POST', // Backblaze b2_upload_file is a POST
134
+ method: 'PUT',
109
135
  headers: {
110
- Authorization: session.authToken,
111
- 'X-Bz-File-Name': encodeURIComponent(session.storagePath),
112
- 'X-Bz-Content-Sha1': sha1,
113
- 'Content-Type': 'application/octet-stream',
136
+ 'Content-Type': session.contentType || 'application/octet-stream',
114
137
  },
115
138
  body: ciphertext,
116
139
  })
117
140
  if (!res.ok) {
118
141
  throw new Error(
119
- `Backblaze upload failed (HTTP ${res.status}): ${await res.text().catch(() => '')}`,
142
+ `Direct upload failed (HTTP ${res.status}): ${await res.text().catch(() => '')}`,
120
143
  )
121
144
  }
122
- const body = await res.json().catch(() => ({}))
123
- if (!body.fileId) {
124
- throw new Error(`Backblaze response missing fileId: ${JSON.stringify(body)}`)
125
- }
126
- return body.fileId
127
145
  }
128
146
 
129
147
  // Ask the server for a fresh b2_upload_part URL + token when the current one
@@ -147,8 +165,11 @@ async function refreshUploadPartUrl({ apiBaseUrl, accessToken, fileId }) {
147
165
  }
148
166
 
149
167
  // One b2_upload_part POST. On an expired token (401/403) refresh the part URL
150
- // once and retry, mirroring uploadMultipartCiphertextPart in the apps. Returns
151
- // the (possibly refreshed) url/token so the caller reuses them for the next part.
168
+ // and retry immediately, mirroring uploadMultipartCiphertextPart in the apps.
169
+ // On a transient failure (5xx / 408 / 429 / network error) refresh the part URL
170
+ // and retry with exponential backoff, up to MAX_PART_ATTEMPTS, so a single blip
171
+ // does not abort the whole upload. Returns the (possibly refreshed) url/token so
172
+ // the caller reuses them for the next part.
152
173
  async function putPartToBackblaze({
153
174
  apiBaseUrl,
154
175
  accessToken,
@@ -173,22 +194,57 @@ async function putPartToBackblaze({
173
194
 
174
195
  let url = uploadUrl
175
196
  let token = authToken
176
- let res = await attempt(url, token)
197
+ let lastError
177
198
 
178
- if (res.status === 401 || res.status === 403) {
179
- const refreshed = await refreshUploadPartUrl({ apiBaseUrl, accessToken, fileId })
180
- url = refreshed.uploadUrl
181
- token = refreshed.authToken
182
- res = await attempt(url, token)
183
- }
199
+ for (let tries = 1; tries <= MAX_PART_ATTEMPTS; tries += 1) {
200
+ let res
201
+ try {
202
+ res = await attempt(url, token)
203
+ } catch (err) {
204
+ // Network/transport failure (reset, timeout, DNS). Retry with a fresh
205
+ // part URL after a backoff.
206
+ lastError = err
207
+ if (tries === MAX_PART_ATTEMPTS) break
208
+ await sleep(partRetryDelayMs(tries))
209
+ ;({ uploadUrl: url, authToken: token } = await refreshUploadPartUrl({
210
+ apiBaseUrl,
211
+ accessToken,
212
+ fileId,
213
+ }))
214
+ continue
215
+ }
184
216
 
185
- if (!res.ok) {
186
- throw new Error(
217
+ if (res.status === 401 || res.status === 403) {
218
+ // Expected token rotation, not a transient failure: refresh and retry
219
+ // immediately without consuming a backoff slot.
220
+ const refreshed = await refreshUploadPartUrl({ apiBaseUrl, accessToken, fileId })
221
+ url = refreshed.uploadUrl
222
+ token = refreshed.authToken
223
+ res = await attempt(url, token)
224
+ }
225
+
226
+ if (res.ok) return { uploadUrl: url, authToken: token }
227
+
228
+ lastError = new Error(
187
229
  `Backblaze part ${partNumber} upload failed (HTTP ${res.status}): ` +
188
230
  `${await res.text().catch(() => '')}`,
189
231
  )
232
+
233
+ // 5xx / 408 / 429 are worth retrying; other 4xx (after the auth refresh
234
+ // above) are terminal.
235
+ const retryable =
236
+ res.status >= 500 || res.status === 429 || res.status === 408
237
+ if (!retryable || tries === MAX_PART_ATTEMPTS) throw lastError
238
+
239
+ await sleep(partRetryDelayMs(tries))
240
+ ;({ uploadUrl: url, authToken: token } = await refreshUploadPartUrl({
241
+ apiBaseUrl,
242
+ accessToken,
243
+ fileId,
244
+ }))
190
245
  }
191
- return { uploadUrl: url, authToken: token }
246
+
247
+ throw lastError ?? new Error(`Backblaze part ${partNumber} upload failed`)
192
248
  }
193
249
 
194
250
  async function finalizeUpload({
@@ -218,7 +274,8 @@ async function finalizeUpload({
218
274
  }
219
275
 
220
276
  // Single-chunk file: encrypt the one chunk (AES-GCM, suite 0x01) + SHA-1 +
221
- // proof, PUT it, finalize. b2FileId is the id Backblaze returns from the PUT.
277
+ // proof, PUT it at the presigned URL, finalize. No b2FileId is sent -- the
278
+ // server HEADs the stored object and derives the storage id itself.
222
279
  async function uploadDirect({ apiBaseUrl, accessToken, session, csk, noncePrefix, path }) {
223
280
  const chunkSize = session.chunkSize ?? CHUNK_SIZE
224
281
  let plaintext = null
@@ -239,13 +296,17 @@ async function uploadDirect({ apiBaseUrl, accessToken, session, csk, noncePrefix
239
296
  chunkSize,
240
297
  })
241
298
 
242
- const b2FileId = await putDirectToBackblaze({ session, ciphertext, sha1: sha1Hex })
299
+ await putDirectPresigned({ session, ciphertext })
243
300
 
301
+ // b2FileId is deliberately omitted. JSON.stringify drops the undefined key,
302
+ // so complete-upload/finalize.ts takes its "no client-supplied id" branch and
303
+ // HEADs the object for the real version id. Forwarding the PUT's ETag instead
304
+ // would be persisted verbatim into files.b2_file_id with no format check --
305
+ // a bogus delete handle, permanently, which is worse than failing.
244
306
  return finalizeUpload({
245
307
  apiBaseUrl,
246
308
  accessToken,
247
309
  fileId: session.fileId,
248
- b2FileId,
249
310
  partSha1Array: [sha1Hex],
250
311
  proof: proofHex,
251
312
  ciphertextHash: sha1Hex,
@@ -258,49 +319,111 @@ async function uploadDirect({ apiBaseUrl, accessToken, session, csk, noncePrefix
258
319
  // bytes of the stored object, which is exactly part 1. b2FileId is the
259
320
  // large-file id from the session; the server calls b2_finish_large_file with
260
321
  // the ordered part hashes at finalize.
322
+ // Bounded concurrency for multipart part uploads. The serial version left the
323
+ // network idle during encrypt and the CPU idle during each PUT; uploading a few
324
+ // parts at once fills the pipe. Backblaze needs one upload URL per concurrent
325
+ // connection, so K concurrent parts draw K URLs from a small pool. Default 3;
326
+ // override with SF_UPLOAD_CONCURRENCY (clamped 1..6). SF_UPLOAD_CONCURRENCY=1
327
+ // reproduces the exact serial behaviour. Memory ceiling ≈ 2 * K * chunkSize
328
+ // (K plaintext chunks + K ciphertexts resident at once).
329
+ const DEFAULT_UPLOAD_CONCURRENCY = 3
330
+ const MAX_UPLOAD_CONCURRENCY = 6
331
+ function resolveUploadConcurrency() {
332
+ const raw = Number.parseInt(process.env.SF_UPLOAD_CONCURRENCY ?? '', 10)
333
+ if (Number.isInteger(raw) && raw >= 1) {
334
+ return Math.min(raw, MAX_UPLOAD_CONCURRENCY)
335
+ }
336
+ return DEFAULT_UPLOAD_CONCURRENCY
337
+ }
338
+
261
339
  async function uploadMultipart({ apiBaseUrl, accessToken, session, csk, noncePrefix, path }) {
262
340
  if (!session.b2FileId) {
263
341
  throw new Error('Multipart upload session is missing storage id (b2FileId).')
264
342
  }
265
343
  const chunkSize = session.chunkSize ?? CHUNK_SIZE
344
+ const concurrency = resolveUploadConcurrency()
266
345
 
267
- let uploadUrl = session.uploadUrl
268
- let authToken = session.authToken
269
- const partSha1Array = []
270
- let proof
271
- let chunkIndex = 0
346
+ // Part SHA-1s keyed by chunkIndex, NOT push order, so parts finishing out of
347
+ // order under concurrency never scramble the manifest the server checks.
348
+ const partSha1ByIndex = []
349
+ let proof // computed from chunk 0 only (the server verifies the proof over part 1)
350
+ let firstError = null
272
351
 
273
- for await (const plaintext of readFileChunks(path, chunkSize)) {
274
- const encrypted = await encryptChunkWithDigest({
275
- cskB64: csk,
276
- noncePrefixB64: noncePrefix,
277
- chunkIndex,
278
- plaintext,
279
- // Only the first chunk carries the proof; the server verifies over part 1.
280
- proofKeyHex: chunkIndex === 0 ? session.proofKey : undefined,
281
- chunkSize: chunkIndex === 0 ? chunkSize : undefined,
282
- })
283
- if (chunkIndex === 0) proof = encrypted.proofHex
352
+ // Pool of b2_upload_part URLs — one per concurrent connection. Seed with the
353
+ // session URL, fetch more on demand up to `concurrency`, and return each after
354
+ // a successful part so URLs are reused rather than re-fetched per part.
355
+ // putPartToBackblaze already self-heals expired tokens and retries 5xx.
356
+ const urlPool = [{ uploadUrl: session.uploadUrl, authToken: session.authToken }]
357
+ const acquireUrl = async () =>
358
+ urlPool.pop() ??
359
+ (await refreshUploadPartUrl({ apiBaseUrl, accessToken, fileId: session.fileId }))
284
360
 
285
- const updated = await putPartToBackblaze({
286
- apiBaseUrl,
287
- accessToken,
288
- fileId: session.fileId,
289
- uploadUrl,
290
- authToken,
291
- partNumber: chunkIndex + 1, // Backblaze parts are 1-based
292
- ciphertext: encrypted.ciphertext,
293
- sha1: encrypted.sha1Hex,
294
- })
295
- uploadUrl = updated.uploadUrl
296
- authToken = updated.authToken
297
- partSha1Array.push(encrypted.sha1Hex)
361
+ const inFlight = new Set()
362
+ const dispatch = (chunkIndex, plaintext) => {
363
+ const run = (async () => {
364
+ try {
365
+ const encrypted = await encryptChunkWithDigest({
366
+ cskB64: csk,
367
+ noncePrefixB64: noncePrefix,
368
+ chunkIndex,
369
+ plaintext,
370
+ proofKeyHex: chunkIndex === 0 ? session.proofKey : undefined,
371
+ chunkSize: chunkIndex === 0 ? chunkSize : undefined,
372
+ })
373
+ if (chunkIndex === 0) proof = encrypted.proofHex
374
+
375
+ const slot = await acquireUrl()
376
+ const updated = await putPartToBackblaze({
377
+ apiBaseUrl,
378
+ accessToken,
379
+ fileId: session.fileId,
380
+ uploadUrl: slot.uploadUrl,
381
+ authToken: slot.authToken,
382
+ partNumber: chunkIndex + 1, // Backblaze parts are 1-based
383
+ ciphertext: encrypted.ciphertext,
384
+ sha1: encrypted.sha1Hex,
385
+ })
386
+ // Return the (possibly refreshed) URL for the next part to reuse. A URL
387
+ // whose part failed is intentionally NOT returned (it may be bad).
388
+ urlPool.push({ uploadUrl: updated.uploadUrl, authToken: updated.authToken })
389
+ partSha1ByIndex[chunkIndex] = encrypted.sha1Hex
390
+ } catch (err) {
391
+ if (!firstError) firstError = err
392
+ }
393
+ })()
394
+ inFlight.add(run)
395
+ run.finally(() => inFlight.delete(run))
396
+ }
397
+
398
+ let chunkIndex = 0
399
+ for await (const plaintext of readFileChunks(path, chunkSize)) {
400
+ if (firstError) break
401
+ // Backpressure: hold at most `concurrency` parts in flight. inFlight is
402
+ // non-empty when this runs (size >= concurrency >= 1), so race never hangs.
403
+ while (inFlight.size >= concurrency) {
404
+ await Promise.race(inFlight)
405
+ if (firstError) break
406
+ }
407
+ if (firstError) break
408
+ dispatch(chunkIndex, plaintext)
298
409
  chunkIndex += 1
299
410
  }
300
411
 
301
- if (!partSha1Array.length || !proof) {
412
+ // Let every in-flight part settle before finalizing or surfacing the error.
413
+ await Promise.allSettled(inFlight)
414
+ if (firstError) throw firstError
415
+
416
+ if (!chunkIndex || !proof) {
302
417
  throw new Error('Multipart upload produced no parts.')
303
418
  }
419
+ // Dense, in chunk order; a gap means a part silently went missing.
420
+ const partSha1Array = Array.from(
421
+ { length: chunkIndex },
422
+ (_value, index) => partSha1ByIndex[index],
423
+ )
424
+ if (partSha1Array.some((sha1) => typeof sha1 !== 'string')) {
425
+ throw new Error('Multipart upload is missing a part.')
426
+ }
304
427
 
305
428
  const ciphertextHash = await getCiphertextHashFromParts(partSha1Array)
306
429
 
@@ -325,8 +448,13 @@ export async function uploadFile({ apiBaseUrl, accessToken, rootKey, name, path,
325
448
  })
326
449
 
327
450
  if (session.uploadKind === 'direct') {
328
- if (!session.uploadUrl || !session.authToken || !session.storagePath) {
329
- throw new Error('Direct upload session is missing credentials.')
451
+ // No authToken and no storagePath check: since web b52b4c8 (PR #726) the
452
+ // direct path is a presigned S3 PUT, so the signature inside uploadUrl is
453
+ // the only credential and it already binds the object key. Requiring the
454
+ // token the server stopped issuing is what made every <= 5 MiB upload fail
455
+ // from 2026-09-03. The 'large' branch below still needs both.
456
+ if (!session.uploadUrl) {
457
+ throw new Error('Direct upload session is missing an upload URL.')
330
458
  }
331
459
  await uploadDirect({ apiBaseUrl, accessToken, session, csk, noncePrefix, path })
332
460
  } else if (session.uploadKind === 'large') {
@@ -10,6 +10,25 @@ import { gcm } from '@noble/ciphers/aes.js'
10
10
  import { argon2id } from '@noble/hashes/argon2.js'
11
11
  import { hmac } from '@noble/hashes/hmac.js'
12
12
  import { sha256 } from '@noble/hashes/sha2.js'
13
+ import sodium from 'libsodium-wrappers-sumo'
14
+
15
+ // libsodium's Argon2id (crypto_pwhash) is byte-identical to @noble's at the same
16
+ // params but ~15x faster (native/WASM vs pure JS). It is used for the ENCRYPT
17
+ // side of the v4 filename metadata — the once-per-file KDF that makes `sf sync`
18
+ // over many files KDF-bound. decryptMetadataV4 stays on @noble (off the hot
19
+ // path, and a second independent implementation guards reads against a
20
+ // libsodium regression). libsodium is already a dependency and initialised in
21
+ // the unlock path; this memoised gate awaits sodium.ready once.
22
+ let sodiumReadyPromise
23
+ const getSodium = async () => {
24
+ if (!sodiumReadyPromise) {
25
+ sodiumReadyPromise = (async () => {
26
+ await sodium.ready
27
+ return sodium
28
+ })()
29
+ }
30
+ return sodiumReadyPromise
31
+ }
13
32
 
14
33
  const AES_GCM_TAG_BYTES = 16
15
34
 
@@ -24,6 +43,30 @@ export function aesGcmEncrypt(key, iv, plaintext) {
24
43
  return gcm(key, iv).encrypt(plaintext)
25
44
  }
26
45
 
46
+ // WebCrypto AES-GCM. Its output is ciphertext||tag with a 16-byte (128-bit)
47
+ // tag — byte-identical to @noble's gcm().encrypt for the same key, 12-byte
48
+ // nonce and no additional data — so ciphertexts, the SHA-1 digests Backblaze
49
+ // checks, and the HMAC upload proof are all unchanged. Native WebCrypto runs
50
+ // at ~5 GiB/s versus @noble's ~70 MiB/s, so it is used for the per-chunk file
51
+ // encryption (the throughput-bound path). @noble stays for the once-per-file
52
+ // key wrap and filename metadata, and for all decryption. It is async because
53
+ // subtle.encrypt is; the chunk path (encryptChunkWithDigest) is already async.
54
+ async function aesGcmEncryptSubtle(key, iv, plaintext) {
55
+ const cryptoKey = await webcrypto.subtle.importKey(
56
+ 'raw',
57
+ key,
58
+ { name: 'AES-GCM' },
59
+ false,
60
+ ['encrypt'],
61
+ )
62
+ const ciphertext = await webcrypto.subtle.encrypt(
63
+ { name: 'AES-GCM', iv },
64
+ cryptoKey,
65
+ plaintext,
66
+ )
67
+ return new Uint8Array(ciphertext)
68
+ }
69
+
27
70
  export function generateRandomKeyB64(bytes = 32) {
28
71
  return bytesToBase64(randomBytes(bytes))
29
72
  }
@@ -63,10 +106,24 @@ const ARGON2_PARAMS = {
63
106
 
64
107
  // v4 filename metadata (Argon2id "interactive" + AES-GCM) — the format the web
65
108
  // reads. rootKeySecret is the base64 string of the vault root key.
66
- export function encryptMetadataV4(plaintext, rootKeySecret) {
109
+ //
110
+ // Derives the metadata key with libsodium crypto_pwhash (ARGON2ID13, opslimit=2,
111
+ // memlimit=64 MiB) — byte-identical to ARGON2_PARAMS.interactive but far faster.
112
+ // The envelope (v/ct/iv/tag/salt/kdf) is unchanged, so it decrypts on the web
113
+ // and mobile clients exactly as before. Async because of the libsodium ready
114
+ // gate; the sole caller (createUploadSession) is already async.
115
+ export async function encryptMetadataV4(plaintext, rootKeySecret) {
116
+ const s = await getSodium()
67
117
  const salt = randomBytes(16)
68
118
  const iv = randomBytes(12)
69
- const metadataKey = argon2id(rootKeySecret, salt, ARGON2_PARAMS.interactive)
119
+ const metadataKey = s.crypto_pwhash(
120
+ 32,
121
+ rootKeySecret,
122
+ salt,
123
+ 2,
124
+ 64 * 1024 * 1024,
125
+ s.crypto_pwhash_ALG_ARGON2ID13,
126
+ )
70
127
  const ciphertextWithTag = aesGcmEncrypt(metadataKey, iv, utf8(plaintext))
71
128
  const tagStart = ciphertextWithTag.length - AES_GCM_TAG_BYTES
72
129
  return {
@@ -104,7 +161,7 @@ function hexToBytes(hex) {
104
161
  }
105
162
 
106
163
  // nonce = 4-byte random prefix || 8-byte big-endian chunk counter.
107
- export function encryptFileChunk({ cskB64, noncePrefixB64, chunkIndex, plaintext }) {
164
+ export async function encryptFileChunk({ cskB64, noncePrefixB64, chunkIndex, plaintext }) {
108
165
  const noncePrefix = base64ToBytes(noncePrefixB64)
109
166
  if (noncePrefix.length !== 4) throw new Error('Invalid nonce prefix')
110
167
  const nonce = new Uint8Array(12)
@@ -114,7 +171,7 @@ export function encryptFileChunk({ cskB64, noncePrefixB64, chunkIndex, plaintext
114
171
  nonce[11 - i] = Number(counter & 0xffn)
115
172
  counter >>= 8n
116
173
  }
117
- return aesGcmEncrypt(base64ToBytes(cskB64), nonce, plaintext)
174
+ return aesGcmEncryptSubtle(base64ToBytes(cskB64), nonce, plaintext)
118
175
  }
119
176
 
120
177
  function buildUploadProofPrefix({ cipherVersion, chunkSize, noncePrefixBytes }) {
@@ -166,7 +223,7 @@ export async function encryptChunkWithDigest({
166
223
  proofKeyHex,
167
224
  chunkSize,
168
225
  }) {
169
- const ciphertext = encryptFileChunk({
226
+ const ciphertext = await encryptFileChunk({
170
227
  cskB64,
171
228
  noncePrefixB64,
172
229
  chunkIndex,