cloudrop 0.2.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/docs/resume.md ADDED
@@ -0,0 +1,253 @@
1
+ # Resume after reload
2
+
3
+ This page is for developers whose users upload large files. It covers how a large upload survives a reload or a closed
4
+ tab: what you write in your page, how the client recognises the same file, what it asks the server, what it keeps in the
5
+ browser and when it deletes it, how two tabs share an upload, how to discard one upload without its file, and what is
6
+ not supported. For uploads in general, see [Client](client.md).
7
+
8
+ ## What resumes
9
+
10
+ A file sent in parts (over the server's `multipartThreshold`, 64 MiB by default) survives a reload or a closed tab. The
11
+ client keeps a record of its upload in the browser's IndexedDB. When the user picks the same file again, only the parts
12
+ the bucket does not hold are sent.
13
+
14
+ It is on by default and needs nothing on the server. Files sent in one `PUT` are not kept: a reload loses them, and the
15
+ page starts them again.
16
+
17
+ ## Resume in your page
18
+
19
+ ```ts
20
+ const client = createCloudropClient<typeof cloudrop>({
21
+ namespace: user.id, // the signed-in user's id: one user's unfinished uploads never mix with another's
22
+ // persist: false, // keeps nothing on disk, and nothing can be resumed after a reload
23
+ });
24
+
25
+ // After the reload: what can be continued, newest first, for one route (every route without an argument). Uploads this
26
+ // client holds itself, running, failed and kept for retry(), or being resumed, are left out.
27
+ const { data: unfinished } = await client.resumable("files"); // null when the browser's storage failed
28
+ const upload = unfinished?.[0]; // { id, route, name, size, type, lastModified, expiresAt }: enough to ask for the file
29
+
30
+ // The user picks the file again, through a file input or a drop.
31
+ if (upload) {
32
+ const job = client.resume(upload, picked, { signal });
33
+ const { data, error } = await job.done; // an UploadResult of one file, as start(...).done gives
34
+ }
35
+
36
+ // The user does not want it any more: no file needed.
37
+ if (upload) await client.discard(upload); // { data: { aborted: true }, error: null }
38
+
39
+ // At sign-out:
40
+ await client.forget();
41
+ ```
42
+
43
+ `user`, `picked` and `signal` stand for your app's signed-in user, the `File` the user picked again, and an optional
44
+ `AbortSignal`.
45
+
46
+ ### The user picks the file again
47
+
48
+ A page cannot get a `File` back after a reload: a browser hands a page a file only when the user picks or drops it, and
49
+ cloudrop does not copy files (see [Not supported](#not-supported)). `resumable()` gives you the name and size to ask
50
+ for.
51
+
52
+ `start()` and `upload()` never match a file against a record: the same file passed to them is uploaded again under a new
53
+ key. Only `resume()` continues an upload.
54
+
55
+ ### What counts as the same file
56
+
57
+ The same name, size, `lastModified` and type, and the same SHA-256 of its first and last 64 KiB (of the whole file, up
58
+ to 128 KiB). It is a heuristic: a file changed only in the middle, with the same timestamp, passes.
59
+
60
+ ## What resume() does
61
+
62
+ `resume(upload, file, options?)` returns an `UploadJob` of one file with everything a job of `start()` has:
63
+ `subscribe`, `getSnapshot`, `pause`, `resume`, `abort`, `retry` and `done`. The route and the metadata come from the
64
+ record; `signal` is the only option.
65
+
66
+ ### Checks before any request
67
+
68
+ The file is `queued` while the record, the file and the lock are checked. When a check fails, `done` resolves
69
+ `{ data: null, error }` before any request, with:
70
+
71
+ | Code | Why |
72
+ | --- | --- |
73
+ | `INVALID_INPUT` | Wrong arguments, or a different file than the one the upload started with. |
74
+ | `NOT_FOUND` | No unexpired record of that upload in the client's namespace: it finished, was cancelled or expired, or persistence is off. |
75
+ | `STORAGE_ERROR` | The browser's storage cannot be read. |
76
+ | `UPLOAD_IN_USE` | The upload is held in this page or another tab (see [Two tabs](#two-tabs)). |
77
+
78
+ A cancel meanwhile resolves `ABORTED` the same way. `retry()` of a refused job runs the checks again: it resolves the
79
+ same refusal while they fail, sending nothing, and resumes once they pass, such as after `UPLOAD_IN_USE` once the other
80
+ tab let go, or after `STORAGE_ERROR` once the storage reads again.
81
+
82
+ ### The server is asked first
83
+
84
+ Once the checks pass, the file is `completing`, and its first request is `/complete` with the saved token. That lists
85
+ the bucket's parts: one `ListParts` per resumed file. On R2, `ListParts` is a **Class A operation**, the more expensive
86
+ of R2's two billed operation classes.
87
+
88
+ - **All parts there, or a `/complete` that finished before the reload:** the file ends done without a `PUT`. The route's
89
+ `afterVerify` and `onComplete` run again then, so make them idempotent by `file.id`, as for any retried request. With
90
+ a [database store](store.md) a finished `/complete` is answered from its row instead, and neither runs again.
91
+ - **Parts missing:** exactly those are sent, then `/complete` again. The parts already stored in the bucket count in `bytes` from that
92
+ answer; before it, `bytes` is 0.
93
+ - **The upload is gone** (`UPLOAD_EXPIRED`, `INVALID_TOKEN` or `NO_SUCH_UPLOAD`, or parts missing in the ticket's
94
+ last 15 minutes, when the server signs no more URLs for it): the file is `queued` again, asks
95
+ `/upload` once for a new ticket with the record's route and metadata, and sends the whole file under a new key, its
96
+ `bytes` from 0, with no failure in between. The old parts wait for the bucket's lifecycle rule. When that `/upload` is
97
+ refused, the file ends `failed` with its error.
98
+ - **Any other failure** ends the file `failed` and keeps its record. `retry()` asks `/complete` again, and so does a
99
+ `resume()` after a reload that left the record (see [When a record is deleted](#when-a-record-is-deleted)).
100
+
101
+ ## What is stored
102
+
103
+ One record per upload, in the IndexedDB database `cloudrop`:
104
+
105
+ - the upload's id, route and the metadata sent to `/upload`;
106
+ - its upload token, key, part size, part count and expiry;
107
+ - the file's name, size, type, `lastModified` and fingerprint;
108
+ - the namespace.
109
+
110
+ Never a presigned URL, an ETag or a part list.
111
+
112
+ The token is a bearer credential for its upload for 24 hours: it signs `PUT`s for its key (until 15 minutes before it
113
+ expires), completes the upload, and
114
+ through `/abort` cancels it or deletes its finished file (not with a [database store](store.md), which keeps a finished
115
+ file). It is signed, not encrypted, so what `authorize` returned and
116
+ the metadata can be read from it.
117
+
118
+ ## When a record is deleted
119
+
120
+ - when its file ends done or is cancelled, also `abort()` of a failed file of `start()` that still holds its ticket;
121
+ - when a new ticket replaces it (except the record of a ticket `pagehide` already gave up, which another tab may hold);
122
+ - when its file fails for a reason the same ticket can never fix (the bucket or the server refused it, or the ticket
123
+ expired), with the `/abort` that gives the ticket up;
124
+ - when it is found expired, by `resumable()`, `resume()` or the next save (a minute early, so a listed upload has time
125
+ to start);
126
+ - on `discard()` of its upload, once the server answered its `/abort` (see [Discard one upload](#discard-one-upload));
127
+ - on `forget()`.
128
+
129
+ Any other failed file keeps its record, such as one the network failed: a failure of `upload()` and `pagehide` give its
130
+ ticket up without `/abort`, so a reload lists it and `resume()` sends only the parts the bucket lacks. So after a
131
+ reload, the uploads left to resume are those that were still uploading, paused or failed for a reason a resume can
132
+ fix. A file given up this way is not cancelled by a later `abort()` from the same page, which sends nothing: another
133
+ tab may have resumed it.
134
+
135
+ A delete issued just before the page unloads (a cancel, then the tab closes) may not commit, so the next page still
136
+ lists that upload. Resuming it finds the upload gone when the `/abort` reached the server, and starts over with a new
137
+ ticket, sending every byte; when the `/abort` did not reach the server, only the missing parts are sent.
138
+
139
+ ## Two tabs
140
+
141
+ The page that holds an upload's ticket holds a Web Lock on it (`cloudrop:upload:<id>`) as long as it keeps the ticket:
142
+ while the file runs, while a failed file waits for `retry()`, and until it gives the ticket up: once the server answered
143
+ its `/abort`, or at once when no `/abort` is sent.
144
+
145
+ - `resume()` of an upload held in another tab, or in this page, ends with `UPLOAD_IN_USE` and sends nothing. Finish or
146
+ cancel it there.
147
+ - `resumable()` leaves out the uploads this client holds: from the ticket's arrival until the page gives it up, and from
148
+ a `resume()` call until its checks refuse it or its upload ends. A failed file of `start()` is continued with its own
149
+ `retry()`, so it is not listed; one of `upload()`, which has no handle to retry with, is listed once it fails. An
150
+ upload another tab holds is still listed, since that tab may let it go; `resume()` of it is `UPLOAD_IN_USE` until
151
+ then.
152
+ - A record is saved only once its lock is held. Without the lock, a second holder could complete an upload whose
153
+ `/abort` this page then sends, deleting the stored file (without a [database store](store.md)).
154
+
155
+ ## Namespace and sign-out
156
+
157
+ Pass the signed-in user's id as `namespace` (a string of up to 256 characters, `""` by default). A client sees only its
158
+ namespace's records. The namespace is fixed when the client is created, so create a new client when the user changes.
159
+
160
+ At sign-out, close the client, then call `forget()`:
161
+
162
+ ```ts
163
+ await client.close({ abort: true }); // cancels every upload of every route, telling the server
164
+ await client.forget(); // deletes the namespace's records
165
+ ```
166
+
167
+ - `forget()` deletes the namespace's records and resolves `{ data: { forgotten }, error }`, counting the records it
168
+ deleted, expired ones included.
169
+ - `forget({ all: true })` deletes every record of this origin. Any other argument is `INVALID_INPUT`.
170
+ - `forget()` stops nothing: uploads that are running keep going. Close the client first, as above: a file that gets
171
+ its ticket after `forget()` saves a new record, and a file still waiting for its ticket asks `/upload` with whatever
172
+ session the browser holds by then. What [`close()`](client.md#close-the-client) does in each phase is in the client
173
+ guide; in React: [Signed-in users](react.md#signed-in-users).
174
+ - When the user changed without signing out here, such as a sign-in in another tab, call `close()` without options on
175
+ the last user's client and leave the records: its multipart uploads stop without an `/abort`, as when a tab closes,
176
+ and that user resumes them on their next sign-in.
177
+
178
+ Without `forget()`, a signed-out user's unfinished uploads, tokens included, stay on the browser profile for up to 24
179
+ hours, and another user signed in on the same profile and namespace could finish them. They stay the first user's files,
180
+ since `authorize` runs only in `/upload`: check the session again in `onComplete` when that matters.
181
+
182
+ ## Discard one upload
183
+
184
+ `discard(upload)` drops one upload from `resumable()` without its file, for a list with a Discard button per row:
185
+
186
+ ```ts
187
+ const { data, error } = await client.discard(upload);
188
+ if (error) showMessage(error.message); // the upload is still listed: try again later
189
+ ```
190
+
191
+ It takes the upload's Web Lock, as a resume would, sends `/abort` with the token the record keeps and `keep: true`,
192
+ deletes the record and, once the delete has run, lets the lock go. `/abort` aborts the multipart upload, so its parts
193
+ leave the bucket at once; with a [database store](store.md) it also ends the upload's row and frees the bytes it
194
+ reserved against the quota. Without a store, `keep` tells the server never to delete an object that landed: a listed
195
+ upload may be one whose `/complete` finished while the page heard nothing (a reload during the last step, or an answer
196
+ lost), and its file stays. With a store the row decides and `keep` changes nothing: a completed row keeps its file,
197
+ and a pending one proves `onComplete` has not committed, so a landed object is deleted with the row and nothing is
198
+ left outside the quota. The server takes an expired token for `/abort`, so any listed upload can be aborted. A server
199
+ older than this release ignores `keep` and, without a store and while the token is valid, deletes such a finished file:
200
+ update the server with the client.
201
+
202
+ | What happens | Result | The record |
203
+ | --- | --- | --- |
204
+ | The server aborted the upload | `{ data: { aborted: true }, error: null }` | deleted |
205
+ | The upload had already finished (its `/complete` went through unheard): the file is kept, nothing is aborted. Without a store, a landed object of a `/complete` that never ran is kept too; with a store, it is deleted with its pending row | `{ data: { aborted: true }, error: null }` | deleted |
206
+ | The server refused the token for good (`INVALID_TOKEN`, `UPLOAD_EXPIRED`, `NO_SUCH_UPLOAD` and the other codes a new ticket would be needed for), for example after the server's secret changed | `{ data: { aborted: false }, error: null }` | deleted |
207
+ | The record expired and the browser already deleted it, with its token (a minute before `expiresAt`) | `{ data: { aborted: false }, error: null }`, no request | already gone |
208
+ | `/abort` did not get through (no connection, a server error, a timeout) | that error, such as a retryable `NETWORK_ERROR` | kept, for another try |
209
+ | Another tab, or this page, holds the upload | `UPLOAD_IN_USE`, no request | kept |
210
+ | No record of that id in this namespace, and not expired: it finished, or was cancelled or discarded | `NOT_FOUND`, no request | none |
211
+ | Anything but an upload with a string `id` | `INVALID_INPUT` | unchanged |
212
+
213
+ With `aborted: false`, the bucket's abort-incomplete-multipart rule removes the parts, and a store's sweep ends the row.
214
+ A tab closed while its `/abort` is on the way keeps the record, so the upload is listed again; discarding it again is
215
+ harmless.
216
+ `forget()` deletes every record of the namespace at once but sends no `/abort`: use it at sign-out, and `discard()` when
217
+ the user drops an upload.
218
+
219
+ ## When persistence is off or unavailable
220
+
221
+ Resuming needs a **secure context** (HTTPS, or `localhost` in development), where browsers have Web Locks.
222
+
223
+ With `persist: false`, or where there is no `indexedDB` (Node, Bun, Deno, Workers) or no Web Locks (a page served over
224
+ plain HTTP, such as a LAN address), nothing is read or written:
225
+
226
+ - `resumable()` resolves `[]`;
227
+ - `resume()` ends with `NOT_FOUND`;
228
+ - `discard()` resolves `NOT_FOUND` (or `{ aborted: false }` for an upload within a minute of `expiresAt`), sending
229
+ nothing;
230
+ - `forget()` resolves `{ forgotten: 0 }`.
231
+
232
+ The same goes where the browser refuses the lock (a document no longer active, an opaque origin): that upload keeps no
233
+ record, and still uploads.
234
+
235
+ The client looks for `indexedDB` and Web Locks once, when it is created: create it after any polyfill for them.
236
+
237
+ `resumable()`, `discard()` and `forget()` never reject. When the browser's storage fails, they resolve `STORAGE_ERROR`.
238
+ A storage failure never fails, pauses or slows an upload.
239
+
240
+ Records live in the origin's best-effort storage, which a browser may evict under pressure. cloudrop never calls
241
+ `navigator.storage.persist()`, since Firefox would prompt the user for it; call it yourself if you want persistent
242
+ storage.
243
+
244
+ ## Not supported
245
+
246
+ - **Resuming without picking the file again.** A copy into the origin-private file system is deferred to a later opt-in:
247
+ it needs Safari 26 or a worker, and writes every file a second time.
248
+ - **Files sent in one `PUT`.**
249
+ - **Resuming on another device or browser.**
250
+ - **Continuing an upload after its 24-hour token expired.** Its record is deleted, the old parts wait for the bucket's
251
+ lifecycle rule, and the file is started again.
252
+
253
+ Why it works this way: the resume decisions `M3-U60` to `M3-U91` in [m3-uploader.md](https://github.com/vcode-sh/cloudrop/blob/main/docs/decisions/m3-uploader.md).