cloudrop 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +55 -0
- package/README.md +29 -1
- package/dist/adapters/sql.d.ts +3 -0
- package/dist/adapters/sql.js +7 -0
- package/dist/adapters/sql.js.map +1 -1
- package/dist/cli.js +2045 -0
- package/dist/{client-ePVHwvMv.js → client-DxP3uv8V.js} +47 -5
- package/dist/client-DxP3uv8V.js.map +1 -0
- package/dist/client.js +1 -1
- package/dist/index.d.ts +25 -1
- package/dist/index.js +86 -45
- package/dist/index.js.map +1 -1
- package/dist/react.js +1 -1
- package/dist/testing.d.ts +81 -0
- package/dist/testing.js +1243 -0
- package/dist/testing.js.map +1 -0
- package/docs/README.md +30 -0
- package/docs/bucket-setup.md +226 -0
- package/docs/client.md +386 -0
- package/docs/command.md +129 -0
- package/docs/deployment.md +169 -0
- package/docs/errors.md +150 -0
- package/docs/hono.md +276 -0
- package/docs/react.md +508 -0
- package/docs/resume.md +253 -0
- package/docs/server.md +603 -0
- package/docs/store.md +804 -0
- package/docs/testing-your-app.md +328 -0
- package/docs/troubleshooting.md +176 -0
- package/package.json +10 -2
- package/dist/client-ePVHwvMv.js.map +0 -1
package/docs/resume.md
ADDED
|
@@ -0,0 +1,253 @@
|
|
|
1
|
+
# Resume after reload
|
|
2
|
+
|
|
3
|
+
This page is for developers whose users upload large files. It covers how a large upload survives a reload or a closed
|
|
4
|
+
tab: what you write in your page, how the client recognises the same file, what it asks the server, what it keeps in the
|
|
5
|
+
browser and when it deletes it, how two tabs share an upload, how to discard one upload without its file, and what is
|
|
6
|
+
not supported. For uploads in general, see [Client](client.md).
|
|
7
|
+
|
|
8
|
+
## What resumes
|
|
9
|
+
|
|
10
|
+
A file sent in parts (over the server's `multipartThreshold`, 64 MiB by default) survives a reload or a closed tab. The
|
|
11
|
+
client keeps a record of its upload in the browser's IndexedDB. When the user picks the same file again, only the parts
|
|
12
|
+
the bucket does not hold are sent.
|
|
13
|
+
|
|
14
|
+
It is on by default and needs nothing on the server. Files sent in one `PUT` are not kept: a reload loses them, and the
|
|
15
|
+
page starts them again.
|
|
16
|
+
|
|
17
|
+
## Resume in your page
|
|
18
|
+
|
|
19
|
+
```ts
|
|
20
|
+
const client = createCloudropClient<typeof cloudrop>({
|
|
21
|
+
namespace: user.id, // the signed-in user's id: one user's unfinished uploads never mix with another's
|
|
22
|
+
// persist: false, // keeps nothing on disk, and nothing can be resumed after a reload
|
|
23
|
+
});
|
|
24
|
+
|
|
25
|
+
// After the reload: what can be continued, newest first, for one route (every route without an argument). Uploads this
|
|
26
|
+
// client holds itself, running, failed and kept for retry(), or being resumed, are left out.
|
|
27
|
+
const { data: unfinished } = await client.resumable("files"); // null when the browser's storage failed
|
|
28
|
+
const upload = unfinished?.[0]; // { id, route, name, size, type, lastModified, expiresAt }: enough to ask for the file
|
|
29
|
+
|
|
30
|
+
// The user picks the file again, through a file input or a drop.
|
|
31
|
+
if (upload) {
|
|
32
|
+
const job = client.resume(upload, picked, { signal });
|
|
33
|
+
const { data, error } = await job.done; // an UploadResult of one file, as start(...).done gives
|
|
34
|
+
}
|
|
35
|
+
|
|
36
|
+
// The user does not want it any more: no file needed.
|
|
37
|
+
if (upload) await client.discard(upload); // { data: { aborted: true }, error: null }
|
|
38
|
+
|
|
39
|
+
// At sign-out:
|
|
40
|
+
await client.forget();
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
`user`, `picked` and `signal` stand for your app's signed-in user, the `File` the user picked again, and an optional
|
|
44
|
+
`AbortSignal`.
|
|
45
|
+
|
|
46
|
+
### The user picks the file again
|
|
47
|
+
|
|
48
|
+
A page cannot get a `File` back after a reload: a browser hands a page a file only when the user picks or drops it, and
|
|
49
|
+
cloudrop does not copy files (see [Not supported](#not-supported)). `resumable()` gives you the name and size to ask
|
|
50
|
+
for.
|
|
51
|
+
|
|
52
|
+
`start()` and `upload()` never match a file against a record: the same file passed to them is uploaded again under a new
|
|
53
|
+
key. Only `resume()` continues an upload.
|
|
54
|
+
|
|
55
|
+
### What counts as the same file
|
|
56
|
+
|
|
57
|
+
The same name, size, `lastModified` and type, and the same SHA-256 of its first and last 64 KiB (of the whole file, up
|
|
58
|
+
to 128 KiB). It is a heuristic: a file changed only in the middle, with the same timestamp, passes.
|
|
59
|
+
|
|
60
|
+
## What resume() does
|
|
61
|
+
|
|
62
|
+
`resume(upload, file, options?)` returns an `UploadJob` of one file with everything a job of `start()` has:
|
|
63
|
+
`subscribe`, `getSnapshot`, `pause`, `resume`, `abort`, `retry` and `done`. The route and the metadata come from the
|
|
64
|
+
record; `signal` is the only option.
|
|
65
|
+
|
|
66
|
+
### Checks before any request
|
|
67
|
+
|
|
68
|
+
The file is `queued` while the record, the file and the lock are checked. When a check fails, `done` resolves
|
|
69
|
+
`{ data: null, error }` before any request, with:
|
|
70
|
+
|
|
71
|
+
| Code | Why |
|
|
72
|
+
| --- | --- |
|
|
73
|
+
| `INVALID_INPUT` | Wrong arguments, or a different file than the one the upload started with. |
|
|
74
|
+
| `NOT_FOUND` | No unexpired record of that upload in the client's namespace: it finished, was cancelled or expired, or persistence is off. |
|
|
75
|
+
| `STORAGE_ERROR` | The browser's storage cannot be read. |
|
|
76
|
+
| `UPLOAD_IN_USE` | The upload is held in this page or another tab (see [Two tabs](#two-tabs)). |
|
|
77
|
+
|
|
78
|
+
A cancel meanwhile resolves `ABORTED` the same way. `retry()` of a refused job runs the checks again: it resolves the
|
|
79
|
+
same refusal while they fail, sending nothing, and resumes once they pass, such as after `UPLOAD_IN_USE` once the other
|
|
80
|
+
tab let go, or after `STORAGE_ERROR` once the storage reads again.
|
|
81
|
+
|
|
82
|
+
### The server is asked first
|
|
83
|
+
|
|
84
|
+
Once the checks pass, the file is `completing`, and its first request is `/complete` with the saved token. That lists
|
|
85
|
+
the bucket's parts: one `ListParts` per resumed file. On R2, `ListParts` is a **Class A operation**, the more expensive
|
|
86
|
+
of R2's two billed operation classes.
|
|
87
|
+
|
|
88
|
+
- **All parts there, or a `/complete` that finished before the reload:** the file ends done without a `PUT`. The route's
|
|
89
|
+
`afterVerify` and `onComplete` run again then, so make them idempotent by `file.id`, as for any retried request. With
|
|
90
|
+
a [database store](store.md) a finished `/complete` is answered from its row instead, and neither runs again.
|
|
91
|
+
- **Parts missing:** exactly those are sent, then `/complete` again. The parts already stored in the bucket count in `bytes` from that
|
|
92
|
+
answer; before it, `bytes` is 0.
|
|
93
|
+
- **The upload is gone** (`UPLOAD_EXPIRED`, `INVALID_TOKEN` or `NO_SUCH_UPLOAD`, or parts missing in the ticket's
|
|
94
|
+
last 15 minutes, when the server signs no more URLs for it): the file is `queued` again, asks
|
|
95
|
+
`/upload` once for a new ticket with the record's route and metadata, and sends the whole file under a new key, its
|
|
96
|
+
`bytes` from 0, with no failure in between. The old parts wait for the bucket's lifecycle rule. When that `/upload` is
|
|
97
|
+
refused, the file ends `failed` with its error.
|
|
98
|
+
- **Any other failure** ends the file `failed` and keeps its record. `retry()` asks `/complete` again, and so does a
|
|
99
|
+
`resume()` after a reload that left the record (see [When a record is deleted](#when-a-record-is-deleted)).
|
|
100
|
+
|
|
101
|
+
## What is stored
|
|
102
|
+
|
|
103
|
+
One record per upload, in the IndexedDB database `cloudrop`:
|
|
104
|
+
|
|
105
|
+
- the upload's id, route and the metadata sent to `/upload`;
|
|
106
|
+
- its upload token, key, part size, part count and expiry;
|
|
107
|
+
- the file's name, size, type, `lastModified` and fingerprint;
|
|
108
|
+
- the namespace.
|
|
109
|
+
|
|
110
|
+
Never a presigned URL, an ETag or a part list.
|
|
111
|
+
|
|
112
|
+
The token is a bearer credential for its upload for 24 hours: it signs `PUT`s for its key (until 15 minutes before it
|
|
113
|
+
expires), completes the upload, and
|
|
114
|
+
through `/abort` cancels it or deletes its finished file (not with a [database store](store.md), which keeps a finished
|
|
115
|
+
file). It is signed, not encrypted, so what `authorize` returned and
|
|
116
|
+
the metadata can be read from it.
|
|
117
|
+
|
|
118
|
+
## When a record is deleted
|
|
119
|
+
|
|
120
|
+
- when its file ends done or is cancelled, also `abort()` of a failed file of `start()` that still holds its ticket;
|
|
121
|
+
- when a new ticket replaces it (except the record of a ticket `pagehide` already gave up, which another tab may hold);
|
|
122
|
+
- when its file fails for a reason the same ticket can never fix (the bucket or the server refused it, or the ticket
|
|
123
|
+
expired), with the `/abort` that gives the ticket up;
|
|
124
|
+
- when it is found expired, by `resumable()`, `resume()` or the next save (a minute early, so a listed upload has time
|
|
125
|
+
to start);
|
|
126
|
+
- on `discard()` of its upload, once the server answered its `/abort` (see [Discard one upload](#discard-one-upload));
|
|
127
|
+
- on `forget()`.
|
|
128
|
+
|
|
129
|
+
Any other failed file keeps its record, such as one the network failed: a failure of `upload()` and `pagehide` give its
|
|
130
|
+
ticket up without `/abort`, so a reload lists it and `resume()` sends only the parts the bucket lacks. So after a
|
|
131
|
+
reload, the uploads left to resume are those that were still uploading, paused or failed for a reason a resume can
|
|
132
|
+
fix. A file given up this way is not cancelled by a later `abort()` from the same page, which sends nothing: another
|
|
133
|
+
tab may have resumed it.
|
|
134
|
+
|
|
135
|
+
A delete issued just before the page unloads (a cancel, then the tab closes) may not commit, so the next page still
|
|
136
|
+
lists that upload. Resuming it finds the upload gone when the `/abort` reached the server, and starts over with a new
|
|
137
|
+
ticket, sending every byte; when the `/abort` did not reach the server, only the missing parts are sent.
|
|
138
|
+
|
|
139
|
+
## Two tabs
|
|
140
|
+
|
|
141
|
+
The page that holds an upload's ticket holds a Web Lock on it (`cloudrop:upload:<id>`) as long as it keeps the ticket:
|
|
142
|
+
while the file runs, while a failed file waits for `retry()`, and until it gives the ticket up: once the server answered
|
|
143
|
+
its `/abort`, or at once when no `/abort` is sent.
|
|
144
|
+
|
|
145
|
+
- `resume()` of an upload held in another tab, or in this page, ends with `UPLOAD_IN_USE` and sends nothing. Finish or
|
|
146
|
+
cancel it there.
|
|
147
|
+
- `resumable()` leaves out the uploads this client holds: from the ticket's arrival until the page gives it up, and from
|
|
148
|
+
a `resume()` call until its checks refuse it or its upload ends. A failed file of `start()` is continued with its own
|
|
149
|
+
`retry()`, so it is not listed; one of `upload()`, which has no handle to retry with, is listed once it fails. An
|
|
150
|
+
upload another tab holds is still listed, since that tab may let it go; `resume()` of it is `UPLOAD_IN_USE` until
|
|
151
|
+
then.
|
|
152
|
+
- A record is saved only once its lock is held. Without the lock, a second holder could complete an upload whose
|
|
153
|
+
`/abort` this page then sends, deleting the stored file (without a [database store](store.md)).
|
|
154
|
+
|
|
155
|
+
## Namespace and sign-out
|
|
156
|
+
|
|
157
|
+
Pass the signed-in user's id as `namespace` (a string of up to 256 characters, `""` by default). A client sees only its
|
|
158
|
+
namespace's records. The namespace is fixed when the client is created, so create a new client when the user changes.
|
|
159
|
+
|
|
160
|
+
At sign-out, close the client, then call `forget()`:
|
|
161
|
+
|
|
162
|
+
```ts
|
|
163
|
+
await client.close({ abort: true }); // cancels every upload of every route, telling the server
|
|
164
|
+
await client.forget(); // deletes the namespace's records
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
- `forget()` deletes the namespace's records and resolves `{ data: { forgotten }, error }`, counting the records it
|
|
168
|
+
deleted, expired ones included.
|
|
169
|
+
- `forget({ all: true })` deletes every record of this origin. Any other argument is `INVALID_INPUT`.
|
|
170
|
+
- `forget()` stops nothing: uploads that are running keep going. Close the client first, as above: a file that gets
|
|
171
|
+
its ticket after `forget()` saves a new record, and a file still waiting for its ticket asks `/upload` with whatever
|
|
172
|
+
session the browser holds by then. What [`close()`](client.md#close-the-client) does in each phase is in the client
|
|
173
|
+
guide; in React: [Signed-in users](react.md#signed-in-users).
|
|
174
|
+
- When the user changed without signing out here, such as a sign-in in another tab, call `close()` without options on
|
|
175
|
+
the last user's client and leave the records: its multipart uploads stop without an `/abort`, as when a tab closes,
|
|
176
|
+
and that user resumes them on their next sign-in.
|
|
177
|
+
|
|
178
|
+
Without `forget()`, a signed-out user's unfinished uploads, tokens included, stay on the browser profile for up to 24
|
|
179
|
+
hours, and another user signed in on the same profile and namespace could finish them. They stay the first user's files,
|
|
180
|
+
since `authorize` runs only in `/upload`: check the session again in `onComplete` when that matters.
|
|
181
|
+
|
|
182
|
+
## Discard one upload
|
|
183
|
+
|
|
184
|
+
`discard(upload)` drops one upload from `resumable()` without its file, for a list with a Discard button per row:
|
|
185
|
+
|
|
186
|
+
```ts
|
|
187
|
+
const { data, error } = await client.discard(upload);
|
|
188
|
+
if (error) showMessage(error.message); // the upload is still listed: try again later
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
It takes the upload's Web Lock, as a resume would, sends `/abort` with the token the record keeps and `keep: true`,
|
|
192
|
+
deletes the record and, once the delete has run, lets the lock go. `/abort` aborts the multipart upload, so its parts
|
|
193
|
+
leave the bucket at once; with a [database store](store.md) it also ends the upload's row and frees the bytes it
|
|
194
|
+
reserved against the quota. Without a store, `keep` tells the server never to delete an object that landed: a listed
|
|
195
|
+
upload may be one whose `/complete` finished while the page heard nothing (a reload during the last step, or an answer
|
|
196
|
+
lost), and its file stays. With a store the row decides and `keep` changes nothing: a completed row keeps its file,
|
|
197
|
+
and a pending one proves `onComplete` has not committed, so a landed object is deleted with the row and nothing is
|
|
198
|
+
left outside the quota. The server takes an expired token for `/abort`, so any listed upload can be aborted. A server
|
|
199
|
+
older than this release ignores `keep` and, without a store and while the token is valid, deletes such a finished file:
|
|
200
|
+
update the server with the client.
|
|
201
|
+
|
|
202
|
+
| What happens | Result | The record |
|
|
203
|
+
| --- | --- | --- |
|
|
204
|
+
| The server aborted the upload | `{ data: { aborted: true }, error: null }` | deleted |
|
|
205
|
+
| The upload had already finished (its `/complete` went through unheard): the file is kept, nothing is aborted. Without a store, a landed object of a `/complete` that never ran is kept too; with a store, it is deleted with its pending row | `{ data: { aborted: true }, error: null }` | deleted |
|
|
206
|
+
| The server refused the token for good (`INVALID_TOKEN`, `UPLOAD_EXPIRED`, `NO_SUCH_UPLOAD` and the other codes a new ticket would be needed for), for example after the server's secret changed | `{ data: { aborted: false }, error: null }` | deleted |
|
|
207
|
+
| The record expired and the browser already deleted it, with its token (a minute before `expiresAt`) | `{ data: { aborted: false }, error: null }`, no request | already gone |
|
|
208
|
+
| `/abort` did not get through (no connection, a server error, a timeout) | that error, such as a retryable `NETWORK_ERROR` | kept, for another try |
|
|
209
|
+
| Another tab, or this page, holds the upload | `UPLOAD_IN_USE`, no request | kept |
|
|
210
|
+
| No record of that id in this namespace, and not expired: it finished, or was cancelled or discarded | `NOT_FOUND`, no request | none |
|
|
211
|
+
| Anything but an upload with a string `id` | `INVALID_INPUT` | unchanged |
|
|
212
|
+
|
|
213
|
+
With `aborted: false`, the bucket's abort-incomplete-multipart rule removes the parts, and a store's sweep ends the row.
|
|
214
|
+
A tab closed while its `/abort` is on the way keeps the record, so the upload is listed again; discarding it again is
|
|
215
|
+
harmless.
|
|
216
|
+
`forget()` deletes every record of the namespace at once but sends no `/abort`: use it at sign-out, and `discard()` when
|
|
217
|
+
the user drops an upload.
|
|
218
|
+
|
|
219
|
+
## When persistence is off or unavailable
|
|
220
|
+
|
|
221
|
+
Resuming needs a **secure context** (HTTPS, or `localhost` in development), where browsers have Web Locks.
|
|
222
|
+
|
|
223
|
+
With `persist: false`, or where there is no `indexedDB` (Node, Bun, Deno, Workers) or no Web Locks (a page served over
|
|
224
|
+
plain HTTP, such as a LAN address), nothing is read or written:
|
|
225
|
+
|
|
226
|
+
- `resumable()` resolves `[]`;
|
|
227
|
+
- `resume()` ends with `NOT_FOUND`;
|
|
228
|
+
- `discard()` resolves `NOT_FOUND` (or `{ aborted: false }` for an upload within a minute of `expiresAt`), sending
|
|
229
|
+
nothing;
|
|
230
|
+
- `forget()` resolves `{ forgotten: 0 }`.
|
|
231
|
+
|
|
232
|
+
The same goes where the browser refuses the lock (a document no longer active, an opaque origin): that upload keeps no
|
|
233
|
+
record, and still uploads.
|
|
234
|
+
|
|
235
|
+
The client looks for `indexedDB` and Web Locks once, when it is created: create it after any polyfill for them.
|
|
236
|
+
|
|
237
|
+
`resumable()`, `discard()` and `forget()` never reject. When the browser's storage fails, they resolve `STORAGE_ERROR`.
|
|
238
|
+
A storage failure never fails, pauses or slows an upload.
|
|
239
|
+
|
|
240
|
+
Records live in the origin's best-effort storage, which a browser may evict under pressure. cloudrop never calls
|
|
241
|
+
`navigator.storage.persist()`, since Firefox would prompt the user for it; call it yourself if you want persistent
|
|
242
|
+
storage.
|
|
243
|
+
|
|
244
|
+
## Not supported
|
|
245
|
+
|
|
246
|
+
- **Resuming without picking the file again.** A copy into the origin-private file system is deferred to a later opt-in:
|
|
247
|
+
it needs Safari 26 or a worker, and writes every file a second time.
|
|
248
|
+
- **Files sent in one `PUT`.**
|
|
249
|
+
- **Resuming on another device or browser.**
|
|
250
|
+
- **Continuing an upload after its 24-hour token expired.** Its record is deleted, the old parts wait for the bucket's
|
|
251
|
+
lifecycle rule, and the file is started again.
|
|
252
|
+
|
|
253
|
+
Why it works this way: the resume decisions `M3-U60` to `M3-U91` in [m3-uploader.md](https://github.com/vcode-sh/cloudrop/blob/main/docs/decisions/m3-uploader.md).
|