cloudrop 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +55 -0
- package/README.md +29 -1
- package/dist/adapters/sql.d.ts +3 -0
- package/dist/adapters/sql.js +7 -0
- package/dist/adapters/sql.js.map +1 -1
- package/dist/cli.js +2045 -0
- package/dist/{client-ePVHwvMv.js → client-DxP3uv8V.js} +47 -5
- package/dist/client-DxP3uv8V.js.map +1 -0
- package/dist/client.js +1 -1
- package/dist/index.d.ts +25 -1
- package/dist/index.js +86 -45
- package/dist/index.js.map +1 -1
- package/dist/react.js +1 -1
- package/dist/testing.d.ts +81 -0
- package/dist/testing.js +1243 -0
- package/dist/testing.js.map +1 -0
- package/docs/README.md +30 -0
- package/docs/bucket-setup.md +226 -0
- package/docs/client.md +386 -0
- package/docs/command.md +129 -0
- package/docs/deployment.md +169 -0
- package/docs/errors.md +150 -0
- package/docs/hono.md +276 -0
- package/docs/react.md +508 -0
- package/docs/resume.md +253 -0
- package/docs/server.md +603 -0
- package/docs/store.md +804 -0
- package/docs/testing-your-app.md +328 -0
- package/docs/troubleshooting.md +176 -0
- package/package.json +10 -2
- package/dist/client-ePVHwvMv.js.map +0 -1
package/docs/client.md
ADDED
|
@@ -0,0 +1,386 @@
|
|
|
1
|
+
# Client
|
|
2
|
+
|
|
3
|
+
This page is for developers writing the browser side without React, or wanting to know what the hooks run on. It covers
|
|
4
|
+
`cloudrop/client`: creating a client, uploading one file or several, following progress, pausing, cancelling and
|
|
5
|
+
retrying, how the client shares connections between uploads, what happens when the page goes offline or a request
|
|
6
|
+
stalls, what it cleans up, and closing it when the user signs out. Resuming after a reload, and `resumable()`, `resume()`, `discard()` and `forget()`, have
|
|
7
|
+
their own page: [Resume after reload](resume.md). Error codes: [Errors](errors.md).
|
|
8
|
+
|
|
9
|
+
## Create a client
|
|
10
|
+
|
|
11
|
+
```ts
|
|
12
|
+
"use client";
|
|
13
|
+
import { createCloudropClient } from "cloudrop/client";
|
|
14
|
+
import type { cloudrop } from "../lib/cloudrop"; // a type import: no server code or secret reaches the browser
|
|
15
|
+
|
|
16
|
+
const client = createCloudropClient<typeof cloudrop>();
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Import the server with `import type`. A value import would pull the server, its storage driver and its secret settings
|
|
20
|
+
into the browser bundle. The type is enough for the client to know your route names, each route's metadata and what
|
|
21
|
+
each route's `onComplete` returns.
|
|
22
|
+
|
|
23
|
+
| Option | What it does | Default |
|
|
24
|
+
| --- | --- | --- |
|
|
25
|
+
| `basePath` | Where the server's handler is mounted: a path on this page's origin, or an absolute URL such as `"https://api.example.com/api/cloudrop"`. No trailing slash, query or fragment. For a handler on another origin, list the page's origin in the server's `trustedOrigins`, or every call gets `ORIGIN_NOT_ALLOWED`. | `"/api/cloudrop"` |
|
|
26
|
+
| `headers` | Headers for the requests to your server (`/upload`, `/sign`, `/complete`, `/abort`), never for bucket `PUT`s: an object, or a function, possibly async, called before each request (see [Send a bearer token](#send-a-bearer-token)). | none |
|
|
27
|
+
| `fetch` | Replaces the global `fetch`, for tests and runtimes that bring their own. Passing it also sends bucket `PUT`s through it (see [Transports](#transports-xhr-or-fetch)). | global `fetch` |
|
|
28
|
+
| `concurrency` | How many bucket `PUT`s the client sends at once, across all its uploads: `"auto"` or a whole number from 1 to 16 (see [Concurrency and order](#concurrency-and-order)). | `"auto"` |
|
|
29
|
+
| `order` | Which `PUT`s go first when they wait for a connection: `"added"` or `"smallest-first"`. | `"added"` |
|
|
30
|
+
| `persist` | Keeps unfinished multipart uploads in IndexedDB so they can resume after a reload. `false` keeps nothing on disk. | `true` |
|
|
31
|
+
| `namespace` | Keeps one signed-in user's unfinished uploads apart from another's: pass the user's id. At most 256 characters. | `""` |
|
|
32
|
+
|
|
33
|
+
A wrong option throws `INVALID_CONFIG` when the client is created. Wire types (`UploadRequest`, `FileTicket`,
|
|
34
|
+
`ErrorBody` and the rest) and the route inference types (`RouteName`, `RouteMetadata`, `RouteResult`,
|
|
35
|
+
`InferableCloudrop`) are imported from `cloudrop/client`, never from `cloudrop`.
|
|
36
|
+
|
|
37
|
+
In React, [`cloudrop/react`](react.md) creates the client for you and gives you hooks for inputs, drop zones and
|
|
38
|
+
progress.
|
|
39
|
+
|
|
40
|
+
### Send a bearer token
|
|
41
|
+
|
|
42
|
+
The client sends cookies to your server. An app that keeps its session in a token instead, such as a desktop or mobile
|
|
43
|
+
shell around a web view, or a session handed over by QR code, adds it with the `headers` option:
|
|
44
|
+
|
|
45
|
+
```ts
|
|
46
|
+
const client = createCloudropClient<typeof cloudrop>({
|
|
47
|
+
basePath: "https://api.example.com/api/cloudrop",
|
|
48
|
+
headers: async () => ({ authorization: `Bearer ${await getToken()}` }), // your app's token, refreshed when needed
|
|
49
|
+
});
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
On the server, `authorize` reads `request.headers.get("authorization")`.
|
|
53
|
+
|
|
54
|
+
- **Only your server gets them.** The headers go with `/upload`, `/sign`, `/complete` and `/abort`, never with a
|
|
55
|
+
bucket `PUT`: a presigned URL must not carry your token, and bucket `PUT`s keep `XMLHttpRequest` and its byte-by-byte
|
|
56
|
+
progress (see [Transports](#transports-xhr-or-fetch)).
|
|
57
|
+
- **A function runs before each request,** so each request carries the token of its moment, and a token that expires
|
|
58
|
+
during a long upload is refreshed for the next `/sign` or `/complete`. An object is sent as it is with every request.
|
|
59
|
+
- **When the function throws or rejects,** that request is not sent and fails with `NETWORK_ERROR`, which the client
|
|
60
|
+
retries as any network failure. Its time counts against the request's timeout, and a cancel stops the wait.
|
|
61
|
+
- **`content-type` stays the client's.** A `content-type` among the headers is replaced by `application/json`.
|
|
62
|
+
- **At sign-out, await [`close()`](#close-the-client)** before the session the function reads ends: each `/abort`
|
|
63
|
+
calls the function too. Return no headers rather than throw when signed out: `/abort` needs only the upload token.
|
|
64
|
+
- **At `pagehide`, an async function may be too late.** The `/abort` a closing page sends for a failed file waits for
|
|
65
|
+
it, and a page that is gone sends nothing. The bucket's lifecycle rule and, with a store, the sweep clean up.
|
|
66
|
+
- **From a page on another origin,** the `authorization` header makes the browser send a preflight first. The handler
|
|
67
|
+
allows the headers a page on a `trustedOrigins` origin asks for.
|
|
68
|
+
- **A request without an `Origin` header,** as a native app sends, needs no `trustedOrigins` entry (see
|
|
69
|
+
[Origins](server.md#origins)).
|
|
70
|
+
|
|
71
|
+
## Upload one file
|
|
72
|
+
|
|
73
|
+
```ts
|
|
74
|
+
const controller = new AbortController();
|
|
75
|
+
const { data, error } = await client.upload("avatar", file, { signal: controller.signal });
|
|
76
|
+
|
|
77
|
+
if (error) {
|
|
78
|
+
showError(error.message); // your app's error display
|
|
79
|
+
} else {
|
|
80
|
+
console.log(data.file.key, data.data); // data.data is what onComplete returned, null without one
|
|
81
|
+
}
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
- `upload(route, file, options?)` sends one `File` and resolves to `{ data, error }`. It never rejects for an expected
|
|
85
|
+
failure.
|
|
86
|
+
- `options.metadata` is validated by the route's schema on the server. It is required when the route's schema requires
|
|
87
|
+
it, and TypeScript enforces that.
|
|
88
|
+
- `options.signal` cancels the upload: requests stop, the server is told, and the result is `ABORTED`.
|
|
89
|
+
- `data.file` is the stored file: `id`, `key`, `name`, `size`, `type` and `url` (the public URL on a public route,
|
|
90
|
+
otherwise `null`).
|
|
91
|
+
- No `this`: `const { upload } = createCloudropClient()` works too.
|
|
92
|
+
|
|
93
|
+
## Upload several files
|
|
94
|
+
|
|
95
|
+
Pass a list, such as an array or `input.files`, and `upload()` resolves to an `UploadResult` once every file has
|
|
96
|
+
settled:
|
|
97
|
+
|
|
98
|
+
```ts
|
|
99
|
+
const files = input.files; // a FileList, or null when the input has none
|
|
100
|
+
if (!files || files.length === 0) return;
|
|
101
|
+
const { data, error } = await client.upload("files", files);
|
|
102
|
+
|
|
103
|
+
for (const outcome of data?.files ?? []) {
|
|
104
|
+
if (outcome.status === "done") console.log(outcome.name, outcome.result.file.key);
|
|
105
|
+
else console.log(outcome.name, outcome.status, outcome.error.message); // "failed" or "aborted"
|
|
106
|
+
}
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
- `data.files` holds one outcome per file, in the order passed: `status: "done"` with its `result`, or `"failed"` or
|
|
110
|
+
`"aborted"` with its `error`.
|
|
111
|
+
- `error` is the error of the first file, in the order passed, that did not end done.
|
|
112
|
+
- Unlike other results, `data` stays when some files failed. It is `null` only when nothing started: wrong arguments, or
|
|
113
|
+
`/upload` refused the whole job (such as `TOO_MANY_FILES`).
|
|
114
|
+
- An error that belongs to one file carries `error.file`, `{ id, name }`.
|
|
115
|
+
- A value typed `File | File[]` fits neither `upload()` signature: pass it to `start()` and await `start(...).done`.
|
|
116
|
+
|
|
117
|
+
### Reset the file input
|
|
118
|
+
|
|
119
|
+
A browser fires no `change` event when the user picks the same file again, so clear the input after each pick. Copy the
|
|
120
|
+
files out of the input before you clear it.
|
|
121
|
+
|
|
122
|
+
```ts
|
|
123
|
+
input.addEventListener("change", () => {
|
|
124
|
+
const files = Array.from(input.files ?? []);
|
|
125
|
+
input.value = ""; // picking the same file again now fires change
|
|
126
|
+
if (files.length > 0) void client.upload("files", files);
|
|
127
|
+
});
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
In React, `getInputProps()` from [`cloudrop/react`](react.md#one-file) does this for you.
|
|
131
|
+
|
|
132
|
+
## Jobs: progress, pause and retry
|
|
133
|
+
|
|
134
|
+
`start()` gives you a handle to follow and control the upload while it runs:
|
|
135
|
+
|
|
136
|
+
```ts
|
|
137
|
+
const files = input.files; // a FileList, or null when the input has none
|
|
138
|
+
if (!files || files.length === 0) return;
|
|
139
|
+
const job = client.start("files", files, { metadata, signal }); // a File, an array or a FileList
|
|
140
|
+
const unsubscribe = job.subscribe(() => render(job.getSnapshot()));
|
|
141
|
+
|
|
142
|
+
const first = job.files[0];
|
|
143
|
+
first?.pause(); first?.resume(); first?.abort(); // one file
|
|
144
|
+
job.pause(); job.resume(); job.abort(); // every file of the job
|
|
145
|
+
|
|
146
|
+
const { data, error } = await job.done;
|
|
147
|
+
const again = await job.retry(); // or job.files[0].retry(): the failed files again
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
`input`, `metadata`, `signal` and `render` stand for your page's file input, the route's metadata, an optional
|
|
151
|
+
`AbortSignal` and your own rendering code.
|
|
152
|
+
|
|
153
|
+
- `start()` sends every file through one `/upload`, then runs each file on its own: one file failing or being
|
|
154
|
+
cancelled never stops another.
|
|
155
|
+
- `job.files` and `getSnapshot().files` are in the order the files were passed. Each file has an `id` (`"1"`, `"2"`, …)
|
|
156
|
+
that is unique per client.
|
|
157
|
+
- `job.done` resolves to the same `UploadResult` as `upload()` with a list, and never rejects.
|
|
158
|
+
|
|
159
|
+
### Snapshots and React
|
|
160
|
+
|
|
161
|
+
`getSnapshot()` returns the same object until something changes, so React's `useSyncExternalStore` works directly:
|
|
162
|
+
|
|
163
|
+
```tsx
|
|
164
|
+
import { useSyncExternalStore } from "react";
|
|
165
|
+
import type { UploadJob } from "cloudrop/client";
|
|
166
|
+
|
|
167
|
+
function Progress({ job }: { job: UploadJob<unknown> }) {
|
|
168
|
+
const snapshot = useSyncExternalStore(job.subscribe, job.getSnapshot);
|
|
169
|
+
return <progress value={snapshot.totals.progress} />;
|
|
170
|
+
}
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
Each file in `snapshot.files` has:
|
|
174
|
+
|
|
175
|
+
| Field | Meaning |
|
|
176
|
+
| --- | --- |
|
|
177
|
+
| `status` | `queued`, `uploading`, `paused`, `completing`, `done`, `failed` or `aborted`. The last three are final. |
|
|
178
|
+
| `bytes`, `size` | Bytes sent and the file's size. |
|
|
179
|
+
| `progress` | From 0 to 1. |
|
|
180
|
+
| `speed` | Bytes per second, averaged over about 3 s. 0 unless uploading. |
|
|
181
|
+
| `eta` | Seconds left at the current speed; `null` while nothing moves. |
|
|
182
|
+
| `error`, `result` | The file's error, or its `/complete` result. |
|
|
183
|
+
|
|
184
|
+
`snapshot.totals` adds up `bytes`, `size`, `progress`, `speed` and `eta` over the files that did not fail or abort, so
|
|
185
|
+
a failed file never holds the job below 1. It also counts the `done`, `failed` and `aborted` files, and `count` is every
|
|
186
|
+
file of the job.
|
|
187
|
+
|
|
188
|
+
Listeners are called at once on a status change, at most every 100 ms while bytes flow, and every second while a file
|
|
189
|
+
uploads.
|
|
190
|
+
|
|
191
|
+
### Statuses
|
|
192
|
+
|
|
193
|
+
```text
|
|
194
|
+
queued -> uploading -> completing -> done
|
|
195
|
+
completing -> uploading (/complete reported parts missing)
|
|
196
|
+
queued or uploading -> paused -> (resume) -> queued
|
|
197
|
+
any status that is not final -> failed or aborted
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
A file is `queued` until its first bucket `PUT` gets a connection, `uploading` while its bytes go to the bucket, and
|
|
201
|
+
`completing` once every byte is sent and `/complete` runs. When `/complete` reports parts missing, the file goes back to
|
|
202
|
+
`uploading` to send them. A file resumed after `pause()` is `queued` again until it gets a connection. A file resumed
|
|
203
|
+
after a reload starts `queued` while its record is checked, then goes to `completing`, since its first request is
|
|
204
|
+
`/complete` (see [Resume after reload](resume.md#the-server-is-asked-first)).
|
|
205
|
+
|
|
206
|
+
### Pause and resume
|
|
207
|
+
|
|
208
|
+
- `pause()` stops a queued or uploading file's requests and keeps the parts already stored in the bucket. A file that
|
|
209
|
+
is completing (its bytes all sent) ignores it.
|
|
210
|
+
- `resume()` sends only the parts not yet stored in the bucket. A single `PUT` starts again from the beginning.
|
|
211
|
+
- `job.pause()` and `job.resume()` do that for every file.
|
|
212
|
+
|
|
213
|
+
### Test pause and progress locally
|
|
214
|
+
|
|
215
|
+
On `localhost` an upload finishes before you can press Pause. Slow it down:
|
|
216
|
+
|
|
217
|
+
- In Chrome's DevTools, open the Network panel and pick a throttling profile such as "Slow 4G", or add a custom one.
|
|
218
|
+
It slows the tab's bucket `PUT`s too, so progress, speed and Pause have time to show. The Offline profile fires the
|
|
219
|
+
page's `offline` event, which pauses the uploads (see [Offline](#offline)).
|
|
220
|
+
- Use a file over the server's `multipartThreshold` to see parts. In development you can lower the threshold, as
|
|
221
|
+
[`examples/next`](https://github.com/vcode-sh/cloudrop/blob/main/examples/next/README.md) does with `multipartThreshold: "8MB"`.
|
|
222
|
+
- The example's `/lab` page can delay every bucket `PUT` and inject failures, and shows each request.
|
|
223
|
+
|
|
224
|
+
### Cancel
|
|
225
|
+
|
|
226
|
+
`abort()` on a file, `job.abort()`, or the `signal` you passed cancels: requests stop, the file ends `aborted`, and the
|
|
227
|
+
server is told to clean up with `/abort`. A file still waiting for its ticket ends `aborted` at once; a ticket that
|
|
228
|
+
arrives for it later only gets its `/abort`. A failed file ends `aborted` too, and the server is told to drop what it
|
|
229
|
+
kept for `retry()`.
|
|
230
|
+
|
|
231
|
+
The one exception: a file after a `/complete` that may have succeeded is kept, never deleted.
|
|
232
|
+
|
|
233
|
+
### Retry
|
|
234
|
+
|
|
235
|
+
- `job.files[n].retry()` sends a failed file again and resolves with its outcome.
|
|
236
|
+
- `job.retry()` does that for every failed file and resolves with an `UploadResult` of the job's current outcomes once
|
|
237
|
+
every file has settled. Its `data` is `null` when its `/upload` for every file was refused, as with `done`.
|
|
238
|
+
- A retry reuses the file's ticket when it can, so the parts already stored in the bucket are not sent again and the
|
|
239
|
+
bytes stay counted.
|
|
240
|
+
- It asks `/upload` for a new ticket when the old one expired or was given up, or when the failure was one a new attempt
|
|
241
|
+
with the same ticket cannot fix: `UPLOAD_EXPIRED`, `INVALID_TOKEN`, `UPLOAD_REJECTED`, `VERIFICATION_FAILED`,
|
|
242
|
+
`NO_SUCH_UPLOAD`, or a file that never got a ticket. `job.retry()` asks for all of those files in one `/upload`. On a
|
|
243
|
+
new ticket the file starts again under a new key, and its `bytes` drop to 0.
|
|
244
|
+
- When a `/complete` may have succeeded, a retry sends `/complete` again first, never a second upload.
|
|
245
|
+
- `retry()` of a file that is still running waits for it to settle. Of a done or aborted file, it answers that outcome
|
|
246
|
+
at once.
|
|
247
|
+
- `job.done` keeps each file's first outcome, whatever a retry does later. The snapshot shows the current one.
|
|
248
|
+
|
|
249
|
+
## How files are sent
|
|
250
|
+
|
|
251
|
+
- Files up to the server's `multipartThreshold` (64 MiB by default) go in one `PUT`. Larger ones go in parts of 8 MiB
|
|
252
|
+
(16 MiB from 256 MiB, larger only when a file would need over 10,000 parts), read with `Blob.slice`, at most four
|
|
253
|
+
parts of one file at a time.
|
|
254
|
+
- Every part except the last has exactly the same size. R2 requires equal part sizes in a multipart upload, so the
|
|
255
|
+
server plans one part size per file and the client cuts the file by that plan.
|
|
256
|
+
- A request is tried up to 6 times with backoff: a random wait up to 0.5 s, then up to 1 s, 2 s and so on, at most 20 s.
|
|
257
|
+
When the answer carries `Retry-After`, the wait is at least that long (at most 20 s).
|
|
258
|
+
- An expired upload URL is renewed once.
|
|
259
|
+
- Parts that `/complete` reports missing are sent again, at most twice.
|
|
260
|
+
- Calls to your server send cookies (`credentials: "include"`), so `authorize` can read the session. Uploads to the
|
|
261
|
+
bucket send none.
|
|
262
|
+
|
|
263
|
+
A browser hides every answer header the bucket's CORS rule does not expose. Expose `Retry-After` in the rule, or the
|
|
264
|
+
client cannot read it and waits its own backoff alone. See [Bucket setup](bucket-setup.md).
|
|
265
|
+
|
|
266
|
+
### When /upload refuses one file
|
|
267
|
+
|
|
268
|
+
`/upload` answers per file. A file it refuses with a retryable error (the storage service failed or was busy while the
|
|
269
|
+
server started it) asks `/upload` again for itself alone, with the backoff above and within the attempts the job's
|
|
270
|
+
`/upload` left, while the job's other files go on.
|
|
271
|
+
|
|
272
|
+
- The file stays `queued` while it waits.
|
|
273
|
+
- Pausing it stops the wait until it resumes. Cancelling it ends it `aborted` at once, without another request.
|
|
274
|
+
- Going offline pauses it while it waits, after a refusal or after a later `/upload` that failed as a whole, whatever
|
|
275
|
+
the job's other files do; going back online asks at once. An `/upload` already sent is not stopped: a ticket it
|
|
276
|
+
brings waits paused.
|
|
277
|
+
- A refusal that is not retryable, such as a file over the size limit, fails the file at once.
|
|
278
|
+
|
|
279
|
+
## Concurrency and order
|
|
280
|
+
|
|
281
|
+
One client shares its connection slots between all its uploads. Each bucket `PUT` holds one slot, and one file has at
|
|
282
|
+
most four parts in flight. Calls to your server take no slot.
|
|
283
|
+
|
|
284
|
+
- **`concurrency: "auto"`** (the default) starts with 4 slots. After each 2 s in which the bytes sent rose by 10 % or
|
|
285
|
+
more while `PUT`s waited, it adds one, up to 8. A network error, a stall, or a `429` or `503` from the bucket halves
|
|
286
|
+
the slots.
|
|
287
|
+
- **A number from 1 to 16** fixes the slots.
|
|
288
|
+
- **`order: "added"`** (the default) sends files added earlier first. **`order: "smallest-first"`** sends smaller files
|
|
289
|
+
first, across every job of the client.
|
|
290
|
+
|
|
291
|
+
## Offline
|
|
292
|
+
|
|
293
|
+
When the page goes offline, every queued and uploading file pauses (`status: "paused"`) and resumes when the page is
|
|
294
|
+
back online.
|
|
295
|
+
|
|
296
|
+
- A file you paused stays paused. Your `pause()` or `resume()` of a file the network paused makes it yours.
|
|
297
|
+
- Files that start while the page is offline pause before any `PUT`.
|
|
298
|
+
- The `/upload` that starts a job, or that a `job.retry()` sends for several files, is not held: with no network it
|
|
299
|
+
fails those files with `NETWORK_ERROR` after its attempts, and `retry()` sends it again. A one-file `/upload` (a
|
|
300
|
+
file asking again after a refusal, a `file.retry()` that needs a new ticket) pauses while it waits.
|
|
301
|
+
|
|
302
|
+
## Stalled requests
|
|
303
|
+
|
|
304
|
+
In a browser, a bucket `PUT` that makes no progress for 30 s, or whose answer takes that long after the last byte,
|
|
305
|
+
counts as a network error and is tried again. Through `fetch`, which reports no progress, a `PUT` gets 30 s plus its
|
|
306
|
+
size at 100 KiB/s.
|
|
307
|
+
|
|
308
|
+
Calls to your server have a deadline for the whole answer: 30 s for `/upload` and `/sign`, 120 s for `/complete`. A
|
|
309
|
+
missed deadline is a retryable `NETWORK_ERROR`, tried again within the same attempts. Each `/upload` ask carries a
|
|
310
|
+
random `idempotencyKey`, the same on all of its attempts, so a server with a database store releases what an attempt
|
|
311
|
+
whose answer never arrived reserved. A `/complete` that timed out counts as one that may have finished, so a cancel or a
|
|
312
|
+
failure afterwards keeps the file. `/abort` gets 10 s and is sent once; if it is missed, the bucket's lifecycle rule
|
|
313
|
+
cleans up.
|
|
314
|
+
|
|
315
|
+
## Transports: XHR or fetch
|
|
316
|
+
|
|
317
|
+
- In a browser, bucket `PUT`s go through `XMLHttpRequest`, which reports bytes while they are sent, so progress moves
|
|
318
|
+
smoothly.
|
|
319
|
+
- Passing the `fetch` option sends everything through it, `PUT`s included. A part's bytes then count only once it is
|
|
320
|
+
stored, so progress moves a part at a time. To add a token or other headers to your server's requests, use the
|
|
321
|
+
`headers` option instead: it leaves `PUT`s on `XMLHttpRequest`.
|
|
322
|
+
- Outside a browser (Node, Bun, Deno, Workers), `PUT`s use `fetch` the same way.
|
|
323
|
+
|
|
324
|
+
## What gets cleaned up
|
|
325
|
+
|
|
326
|
+
- **Cancelling** returns `ABORTED` and tells the server to clean up with `/abort`.
|
|
327
|
+
- **A single-`PUT` file of `upload()` that fails after `/upload`** is cleaned up the same way, in the background, so
|
|
328
|
+
`upload()` answers the error at once. `upload()` returns no handle to retry with, so there is nothing to keep.
|
|
329
|
+
- **A multipart file of `upload()` that fails** for a reason a later attempt can fix, such as the network, keeps its
|
|
330
|
+
parts and its resume record, so a reload can [resume](resume.md) it and send only the parts the bucket lacks; no
|
|
331
|
+
`/abort` is sent. One the bucket or the server refused, or whose ticket expired, is cleaned up as a single `PUT` is,
|
|
332
|
+
and so is every failed file with persistence off or unavailable.
|
|
333
|
+
- **A failed file of `start()`** keeps its ticket and the parts already stored in the bucket, so `retry()` can reuse
|
|
334
|
+
them, until `abort()` of the file or the job (or `options.signal`), `pagehide`, or the ticket's expiry (24 hours).
|
|
335
|
+
`abort()` tells the server with `/abort`. `pagehide` does too, except for a multipart file `upload()` would keep,
|
|
336
|
+
which keeps its parts and its resume record for a reload. The bucket's lifecycle rule removes unfinished parts nobody
|
|
337
|
+
resumes. A failed single-`PUT` object kept until expiry stays in the bucket: call `job.abort()` once you will not
|
|
338
|
+
retry.
|
|
339
|
+
- **A file after a `/complete` that may have succeeded** is always kept, never deleted.
|
|
340
|
+
- **[`close()`](#close-the-client)** cancels every file, or, without `abort`, keeps each multipart upload's record for
|
|
341
|
+
its user's next client.
|
|
342
|
+
|
|
343
|
+
## Close the client
|
|
344
|
+
|
|
345
|
+
`close()` stops a client for good, which an app with signed-in users needs when the user changes:
|
|
346
|
+
|
|
347
|
+
```ts
|
|
348
|
+
await client.close({ abort: true }); // at sign-out: cancel every upload, telling the server
|
|
349
|
+
await client.forget(); // then delete the user's unfinished uploads
|
|
350
|
+
```
|
|
351
|
+
|
|
352
|
+
- **Afterwards** `start()`, `upload()` and `resume()` answer `CLIENT_CLOSED` before any request, and a job's or a
|
|
353
|
+
file's `retry()` sends nothing. `resumable()`, `discard()` and `forget()` keep working. Create a new client for the
|
|
354
|
+
next user.
|
|
355
|
+
- **Every file of every route stops,** queued, uploading, paused, failed or waiting for its ticket, in any phase. A file
|
|
356
|
+
waiting for its ticket ends `aborted` at once, and its `/upload`, if in flight, is cancelled.
|
|
357
|
+
- **`close({ abort: true })`,** at sign-out, cancels each file as `abort()` does: its `/abort` goes with its own
|
|
358
|
+
upload token and `keepalive`, so a reload right after does not stop it, and the records of those uploads go.
|
|
359
|
+
- **`close()` without options,** when another user signed in in another tab, stops each file as closing the tab would.
|
|
360
|
+
A multipart upload sends no `/abort` and keeps its record, which the same user [resumes](resume.md) on a new client
|
|
361
|
+
after signing in again, in this page or after a reload. A single `PUT`, which cannot resume, is cancelled, and so is
|
|
362
|
+
every file when persistence is off or unavailable.
|
|
363
|
+
- **A file whose `/complete` may have finished** is kept, never deleted, either way.
|
|
364
|
+
- **It resolves** once every file it stopped has ended and each `/abort` it sent was answered; it never rejects. A file
|
|
365
|
+
in the middle of `/complete` waits for that answer, at most 2 minutes. Await it before a sign-out ends what a
|
|
366
|
+
`headers` function reads, or that function cannot give the `/abort`s their headers.
|
|
367
|
+
- **The first call decides,** and every later call does nothing but resolve with the first.
|
|
368
|
+
|
|
369
|
+
In React, the factory returns the same `close()`: [Signed-in users](react.md#signed-in-users).
|
|
370
|
+
|
|
371
|
+
## Errors that only the client produces
|
|
372
|
+
|
|
373
|
+
Five codes come from the client and never from the server:
|
|
374
|
+
|
|
375
|
+
- `UPLOAD_REJECTED`: the bucket refused the bytes as signed (a different size, type or checksum).
|
|
376
|
+
- `ABORTED`: a cancel.
|
|
377
|
+
- `NETWORK_ERROR`: no answer after every attempt. Retryable.
|
|
378
|
+
- `UPLOAD_IN_USE`: `resume()` or `discard()` found the upload held in this browser (see
|
|
379
|
+
[Resume after reload](resume.md#two-tabs)).
|
|
380
|
+
- `CLIENT_CLOSED`: `start()`, `upload()` or `resume()` after [`close()`](#close-the-client).
|
|
381
|
+
|
|
382
|
+
Every other code, and what to do about it: [Errors](errors.md).
|
|
383
|
+
|
|
384
|
+
Why the client behaves as it does: the [M3-S decisions](https://github.com/vcode-sh/cloudrop/blob/main/docs/decisions/m3-slice.md), the
|
|
385
|
+
[M3-U decisions](https://github.com/vcode-sh/cloudrop/blob/main/docs/decisions/m3-uploader.md) and, for `close()` and `headers`, the
|
|
386
|
+
[M8-L decisions](https://github.com/vcode-sh/cloudrop/blob/main/docs/decisions/m8-client.md).
|
package/docs/command.md
ADDED
|
@@ -0,0 +1,129 @@
|
|
|
1
|
+
# The cloudrop command
|
|
2
|
+
|
|
3
|
+
This page is for whoever sets up an app's storage. The package ships a `cloudrop` command with two subcommands:
|
|
4
|
+
`doctor` tells you in plain words what is missing in your settings and your bucket, and `setup` gives you the bucket's
|
|
5
|
+
CORS and lifecycle rules, or adds them to the bucket for you. It runs on Node 24 or later and on Bun.
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
npx cloudrop doctor --origin https://app.example.com --origin http://localhost:5173
|
|
9
|
+
npx cloudrop setup --origin https://app.example.com # prints the rules and the commands
|
|
10
|
+
npx cloudrop setup --origin https://app.example.com --dry-run # shows what --apply would change
|
|
11
|
+
npx cloudrop setup --origin https://app.example.com --apply # adds them, keeping every other rule
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
`bunx cloudrop` works the same way. `--origin` names an origin your pages upload from (the page's, not your API's) and
|
|
15
|
+
is required; repeat it for each origin. An origin is exactly what the browser sends: scheme, host and port, in lower
|
|
16
|
+
case, with no path, not even a trailing `/`, and no `*`. Given `http://localhost:5173/`, the command answers with the
|
|
17
|
+
origin to use instead, `http://localhost:5173`; it refuses a pattern such as `https://*.example.com`, which no browser
|
|
18
|
+
sends: name each origin.
|
|
19
|
+
|
|
20
|
+
## Settings
|
|
21
|
+
|
|
22
|
+
The command reads the same variables as the library: `CLOUDROP_R2_ACCOUNT_ID`, `CLOUDROP_R2_BUCKET`,
|
|
23
|
+
`CLOUDROP_R2_ACCESS_KEY_ID` and `CLOUDROP_R2_SECRET_ACCESS_KEY` for R2, or the `CLOUDROP_S3_*` variables for another
|
|
24
|
+
S3-compatible service, and `CLOUDROP_SECRET`. When any `CLOUDROP_R2_*` variable is set it checks R2, otherwise S3.
|
|
25
|
+
|
|
26
|
+
- `--env-file <path>` also reads a file of `NAME=value` lines, such as `.env.local`, the way Node's `--env-file` does:
|
|
27
|
+
`export` in front of a name is allowed, a value in single quotes, double quotes or backticks is taken as written
|
|
28
|
+
and may run over several lines (`\n` in double quotes is a line break), and in an unquoted value `#` starts a
|
|
29
|
+
comment, so quote a value that contains one. Nothing in the file is run or expanded. A variable set in the environment
|
|
30
|
+
wins over the file, as with Node, and the command says so for each cloudrop variable set in both.
|
|
31
|
+
- **On Node, a missing `--env-file` stops before the command runs.** Node itself checks every `--env-file` on its
|
|
32
|
+
command line, the command's own included, and exits with code 9 and `node: .env.local: not found` (or
|
|
33
|
+
`invalid format` for a directory). Check the path. Bun runs the command, which reports the file with exit code 2.
|
|
34
|
+
- `--jurisdiction eu` (or `fedramp`) selects the endpoint of an R2 bucket in a jurisdiction, as `r2({ jurisdiction })`
|
|
35
|
+
does.
|
|
36
|
+
|
|
37
|
+
A name that is not a setting but looks like one, such as `CLOUDROP_R2_BUCKT` or `cloudrop_secret`, gets a `note` naming
|
|
38
|
+
the setting it was probably meant to be, among the settings of the service its name names. `CLOUDROP_R2_ENDPOINT` and
|
|
39
|
+
`CLOUDROP_R2_REGION` get a note that R2 needs neither. When settings for R2 and S3 are both set, a `note` says that R2
|
|
40
|
+
is checked.
|
|
41
|
+
|
|
42
|
+
The command never prints a setting's value, only its name. The commands it prints name the bucket as
|
|
43
|
+
`"$CLOUDROP_R2_BUCKET"` (or `"$CLOUDROP_S3_BUCKET"`), so they work in a shell that has the variable set.
|
|
44
|
+
|
|
45
|
+
## doctor
|
|
46
|
+
|
|
47
|
+
`doctor` checks, in order:
|
|
48
|
+
|
|
49
|
+
1. that every storage setting is present and valid, naming the variable to fix;
|
|
50
|
+
2. that `CLOUDROP_SECRET` is set and has at least 32 characters;
|
|
51
|
+
3. that the bucket answers and the credentials may list it (a listing, which tells a missing bucket from a missing
|
|
52
|
+
object). When it does not, the finding names the settings to check: the account ID (R2) or the endpoint (S3) when
|
|
53
|
+
the service cannot be reached, the keys, the bucket and `--jurisdiction` when it refuses them, the region (S3) when
|
|
54
|
+
it answers that the bucket is in another one;
|
|
55
|
+
4. the bucket's CORS rule for each `--origin` and its lifecycle rule, as `cloudrop.api.checkBucket` does.
|
|
56
|
+
|
|
57
|
+
Each line is one finding: `ok`, `PROBLEM` with what to do, `NOT CHECKED` with how to check by hand, or a `note`. The
|
|
58
|
+
last line sums them up. Against an R2 bucket with an Object Read & Write token it prints:
|
|
59
|
+
|
|
60
|
+
```text
|
|
61
|
+
cloudrop doctor: checking R2 settings, the bucket and its rules for http://localhost:5173.
|
|
62
|
+
ok R2 settings are present and valid (CLOUDROP_R2_*).
|
|
63
|
+
ok CLOUDROP_SECRET is set.
|
|
64
|
+
ok The bucket answers, and these credentials may list it.
|
|
65
|
+
NOT CHECKED CORS: These storage credentials may not read the bucket's CORS rules (R2 object tokens cannot). Check it with an admin read token or in the dashboard. By hand: wrangler r2 bucket cors list "$CLOUDROP_R2_BUCKET"
|
|
66
|
+
NOT CHECKED Lifecycle: These storage credentials may not read the bucket's lifecycle rules (R2 object tokens cannot). Check it with an admin read token or in the dashboard. By hand: wrangler r2 bucket lifecycle list "$CLOUDROP_R2_BUCKET"
|
|
67
|
+
No problems found. 2 checks could not run; see above.
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
An object token cannot read bucket settings, so those two checks are reported as not checked, never as a failure. Run
|
|
71
|
+
`doctor` with an admin token to check them too.
|
|
72
|
+
|
|
73
|
+
`doctor` writes nothing, so it does not prove that the credentials may upload: a read-only token lists the bucket as
|
|
74
|
+
well as a read-write one does. Give the app a token with Object Read & Write
|
|
75
|
+
([Credentials and permissions](bucket-setup.md#credentials-and-permissions)).
|
|
76
|
+
|
|
77
|
+
Your code may pass settings to `r2()`, `s3()` or `createCloudrop()` instead of the environment; `doctor` reads only the
|
|
78
|
+
environment and the env file, so it reports those settings as missing.
|
|
79
|
+
|
|
80
|
+
## setup
|
|
81
|
+
|
|
82
|
+
Without `--apply`, `setup` reaches nothing. It prints the CORS rule in R2's format (`cors.json` for Wrangler) and in the
|
|
83
|
+
S3 format (`cors-s3.json` for the AWS CLI), the lifecycle rule (`lifecycle.json`), and the commands that set them: the
|
|
84
|
+
Wrangler ones for R2, the AWS CLI ones for other services, both when no settings name the service. The rules are those
|
|
85
|
+
of [Bucket setup](bucket-setup.md): `PUT`, `GET` and `HEAD` from your origins with the four headers cloudrop signs,
|
|
86
|
+
`ETag` and `Retry-After` exposed, and an abort of incomplete multipart uploads after 1 day across the bucket.
|
|
87
|
+
|
|
88
|
+
Setting CORS with Wrangler or the AWS CLI replaces every rule the bucket has. List the current rules first and add the
|
|
89
|
+
ones you keep to the file, or use `--apply`.
|
|
90
|
+
|
|
91
|
+
With `--apply`, `setup` writes the rules through the S3 API (`PutBucketCors` and `PutBucketLifecycleConfiguration`):
|
|
92
|
+
|
|
93
|
+
- **It merges, it never replaces.** It reads the current rules and adds cloudrop's own rule: `cloudrop-uploads` for
|
|
94
|
+
CORS, `cloudrop-abort-incomplete` for lifecycle. Every other rule is written back as it was read, a lifecycle rule's
|
|
95
|
+
expirations, transitions and filters included, and so is the bucket's transition default on AWS.
|
|
96
|
+
- **It only adds.** When `cloudrop-uploads` exists, it gains what it lacks (an origin you pass, a method or header
|
|
97
|
+
cloudrop needs) where it stands; origins, headers and the max age already on it stay. To remove an origin, edit the
|
|
98
|
+
rule by hand.
|
|
99
|
+
- When another enabled rule already aborts incomplete multipart uploads across the bucket (R2 creates one by default),
|
|
100
|
+
the lifecycle rules are left as they are. When `cloudrop-abort-incomplete` exists but no longer does that (someone
|
|
101
|
+
disabled it or gave it a prefix), `setup` leaves the lifecycle rules alone, still writes the CORS rule, and reports
|
|
102
|
+
the problem (exit 1): fix or delete that rule by hand.
|
|
103
|
+
- It writes nothing for a setting that would not change.
|
|
104
|
+
- It needs credentials that may read and change bucket settings, such as an R2 Admin Read & Write token. Use it for
|
|
105
|
+
this step only; the app needs no more than Object Read & Write.
|
|
106
|
+
- When it cannot read the current rules, it writes nothing and exits 1. It also lists the bucket, as `doctor` does.
|
|
107
|
+
When the listing fails too, the keys, the bucket name or the network is the problem, and it names the settings to
|
|
108
|
+
check. When the listing works and the read was refused, the credentials may not read bucket settings, as an R2 object
|
|
109
|
+
token cannot; it names an R2 Admin Read & Write token, or S3's two permissions for the setting. `--replace` (with
|
|
110
|
+
`--apply` or `--dry-run`) writes cloudrop's rules alone instead, replacing what the bucket has, but only when the
|
|
111
|
+
read was refused (`403`). A read that failed for any other reason, such as a timeout or a service that does not
|
|
112
|
+
support it, never leads to a write. Neither does a configuration written in XML the command does not read (CDATA,
|
|
113
|
+
comments or namespace prefixes): check that bucket's rules by hand.
|
|
114
|
+
|
|
115
|
+
`--dry-run` does everything `--apply` does except the writes, and shows the result: which rule it would add or update
|
|
116
|
+
and which rules it would keep. R2 can take up to 30 seconds to apply a change; run `doctor` afterwards to check.
|
|
117
|
+
|
|
118
|
+
Wrangler's local R2 cannot read or write bucket settings (it answers `501`), and needs no CORS rule.
|
|
119
|
+
|
|
120
|
+
## Exit codes
|
|
121
|
+
|
|
122
|
+
| Code | Meaning |
|
|
123
|
+
| --- | --- |
|
|
124
|
+
| `0` | Nothing is wrong: every check passed or could not run, or `setup` printed, wrote or would write the rules. |
|
|
125
|
+
| `1` | `doctor` found a problem, or `setup` could not read, could not write, or found the settings incomplete. |
|
|
126
|
+
| `2` | A usage error: an unknown command or option, a missing `--origin`, a value that is not an exact origin, `--replace` without `--apply` or `--dry-run`, or an `--env-file` that cannot be read. |
|
|
127
|
+
| `9` | Node's own, not the command's: an `--env-file` that does not exist or is a directory (see [Settings](#settings)). |
|
|
128
|
+
|
|
129
|
+
In CI, `cloudrop doctor` fails the job on a problem and passes when it could only not check something.
|