cloudrop 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +55 -0
- package/README.md +29 -1
- package/dist/adapters/sql.d.ts +3 -0
- package/dist/adapters/sql.js +7 -0
- package/dist/adapters/sql.js.map +1 -1
- package/dist/cli.js +2045 -0
- package/dist/{client-ePVHwvMv.js → client-DxP3uv8V.js} +47 -5
- package/dist/client-DxP3uv8V.js.map +1 -0
- package/dist/client.js +1 -1
- package/dist/index.d.ts +25 -1
- package/dist/index.js +86 -45
- package/dist/index.js.map +1 -1
- package/dist/react.js +1 -1
- package/dist/testing.d.ts +81 -0
- package/dist/testing.js +1243 -0
- package/dist/testing.js.map +1 -0
- package/docs/README.md +30 -0
- package/docs/bucket-setup.md +226 -0
- package/docs/client.md +386 -0
- package/docs/command.md +129 -0
- package/docs/deployment.md +169 -0
- package/docs/errors.md +150 -0
- package/docs/hono.md +276 -0
- package/docs/react.md +508 -0
- package/docs/resume.md +253 -0
- package/docs/server.md +603 -0
- package/docs/store.md +804 -0
- package/docs/testing-your-app.md +328 -0
- package/docs/troubleshooting.md +176 -0
- package/package.json +10 -2
- package/dist/client-ePVHwvMv.js.map +0 -1
package/docs/store.md
ADDED
|
@@ -0,0 +1,804 @@
|
|
|
1
|
+
# Database store
|
|
2
|
+
|
|
3
|
+
This page is for developers who want a record of every upload in their own database. cloudrop's server is stateless by
|
|
4
|
+
default: the signed upload token carries everything, and nothing is written anywhere. Pass a database store and cloudrop
|
|
5
|
+
keeps one row per upload in two tables of your database.
|
|
6
|
+
|
|
7
|
+
## Quick start
|
|
8
|
+
|
|
9
|
+
1. **Create the two tables.** [Create the tables](#create-the-tables) has the SQL for Postgres and SQLite, the Drizzle
|
|
10
|
+
tables and the Prisma models. Until they exist, every upload fails with a retryable `SERVER_ERROR`, and the log
|
|
11
|
+
names the missing table (`INVALID_CONFIG`).
|
|
12
|
+
2. **Build the store** with the adapter for the driver your app already uses ([Set it up](#set-it-up)). Here,
|
|
13
|
+
node-postgres with a transaction, so `onComplete` can write through `tx`:
|
|
14
|
+
|
|
15
|
+
```ts
|
|
16
|
+
import { pgExecutor, sqlStore } from "cloudrop/adapters/sql";
|
|
17
|
+
import pg from "pg";
|
|
18
|
+
|
|
19
|
+
const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });
|
|
20
|
+
const store = sqlStore(pgExecutor(pool), { transaction: true });
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
3. **Pass it to `createCloudrop`**, and give each route an `owner`:
|
|
24
|
+
|
|
25
|
+
```ts
|
|
26
|
+
// lib/cloudrop.ts
|
|
27
|
+
import { createCloudrop, r2, unauthorized } from "cloudrop";
|
|
28
|
+
|
|
29
|
+
export const cloudrop = createCloudrop({
|
|
30
|
+
storage: r2({ bucket: "media" }),
|
|
31
|
+
store,
|
|
32
|
+
quota: () => 1024 * 1024 * 1024, // 1 GiB per owner
|
|
33
|
+
routes: {
|
|
34
|
+
avatar: store.route(
|
|
35
|
+
{
|
|
36
|
+
accept: "image/*",
|
|
37
|
+
maxSize: "5MB",
|
|
38
|
+
replace: true, // one avatar per user: the next one deletes this one
|
|
39
|
+
authorize: async ({ request }) => {
|
|
40
|
+
const user = await getUser(request); // your app's session lookup
|
|
41
|
+
return user ? { userId: user.id } : unauthorized();
|
|
42
|
+
},
|
|
43
|
+
},
|
|
44
|
+
{
|
|
45
|
+
owner: ({ userId }) => userId,
|
|
46
|
+
onComplete: async ({ tx, userId, file }) => {
|
|
47
|
+
await tx.query("UPDATE users SET avatar_key = $1 WHERE id = $2", [file.key, userId]);
|
|
48
|
+
return { key: file.key };
|
|
49
|
+
},
|
|
50
|
+
afterComplete: async ({ userId, file }) => {
|
|
51
|
+
await notifyFollowers(userId, file.key); // network calls belong here
|
|
52
|
+
},
|
|
53
|
+
},
|
|
54
|
+
),
|
|
55
|
+
},
|
|
56
|
+
});
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
The client needs no change: it speaks the same protocol, and gets two more answers (`QUOTA_EXCEEDED`, and a busy
|
|
60
|
+
`SERVER_ERROR` while another request completes the same file) that it already handles.
|
|
61
|
+
|
|
62
|
+
This page covers what the store guarantees, how to set it up with each adapter, how to create the tables, the
|
|
63
|
+
callbacks and file operations it adds, the sweep that cleans up after abandoned uploads, and what it costs. The
|
|
64
|
+
[Next.js example](https://github.com/vcode-sh/cloudrop/blob/main/examples/next/README.md#run-with-a-database-store) runs it on SQLite.
|
|
65
|
+
|
|
66
|
+
## Why a store
|
|
67
|
+
|
|
68
|
+
Without one, cloudrop already uploads safely: the token is the authority, and `onComplete` runs at least once per file
|
|
69
|
+
with a stable `file.id`. What it cannot do without a record of its own:
|
|
70
|
+
|
|
71
|
+
- **Run `onComplete` once.** A client that lost the answer to `/complete` asks again, and without a row the server
|
|
72
|
+
cannot tell that the file is done. Your `onComplete` must then be idempotent.
|
|
73
|
+
- **Enforce a quota that cannot be overshot.** Checking a total at `/upload` lets several uploads started at once all
|
|
74
|
+
pass; a store reserves the bytes at `/upload` and counts them when the upload completes.
|
|
75
|
+
- **Stop an upload.** A signed token stays valid for 24 hours, even after the user signs out. A row can be ended.
|
|
76
|
+
- **Keep `/abort` from deleting a finished file.** Without a store, whoever holds the token can delete the file through
|
|
77
|
+
`/abort` until the token expires, even after `onComplete` ran.
|
|
78
|
+
- **Clean up after abandoned uploads.** A row says which objects belong to an upload that never finished.
|
|
79
|
+
|
|
80
|
+
## What it guarantees
|
|
81
|
+
|
|
82
|
+
- **A row before anything is signed.** `/upload` writes one row per accepted file before it signs a URL or creates a
|
|
83
|
+
multipart upload. When the rows cannot be written, every file of that `/upload` is refused with a retryable error.
|
|
84
|
+
- **`onComplete` once, exactly once for database work on Postgres with a transaction.** With a store built with
|
|
85
|
+
`transaction: true` (Postgres only, on any of the three adapters), the claim on the file, your `onComplete`'s writes
|
|
86
|
+
through `tx` and the completion's record commit together, or not at all. A crash, a throw or a timeout rolls
|
|
87
|
+
everything back, and the next `/complete` runs `onComplete` again from a clean state. Writes through anything other
|
|
88
|
+
than `tx` (your global database client, an API call) are outside that guarantee.
|
|
89
|
+
- **Otherwise, once per file under a lease.** On SQLite and D1, and with any store built without `transaction: true`,
|
|
90
|
+
a `/complete` claims the file for `completionLeaseMs` while `onComplete` runs. Another `/complete` of the same file
|
|
91
|
+
is answered busy, and the client retries. `onComplete` runs again in three cases only:
|
|
92
|
+
- the server stops, or the completion's record fails, between the claim and that record, and the lease then ends;
|
|
93
|
+
- `onComplete` runs longer than the lease, which is never extended: the next `/complete` takes the file over and
|
|
94
|
+
runs `onComplete` beside the first run, whose record is then refused;
|
|
95
|
+
- `onComplete` throws: the claim is handed back at once, so the retry runs it without waiting.
|
|
96
|
+
|
|
97
|
+
`completionLeaseMs` is a `createCloudrop` option, 300000 (5 minutes) by default: set it above your slowest
|
|
98
|
+
`onComplete`. Keep `onComplete` idempotent in this mode: write with `INSERT … ON CONFLICT`, as the example's
|
|
99
|
+
`recordUpload` does.
|
|
100
|
+
- **Retries are answered from the row.** A `/complete` of a completed file answers the stored result with no storage
|
|
101
|
+
call and no hook, also after the token expired. So does a file deleted with `api.deleteFile`: its answer still names
|
|
102
|
+
the deleted object, so a resumed client does not upload it again.
|
|
103
|
+
- **A hard quota.** Concurrent uploads of one owner cannot pass their limit.
|
|
104
|
+
- **`/abort` never deletes a completed file**, and an ended row refuses `/sign` and `/complete`.
|
|
105
|
+
- **The sweep deletes only what it may.** It removes an object only when the object's `cloudrop-id` names a row that
|
|
106
|
+
never completed, and never touches completed files or keys without a row.
|
|
107
|
+
|
|
108
|
+
## Set it up
|
|
109
|
+
|
|
110
|
+
Every adapter builds the same store over one SQL core; pick the one that matches how your app reaches its database.
|
|
111
|
+
Each method of the store is one atomic unit (one statement on Postgres, one batch on SQLite), and the store checks
|
|
112
|
+
its tables on its first call: a missing table or column is logged as `INVALID_CONFIG` naming it, and clients get a
|
|
113
|
+
retryable `SERVER_ERROR` until the tables are fixed.
|
|
114
|
+
|
|
115
|
+
| Adapter | Databases | Completions | Limits |
|
|
116
|
+
| --- | --- | --- | --- |
|
|
117
|
+
| `sqlStore` (`cloudrop/adapters/sql`) | Postgres through node-postgres or postgres.js; SQLite through `node:sqlite`; Cloudflare D1 | Transaction with `{ transaction: true }` on Postgres, otherwise lease | A node-postgres `Pool`, never a single `Client` |
|
|
118
|
+
| `drizzleStore` (`cloudrop/adapters/drizzle`) | Postgres through node-postgres or postgres.js; SQLite through better-sqlite3, libsql (Turso) or D1 | Transaction with `transaction: true` on Postgres, otherwise lease | A single node-postgres `Client` is refused |
|
|
119
|
+
| `prismaStore` (`cloudrop/adapters/prisma`) | Prisma 7 with a driver adapter, on Postgres or SQLite | Transaction with `transaction: true` on Postgres, otherwise lease | Prisma's 5-second transaction timeout includes `onComplete`; SQLite for one process |
|
|
120
|
+
|
|
121
|
+
`store.route`, and with it `tx` in `onComplete`, exists only on a store built with `transaction: true`. SQLite and D1
|
|
122
|
+
have no transaction mode: an interactive transaction would hold their only writer while `onComplete` awaits.
|
|
123
|
+
|
|
124
|
+
### Plain SQL drivers: `cloudrop/adapters/sql`
|
|
125
|
+
|
|
126
|
+
```ts
|
|
127
|
+
import { d1Executor, pgExecutor, postgresJsExecutor, sqliteExecutor, sqlStore } from "cloudrop/adapters/sql";
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
| Driver | Store | With `{ transaction: true }` |
|
|
131
|
+
| --- | --- | --- |
|
|
132
|
+
| node-postgres | `sqlStore(pgExecutor(new pg.Pool({ connectionString })))` | `tx` is the transaction's client (`tx.query`) |
|
|
133
|
+
| postgres.js | `sqlStore(postgresJsExecutor(postgres(connectionString)))` | `tx` is postgres.js's transaction (`` tx`…` ``) |
|
|
134
|
+
| `node:sqlite` | `sqlStore(sqliteExecutor(new DatabaseSync(path, { timeout: 5000 })))` | Refused: lease only |
|
|
135
|
+
| Cloudflare D1 | `sqlStore(d1Executor(env.DB))` | Refused: lease only (D1 has no interactive transaction) |
|
|
136
|
+
|
|
137
|
+
Pass a node-postgres `Pool`, never a single `Client`: the store runs a transaction on a connection of its own while
|
|
138
|
+
other requests run their statements. Give the pool an error listener, `pool.on("error", (error) => log(error))`, as
|
|
139
|
+
node-postgres asks: without one, a connection that dies while idle in the pool, as in a database restart, stops
|
|
140
|
+
the process. `pgExecutor` handles the connections it has checked out itself. Any other driver fits too: an executor
|
|
141
|
+
is an object with `dialect`, `query` and an atomic, ordered `batch` (`SqlExecutor`), plus `transaction` for
|
|
142
|
+
`{ transaction: true }`.
|
|
143
|
+
|
|
144
|
+
On Cloudflare Workers, the D1 binding is reachable at module scope through `cloudflare:workers`:
|
|
145
|
+
|
|
146
|
+
```ts
|
|
147
|
+
// src/cloudrop.ts, in a Worker
|
|
148
|
+
import { env } from "cloudflare:workers";
|
|
149
|
+
import { createCloudrop, r2, route, unauthorized } from "cloudrop";
|
|
150
|
+
import { d1Executor, sqlStore } from "cloudrop/adapters/sql";
|
|
151
|
+
|
|
152
|
+
export const cloudrop = createCloudrop({
|
|
153
|
+
storage: r2({ bucket: "media" }),
|
|
154
|
+
store: sqlStore(d1Executor(env.DB)),
|
|
155
|
+
routes: {
|
|
156
|
+
files: route(
|
|
157
|
+
{
|
|
158
|
+
accept: "image/*",
|
|
159
|
+
maxSize: "10MB",
|
|
160
|
+
authorize: async ({ request }) => {
|
|
161
|
+
const user = await getUser(request);
|
|
162
|
+
return user ? { userId: user.id } : unauthorized();
|
|
163
|
+
},
|
|
164
|
+
},
|
|
165
|
+
{ owner: ({ userId }) => userId },
|
|
166
|
+
),
|
|
167
|
+
},
|
|
168
|
+
});
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
### Drizzle: `cloudrop/adapters/drizzle`
|
|
172
|
+
|
|
173
|
+
`drizzle-orm` is an optional peer dependency, `>=0.45.0 <1 || >=1.0.0-rc.5 <2`: 0.45 and later 0.x releases, and
|
|
174
|
+
1.0 from its fifth release candidate. Pass your app's Drizzle database:
|
|
175
|
+
|
|
176
|
+
```ts
|
|
177
|
+
import { drizzleStore } from "cloudrop/adapters/drizzle";
|
|
178
|
+
import { drizzle } from "drizzle-orm/node-postgres";
|
|
179
|
+
import { Pool } from "pg";
|
|
180
|
+
|
|
181
|
+
const db = drizzle(new Pool({ connectionString: process.env.DATABASE_URL }));
|
|
182
|
+
export const store = drizzleStore(db, { dialect: "postgres", transaction: true });
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
- **Postgres:** `drizzle-orm/node-postgres` over a `Pool` (a single `Client` is refused), or `drizzle-orm/postgres-js`.
|
|
186
|
+
With `transaction: true`, `store.route` gives `onComplete` Drizzle's transaction as `tx`:
|
|
187
|
+
`await tx.update(users).set({ avatarKey: file.key }).where(eq(users.id, userId))`. On drizzle-orm 0.45 with
|
|
188
|
+
node-postgres, Drizzle's transaction does not listen for its connection's errors, so a database restart during
|
|
189
|
+
`onComplete` stops the process: give every connection a listener,
|
|
190
|
+
`pool.on("connect", (client) => client.on("error", (error) => log(error)))` (the next statement still fails, and
|
|
191
|
+
the completion answers a retryable error), or use 1.0.0-rc.5 or later, which listens itself.
|
|
192
|
+
- **SQLite:** `drizzle-orm/better-sqlite3`, `drizzle-orm/libsql` (Turso) or `drizzle-orm/d1`, with
|
|
193
|
+
`{ dialect: "sqlite" }`. No transaction mode: an interactive transaction would hold SQLite's only writer while
|
|
194
|
+
`onComplete` awaits.
|
|
195
|
+
|
|
196
|
+
### Prisma: `cloudrop/adapters/prisma`
|
|
197
|
+
|
|
198
|
+
For Prisma 7 with a driver adapter. cloudrop imports nothing from Prisma:
|
|
199
|
+
|
|
200
|
+
```ts
|
|
201
|
+
import { PrismaPg } from "@prisma/adapter-pg";
|
|
202
|
+
import { prismaStore } from "cloudrop/adapters/prisma";
|
|
203
|
+
import { PrismaClient } from "./generated/prisma/client";
|
|
204
|
+
|
|
205
|
+
const prisma = new PrismaClient({ adapter: new PrismaPg({ connectionString: process.env.DATABASE_URL }) });
|
|
206
|
+
export const store = prismaStore(prisma, { dialect: "postgres", transaction: true });
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
With `transaction: true` (Postgres only), `store.route` gives `onComplete` Prisma's interactive-transaction client as
|
|
210
|
+
`tx`: `await tx.avatar.create({ data: { userId, key: file.key } })`. Prisma rolls back an interactive transaction after
|
|
211
|
+
5 seconds by default, and that time includes your `onComplete`; a completion that runs out answers a retryable error
|
|
212
|
+
and keeps nothing. Raise it when you need more: `prismaStore(prisma, { dialect: "postgres", transaction: true,
|
|
213
|
+
transactionOptions: { timeout: 15_000 } })`. On SQLite, use `{ dialect: "sqlite" }` without a transaction.
|
|
214
|
+
|
|
215
|
+
Add the two models to your Prisma schema: copy them from the shipped files, which the test suite generates, migrates
|
|
216
|
+
and round-trips, so they match the store:
|
|
217
|
+
|
|
218
|
+
- Postgres: [`cloudrop.prisma`](https://github.com/vcode-sh/cloudrop/blob/main/packages/cloudrop/src/testing/db/prisma/postgres/cloudrop.prisma)
|
|
219
|
+
- SQLite: [`cloudrop.prisma`](https://github.com/vcode-sh/cloudrop/blob/main/packages/cloudrop/src/testing/db/prisma/sqlite/cloudrop.prisma)
|
|
220
|
+
|
|
221
|
+
The model and field names are yours to change. The column names (`@map`) are fixed; so are the table names
|
|
222
|
+
(`@@map`), unless you pass the same names to `prismaStore` as `tables`.
|
|
223
|
+
|
|
224
|
+
## Create the tables
|
|
225
|
+
|
|
226
|
+
Use the tool you query with:
|
|
227
|
+
|
|
228
|
+
- **Prisma:** `prisma migrate dev --create-only` from the two models above, add the CHECK line (below) to the
|
|
229
|
+
generated `migration.sql`, then apply it with `prisma migrate dev`.
|
|
230
|
+
- **Drizzle:** export the tables from your schema file and let drizzle-kit create them:
|
|
231
|
+
|
|
232
|
+
```ts
|
|
233
|
+
// db/schema.ts
|
|
234
|
+
import { cloudropPgTables } from "cloudrop/adapters/drizzle";
|
|
235
|
+
|
|
236
|
+
export const { files: cloudropFiles, usage: cloudropUsage } = cloudropPgTables();
|
|
237
|
+
```
|
|
238
|
+
|
|
239
|
+
On SQLite, `cloudropSqliteTables()`. Both take `{ tables }`, the same names you pass to `drizzleStore`.
|
|
240
|
+
`api.files({ owner })` lists a user's files; for anything else, read the tables freely
|
|
241
|
+
(`db.select().from(cloudropFiles)`): they are your app's. Write to them only through the store, which keeps the
|
|
242
|
+
usage counters in step with the files.
|
|
243
|
+
- **Everyone else:** run the SQL below in a migration. On D1: `wrangler d1 migrations create DB cloudrop`, paste the
|
|
244
|
+
SQLite statements into the new file, then `wrangler d1 migrations apply DB` (it applies to the local database; add
|
|
245
|
+
`--remote` for the deployed one).
|
|
246
|
+
|
|
247
|
+
Postgres times are `timestamptz(3)`: the store writes milliseconds, the precision of a JavaScript `Date`, so a
|
|
248
|
+
microsecond column would only hold zeros. Tables created earlier with plain `timestamptz` keep working unchanged;
|
|
249
|
+
drizzle-kit and Prisma then propose altering the columns to `timestamptz(3)`, a change that is safe to apply or to leave
|
|
250
|
+
out.
|
|
251
|
+
|
|
252
|
+
<!-- ddl:postgres -->
|
|
253
|
+
```sql
|
|
254
|
+
CREATE TABLE "cloudrop_files" (
|
|
255
|
+
id text NOT NULL PRIMARY KEY,
|
|
256
|
+
route text NOT NULL,
|
|
257
|
+
key text NOT NULL,
|
|
258
|
+
owner text,
|
|
259
|
+
status text NOT NULL,
|
|
260
|
+
name text NOT NULL,
|
|
261
|
+
type text NOT NULL,
|
|
262
|
+
size bigint NOT NULL,
|
|
263
|
+
upload_id text,
|
|
264
|
+
metadata text,
|
|
265
|
+
result text,
|
|
266
|
+
claim text,
|
|
267
|
+
claimed_at timestamptz(3),
|
|
268
|
+
expires_at timestamptz(3) NOT NULL,
|
|
269
|
+
created_at timestamptz(3) NOT NULL,
|
|
270
|
+
updated_at timestamptz(3) NOT NULL,
|
|
271
|
+
completed_at timestamptz(3),
|
|
272
|
+
ended_at timestamptz(3),
|
|
273
|
+
cleaned_at timestamptz(3),
|
|
274
|
+
request_key text
|
|
275
|
+
);
|
|
276
|
+
|
|
277
|
+
CREATE INDEX "cloudrop_files_status_expires_at_idx" ON "cloudrop_files" (status, expires_at);
|
|
278
|
+
|
|
279
|
+
CREATE INDEX "cloudrop_files_status_claimed_at_idx" ON "cloudrop_files" (status, claimed_at);
|
|
280
|
+
|
|
281
|
+
CREATE INDEX "cloudrop_files_status_cleaned_at_ended_at_idx" ON "cloudrop_files" (status, cleaned_at, ended_at);
|
|
282
|
+
|
|
283
|
+
CREATE INDEX "cloudrop_files_owner_status_created_at_idx" ON "cloudrop_files" (owner, status, created_at);
|
|
284
|
+
|
|
285
|
+
CREATE INDEX "cloudrop_files_key_idx" ON "cloudrop_files" (key);
|
|
286
|
+
|
|
287
|
+
CREATE INDEX "cloudrop_files_route_request_key_idx" ON "cloudrop_files" (route, request_key);
|
|
288
|
+
|
|
289
|
+
CREATE TABLE "cloudrop_usage" (
|
|
290
|
+
owner text NOT NULL PRIMARY KEY,
|
|
291
|
+
reserved_bytes bigint NOT NULL CHECK (reserved_bytes >= 0),
|
|
292
|
+
active_bytes bigint NOT NULL CHECK (active_bytes >= 0),
|
|
293
|
+
files integer NOT NULL CHECK (files >= 0),
|
|
294
|
+
updated_at timestamptz(3) NOT NULL
|
|
295
|
+
);
|
|
296
|
+
```
|
|
297
|
+
|
|
298
|
+
SQLite and D1 store times as integer milliseconds:
|
|
299
|
+
|
|
300
|
+
<!-- ddl:sqlite -->
|
|
301
|
+
```sql
|
|
302
|
+
CREATE TABLE "cloudrop_files" (
|
|
303
|
+
id TEXT NOT NULL PRIMARY KEY,
|
|
304
|
+
route TEXT NOT NULL,
|
|
305
|
+
key TEXT NOT NULL,
|
|
306
|
+
owner TEXT,
|
|
307
|
+
status TEXT NOT NULL,
|
|
308
|
+
name TEXT NOT NULL,
|
|
309
|
+
type TEXT NOT NULL,
|
|
310
|
+
size INTEGER NOT NULL,
|
|
311
|
+
upload_id TEXT,
|
|
312
|
+
metadata TEXT,
|
|
313
|
+
result TEXT,
|
|
314
|
+
claim TEXT,
|
|
315
|
+
claimed_at INTEGER,
|
|
316
|
+
expires_at INTEGER NOT NULL,
|
|
317
|
+
created_at INTEGER NOT NULL,
|
|
318
|
+
updated_at INTEGER NOT NULL,
|
|
319
|
+
completed_at INTEGER,
|
|
320
|
+
ended_at INTEGER,
|
|
321
|
+
cleaned_at INTEGER,
|
|
322
|
+
request_key TEXT
|
|
323
|
+
);
|
|
324
|
+
|
|
325
|
+
CREATE INDEX "cloudrop_files_status_expires_at_idx" ON "cloudrop_files" (status, expires_at);
|
|
326
|
+
|
|
327
|
+
CREATE INDEX "cloudrop_files_status_claimed_at_idx" ON "cloudrop_files" (status, claimed_at);
|
|
328
|
+
|
|
329
|
+
CREATE INDEX "cloudrop_files_status_cleaned_at_ended_at_idx" ON "cloudrop_files" (status, cleaned_at, ended_at);
|
|
330
|
+
|
|
331
|
+
CREATE INDEX "cloudrop_files_owner_status_created_at_idx" ON "cloudrop_files" (owner, status, created_at);
|
|
332
|
+
|
|
333
|
+
CREATE INDEX "cloudrop_files_key_idx" ON "cloudrop_files" (key);
|
|
334
|
+
|
|
335
|
+
CREATE INDEX "cloudrop_files_route_request_key_idx" ON "cloudrop_files" (route, request_key);
|
|
336
|
+
|
|
337
|
+
CREATE TABLE "cloudrop_usage" (
|
|
338
|
+
owner TEXT NOT NULL PRIMARY KEY,
|
|
339
|
+
reserved_bytes INTEGER NOT NULL,
|
|
340
|
+
active_bytes INTEGER NOT NULL,
|
|
341
|
+
files INTEGER NOT NULL,
|
|
342
|
+
updated_at INTEGER NOT NULL
|
|
343
|
+
);
|
|
344
|
+
```
|
|
345
|
+
|
|
346
|
+
Other table names: pass `tables: { files: "uploads", usage: "upload_usage" }` to the adapter (letters, digits and `_`)
|
|
347
|
+
and rename them in the SQL, including the index and constraint names if you like.
|
|
348
|
+
|
|
349
|
+
**Prisma and the CHECK constraints.** Prisma's schema language has no CHECK, so `prisma migrate` creates the usage
|
|
350
|
+
table without the three `CHECK (… >= 0)` on Postgres. Run `prisma migrate dev --create-only`, add this line to the
|
|
351
|
+
generated `migration.sql`, then apply it:
|
|
352
|
+
|
|
353
|
+
```sql
|
|
354
|
+
ALTER TABLE cloudrop_usage ADD CONSTRAINT cloudrop_usage_reserved_bytes_check CHECK (reserved_bytes >= 0), ADD CONSTRAINT cloudrop_usage_active_bytes_check CHECK (active_bytes >= 0), ADD CONSTRAINT cloudrop_usage_files_check CHECK (files >= 0);
|
|
355
|
+
```
|
|
356
|
+
|
|
357
|
+
The names are the ones the plain SQL gets by default, so both routes end with the same schema. The store works without
|
|
358
|
+
them; they only guard the counters against a bug or a hand edit.
|
|
359
|
+
|
|
360
|
+
**Moving a SQLite database from the SQL to Prisma.** Prisma declares its `BigInt` columns `BIGINT` where the SQL says
|
|
361
|
+
`INTEGER`. Both have INTEGER affinity, so storage and behaviour are the same, but Prisma's first `migrate dev` rebuilds
|
|
362
|
+
both tables (the data is kept). To skip the rebuild, write the baseline yourself:
|
|
363
|
+
|
|
364
|
+
```sh
|
|
365
|
+
mkdir -p prisma/migrations/0_cloudrop
|
|
366
|
+
prisma migrate diff --from-empty --to-schema prisma/schema.prisma --script > prisma/migrations/0_cloudrop/migration.sql
|
|
367
|
+
prisma migrate resolve --applied 0_cloudrop
|
|
368
|
+
```
|
|
369
|
+
|
|
370
|
+
On Postgres Prisma finds nothing to change in tables made from the SQL.
|
|
371
|
+
|
|
372
|
+
### Upgrading from 0.1
|
|
373
|
+
|
|
374
|
+
0.2.0 replaces the index on `(owner, status)` with one on `(owner, status, created_at)`, so
|
|
375
|
+
[`api.files`](#list-an-owners-files) reads one page of an owner's files without sorting all of them. The store works
|
|
376
|
+
without it, only slower for owners with many files. Drizzle and Prisma users regenerate their migration from the
|
|
377
|
+
updated tables (`drizzle-kit generate` asks whether the new index renames the old one: answer that it is created).
|
|
378
|
+
Everyone else runs, on Postgres or SQLite, in a migration:
|
|
379
|
+
|
|
380
|
+
```sql
|
|
381
|
+
CREATE INDEX "cloudrop_files_owner_status_created_at_idx" ON "cloudrop_files" (owner, status, created_at);
|
|
382
|
+
DROP INDEX "cloudrop_files_owner_status_idx";
|
|
383
|
+
```
|
|
384
|
+
|
|
385
|
+
On a large Postgres table, `CREATE INDEX CONCURRENTLY` (outside a transaction) avoids blocking uploads while it builds.
|
|
386
|
+
With [`tables.schema`](#tables-in-another-schema) or table names of your own, put those names in both statements.
|
|
387
|
+
|
|
388
|
+
### Tables in another schema
|
|
389
|
+
|
|
390
|
+
On Postgres, pass the schema with the names: `tables: { schema: "app" }` (letters, digits and `_`). Every statement
|
|
391
|
+
then names `"app"."cloudrop_files"` and `"app"."cloudrop_usage"`, the first-call table check included, whatever the
|
|
392
|
+
connection's search path is. SQLite has no schemas, so every adapter refuses the option there with `INVALID_CONFIG`.
|
|
393
|
+
|
|
394
|
+
- **Plain SQL:** `CREATE SCHEMA app;`, then the SQL above with each table name written `app.cloudrop_files` and
|
|
395
|
+
`app.cloudrop_usage`. Pass `sqlStore(executor, { tables: { schema: "app" } })`.
|
|
396
|
+
- **Drizzle:** pass the same `tables` to the tables and the store, and export the schema from your schema file too, as
|
|
397
|
+
Drizzle's [schemas page](https://orm.drizzle.team/docs/schemas) does, whose SQL starts with `CREATE SCHEMA`:
|
|
398
|
+
|
|
399
|
+
```ts
|
|
400
|
+
// db/schema.ts
|
|
401
|
+
import { cloudropPgTables } from "cloudrop/adapters/drizzle";
|
|
402
|
+
import { pgSchema } from "drizzle-orm/pg-core";
|
|
403
|
+
|
|
404
|
+
export const app = pgSchema("app");
|
|
405
|
+
export const cloudropTables = { schema: app.schemaName };
|
|
406
|
+
export const { files: cloudropFiles, usage: cloudropUsage } = cloudropPgTables({ tables: cloudropTables });
|
|
407
|
+
```
|
|
408
|
+
|
|
409
|
+
and build the store with them: `drizzleStore(db, { dialect: "postgres", tables: cloudropTables })`. `drizzle-kit
|
|
410
|
+
push` and `pull` manage all schemas by default; if your `drizzle.config.ts` sets `schemaFilter`, list `app` there
|
|
411
|
+
([config file](https://orm.drizzle.team/docs/drizzle-config-file)).
|
|
412
|
+
|
|
413
|
+
- **Prisma:** list the schema in the datasource and put both models in it, then pass
|
|
414
|
+
`prismaStore(prisma, { dialect: "postgres", tables: { schema: "app" } })`. Prisma 7 needs no preview feature for
|
|
415
|
+
this. Once the datasource lists `schemas`, every model needs its `@@schema`, your own included
|
|
416
|
+
(`@@schema("public")`), and the CHECK line above names `app.cloudrop_usage`. PrismaPg's own `schema` option is not
|
|
417
|
+
needed and does not reach the store.
|
|
418
|
+
|
|
419
|
+
```prisma
|
|
420
|
+
datasource db {
|
|
421
|
+
provider = "postgresql"
|
|
422
|
+
schemas = ["public", "app"]
|
|
423
|
+
}
|
|
424
|
+
|
|
425
|
+
model CloudropFile {
|
|
426
|
+
// the fields as shipped
|
|
427
|
+
@@map("cloudrop_files")
|
|
428
|
+
@@schema("app")
|
|
429
|
+
}
|
|
430
|
+
|
|
431
|
+
model CloudropUsage {
|
|
432
|
+
// the fields as shipped
|
|
433
|
+
@@map("cloudrop_usage")
|
|
434
|
+
@@schema("app")
|
|
435
|
+
}
|
|
436
|
+
```
|
|
437
|
+
|
|
438
|
+
The alternative is to leave `schema` out and let the search path find the tables: set it once for the database role,
|
|
439
|
+
which also works behind connection poolers:
|
|
440
|
+
|
|
441
|
+
```sql
|
|
442
|
+
ALTER ROLE app_user SET search_path = app, public;
|
|
443
|
+
```
|
|
444
|
+
|
|
445
|
+
or per connection with `?options=-c%20search_path%3Dapp` in the connection string, which some poolers drop or refuse
|
|
446
|
+
(PgBouncer without `ignore_startup_parameters`, some hosted pooled endpoints). With Prisma, the search path reaches the
|
|
447
|
+
store's SQL, and PrismaPg's `schema` option (`new PrismaPg({ connectionString }, { schema: "app" })`) reaches model
|
|
448
|
+
queries; set both. If `public` also has tables with the same names, the store uses whichever comes first in the search
|
|
449
|
+
path.
|
|
450
|
+
|
|
451
|
+
## Owners and quota
|
|
452
|
+
|
|
453
|
+
```ts
|
|
454
|
+
createCloudrop({
|
|
455
|
+
storage,
|
|
456
|
+
store,
|
|
457
|
+
// Bytes each owner may keep, or null for no limit. Asked once per /upload.
|
|
458
|
+
quota: async (owner) => (await planOf(owner)).storageBytes,
|
|
459
|
+
routes: {
|
|
460
|
+
files: route(
|
|
461
|
+
{ accept: "*/*", maxSize: "100MB", authorize },
|
|
462
|
+
{ owner: ({ userId }) => userId },
|
|
463
|
+
),
|
|
464
|
+
},
|
|
465
|
+
});
|
|
466
|
+
```
|
|
467
|
+
|
|
468
|
+
- `owner` names who a file counts against: a user, team or workspace id of 1 to 255 characters without NUL (U+0000,
|
|
469
|
+
which Postgres text cannot hold), or null for no one. It runs once per `/upload`, before plugins' `beforeUpload`,
|
|
470
|
+
and sees the authorize result, the request and the metadata, not the files. It is not the authorize result signed
|
|
471
|
+
into the token.
|
|
472
|
+
- The store counts each owner's usage whether or not you set `quota`, so turning a quota on later needs no backfill.
|
|
473
|
+
- The rule is `reserved + stored + size <= limit` (stored is `activeBytes` in `api.usage`), checked file by file in
|
|
474
|
+
the order sent: a file that does not fit is refused with `QUOTA_EXCEEDED` (HTTP 403, not retryable), and the files
|
|
475
|
+
of the same `/upload` that fit go on. Bytes are reserved at `/upload`, move to stored when the upload completes, and
|
|
476
|
+
are released when it is cancelled, refused, revoked, expires or the file is deleted.
|
|
477
|
+
- A `quota` that throws, or answers something other than a non-negative whole number or null, refuses the upload with a
|
|
478
|
+
retryable `SERVER_ERROR`: it fails closed.
|
|
479
|
+
- With `quota` set, a route without `owner` logs a warning at startup: its uploads count against no one.
|
|
480
|
+
|
|
481
|
+
## `onComplete` with `tx`
|
|
482
|
+
|
|
483
|
+
On a store built with `transaction: true`, build routes with `store.route` instead of `route` to get the transaction
|
|
484
|
+
in `onComplete`:
|
|
485
|
+
|
|
486
|
+
```ts
|
|
487
|
+
const avatar = store.route(policy, {
|
|
488
|
+
owner: ({ userId }) => userId,
|
|
489
|
+
onComplete: async ({ tx, userId, file }) => {
|
|
490
|
+
// Commits with the upload's completion, or not at all.
|
|
491
|
+
await tx.query("INSERT INTO avatars (user_id, key) VALUES ($1, $2)", [userId, file.key]);
|
|
492
|
+
return { key: file.key };
|
|
493
|
+
},
|
|
494
|
+
});
|
|
495
|
+
```
|
|
496
|
+
|
|
497
|
+
- `tx` is the adapter's own transaction handle, typed: node-postgres's client, postgres.js's transaction, Drizzle's or
|
|
498
|
+
Prisma's transaction client. A route from `store.route` works only with that store; `createCloudrop` refuses it with
|
|
499
|
+
any other store or none. Routes from plain `route()` run inside the transaction too, without `tx`.
|
|
500
|
+
- **Keep network calls out.** No storage calls, emails or HTTP requests inside `onComplete` in transaction mode:
|
|
501
|
+
they hold a database connection and a row lock while they wait, and a rollback cannot undo them. They belong in
|
|
502
|
+
`afterComplete`.
|
|
503
|
+
- **Never swallow an error from `tx`.** On Postgres a failed statement aborts the whole transaction. If `onComplete`
|
|
504
|
+
catches the error and returns normally, Postgres refuses every later statement of the transaction, including
|
|
505
|
+
cloudrop's record of the completion, so the completion rolls back and the client gets a retryable `SERVER_ERROR`.
|
|
506
|
+
Every retry fails the same way, until the upload expires and the sweep deletes its file. Let the error propagate, or
|
|
507
|
+
catch it inside a savepoint (Drizzle's nested `tx.transaction`, `SAVEPOINT` with node-postgres).
|
|
508
|
+
- The store waits at most 2 seconds for the file's row lock, so a second `/complete` of the same file is answered busy
|
|
509
|
+
rather than waiting. Your own statements through `tx` keep your session's lock timeout.
|
|
510
|
+
- `revokeUploads`, `eraseOwner` and the sweep wait for the row lock too: one that reaches a file whose completion is
|
|
511
|
+
running `onComplete` returns once that completion commits or rolls back. Another reason to keep `onComplete` short.
|
|
512
|
+
- Postgres must run at its default isolation level, READ COMMITTED. Under REPEATABLE READ or SERIALIZABLE, concurrent
|
|
513
|
+
uploads fail with serialization errors instead of waiting their turn.
|
|
514
|
+
- `onComplete`'s result must be JSON of at most 16 KiB with a store: it is stored and replayed to every retry. A larger
|
|
515
|
+
one fails the completion like a throw.
|
|
516
|
+
|
|
517
|
+
## `afterComplete`
|
|
518
|
+
|
|
519
|
+
```ts
|
|
520
|
+
store.route(policy, {
|
|
521
|
+
onComplete: async ({ tx, file }) => {
|
|
522
|
+
await tx.query("INSERT INTO outbox (kind, payload) VALUES ('avatar-changed', $1)", [file.key]);
|
|
523
|
+
return { key: file.key };
|
|
524
|
+
},
|
|
525
|
+
// Written after onComplete, so its `data` is typed from onComplete's result.
|
|
526
|
+
afterComplete: async ({ data, userId }) => {
|
|
527
|
+
await sendEmail(userId, `Your new avatar: ${data.key}`);
|
|
528
|
+
},
|
|
529
|
+
});
|
|
530
|
+
```
|
|
531
|
+
|
|
532
|
+
`afterComplete` runs after the completion is recorded, at most once per file and best effort: a crash right after the
|
|
533
|
+
record skips it, and a throw is logged while the client still gets its answer. The answer waits for it, so keep it
|
|
534
|
+
short. For work that must happen, write an outbox row through `tx` in `onComplete` and process the outbox elsewhere.
|
|
535
|
+
`afterComplete` needs a store, like `owner`.
|
|
536
|
+
|
|
537
|
+
## One file per owner: `replace`
|
|
538
|
+
|
|
539
|
+
```ts
|
|
540
|
+
avatar: store.route(
|
|
541
|
+
{ accept: "image/*", maxSize: "5MB", replace: true, authorize },
|
|
542
|
+
{
|
|
543
|
+
owner: ({ userId }) => userId,
|
|
544
|
+
onComplete: async ({ tx, userId, file }) => {
|
|
545
|
+
await tx.query("UPDATE users SET avatar_key = $1 WHERE id = $2", [file.key, userId]);
|
|
546
|
+
},
|
|
547
|
+
},
|
|
548
|
+
),
|
|
549
|
+
```
|
|
550
|
+
|
|
551
|
+
With `replace: true`, a route keeps one file per owner, such as an avatar or a profile banner. When a file completes,
|
|
552
|
+
the owner's previous completed files on that route are deleted in the same database write that records the
|
|
553
|
+
completion: their rows become `deleted` and their bytes leave the quota. Then, before `/complete` answers, their
|
|
554
|
+
objects are removed from the bucket; a removal that fails is logged and left to [the sweep](#the-sweep), which finds
|
|
555
|
+
the row deleted but not cleaned up.
|
|
556
|
+
|
|
557
|
+
- **The previous file goes only after the next one completes.** An upload that is cancelled, refused or never
|
|
558
|
+
finished deletes nothing, and the previous file stays.
|
|
559
|
+
- **Two uploads completing at once leave exactly one file:** the one recorded last. On Postgres the two completions
|
|
560
|
+
take turns on a transaction-scoped advisory lock for the owner and route; SQLite and D1 have one writer at a time.
|
|
561
|
+
- **What the hooks see.** `onComplete` runs before the completion is recorded, so the previous file is still
|
|
562
|
+
completed there (`api.files` lists it). `afterComplete` runs after: the previous file's row is deleted and the removal
|
|
563
|
+
of its object was tried. Neither gets a new field.
|
|
564
|
+
- **Read the kept file; do not keep a pointer of your own.** `api.files({ owner, route, limit: 1 })` answers the file
|
|
565
|
+
that survived. A pointer such as `users.avatar_key` written in `onComplete` can name the file that was deleted: without
|
|
566
|
+
`transaction: true`, `onComplete` runs before the completion is recorded and outside any lock, so two uploads at once
|
|
567
|
+
can set the pointer in one order and record their completions in the other. A pointer is safe only on Postgres with
|
|
568
|
+
`transaction: true`, written through `tx` by a statement that locks the owner's row (an `UPDATE users … WHERE id =`
|
|
569
|
+
does), so the two completions take turns from that statement on.
|
|
570
|
+
- **It needs a store and `owner`**, and takes one file per `/upload`: `maxFiles` is 1, and `createCloudrop` refuses a
|
|
571
|
+
replacing route without a store, without `owner`, or with another `maxFiles`, with `INVALID_CONFIG`. A file whose
|
|
572
|
+
`owner` answered null replaces nothing.
|
|
573
|
+
- Only files of the same owner and route go. Files uploaded before you turned `replace` on are replaced too, by the
|
|
574
|
+
owner's next completion on the route.
|
|
575
|
+
- **Under a quota, the owner needs room for both files** until the replacement completes: the new file's bytes are
|
|
576
|
+
reserved at `/upload` while the previous file still counts. An owner at their limit is refused `QUOTA_EXCEEDED` for a
|
|
577
|
+
new avatar; give replacing routes room in your `quota`, or keep their files out of it.
|
|
578
|
+
|
|
579
|
+
## What changes for the client
|
|
580
|
+
|
|
581
|
+
- **`/upload`** refuses a file over the quota with `QUOTA_EXCEEDED`, and answers a retryable `SERVER_ERROR` when the
|
|
582
|
+
store cannot be reached ("The server could not reach its upload records. Try again in a moment.").
|
|
583
|
+
- **`/sign` and `/complete`** refuse an upload whose row was ended (cancelled, revoked, refused, failed or expired)
|
|
584
|
+
with `UPLOAD_EXPIRED`: "This upload was cancelled or has expired. Start it again." The file fails with that code and
|
|
585
|
+
the client sends its `/abort`. `retry()` asks for a new ticket through `/upload` and `authorize`, so a signed-out
|
|
586
|
+
user is refused. A `resume()` after a reload asks for a new ticket by itself and starts the file again while the
|
|
587
|
+
user is still signed in ([Resume](resume.md)).
|
|
588
|
+
- **A lost `/upload` answer** no longer holds its reservation. Each ask of `/upload` carries a random
|
|
589
|
+
`idempotencyKey`, the same on every automatic retry of that ask. When the client gives up waiting (30 seconds, or a
|
|
590
|
+
dropped connection) and asks again, the retry ends the earlier attempt's rows as `aborted`, releasing their bytes
|
|
591
|
+
before the quota is checked, so a user near their limit is not refused for files they never got tickets for. The
|
|
592
|
+
sweep's cleanup pass aborts any multipart upload those rows had started. Only `pending` rows with the same key,
|
|
593
|
+
route and owner are ended; plugins hear no `onAbort` for them, as with the sweep and `revokeUploads`. Rows without
|
|
594
|
+
an owner (a route without `owner`, or one that answered null) hold no quota, so there the key does nothing.
|
|
595
|
+
- **A second `/complete` of a file being completed** is answered busy: `SERVER_ERROR`, retryable, "Another request is
|
|
596
|
+
completing this upload. Try again in a moment.", with `Retry-After: 2`. The client waits and asks again, and gets
|
|
597
|
+
the stored answer. Busy is not logged as an error.
|
|
598
|
+
- **`/abort`** of a completed file answers `ok` and deletes nothing. An unfinished upload is aborted and its object
|
|
599
|
+
deleted as before.
|
|
600
|
+
- **A refused file whose delete fails** (the object did not match what was declared, and the storage service did not
|
|
601
|
+
answer the delete) is still answered with the refusal, `VERIFICATION_FAILED`. Its row already says `rejected`, so a
|
|
602
|
+
retry of the same token answers `UPLOAD_EXPIRED`, and the sweep's cleanup pass deletes the object.
|
|
603
|
+
|
|
604
|
+
These words map to the `status` of [`api.file(id)`](#file-operations): cancelled and revoked uploads are `aborted`,
|
|
605
|
+
refused ones `rejected`, uploads that could not start and stuck completions `failed`, and expired ones `expired`; a
|
|
606
|
+
file removed with `deleteFile` is `deleted`.
|
|
607
|
+
|
|
608
|
+
[Errors](errors.md#upload-endpoint-errors) lists every code.
|
|
609
|
+
|
|
610
|
+
## Revoke a user's uploads
|
|
611
|
+
|
|
612
|
+
```ts
|
|
613
|
+
// At sign-out, or when an account is suspended:
|
|
614
|
+
const { data, error } = await cloudrop.api.revokeUploads({ owner: userId });
|
|
615
|
+
// data.revoked: how many unfinished uploads were ended
|
|
616
|
+
```
|
|
617
|
+
|
|
618
|
+
Each unfinished upload of that owner that is not already being completed is ended and its reservation released; its
|
|
619
|
+
next `/sign` or `/complete` is refused with `UPLOAD_EXPIRED`, and its client's `/abort` removes what it had uploaded.
|
|
620
|
+
A completion already under way finishes, and completed files are not touched.
|
|
621
|
+
|
|
622
|
+
## List an owner's files
|
|
623
|
+
|
|
624
|
+
```ts
|
|
625
|
+
const { data, error } = await cloudrop.api.files({ owner: userId, route: "attachments", limit: 50 });
|
|
626
|
+
if (error) throw error;
|
|
627
|
+
// data.items: FileRecord[], newest upload first. data.cursor: pass it back for the next page; null on the last one.
|
|
628
|
+
if (data.cursor) {
|
|
629
|
+
const next = await cloudrop.api.files({ owner: userId, route: "attachments", limit: 50, cursor: data.cursor });
|
|
630
|
+
}
|
|
631
|
+
```
|
|
632
|
+
|
|
633
|
+
Completed files only, newest upload first (by when `/upload` accepted them), one page of `limit` files (1 to 1,000,
|
|
634
|
+
default 100) at a time. `route` must be one of the server's routes; leave it out for all of them. The cursor is
|
|
635
|
+
opaque: pass back only what a page answered. Files deleted between two pages make the next page skip nothing.
|
|
636
|
+
|
|
637
|
+
## Erase an owner
|
|
638
|
+
|
|
639
|
+
For an account deletion or an erasure request: everything an owner uploaded goes, driven by the owner's rows, so it
|
|
640
|
+
needs no knowledge of your key layout.
|
|
641
|
+
|
|
642
|
+
```ts
|
|
643
|
+
// Stop the owner's access first (delete the account or its sessions), then run it, now and on a schedule, until done.
|
|
644
|
+
const { data, error } = await cloudrop.api.eraseOwner(userId);
|
|
645
|
+
// data: { revoked, deleted, cleaned, failed, waiting, done }
|
|
646
|
+
if (data?.done) await markErased(userId); // your app's record
|
|
647
|
+
```
|
|
648
|
+
|
|
649
|
+
Each run revokes the owner's unfinished uploads, deletes every completed file (row and object, as `deleteFile` does),
|
|
650
|
+
and checks the key of every ended upload the store cannot already prove clean, removing what is still the upload's
|
|
651
|
+
own. It reports:
|
|
652
|
+
|
|
653
|
+
| Field | Meaning |
|
|
654
|
+
| --- | --- |
|
|
655
|
+
| `revoked` | Unfinished uploads ended in this run. |
|
|
656
|
+
| `deleted` | Completed files deleted in this run. |
|
|
657
|
+
| `cleaned` | Ended uploads whose key this run checked once their URLs and the grace had run out. |
|
|
658
|
+
| `failed` | Files a storage or database error kept from being handled; each is logged with its id. |
|
|
659
|
+
| `waiting` | Files that cannot be proven yet: a completion still under way, an upload that started while it ran, or an ended upload whose URLs, or the grace after them, have not run out. What they had stored was removed. |
|
|
660
|
+
| `done` | Nothing of the owner was unfinished or completed, and every ended upload had its key checked after its URLs and the grace ran out. |
|
|
661
|
+
|
|
662
|
+
**Why it takes two runs.** A `PUT` with a URL signed before the revocation can still land afterwards. A URL works for
|
|
663
|
+
15 minutes and never past the upload's token, so the last `PUT` can start 15 minutes after the upload stopped being
|
|
664
|
+
unfinished (it completed, or ended), or at the token's deadline if that comes first; a key is checked for the last
|
|
665
|
+
time `graceMs` after that (default 1 hour, at least 15 minutes), the time a `PUT` already under way gets to land. Files
|
|
666
|
+
completed or ended longer ago than that are proven in the first run; a recent one makes the first run answer
|
|
667
|
+
`waiting`, and a run 1 hour 15 minutes later `done`. Run it again until it reports `done`: it is idempotent, and two
|
|
668
|
+
runs at once are safe.
|
|
669
|
+
|
|
670
|
+
- **Stuck completions too.** The sweep keeps the object of a completion it ended as `failed`, since its `onComplete`
|
|
671
|
+
may have recorded it; `eraseOwner` removes it.
|
|
672
|
+
- **`signal`** (an `AbortSignal`) stops a run before the next file, with `done: false`.
|
|
673
|
+
- **`done` is about the moment it ran.** It reads the owner's unfinished and completed files in one statement before
|
|
674
|
+
it checks the ended ones, so a file that changes status while it runs is not missed, and at the end it is not done
|
|
675
|
+
when any upload of the owner started after the run did, even one already cancelled. An
|
|
676
|
+
`/upload` that passed `authorize` before you revoked the owner's access can still reserve a file after the last read;
|
|
677
|
+
a run that sees it answers `waiting`. Revoke access first, and if a request may still be in flight, run it once more
|
|
678
|
+
after it.
|
|
679
|
+
- The usage row stays, at zeros. The rows stay too, as `deleted` and the other ended statuses, which name the file
|
|
680
|
+
(`name`, `key`); delete them with your own SQL once `done` if your erasure policy requires it, never before.
|
|
681
|
+
|
|
682
|
+
## File operations
|
|
683
|
+
|
|
684
|
+
With a store, `cloudrop.api` gains these, each returning `{ data, error }`. Without one they answer `INVALID_CONFIG`.
|
|
685
|
+
|
|
686
|
+
| Call | What it does |
|
|
687
|
+
| --- | --- |
|
|
688
|
+
| `file(id)` | The file's row (`FileRecord`: status, key, owner, size, result, times), or `null`. |
|
|
689
|
+
| `files({ owner, route?, limit?, cursor? })` | An owner's completed files, newest first, a page at a time ([above](#list-an-owners-files)). |
|
|
690
|
+
| `usage(owner)` | `{ reservedBytes, activeBytes, files }` for an owner, with or without a quota. |
|
|
691
|
+
| `revokeUploads({ owner })` | Ends the owner's unfinished uploads (above). |
|
|
692
|
+
| `deleteFile(id)` | Deletes a completed file: its row becomes `deleted`, its bytes and file count are released, and its object is removed when it is still this upload's. Within 1 hour 15 minutes of the file's completion (its URLs' 15 minutes, then an hour), the row is left for the sweep to check once more, in case a `PUT` with one of its URLs lands later. Idempotent; `NOT_FOUND` for a file that never completed. |
|
|
693
|
+
| `eraseOwner(owner, { graceMs?, signal? })` | Removes everything an owner uploaded, and reports when that is proven ([above](#erase-an-owner)). |
|
|
694
|
+
| `reconcile({ owner? })` | Recounts usage from the rows, for one owner or all of them, and answers `{ checked, repaired }`. Only needed after editing the tables by hand. |
|
|
695
|
+
| `sweep(options?)` | One bounded round of cleanup, or every round with `all: true` (below). |
|
|
696
|
+
|
|
697
|
+
`api.delete(key)` still deletes an object by key, but leaves the row and the owner's usage as they were: use
|
|
698
|
+
`deleteFile` for uploaded files.
|
|
699
|
+
|
|
700
|
+
## The sweep
|
|
701
|
+
|
|
702
|
+
Uploads that never finish leave rows, reservations and sometimes parts or objects behind. `api.sweep()` cleans them up
|
|
703
|
+
in bounded rounds; run it on a schedule, once an hour is plenty. `all: true` runs every round in one call, `limit` rows
|
|
704
|
+
(default 100) at a time, and answers their reports summed; its `signal` is checked before each round, and once it
|
|
705
|
+
aborts, the answer's `cursor` says where to continue (it is null when every row was seen).
|
|
706
|
+
|
|
707
|
+
On Cloudflare Workers, a [cron trigger](https://developers.cloudflare.com/workers/configuration/cron-triggers/)
|
|
708
|
+
(`[triggers] crons = ["41 * * * *"]` in `wrangler.toml`) calls `scheduled`:
|
|
709
|
+
|
|
710
|
+
```ts
|
|
711
|
+
export default {
|
|
712
|
+
async scheduled(controller, env, ctx) {
|
|
713
|
+
const cloudrop = createUploads(env); // your createCloudrop, with the store over env.DB
|
|
714
|
+
const { data, error } = await cloudrop.api.sweep({ all: true, signal: AbortSignal.timeout(60_000) });
|
|
715
|
+
if (error) throw error; // shown as a failed run; the next one starts over
|
|
716
|
+
for (const stuck of data.reported) console.warn(`cloudrop: completion of ${stuck.id} never finished; check it`);
|
|
717
|
+
},
|
|
718
|
+
} satisfies ExportedHandler<Env>;
|
|
719
|
+
```
|
|
720
|
+
|
|
721
|
+
On Node, from a long-running process (or any job scheduler that runs a script, such as a Kubernetes `CronJob`):
|
|
722
|
+
|
|
723
|
+
```ts
|
|
724
|
+
async function sweepUploads() {
|
|
725
|
+
const { data, error } = await cloudrop.api.sweep({ all: true, signal: AbortSignal.timeout(10 * 60_000) });
|
|
726
|
+
if (error) return console.error(`Sweep stopped: ${error.message} The next run retries.`);
|
|
727
|
+
for (const stuck of data.reported) console.warn(`cloudrop: completion of ${stuck.id} never finished; check it`);
|
|
728
|
+
}
|
|
729
|
+
setInterval(sweepUploads, 60 * 60_000);
|
|
730
|
+
```
|
|
731
|
+
|
|
732
|
+
Without `all`, each call handles up to `limit` rows (default 100, at most 1,000) and answers `{ expired, cleaned,
|
|
733
|
+
failed, reported, cursor }`; pass `cursor` back to continue. It runs two passes:
|
|
734
|
+
|
|
735
|
+
- **Expire:** an unfinished upload past its token's deadline plus `graceMs` is ended as `expired` first, its
|
|
736
|
+
reservation released; then its multipart upload is aborted and its object deleted only when the object's
|
|
737
|
+
`cloudrop-id` is this upload's. A completion stuck past its lease, and past its deadline plus `graceMs`, is ended as
|
|
738
|
+
`failed` and **reported**, and its object is kept: its `onComplete` may have recorded it, so check each reported
|
|
739
|
+
file.
|
|
740
|
+
- **Cleanup:** an ended upload (`aborted`, `rejected`, `failed`, `expired` or `deleted`) not yet cleaned up, and ended
|
|
741
|
+
more than `graceMs` ago, has its late parts and object removed, again only when the object is its own. This catches
|
|
742
|
+
a `PUT` that landed after an `/abort`, an upload that could not start, and an expired one whose first cleanup
|
|
743
|
+
failed. The row is marked cleaned up, and counted in `cleaned`, only once no `PUT` can still land: `graceMs` after
|
|
744
|
+
its last upload URL ran out, which for a row ended within 15 minutes of leaving `pending` is up to 15 minutes later
|
|
745
|
+
than `graceMs` after its end. Until then each sweep checks its key again.
|
|
746
|
+
|
|
747
|
+
`graceMs` (default 1 hour) must be at least 15 minutes, the life of an upload URL, so no new `PUT` can start once a
|
|
748
|
+
row may be cleaned up. A failing row is logged, counted in `failed` and left for the next sweep; the rows after it are
|
|
749
|
+
still swept. Two sweeps at once are safe, and so is a sweep racing a `/complete`: the sweep takes the row before it
|
|
750
|
+
deletes anything. Each sweep counts in `cleaned` the rows it marked, so two at once may both count the same row.
|
|
751
|
+
|
|
752
|
+
The sweep never deletes a completed file, never touches an object whose `cloudrop-id` belongs to another upload or
|
|
753
|
+
that has no row, and runs no plugin `onAbort` (rows carry no authorize result). Keep the bucket's lifecycle rule that
|
|
754
|
+
aborts incomplete multipart uploads ([Bucket setup](bucket-setup.md)): it covers a server that crashes between creating
|
|
755
|
+
a multipart upload and recording it, which leaves no row to sweep. With a store, `api.checkBucket()` reports that rule
|
|
756
|
+
as required.
|
|
757
|
+
|
|
758
|
+
## Postgres notes
|
|
759
|
+
|
|
760
|
+
- **Isolation:** READ COMMITTED, the default. See [`onComplete` with `tx`](#oncomplete-with-tx).
|
|
761
|
+
- **Tables outside `public`:** pass `tables: { schema: "app" }`, or set the search path; see
|
|
762
|
+
[Tables in another schema](#tables-in-another-schema).
|
|
763
|
+
- **Connection pools:** with `transaction: true`, a completion holds one connection and the file's row lock for the
|
|
764
|
+
length of `onComplete`, plain `route()`s included. Size the pool for your concurrent completions plus your other
|
|
765
|
+
traffic, or leave `transaction` off and completions use a lease, holding no connection.
|
|
766
|
+
|
|
767
|
+
## SQLite notes
|
|
768
|
+
|
|
769
|
+
- **One writer at a time.** Every store method is a short batch, and `/complete` holds its file with a lease rather
|
|
770
|
+
than a transaction, so `onComplete` never blocks other uploads.
|
|
771
|
+
- **Several connections or processes on one file** need a busy timeout, or a second writer fails at once with
|
|
772
|
+
"database is locked": `new DatabaseSync(path, { timeout: 5000 })`.
|
|
773
|
+
- **Prisma on SQLite is for one process**, such as a single server or local development. Its adapter starts batches
|
|
774
|
+
with a deferred `BEGIN`, so two processes writing one file may see `SQLITE_BUSY` (not measured).
|
|
775
|
+
- **D1** batches are atomic and serialised by Cloudflare; nothing to configure.
|
|
776
|
+
|
|
777
|
+
## Known costs
|
|
778
|
+
|
|
779
|
+
- **Some lost `/upload` answers still hold their reservation.** A client that sends no `idempotencyKey` (your own
|
|
780
|
+
code calling `/upload`) cannot be told apart from a new request: its first attempt's rows reserve their bytes until
|
|
781
|
+
the sweep expires them (deadline plus `graceMs`) or `revokeUploads` releases them. A retry whose `beforeUpload`
|
|
782
|
+
refuses the request reaches no store call, so it supersedes nothing either. Nor does a retry that reaches the store
|
|
783
|
+
before its first attempt does (a first request still being worked on, its `authorize` slow, when the client gives up
|
|
784
|
+
on it): that attempt's rows are written after the retry's and keep their bytes until the sweep. An ask whose every
|
|
785
|
+
attempt lost its answer: its last attempt's rows hold their bytes until the sweep, since `retry()` and `resume()` ask
|
|
786
|
+
again with a new key. Plugin reservations of a superseded attempt need their own expiry.
|
|
787
|
+
- **A `PUT` that lands after cleanup stays.** The sweep cleans a row once. A `PUT` already under way that lands after
|
|
788
|
+
that is left in the bucket; with the default hour of grace this is unlikely. `eraseOwner` checks such keys again.
|
|
789
|
+
- **A completion's record that fails after `onComplete` ran, in lease mode**, keeps the file busy until the lease ends
|
|
790
|
+
(5 minutes by default), longer than the client retries. The client then reports a failure but keeps the file. Once
|
|
791
|
+
the lease has passed, the next `/complete` of it (a `retry()` or a `resume()`) takes the file over and runs
|
|
792
|
+
`onComplete` again: one more reason to keep it idempotent in this mode.
|
|
793
|
+
- **Clock skew.** The lease and the grace periods are minutes, far above normal skew between servers. A server whose
|
|
794
|
+
clock runs ahead can delay `eraseOwner`'s `done` by that much: a file it wrote looks newer than the run. Keep server
|
|
795
|
+
clocks in sync (NTP), as SigV4 signing already requires.
|
|
796
|
+
- **Bundle size.** The store paths add about 6.7 kB (gzip) to `cloudrop`'s server code whether or not you use a store;
|
|
797
|
+
the browser bundle does not change.
|
|
798
|
+
|
|
799
|
+
## Turning it on, or off, in a running app
|
|
800
|
+
|
|
801
|
+
Tokens signed with a store carry a `stored` mark. During a rolling deploy, a token from an instance without the store
|
|
802
|
+
takes the stateless path on an instance with one, until it expires (24 hours); a stored token whose row is missing is
|
|
803
|
+
refused as cancelled. Turning the store off makes stored tokens stateless again. So you can enable the store with a
|
|
804
|
+
normal deploy: uploads started before it finish as they began, and their files count toward no owner's usage.
|