@tapedb/tapedb 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/API.md +301 -0
- package/LICENSE +21 -0
- package/LIMITATIONS.md +34 -0
- package/PRIVACY.md +119 -0
- package/README.md +720 -2
- package/RELEASING.md +117 -0
- package/SECURITY.md +121 -0
- package/dist/chunk-3AL32Q4K.js +241 -0
- package/dist/chunk-3AL32Q4K.js.map +1 -0
- package/dist/chunk-CYDXFAEB.js +33 -0
- package/dist/chunk-CYDXFAEB.js.map +1 -0
- package/dist/chunk-UYZJAMBR.js +27 -0
- package/dist/chunk-UYZJAMBR.js.map +1 -0
- package/dist/chunk-VWW5OS5B.js +841 -0
- package/dist/chunk-VWW5OS5B.js.map +1 -0
- package/dist/cli.d.ts +44 -0
- package/dist/cli.js +1501 -0
- package/dist/cli.js.map +1 -0
- package/dist/client-DgSnn9BE.d.ts +646 -0
- package/dist/index.d.ts +24 -0
- package/dist/index.js +3007 -0
- package/dist/index.js.map +1 -0
- package/dist/react.d.ts +41 -0
- package/dist/react.js +59 -0
- package/dist/react.js.map +1 -0
- package/dist/schema-CVcGZJfR.d.ts +691 -0
- package/dist/schema-entry-CpO_IGFv.d.ts +522 -0
- package/dist/schema.d.ts +2 -0
- package/dist/schema.js +22 -0
- package/dist/schema.js.map +1 -0
- package/dist/worker.d.ts +437 -0
- package/dist/worker.js +9270 -0
- package/dist/worker.js.map +1 -0
- package/package.json +93 -4
package/README.md
CHANGED
|
@@ -1,3 +1,721 @@
|
|
|
1
|
-
#
|
|
1
|
+
# tapedb
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
**A typed database for Cloudflare.** Describe your models once; `tapedb init`
|
|
4
|
+
generates the Worker and configuration for one Durable Object class and two R2
|
|
5
|
+
buckets in your own account. Create the buckets with the commands printed by
|
|
6
|
+
`init`, set the deployment secrets, and deploy the Worker.
|
|
7
|
+
Each workspace gets its own SQLite database inside a Durable Object, every
|
|
8
|
+
write is appended to a transaction log, the log ships to R2, and a cold start
|
|
9
|
+
replays it. Reads can be awaited once or kept live over a WebSocket; writes
|
|
10
|
+
are optimistic and survive going offline.
|
|
11
|
+
|
|
12
|
+
Think of it as an ORM whose database is Cloudflare: schema in TypeScript,
|
|
13
|
+
committed migrations, typed queries — plus live queries and offline writes,
|
|
14
|
+
because the engine underneath is built for them. It is a library, not a
|
|
15
|
+
service: there is no hosted control plane, and your data never leaves your
|
|
16
|
+
account. [PRIVACY.md](PRIVACY.md) covers what is stored where, for how long,
|
|
17
|
+
and how to answer GDPR access and erasure requests.
|
|
18
|
+
|
|
19
|
+
## Project status
|
|
20
|
+
|
|
21
|
+
0.1.0 is the first public release. The API is described, with what is stable
|
|
22
|
+
and what is experimental, in [API.md](API.md); the design boundaries are in
|
|
23
|
+
[LIMITATIONS.md](LIMITATIONS.md). The packaged CLI requires Node 22.18 or
|
|
24
|
+
newer; the library itself runs in Cloudflare Workers.
|
|
25
|
+
|
|
26
|
+
```bash
|
|
27
|
+
npx @tapedb/tapedb init # schema, worker, wrangler config, first migration
|
|
28
|
+
npx @tapedb/tapedb dev # local database with a dev claim route
|
|
29
|
+
npx @tapedb/tapedb studio # local table browser on http://127.0.0.1:4400
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
## The schema
|
|
33
|
+
|
|
34
|
+
```ts
|
|
35
|
+
// tapedb.schema.ts
|
|
36
|
+
import { check, filter, model, role, schema, t } from "@tapedb/tapedb/schema";
|
|
37
|
+
|
|
38
|
+
export default schema({
|
|
39
|
+
board: model({
|
|
40
|
+
name: t.string(),
|
|
41
|
+
ownerId: t.string().index(),
|
|
42
|
+
archived: t.boolean().default(false),
|
|
43
|
+
columns: t.hasMany("column"),
|
|
44
|
+
}).access({
|
|
45
|
+
read: filter(({ auth }) => ({ ownerId: auth.sub })),
|
|
46
|
+
update: check(({ auth, row }) => row?.ownerId === auth.sub),
|
|
47
|
+
delete: role("moderator"),
|
|
48
|
+
}),
|
|
49
|
+
|
|
50
|
+
column: model({
|
|
51
|
+
name: t.string(),
|
|
52
|
+
position: t.number().default(0).index(),
|
|
53
|
+
board: t.belongsTo("board", { onDelete: "cascade" }),
|
|
54
|
+
tasks: t.hasMany("task"),
|
|
55
|
+
}),
|
|
56
|
+
|
|
57
|
+
task: model({
|
|
58
|
+
title: t.string(),
|
|
59
|
+
done: t.boolean().default(false),
|
|
60
|
+
due: t.date().optional(),
|
|
61
|
+
meta: t.json().optional(),
|
|
62
|
+
column: t.belongsTo("column").optional(),
|
|
63
|
+
}),
|
|
64
|
+
});
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
One `schema()` call produces the engine's table registry, the SQL migrations
|
|
68
|
+
and the client's types. Fields: `t.string()`, `t.number()`, `t.boolean()`,
|
|
69
|
+
`t.date()` (stored as epoch milliseconds), `t.json()`. Modifiers:
|
|
70
|
+
`.optional()`, `.default(v)`, `.index()`, `.unique()`. Relations:
|
|
71
|
+
`t.belongsTo("model")` owns the foreign key (`board` stores `boardId`);
|
|
72
|
+
`t.hasMany("model")` is its inverse. Every row carries `id` (client-generated,
|
|
73
|
+
time-ordered), `createdAt` and `updatedAt`.
|
|
74
|
+
|
|
75
|
+
## Reads
|
|
76
|
+
|
|
77
|
+
```ts
|
|
78
|
+
import { createClient } from "@tapedb/tapedb";
|
|
79
|
+
import schema from "./tapedb.schema";
|
|
80
|
+
|
|
81
|
+
const db = createClient({ schema, url, workspace, userId, getToken });
|
|
82
|
+
|
|
83
|
+
const open = await db.task.findMany({
|
|
84
|
+
where: { done: false, due: { lt: Date.now() } },
|
|
85
|
+
orderBy: { due: "asc" },
|
|
86
|
+
take: 50,
|
|
87
|
+
include: { column: { include: { board: true } } },
|
|
88
|
+
});
|
|
89
|
+
|
|
90
|
+
const one = await db.task.findUnique({ where: { id } }); // row or null
|
|
91
|
+
const total = await db.task.count({ where: { done: false } });
|
|
92
|
+
|
|
93
|
+
const page = await db.task.findPage({ orderBy: { due: "asc" }, take: 50 });
|
|
94
|
+
// { rows, hasMore, cursor } — pass cursor back as `after`
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
`findMany` returns every matching row unless you pass `take`; `take` and
|
|
98
|
+
`findPage` sizes have no ceiling. Wide reads run as bounded statements under
|
|
99
|
+
the hood (keyset chunks of 1000 rows, includes hydrated 1000 parents at a
|
|
100
|
+
time), so nothing grows into one oversized query.
|
|
101
|
+
|
|
102
|
+
`select` returns only the fields you name, and relations named in it are
|
|
103
|
+
included with their own `select`:
|
|
104
|
+
|
|
105
|
+
```ts
|
|
106
|
+
const titles = await db.task.findMany({
|
|
107
|
+
select: { title: true, column: { select: { name: true } } },
|
|
108
|
+
});
|
|
109
|
+
// { title: string; column: { name: string } | null }[]
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
The engine reads and sends only those columns (plus the row's id, which live
|
|
113
|
+
updates key on), and the result type narrows to match. A live select is only
|
|
114
|
+
pushed a change when a selected field changed.
|
|
115
|
+
It works the same for `findPage` and live subscriptions. `select` and
|
|
116
|
+
`include` combine: the selected fields plus the included relations, and a
|
|
117
|
+
relation named in both gets the arguments of both.
|
|
118
|
+
|
|
119
|
+
Aggregates take the same `where` and return only the measures asked for:
|
|
120
|
+
|
|
121
|
+
```ts
|
|
122
|
+
const stats = await db.task.aggregate({
|
|
123
|
+
where: { done: false },
|
|
124
|
+
_count: true, // or { _all: true, due: true } for non-null counts
|
|
125
|
+
_sum: { points: true },
|
|
126
|
+
_avg: { points: true },
|
|
127
|
+
_min: { due: true },
|
|
128
|
+
_max: { due: true },
|
|
129
|
+
});
|
|
130
|
+
|
|
131
|
+
const perColumn = await db.task.groupBy({
|
|
132
|
+
by: ["columnId"],
|
|
133
|
+
_count: true,
|
|
134
|
+
_sum: { points: true },
|
|
135
|
+
orderBy: { columnId: "asc" },
|
|
136
|
+
take: 20,
|
|
137
|
+
});
|
|
138
|
+
// [{ columnId, _count, _sum: { points } }, ...]
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
They run as one `GROUP BY` statement over the rows the caller may read: a
|
|
142
|
+
`filter()` read rule is compiled into it, and a model whose read rule is a
|
|
143
|
+
function is folded row by row after the rule runs. Both are live with
|
|
144
|
+
`subscribe`, like `count`.
|
|
145
|
+
|
|
146
|
+
Everything is typed from the schema: a misspelt field in `where` is a compile
|
|
147
|
+
error, and `include` / `select` shape the result type. Filters take `not`, `in`,
|
|
148
|
+
`notIn`, `gt`, `gte`, `lt`, `lte`, and `AND` / `OR` / `NOT` over whole
|
|
149
|
+
filters. Pagination is keyset, so an insert above the window cannot repeat a
|
|
150
|
+
row.
|
|
151
|
+
|
|
152
|
+
Filters can also reach through a relation. A to-one link takes a filter on
|
|
153
|
+
the row it points at; a to-many link takes `some`, `every` or `none`:
|
|
154
|
+
|
|
155
|
+
```ts
|
|
156
|
+
await db.column.findMany({ where: { board: { archived: false } } });
|
|
157
|
+
await db.board.findMany({ where: { columns: { some: { tasks: { none: {} } } } } });
|
|
158
|
+
await db.task.findMany({ where: { column: null } }); // no parent
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
These compile to correlated subqueries, so the linked rows are never fetched.
|
|
162
|
+
The linked model's own read rule is compiled into the subquery, so a filter
|
|
163
|
+
cannot answer a question about rows the caller may not see; a model whose read
|
|
164
|
+
rule is a function (which only runs after fetch) is applied to the linked
|
|
165
|
+
rows before they count, so a filter never matches through a row the rule
|
|
166
|
+
hides. The types follow every hop.
|
|
167
|
+
Includes and relation filters nest without limit, and filters can be as wide
|
|
168
|
+
as you like. A Durable Object's SQLite refuses expressions deeper than 100 and
|
|
169
|
+
very long statements, so past a few hops, or past that length, the engine runs
|
|
170
|
+
part of the filter first into a scratch table and tests membership in it;
|
|
171
|
+
every statement stays shallow and short. The results are the same either way.
|
|
172
|
+
|
|
173
|
+
Building a query validates it against the schema, so an unknown field,
|
|
174
|
+
operator or relation throws there and then, at the call, rather than
|
|
175
|
+
resolving to an error later. Everything the engine decides — permissions, a
|
|
176
|
+
missing row, a conflict — surfaces when you await the query or in a live
|
|
177
|
+
query's `error`.
|
|
178
|
+
|
|
179
|
+
### Live
|
|
180
|
+
|
|
181
|
+
The same query object becomes live when you subscribe to it:
|
|
182
|
+
|
|
183
|
+
```ts
|
|
184
|
+
const stop = db.task
|
|
185
|
+
.findMany({ where: { done: false }, orderBy: { due: "asc" } })
|
|
186
|
+
.subscribe(({ data }) => render(data));
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
```tsx
|
|
190
|
+
// React (@tapedb/tapedb/react)
|
|
191
|
+
const { data, isLoading } = useLive(
|
|
192
|
+
db.task.findMany({ where: { columnId }, orderBy: { position: "asc" } })
|
|
193
|
+
);
|
|
194
|
+
|
|
195
|
+
const feed = useLivePage(db.task.findPage({ orderBy: { id: "desc" }, take: 50 }));
|
|
196
|
+
// feed.rows, feed.hasMore, feed.loadMore() — every loaded page stays live and
|
|
197
|
+
// survives reconnects, however many are loaded
|
|
198
|
+
|
|
199
|
+
const peers = usePresence(db, "board:" + boardId, { name, color });
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
After a commit, only subscriptions whose query touches a changed model re-run,
|
|
203
|
+
batched at 50 ms, and clients receive row-level diffs. Counts and aggregates stay live too.
|
|
204
|
+
|
|
205
|
+
### Across workspaces
|
|
206
|
+
|
|
207
|
+
```ts
|
|
208
|
+
const across = db.$across([teamA, teamB, teamC], {
|
|
209
|
+
getToken: (workspaceId) => fetchClaim(workspaceId),
|
|
210
|
+
});
|
|
211
|
+
|
|
212
|
+
const due = await across.task.findMany({ where: { done: false }, orderBy: { due: "asc" }, take: 50 });
|
|
213
|
+
// each row carries its workspaceId
|
|
214
|
+
|
|
215
|
+
const page = await across.task.findPage({ orderBy: { id: "desc" }, take: 50, after });
|
|
216
|
+
const open = await across.task.count({ where: { done: false } });
|
|
217
|
+
|
|
218
|
+
const { data, errors } = useLive(across.task.findMany({ orderBy: { title: "asc" } }));
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
`findMany`, `findFirst`, `findPage` and `count` read every listed workspace,
|
|
222
|
+
each with its own claim, and merge the rows in `orderBy` order (the same order
|
|
223
|
+
SQLite uses). Without `orderBy`, rows come workspace by workspace. A page's
|
|
224
|
+
`cursor` keeps a position in each workspace. Live queries hold one socket per
|
|
225
|
+
workspace and update as any of them changes. If a workspace fails, a one-off
|
|
226
|
+
read rejects with `AcrossError`: its `errors` name the failed workspaces and
|
|
227
|
+
`partial` holds the rest. A live state keeps the other workspaces' rows and
|
|
228
|
+
lists the failures in `errors`. Writes and transactions stay in one workspace.
|
|
229
|
+
|
|
230
|
+
## Writes
|
|
231
|
+
|
|
232
|
+
```ts
|
|
233
|
+
const { id } = await db.task.create({
|
|
234
|
+
data: { title: "Ship it", column: { connect: columnId } },
|
|
235
|
+
});
|
|
236
|
+
|
|
237
|
+
await db.task.update({ where: { id }, data: { done: true } });
|
|
238
|
+
await db.task.upsert({ where: { id }, create: { title: "A" }, update: { title: "B" } });
|
|
239
|
+
await db.task.delete({ where: { id } });
|
|
240
|
+
|
|
241
|
+
await db.column.update({
|
|
242
|
+
where: { id: columnId },
|
|
243
|
+
data: { tasks: { connect: [a, b], disconnect: [c] } },
|
|
244
|
+
});
|
|
245
|
+
|
|
246
|
+
// Several writes, one log entry, all or nothing
|
|
247
|
+
await db.$transaction([
|
|
248
|
+
db.task.update({ where: { id: a }, data: { position: 2 } }),
|
|
249
|
+
db.task.update({ where: { id: b }, data: { position: 1 } }),
|
|
250
|
+
]);
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
`create` fails if the id exists; `update` fails if the row is missing;
|
|
254
|
+
`upsert` chooses inside the engine's transaction, so it cannot race. Writes
|
|
255
|
+
apply optimistically, queue in an IndexedDB outbox while offline, and resolve
|
|
256
|
+
to `{ id, status: "applied" | "queued" }`. A write the engine refuses for
|
|
257
|
+
good during a background flush reaches `onWriteRejected`.
|
|
258
|
+
|
|
259
|
+
Retries are deduplicated by client transaction id without an age limit.
|
|
260
|
+
Receipts survive snapshots, replay-log pruning and recovery from R2. Each
|
|
261
|
+
transaction keeps a small row containing its random id, sequence and time for
|
|
262
|
+
the workspace's lifetime. Its operation hash is removed after half the
|
|
263
|
+
[retention](#history-retention), 15 days by default. Until then, reusing an id
|
|
264
|
+
with different operations is rejected; afterwards, the id acknowledges the
|
|
265
|
+
original write regardless of its body and never applies another write.
|
|
266
|
+
|
|
267
|
+
An upgrade or restore from an engine that discarded receipts keeps a fixed
|
|
268
|
+
boundary for missing receipts. An unknown retry first issued before that
|
|
269
|
+
boundary is refused with `retry_expired` (HTTP 410) and reaches
|
|
270
|
+
`onWriteRejected`; the app can send it again as a new write. The boundary does
|
|
271
|
+
not advance with time. The client sends each write's original `issuedAt`.
|
|
272
|
+
|
|
273
|
+
|
|
274
|
+
Related rows can be created with the row that needs them:
|
|
275
|
+
|
|
276
|
+
```ts
|
|
277
|
+
await db.board.create({
|
|
278
|
+
data: {
|
|
279
|
+
name: "Launch",
|
|
280
|
+
columns: { create: [{ name: "Todo", position: 1, tasks: { create: { title: "Ship" } } }] },
|
|
281
|
+
},
|
|
282
|
+
});
|
|
283
|
+
|
|
284
|
+
await db.column.create({ data: { name: "Solo", board: { create: { name: "New board" } } } });
|
|
285
|
+
```
|
|
286
|
+
|
|
287
|
+
It is one transaction, ordered so each row exists before anything points at
|
|
288
|
+
it, and one failure rolls the whole tree back. The types follow every level.
|
|
289
|
+
|
|
290
|
+
### Writing many rows
|
|
291
|
+
|
|
292
|
+
```ts
|
|
293
|
+
const { count } = await db.task.updateMany({ where: { done: true }, data: { archived: true } });
|
|
294
|
+
await db.task.deleteMany({ where: { columnId: null } });
|
|
295
|
+
```
|
|
296
|
+
|
|
297
|
+
The engine resolves which rows match, inside the transaction, and logs the
|
|
298
|
+
rows rather than the filter — a replay has to change what the original
|
|
299
|
+
changed, and re-running a filter later cannot promise that. A bulk write
|
|
300
|
+
reaches only rows the caller may read, and still runs the row rules for
|
|
301
|
+
`update` and `delete` on each one.
|
|
302
|
+
|
|
303
|
+
Every matching row is written in one transaction, however many match: the
|
|
304
|
+
write commits whole or not at all, and one row refused by a rule rolls it all
|
|
305
|
+
back. The engine resolves the filter once, against the database before the
|
|
306
|
+
write, then checks, applies and logs the rows a chunk at a time, so memory
|
|
307
|
+
stays flat however large the write is; a log entry larger than a Durable
|
|
308
|
+
Object row holds is stored in parts.
|
|
309
|
+
|
|
310
|
+
Like single-row writes, a bulk write is applied to the cache at once and
|
|
311
|
+
queued offline. The cache changes the rows it holds that certainly match; a
|
|
312
|
+
row whose match depends on a relation, or on a field the cache does not hold,
|
|
313
|
+
waits for the engine. The engine resolves the filter when the write reaches
|
|
314
|
+
it, so a write queued offline applies to the rows matching then. Queued, the
|
|
315
|
+
result is `{ count, queued: true }`, `count` being the cached rows changed.
|
|
316
|
+
|
|
317
|
+
## Access rules
|
|
318
|
+
|
|
319
|
+
Each model's actions take one of three decisions:
|
|
320
|
+
|
|
321
|
+
- `role("member" | "editor" | "moderator" | ...)` — a minimum role, checked
|
|
322
|
+
before any SQL. Defaults: read `member`, create/update `editor`, delete
|
|
323
|
+
`moderator`.
|
|
324
|
+
- `filter(({ auth }) => where)` — `read` only. Compiled into the SQL of every
|
|
325
|
+
query that touches the model, so it uses indexes, fills every page, and
|
|
326
|
+
counts correctly. An array is OR; the caller's own filter ANDs with it.
|
|
327
|
+
- `check(({ auth, row, data, changed }) => boolean)` — any action, arbitrary
|
|
328
|
+
TypeScript. Write checks run inside the transaction next to the write they
|
|
329
|
+
gate. On `read` they scan matching rows until the page is full or the
|
|
330
|
+
result set is exhausted. Counts apply the same visibility rule.
|
|
331
|
+
A rule that throws denies.
|
|
332
|
+
|
|
333
|
+
## Migrations
|
|
334
|
+
|
|
335
|
+
```bash
|
|
336
|
+
# edit tapedb.schema.ts, then
|
|
337
|
+
npx @tapedb/tapedb migrate dev --name add_task_priority
|
|
338
|
+
```
|
|
339
|
+
|
|
340
|
+
`migrate dev` diffs the schema against `tapedb/schema.snapshot.json` and
|
|
341
|
+
writes the next numbered SQL file — reviewed and committed like any ORM
|
|
342
|
+
migration — plus the bundle `worker.ts` imports. Each workspace's Durable
|
|
343
|
+
Object applies pending migrations on its next wake, one transaction per
|
|
344
|
+
migration, with foreign keys switched off around it and checked before it
|
|
345
|
+
commits (SQLite's documented procedure for schema changes): a migration that
|
|
346
|
+
would leave a row pointing at nothing rolls back.
|
|
347
|
+
|
|
348
|
+
- A new required field needs `.default(...)` or `.optional()`.
|
|
349
|
+
- Drops need `--accept-data-loss`; renames need `--rename old=new` or
|
|
350
|
+
`--rename model.old=new`, otherwise they read as a drop plus an add.
|
|
351
|
+
- Type, default and reference changes rebuild the table, including tables
|
|
352
|
+
other tables reference: child rows keep their keys and nothing cascades.
|
|
353
|
+
- Migrations declare what they changed, so recovery can replay log segments
|
|
354
|
+
and snapshots written under an older schema. Undeclared drift fails closed.
|
|
355
|
+
|
|
356
|
+
```bash
|
|
357
|
+
npx @tapedb/tapedb migrate rollback # back to the schema before the last migration
|
|
358
|
+
npx @tapedb/tapedb migrate rollback --to 0004_tags # back to the schema after 0004_tags
|
|
359
|
+
```
|
|
360
|
+
|
|
361
|
+
`migrate rollback` writes the migration that takes the database back to an
|
|
362
|
+
earlier schema, as the next numbered file: workspaces apply it like any other,
|
|
363
|
+
and recovery reads data written on either side of it. Renames are undone with
|
|
364
|
+
their values; undoing an added field or model discards what was written to it
|
|
365
|
+
since, so it needs `--accept-data-loss`. A migration that dropped data cannot
|
|
366
|
+
be undone, and rollback says so. It then lists what to change in the schema
|
|
367
|
+
code; `migrate check` passes once the code matches. The schema after each
|
|
368
|
+
migration is kept in `tapedb/snapshots/`; commit it with the migrations.
|
|
369
|
+
|
|
370
|
+
`migrate check` fails CI (or a pre-commit hook) when the schema has changes
|
|
371
|
+
with no migration. `tapedb deploy` refuses to ship a dev configuration or an
|
|
372
|
+
unmigrated schema, then runs `wrangler deploy`.
|
|
373
|
+
|
|
374
|
+
## Files and presence
|
|
375
|
+
|
|
376
|
+
```ts
|
|
377
|
+
const file = await db.$files.upload(blob, { path: "avatars/me.png" });
|
|
378
|
+
const all = await db.$files.list({ prefix: "avatars/" });
|
|
379
|
+
await db.$files.delete("avatars/me.png");
|
|
380
|
+
|
|
381
|
+
const room = db.$presence("board:" + boardId);
|
|
382
|
+
room.set({ name, color });
|
|
383
|
+
room.cursor({ x, y });
|
|
384
|
+
const stop = room.subscribe((peers) => render(peers));
|
|
385
|
+
```
|
|
386
|
+
|
|
387
|
+
File bytes live in the `FILES` bucket (any size, uploaded in parts past
|
|
388
|
+
16 MiB); their index, with who uploaded each file, is written through the
|
|
389
|
+
transaction log, so a restore keeps it. Presence rides the
|
|
390
|
+
workspace socket.
|
|
391
|
+
|
|
392
|
+
Who may read, write and delete each file is set in the schema. Without
|
|
393
|
+
rules: read `member`, write and delete `editor`. Each rule is a `role()`, a
|
|
394
|
+
`fileCheck()` over the caller and the file, or `attachedTo(model, path => id)`,
|
|
395
|
+
which makes a file follow its row: readable by whoever can read the row,
|
|
396
|
+
writable and deletable by whoever may update it.
|
|
397
|
+
|
|
398
|
+
```ts
|
|
399
|
+
export default schema({ task, project }, {
|
|
400
|
+
files: {
|
|
401
|
+
read: attachedTo("task", (path) => path.split("/")[1]), // tasks/<id>/...
|
|
402
|
+
write: attachedTo("task", (path) => path.split("/")[1]),
|
|
403
|
+
delete: fileCheck(({ auth, file }) => file.uploadedBy === auth.sub),
|
|
404
|
+
},
|
|
405
|
+
});
|
|
406
|
+
```
|
|
407
|
+
|
|
408
|
+
The workspace decides each operation before any byte moves, and `list()`
|
|
409
|
+
returns only the files the caller may read. A file the caller may not read
|
|
410
|
+
answers 404, like one that does not exist. A rule that throws denies, and an
|
|
411
|
+
editor's role does not override a rule.
|
|
412
|
+
|
|
413
|
+
### Broadcast
|
|
414
|
+
|
|
415
|
+
Fire-and-forget messages between the clients of a workspace, on the same
|
|
416
|
+
socket:
|
|
417
|
+
|
|
418
|
+
```ts
|
|
419
|
+
const board = db.$broadcast<{ kind: "ping"; at: number }>("board:" + boardId);
|
|
420
|
+
const stop = board.subscribe((message, from) => flash(from.userId, message));
|
|
421
|
+
board.send({ kind: "ping", at: Date.now() }); // false (dropped) while offline
|
|
422
|
+
|
|
423
|
+
// React (@tapedb/tapedb/react)
|
|
424
|
+
const send = useBroadcast(db, "board:" + boardId, (message, from) => flash(from.userId, message));
|
|
425
|
+
```
|
|
426
|
+
|
|
427
|
+
A message reaches every other client listening on the topic, with the
|
|
428
|
+
sender's client id and user id; your own messages are not echoed back.
|
|
429
|
+
A dropped connection loses nothing recent. Sends made while the socket is
|
|
430
|
+
down wait (up to 100 per topic, for 60 seconds; `send` returns `"queued"`)
|
|
431
|
+
and go out after reconnecting. The workspace keeps the last 60 seconds of
|
|
432
|
+
each topic (`topics.replaySeconds`, at most 1,000 messages), and a listener
|
|
433
|
+
that reconnects gets what it missed before new messages, each once, in
|
|
434
|
+
order. Presence converges the same way: after a reconnect the client
|
|
435
|
+
publishes its state again and receives everyone else's. Messages are limited to
|
|
436
|
+
4 MiB of encoded JSON, including multipart messages. The workspace relays parts
|
|
437
|
+
as they arrive, and the receiving client joins them. Pending assembly is bounded
|
|
438
|
+
to 8 MiB per socket and 32 MiB per workspace; excess messages are rejected. By default anyone with a claim for the workspace may listen and send, guests included; decide
|
|
439
|
+
per topic on the Worker:
|
|
440
|
+
|
|
441
|
+
```ts
|
|
442
|
+
defineTapeDB({
|
|
443
|
+
schema,
|
|
444
|
+
topics: {
|
|
445
|
+
authorize: ({ auth, topic, action }) =>
|
|
446
|
+
!topic.startsWith("admin:") || auth.role >= WORKSPACE_ROLE_LEVELS.ADMIN,
|
|
447
|
+
},
|
|
448
|
+
});
|
|
449
|
+
```
|
|
450
|
+
|
|
451
|
+
## Export
|
|
452
|
+
|
|
453
|
+
```ts
|
|
454
|
+
for await (const record of db.$export()) {
|
|
455
|
+
if (record.type === "row") save(record.model, record.data); // typed per model
|
|
456
|
+
}
|
|
457
|
+
```
|
|
458
|
+
|
|
459
|
+
```bash
|
|
460
|
+
npx @tapedb/tapedb export --workspace <uuid> --out workspace.ndjson # TAPEDB_TOKEN, or a dev claim
|
|
461
|
+
```
|
|
462
|
+
|
|
463
|
+
The engine streams NDJSON: a `header` line (the models exported, and any the
|
|
464
|
+
caller may not read), one `row` line per row — model by model, in id order,
|
|
465
|
+
through the same read rules as any query — one `file` line per file in the
|
|
466
|
+
index (bytes stay in R2), and an `end` line with counts. Rows are read 1000 at
|
|
467
|
+
a time as the stream is consumed, so a large workspace never sits in memory.
|
|
468
|
+
Writes keep landing during an export; `end.consistent` is true when none did,
|
|
469
|
+
and the export is then the workspace exactly at `end.seq`. An export that
|
|
470
|
+
stops early ends with an `error` line instead of `end` (the CLI exits 1).
|
|
471
|
+
|
|
472
|
+
## Audit
|
|
473
|
+
|
|
474
|
+
```ts
|
|
475
|
+
export default defineTapeDB({ schema, audit: true });
|
|
476
|
+
```
|
|
477
|
+
|
|
478
|
+
```ts
|
|
479
|
+
const changes = await db.$history({ model: "task", id: task.id });
|
|
480
|
+
// [{ seq, at, sub, role, action: "insert" | "update" | "delete" }, ...]
|
|
481
|
+
```
|
|
482
|
+
|
|
483
|
+
With `audit: true` each write records the claim's `sub` and role, in the
|
|
484
|
+
local log and in the log lines in R2, and recovery restores them. Every row a
|
|
485
|
+
write changes, cascades included, gets a history entry. `$history` returns a
|
|
486
|
+
row's entries, oldest first, to admins and owners. The history follows the
|
|
487
|
+
[history retention](#history-retention): entries are kept for half of it (15
|
|
488
|
+
days by default) and are gone from every copy within it. Raise
|
|
489
|
+
`retention.days` for a longer audit. Off by default: without it, no writer
|
|
490
|
+
identity is stored, and `$history` answers 409 `audit_disabled`.
|
|
491
|
+
|
|
492
|
+
## Personal fields
|
|
493
|
+
|
|
494
|
+
```ts
|
|
495
|
+
const person = model({
|
|
496
|
+
email: t.string().personal({ subject: "self" }),
|
|
497
|
+
details: t.json().personal({ subject: "self" }).optional(),
|
|
498
|
+
});
|
|
499
|
+
// Or use an ordinary string field such as userId as the subject.
|
|
500
|
+
await db.$erasePerson(personId); // owner only
|
|
501
|
+
```
|
|
502
|
+
|
|
503
|
+
Personal string and JSON values use AES-GCM with one key per subject. Set
|
|
504
|
+
`TAPEDB_KEY_SECRET` to a stable random secret of at least 32 characters with
|
|
505
|
+
`wrangler secret put TAPEDB_KEY_SECRET`; `tapedb init` generates a local key.
|
|
506
|
+
Wrapped subject keys live in R2 `DATA` under `keys/<workspace>/`; their names
|
|
507
|
+
are HMACs of the subject. SQL, snapshots and transaction logs store ciphertext.
|
|
508
|
+
Queries and exports decrypt values after applying the caller's access rules.
|
|
509
|
+
Personal fields cannot be indexed, filtered, ordered or aggregated.
|
|
510
|
+
|
|
511
|
+
`$erasePerson(subject)` deletes the subject key. Existing ciphertext becomes
|
|
512
|
+
unreadable in current storage, old snapshots, logs and Durable Object recovery
|
|
513
|
+
copies. Reads return `null` and list affected fields in `$erased`. Writing new
|
|
514
|
+
data for that subject creates a new key that cannot decrypt the erased data.
|
|
515
|
+
Files and ordinary fields are removed separately. Keep the wrapping secret
|
|
516
|
+
stable across deployments and workspace moves; losing it loses access to all
|
|
517
|
+
personal values. Copies of subject keys outside TapeDB can undo erasure.
|
|
518
|
+
|
|
519
|
+
Enabling `.personal()` encrypts existing rows in resumable batches before the
|
|
520
|
+
workspace serves requests. Older plaintext logs, snapshots and Cloudflare
|
|
521
|
+
recovery copies still follow their original retention windows; encryption
|
|
522
|
+
cannot erase plaintext already held in those copies.
|
|
523
|
+
|
|
524
|
+
## Erasing a workspace
|
|
525
|
+
|
|
526
|
+
```bash
|
|
527
|
+
npx @tapedb/tapedb erase --workspace <uuid> --confirm <uuid> # owner claim: TAPEDB_TOKEN, or a dev claim
|
|
528
|
+
```
|
|
529
|
+
|
|
530
|
+
`DELETE /engine/workspaces/:id` with an owner's claim deletes everything the
|
|
531
|
+
workspace holds: its R2 history (the manifest first, so nothing can be
|
|
532
|
+
recovered from what remains), its uploaded files, then its Durable Object
|
|
533
|
+
storage and alarm. Every step is idempotent: a failed erase (HTTP 500) is
|
|
534
|
+
finished by calling it again. Afterwards the workspace answers
|
|
535
|
+
`410 workspace_erased` to every request, so a client reconnecting with
|
|
536
|
+
queued writes cannot recreate it under the same id. Cloudflare keeps 30 days
|
|
537
|
+
of point-in-time recovery for every SQLite-backed Durable Object; tapedb never
|
|
538
|
+
restores from it. Ordinary plaintext values remain restorable until that
|
|
539
|
+
history ages out. Personal ciphertext loses its key during workspace erasure. That is why the default history
|
|
540
|
+
retention is also 30 days.
|
|
541
|
+
|
|
542
|
+
## Data residency
|
|
543
|
+
|
|
544
|
+
```bash
|
|
545
|
+
npx @tapedb/tapedb init --jurisdiction eu
|
|
546
|
+
```
|
|
547
|
+
|
|
548
|
+
writes `"TAPEDB_JURISDICTION": "eu"` into `wrangler.jsonc` vars and
|
|
549
|
+
`"jurisdiction": "eu"` on both R2 buckets (create them with
|
|
550
|
+
`wrangler r2 bucket create <name> --jurisdiction eu`). Every workspace's
|
|
551
|
+
Durable Object is then created in Cloudflare's EU jurisdiction, where it runs
|
|
552
|
+
and stores data only in the EU, and the R2 history and files stay there too.
|
|
553
|
+
A room reached outside the configured jurisdiction answers
|
|
554
|
+
`500 jurisdiction_mismatch` and stores nothing. `tapedb deploy` refuses a
|
|
555
|
+
config where the objects and the buckets disagree. `fedramp` works the same
|
|
556
|
+
way.
|
|
557
|
+
|
|
558
|
+
### Moving existing workspaces
|
|
559
|
+
|
|
560
|
+
Create destination DATA and FILES buckets in the new jurisdiction. Bind them
|
|
561
|
+
as `DATA` and `FILES`, and retain the original buckets as `DATA_FROM` and
|
|
562
|
+
`FILES_FROM` with their original jurisdiction. Set `TAPEDB_JURISDICTION` to
|
|
563
|
+
the destination and `TAPEDB_MOVE_FROM` to the source (`none`, `eu` or
|
|
564
|
+
`fedramp`), then deploy this move-aware configuration. Keep the same namespace;
|
|
565
|
+
`WORKSPACE_FROM` is needed only when the source uses a different namespace.
|
|
566
|
+
|
|
567
|
+
```sh
|
|
568
|
+
npx @tapedb/tapedb move --to eu --url https://db.example.com --workspaces workspaces.txt
|
|
569
|
+
```
|
|
570
|
+
|
|
571
|
+
The file contains one workspace UUID per line. `--workspace <uuid>` moves one.
|
|
572
|
+
Use an owner claim in `TAPEDB_TOKEN`; for several workspaces, `--tokens tokens.json`
|
|
573
|
+
reads a local object mapping workspace ids to owner tokens. Development uses
|
|
574
|
+
the dev claim route. Discovery uses `--discover --account <id> --source-bucket
|
|
575
|
+
<name>` and `CLOUDFLARE_API_TOKEN`; `--from eu` selects an EU source bucket.
|
|
576
|
+
Include never-flushed workspaces in the id file as they have no R2 manifest.
|
|
577
|
+
|
|
578
|
+
Until a workspace is ready, reads go to its original room. The move freezes
|
|
579
|
+
writes, checkpoints rows and receipts, streams the durable state and indexed
|
|
580
|
+
files to the destination, restores it, then switches traffic and removes the
|
|
581
|
+
source data. Writes receive retryable `workspace_moving` and remain in the
|
|
582
|
+
client outbox. A failed command can be run again to resume; a nonempty
|
|
583
|
+
destination is refused. Remove `TAPEDB_MOVE_FROM` and the `_FROM` bindings only
|
|
584
|
+
after every workspace has completed its move.
|
|
585
|
+
|
|
586
|
+
`POST /engine/workspaces/:id/import` imports a complete, consistent NDJSON
|
|
587
|
+
export into an empty workspace with an owner claim. Rows and file metadata
|
|
588
|
+
are validated, applied and logged atomically. File bytes are transferred by
|
|
589
|
+
`tapedb move`. Import preserves row timestamps and records a fixed boundary
|
|
590
|
+
for transaction receipts absent from the export.
|
|
591
|
+
|
|
592
|
+
Locally workerd has no jurisdictions; `tapedb dev` leaves the setting out.
|
|
593
|
+
|
|
594
|
+
## Claims and identity
|
|
595
|
+
|
|
596
|
+
The engine verifies a short-lived signed claim and never issues one. Identity
|
|
597
|
+
stays yours: implement two methods and the library serves the claim route.
|
|
598
|
+
|
|
599
|
+
```ts
|
|
600
|
+
import { env } from "cloudflare:workers";
|
|
601
|
+
import { defineTapeDB, WORKSPACE_ROLE_LEVELS } from "@tapedb/tapedb/worker";
|
|
602
|
+
import schema from "./tapedb.schema";
|
|
603
|
+
import { migrations } from "./tapedb/migrations.gen";
|
|
604
|
+
|
|
605
|
+
const tapedb = defineTapeDB({
|
|
606
|
+
schema,
|
|
607
|
+
migrations,
|
|
608
|
+
identity: {
|
|
609
|
+
async getSubject(headers) {
|
|
610
|
+
const session = await auth.api.getSession({ headers });
|
|
611
|
+
return session ? { sub: session.user.id } : null;
|
|
612
|
+
},
|
|
613
|
+
async getRole(workspaceId, sub) {
|
|
614
|
+
const member = await memberships.get(workspaceId, sub);
|
|
615
|
+
return member ? WORKSPACE_ROLE_LEVELS[member.role] : null;
|
|
616
|
+
},
|
|
617
|
+
// Optional; these are the defaults.
|
|
618
|
+
limits: {
|
|
619
|
+
ip: { requests: 300, perSeconds: 60 },
|
|
620
|
+
subject: { requests: 30, perSeconds: 60 },
|
|
621
|
+
},
|
|
622
|
+
},
|
|
623
|
+
});
|
|
624
|
+
|
|
625
|
+
export const WorkspaceRoom = tapedb.WorkspaceRoom;
|
|
626
|
+
export default tapedb.worker;
|
|
627
|
+
```
|
|
628
|
+
|
|
629
|
+
`POST /_tapedb/claim` returns `{ token, workspaceId, roleLevel, expiresAt, expiresIn }`.
|
|
630
|
+
The route is rate-limited out of the box: a token bucket per caller IP, checked
|
|
631
|
+
before your session lookup, and one per subject, checked after it. The buckets
|
|
632
|
+
live in Durable Objects, so a count is exact and global rather than per
|
|
633
|
+
location, and they store HMACs of the IP and subject, never the values. Past a
|
|
634
|
+
limit the route answers 429 `rate_limited` with `Retry-After`. Set a limit to
|
|
635
|
+
`false` (or `limits: false`) to turn it off, and add `allowRequest(key)` for a
|
|
636
|
+
check of your own, run after the IP limit with `key` = `claim:<ip>`. If the
|
|
637
|
+
limiter cannot be reached, claims go through and the Worker logs
|
|
638
|
+
`claim_limit_unavailable`.
|
|
639
|
+
The client
|
|
640
|
+
sends it as `Authorization: Bearer <token>` and renews on
|
|
641
|
+
reconnect. On the WebSocket, where browsers cannot set that header, it rides
|
|
642
|
+
in `Sec-WebSocket-Protocol` as `tapedb.claim.<token>` next to `tapedb`, which
|
|
643
|
+
the room selects. A token is never read from the URL, so it stays out of
|
|
644
|
+
request logs and browser history. In development, `tapedb dev` enables a dev claim route so the
|
|
645
|
+
client needs no session at all; `ENVIRONMENT=production` refuses that route,
|
|
646
|
+
and `tapedb deploy` refuses to ship the flag. A deployed test Worker that
|
|
647
|
+
needs the route sets a random `TAPEDB_DEV_CLAIMS_TOKEN` secret first; the route
|
|
648
|
+
then mints only for requests carrying it in `x-tapedb-dev-token`.
|
|
649
|
+
|
|
650
|
+
## Architecture
|
|
651
|
+
|
|
652
|
+
One Durable Object per workspace is the only writer, which is what makes
|
|
653
|
+
transactions serializable without a lock manager. The log is the source of
|
|
654
|
+
truth; snapshots are a checkpoint over it, and the local log is trimmed to
|
|
655
|
+
the same window R2 keeps.
|
|
656
|
+
|
|
657
|
+
```
|
|
658
|
+
client ──HTTP /transact──▶ Worker ──claim──▶ WorkspaceRoom (SQLite)
|
|
659
|
+
▲ │
|
|
660
|
+
└──WebSocket subscribe / presence ◀───────────┘
|
|
661
|
+
│
|
|
662
|
+
R2 DATA (log + snapshots)
|
|
663
|
+
```
|
|
664
|
+
|
|
665
|
+
Durability is tested by deleting the Durable Object's entire SQLite and
|
|
666
|
+
rebuilding from R2: rows, sequence numbers, idempotency keys and the file
|
|
667
|
+
index all survive, including across schema migrations made since the
|
|
668
|
+
snapshot. Recovery runs in steps that commit their progress, a snapshot
|
|
669
|
+
chunk, a log segment or a 256 KiB part of a large write at a time. The first
|
|
670
|
+
boot spends at most 15 seconds on it, well inside the 30 seconds Cloudflare
|
|
671
|
+
allows; alarms carry on in steps of about 10 seconds, each stopping before a
|
|
672
|
+
unit as slow as the slowest so far would overrun it. So a workspace of any
|
|
673
|
+
size recovers, and a recovery cut off by a reset or by an R2 read that fails
|
|
674
|
+
resumes rather than starting over. Until it is complete the workspace answers
|
|
675
|
+
503 `workspace_recovering` with `Retry-After`, never a partial result. The
|
|
676
|
+
client waits it out: reads, writes and file requests retry as `Retry-After`
|
|
677
|
+
says for up to a minute, and live queries reconnect once it is done.
|
|
678
|
+
|
|
679
|
+
### History retention
|
|
680
|
+
|
|
681
|
+
Old values live on in the workspace's history: the R2 log segments, older
|
|
682
|
+
snapshots and the local replay log. tapedb bounds how long. A value deleted or
|
|
683
|
+
overwritten at time T is gone from every copy by T + 30 days, also in a
|
|
684
|
+
workspace that sees few writes: a workspace with changes snapshots once its
|
|
685
|
+
newest snapshot is half the retention old, and superseded snapshots, the log
|
|
686
|
+
they cover and covered local log rows are deleted once older than that.
|
|
687
|
+
Recovery always keeps the newest snapshot and the log after it.
|
|
688
|
+
|
|
689
|
+
```ts
|
|
690
|
+
export default defineTapeDB({ schema, retention: { days: 14 } }); // at least 2
|
|
691
|
+
```
|
|
692
|
+
|
|
693
|
+
## Entry points
|
|
694
|
+
|
|
695
|
+
| import | contains | React |
|
|
696
|
+
| --- | --- | --- |
|
|
697
|
+
| `@tapedb/tapedb` | `createClient`, schema builders, `id` | no (zero runtime deps) |
|
|
698
|
+
| `@tapedb/tapedb/schema` | `schema`, `model`, `t`, `role`, `filter`, `check` | no |
|
|
699
|
+
| `@tapedb/tapedb/react` | `useLive`, `useLivePage`, `usePresence` | peer (React 19) |
|
|
700
|
+
| `@tapedb/tapedb/worker` | `defineTapeDB`, claim helpers | no |
|
|
701
|
+
|
|
702
|
+
The CLI is a command, not an import: `npx @tapedb/tapedb init | migrate | dev | studio
|
|
703
|
+
| deploy | export | erase | move`.
|
|
704
|
+
|
|
705
|
+
[API.md](API.md) is the reference: every export and signature, what is stable
|
|
706
|
+
and what is experimental, the HTTP routes, the export format and the error
|
|
707
|
+
codes. A test checks the package against it.
|
|
708
|
+
|
|
709
|
+
## Security
|
|
710
|
+
|
|
711
|
+
Report vulnerabilities privately as described in [SECURITY.md](SECURITY.md).
|
|
712
|
+
Do not open a public issue for a security report.
|
|
713
|
+
|
|
714
|
+
## License
|
|
715
|
+
|
|
716
|
+
MIT
|
|
717
|
+
|
|
718
|
+
Browser clients using the default persistent outbox must pass `userId`, matching
|
|
719
|
+
the claim subject (`dev-user` with `tapedb dev`). Queues are isolated by backend,
|
|
720
|
+
user and workspace. Close and recreate the client when switching accounts. Old
|
|
721
|
+
unscoped queues remain quarantined and are never automatically replayed.
|