toiljs 0.0.85 → 0.0.87
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +5 -0
- package/README.md +2 -2
- package/build/cli/.tsbuildinfo +1 -1
- package/build/cli/index.js +303 -293
- package/build/compiler/.tsbuildinfo +1 -1
- package/build/compiler/config.d.ts +2 -0
- package/build/compiler/config.js +1 -0
- package/build/compiler/docs.js +8 -24
- package/build/compiler/generate.js +4 -2
- package/build/compiler/index.d.ts +1 -1
- package/build/compiler/index.js +69 -6
- package/build/compiler/toil-docs.generated.js +64 -22
- package/build/devserver/.tsbuildinfo +1 -1
- package/build/devserver/analytics/index.js +7 -3
- package/build/devserver/db/database.d.ts +4 -0
- package/build/devserver/db/database.js +43 -1
- package/build/devserver/db/index.d.ts +1 -1
- package/build/devserver/db/index.js +1 -1
- package/build/devserver/db/types.d.ts +3 -0
- package/build/devserver/db/types.js +2 -0
- package/build/devserver/runtime/module.js +4 -1
- package/docs/README.md +104 -65
- package/docs/auth/README.md +102 -0
- package/docs/auth/configuration.md +94 -0
- package/docs/auth/extending.md +202 -0
- package/docs/auth/how-it-works.md +138 -0
- package/docs/auth/usage.md +188 -0
- package/docs/backend/README.md +143 -0
- package/docs/backend/data.md +351 -0
- package/docs/backend/rest.md +402 -0
- package/docs/backend/rpc.md +226 -0
- package/docs/background/README.md +114 -0
- package/docs/background/daemons.md +230 -0
- package/docs/background/derive.md +179 -0
- package/docs/cli/README.md +377 -0
- package/docs/concepts/config.md +416 -0
- package/docs/concepts/decorators.md +127 -0
- package/docs/concepts/security.md +108 -0
- package/docs/concepts/tiers.md +166 -0
- package/docs/concepts/types.md +216 -0
- package/docs/database/README.md +143 -0
- package/docs/database/capacity.md +350 -0
- package/docs/database/counters.md +174 -0
- package/docs/database/documents.md +255 -0
- package/docs/database/events.md +307 -0
- package/docs/database/membership.md +246 -0
- package/docs/database/setup.md +155 -0
- package/docs/database/unique.md +216 -0
- package/docs/database/views.md +246 -0
- package/docs/frontend/README.md +101 -0
- package/docs/frontend/data-fetching.md +243 -0
- package/docs/frontend/images.md +148 -0
- package/docs/frontend/metadata.md +344 -0
- package/docs/frontend/rendering.md +236 -0
- package/docs/frontend/routing.md +344 -0
- package/docs/frontend/scripts.md +118 -0
- package/docs/frontend/search.md +191 -0
- package/docs/frontend/styling.md +147 -0
- package/docs/getting-started/README.md +81 -0
- package/docs/getting-started/create-project.md +131 -0
- package/docs/getting-started/deploy.md +101 -0
- package/docs/getting-started/first-app.md +215 -0
- package/docs/getting-started/installation.md +106 -0
- package/docs/getting-started/migrating.md +125 -0
- package/docs/getting-started/project-structure.md +163 -0
- package/docs/introduction/README.md +41 -0
- package/docs/introduction/design-principles.md +55 -0
- package/docs/introduction/distributed.md +74 -0
- package/docs/introduction/how-it-works.md +74 -0
- package/docs/introduction/hyperscale.md +51 -0
- package/docs/introduction/modern-stack.md +36 -0
- package/docs/introduction/vs-other-frameworks.md +48 -0
- package/docs/introduction/why-toil.md +93 -0
- package/docs/llms.txt +90 -0
- package/docs/realtime/README.md +102 -0
- package/docs/realtime/channels.md +211 -0
- package/docs/realtime/streams.md +369 -0
- package/docs/services/README.md +60 -0
- package/docs/services/analytics.md +268 -0
- package/docs/services/caching.md +175 -0
- package/docs/services/cookies.md +235 -0
- package/docs/services/crypto.md +209 -0
- package/docs/services/email.md +289 -0
- package/docs/services/environment.md +141 -0
- package/docs/services/ratelimit.md +174 -0
- package/docs/services/time.md +85 -0
- package/examples/basic/client/routes/analytics.tsx +13 -12
- package/examples/basic/server/models/SiteAnalytics.ts +29 -17
- package/examples/basic/server/routes/Analytics.ts +15 -17
- package/examples/basic/server/routes/UserId.ts +22 -0
- package/package.json +15 -11
- package/scripts/gen-toil-docs.mjs +24 -35
- package/server/auth/AuthController.ts +336 -0
- package/server/auth/AuthUser.ts +23 -0
- package/server/auth/index.ts +16 -0
- package/server/globals/auth.ts +31 -0
- package/server/globals/userid.ts +107 -0
- package/src/compiler/config.ts +13 -0
- package/src/compiler/docs.ts +16 -33
- package/src/compiler/generate.ts +6 -2
- package/src/compiler/index.ts +114 -6
- package/src/compiler/toil-docs.generated.ts +64 -22
- package/src/devserver/analytics/index.ts +10 -4
- package/src/devserver/db/database.ts +67 -1
- package/src/devserver/db/index.ts +1 -0
- package/src/devserver/db/types.ts +13 -0
- package/src/devserver/runtime/module.ts +7 -0
- package/test/analytics-dev.test.ts +2 -1
- package/test/devserver-database.test.ts +113 -0
- package/docs/auth-todo.md +0 -149
- package/docs/auth.md +0 -322
- package/docs/caching.md +0 -115
- package/docs/cli.md +0 -17
- package/docs/client.md +0 -39
- package/docs/cookies.md +0 -457
- package/docs/crypto.md +0 -130
- package/docs/daemon.md +0 -123
- package/docs/data.md +0 -131
- package/docs/derive.md +0 -159
- package/docs/email.md +0 -326
- package/docs/environment.md +0 -97
- package/docs/getting-started.md +0 -128
- package/docs/index.md +0 -30
- package/docs/ratelimit.md +0 -95
- package/docs/routing.md +0 -259
- package/docs/rpc.md +0 -149
- package/docs/server.md +0 -61
- package/docs/ssr.md +0 -632
- package/docs/streams.md +0 -178
- package/docs/styling.md +0 -22
- package/docs/tiers.md +0 -133
- package/docs/time.md +0 -43
|
@@ -0,0 +1,246 @@
|
|
|
1
|
+
# Membership (sets and relationships)
|
|
2
|
+
|
|
3
|
+
A `Membership` collection stores **sets**: unordered groups of members under a
|
|
4
|
+
key. You add and remove members, ask whether a member is in the set, and list a
|
|
5
|
+
set's members. It is the natural fit for many-to-many relationships like group
|
|
6
|
+
members, followers, and tags.
|
|
7
|
+
|
|
8
|
+
## What and why
|
|
9
|
+
|
|
10
|
+
A "set" is a collection with no duplicates and no order: a member is either in it
|
|
11
|
+
or not. `Membership` gives you one set per key. The key names the set; each member
|
|
12
|
+
is one item in it.
|
|
13
|
+
|
|
14
|
+
This maps directly onto **many-to-many relationships**, where each thing on one
|
|
15
|
+
side can relate to many things on the other:
|
|
16
|
+
|
|
17
|
+
- **Group members:** the key is a group id, the members are user ids.
|
|
18
|
+
- **Followers:** the key is a user id, the members are the ids of their followers.
|
|
19
|
+
- **Tags:** the key is an article id, the members are tag names.
|
|
20
|
+
- **Access control:** the key is a resource id, the members are the users allowed
|
|
21
|
+
in.
|
|
22
|
+
|
|
23
|
+
You could try to store these as an array field inside a [Document](./documents.md)
|
|
24
|
+
(for example `group.memberIds: string[]`). That works for **small, rarely-changed**
|
|
25
|
+
sets, but it has real limits:
|
|
26
|
+
|
|
27
|
+
| | Array in a Document | `Membership` set |
|
|
28
|
+
| --- | --- | --- |
|
|
29
|
+
| Add/remove one member | Read the whole array, edit it, write it all back | One direct `add`/`remove`, no read-modify-write |
|
|
30
|
+
| Concurrent edits | Two writers can clobber each other's array | Each `add`/`remove` is its own operation |
|
|
31
|
+
| Big sets | The whole array is loaded on every read/write | Members are stored separately; you read a bounded page |
|
|
32
|
+
| "Is X a member?" | Load the array and search it | One direct `contains` check |
|
|
33
|
+
|
|
34
|
+
Rule of thumb: a **handful** of stable items (a user's two or three roles) is fine
|
|
35
|
+
as an array in a Document. A set that **grows** or is **edited concurrently**
|
|
36
|
+
(followers, group members, tags across many articles) should be a `Membership`.
|
|
37
|
+
|
|
38
|
+
```mermaid
|
|
39
|
+
flowchart LR
|
|
40
|
+
K["Group key: 'eng'"] --> S[("Membership set")]
|
|
41
|
+
S --> M1["ada"]
|
|
42
|
+
S --> M2["grace"]
|
|
43
|
+
S --> M3["linus"]
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
## The type
|
|
47
|
+
|
|
48
|
+
```ts
|
|
49
|
+
Membership<K, M>
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
- `K` is the key type: it picks *which* set. For group membership, the key is the
|
|
53
|
+
group id.
|
|
54
|
+
- `M` is the member type: the shape of one member. For user membership, that is a
|
|
55
|
+
user id. Both are [`@data`](../concepts/types.md) classes.
|
|
56
|
+
|
|
57
|
+
Declare it as a `@collection` field on a `@database` class:
|
|
58
|
+
|
|
59
|
+
```ts
|
|
60
|
+
@data
|
|
61
|
+
class GroupKey {
|
|
62
|
+
group: string = '';
|
|
63
|
+
constructor(group: string = '') { this.group = group; }
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
@data
|
|
67
|
+
class Member {
|
|
68
|
+
userId: string = '';
|
|
69
|
+
constructor(userId: string = '') { this.userId = userId; }
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
@database
|
|
73
|
+
class GroupsDb {
|
|
74
|
+
@collection static members: Membership<GroupKey, Member>;
|
|
75
|
+
}
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
## Operations
|
|
79
|
+
|
|
80
|
+
Four operations, matching the toilscript API exactly. Exact signatures:
|
|
81
|
+
|
|
82
|
+
| Operation | Signature | What it does |
|
|
83
|
+
| --- | --- | --- |
|
|
84
|
+
| `contains` | `contains(key: K, member: M): bool` | Is `member` in the set? |
|
|
85
|
+
| `add` | `add(key: K, member: M): void` | Put `member` in the set (idempotent). |
|
|
86
|
+
| `remove` | `remove(key: K, member: M): void` | Take `member` out of the set (idempotent). |
|
|
87
|
+
| `list` | `list(key: K, limit: i32): M[]` | Up to `limit` members of the set. |
|
|
88
|
+
|
|
89
|
+
### `contains`
|
|
90
|
+
|
|
91
|
+
A direct membership check. It is a keyed read, so it is allowed from any handler,
|
|
92
|
+
including a read-only `@get`.
|
|
93
|
+
|
|
94
|
+
```ts
|
|
95
|
+
if (GroupsDb.members.contains(new GroupKey('eng'), new Member('ada'))) {
|
|
96
|
+
// ada is in the 'eng' group
|
|
97
|
+
}
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
### `add` and `remove`
|
|
101
|
+
|
|
102
|
+
Both are **writes**, so you call them from an action handler (`@post`, `@put`,
|
|
103
|
+
`@patch`, `@del`), not from a `@get`. Both are **idempotent**: "idempotent"
|
|
104
|
+
means doing it again has no extra effect.
|
|
105
|
+
|
|
106
|
+
- `add` on a member already in the set: no change, no error.
|
|
107
|
+
- `remove` on a member not in the set: no change, no error.
|
|
108
|
+
|
|
109
|
+
```ts
|
|
110
|
+
GroupsDb.members.add(new GroupKey('eng'), new Member('ada')); // ada joins
|
|
111
|
+
GroupsDb.members.add(new GroupKey('eng'), new Member('ada')); // no-op, still one 'ada'
|
|
112
|
+
GroupsDb.members.remove(new GroupKey('eng'), new Member('linus')); // linus leaves (or no-op)
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
Because they are idempotent, you do not need to check `contains` before calling
|
|
116
|
+
them. Just `add` to join and `remove` to leave.
|
|
117
|
+
|
|
118
|
+
### `list`
|
|
119
|
+
|
|
120
|
+
Returns up to `limit` members of a set.
|
|
121
|
+
|
|
122
|
+
```ts
|
|
123
|
+
const roster = GroupsDb.members.list(new GroupKey('eng'), 100);
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
`list` is a **scan** (it can walk many rows), so it is barred on the request
|
|
127
|
+
path: you **cannot** call it from a `@get` or a `@post`. Like reading a whole
|
|
128
|
+
event log, listing a whole set belongs off the request path, in a
|
|
129
|
+
[`@derive`](../background/derive.md) or a `@job`, which publishes a
|
|
130
|
+
[View](./views.md) your routes read. The [worked example](#worked-example-group-membership)
|
|
131
|
+
below shows this.
|
|
132
|
+
|
|
133
|
+
`list` returns up to `limit` members. A set can be larger than one page, so treat
|
|
134
|
+
the result as "a bounded page of members", not necessarily "every member".
|
|
135
|
+
|
|
136
|
+
## Worked example: group membership
|
|
137
|
+
|
|
138
|
+
A user joins or leaves a group from a route; a derive lists the roster into a
|
|
139
|
+
view; a route reads the view. Membership checks happen directly in the route.
|
|
140
|
+
|
|
141
|
+
```ts
|
|
142
|
+
import { GroupKey } from '../models/GroupKey';
|
|
143
|
+
import { Member } from '../models/Member';
|
|
144
|
+
import { Roster } from '../models/Roster';
|
|
145
|
+
import { JoinRequest } from '../models/JoinRequest';
|
|
146
|
+
|
|
147
|
+
@database
|
|
148
|
+
class GroupsDb {
|
|
149
|
+
// The set: who is in each group.
|
|
150
|
+
@collection static members: Membership<GroupKey, Member>;
|
|
151
|
+
// A precomputed roster for the GET (listing is a scan, barred on routes).
|
|
152
|
+
@collection static roster: View<GroupKey, Roster>;
|
|
153
|
+
|
|
154
|
+
@derive
|
|
155
|
+
rebuild(): void {
|
|
156
|
+
const key = new GroupKey('eng');
|
|
157
|
+
const r = new Roster();
|
|
158
|
+
r.users = GroupsDb.members.list(key, 500); // scan, allowed in a derive
|
|
159
|
+
GroupsDb.roster.publish(key, r);
|
|
160
|
+
}
|
|
161
|
+
}
|
|
162
|
+
|
|
163
|
+
@rest('groups')
|
|
164
|
+
class Groups {
|
|
165
|
+
// GET the roster from the view (a keyed read, not a scan).
|
|
166
|
+
@get('/eng')
|
|
167
|
+
public list(): Roster {
|
|
168
|
+
const r = GroupsDb.roster.get(new GroupKey('eng'));
|
|
169
|
+
return r == null ? new Roster() : r;
|
|
170
|
+
}
|
|
171
|
+
|
|
172
|
+
// Direct membership check: `contains` is a keyed read, legal in a GET.
|
|
173
|
+
@get('/eng/is-member')
|
|
174
|
+
public isMember(userId: string): bool {
|
|
175
|
+
return GroupsDb.members.contains(new GroupKey('eng'), new Member(userId));
|
|
176
|
+
}
|
|
177
|
+
|
|
178
|
+
// Join: `add` is idempotent, so joining twice is harmless.
|
|
179
|
+
@post('/eng/join')
|
|
180
|
+
public join(input: JoinRequest): bool {
|
|
181
|
+
GroupsDb.members.add(new GroupKey('eng'), new Member(input.userId));
|
|
182
|
+
return true; // the @derive rebuilds the roster view right after
|
|
183
|
+
}
|
|
184
|
+
|
|
185
|
+
// Leave: `remove` is idempotent, so leaving when not a member is harmless.
|
|
186
|
+
@post('/eng/leave')
|
|
187
|
+
public leave(input: JoinRequest): bool {
|
|
188
|
+
GroupsDb.members.remove(new GroupKey('eng'), new Member(input.userId));
|
|
189
|
+
return true;
|
|
190
|
+
}
|
|
191
|
+
}
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
The models:
|
|
195
|
+
|
|
196
|
+
```ts
|
|
197
|
+
@data
|
|
198
|
+
export class Member {
|
|
199
|
+
userId: string = '';
|
|
200
|
+
}
|
|
201
|
+
|
|
202
|
+
@data
|
|
203
|
+
export class Roster {
|
|
204
|
+
users: Member[] = [];
|
|
205
|
+
}
|
|
206
|
+
```
|
|
207
|
+
|
|
208
|
+
The split is the point: `contains`/`add`/`remove` act on a single member and are
|
|
209
|
+
legal on the request path; `list` scans the whole set and lives in a derive that
|
|
210
|
+
feeds a view.
|
|
211
|
+
|
|
212
|
+
## Consistency
|
|
213
|
+
|
|
214
|
+
- **Add and remove serialize at the set's home.** Each set key has one home
|
|
215
|
+
location where its writes are applied in order, so an `add` followed by a
|
|
216
|
+
`remove` of the same member lands in that order and the set ends up correct.
|
|
217
|
+
- **Reads can lag slightly across regions.** ToilDB is worldwide. A change made
|
|
218
|
+
at a set's home is copied to other regions in the background (asynchronous
|
|
219
|
+
replication, giving eventual consistency: every region converges after a short
|
|
220
|
+
delay). A `contains` or a `list` served from a far region may briefly miss a
|
|
221
|
+
just-made change.
|
|
222
|
+
- **`add`/`remove` are idempotent**, so retries are safe: re-adding an existing
|
|
223
|
+
member or re-removing an absent one does nothing.
|
|
224
|
+
- **A set can grow without bound.** Adding never shrinks it. Read it through a
|
|
225
|
+
bounded `list(key, N)` in a derive, not all at once.
|
|
226
|
+
|
|
227
|
+
## Gotchas
|
|
228
|
+
|
|
229
|
+
- **You cannot `list` from a route.** `list` is a scan, legal only in a `@derive`
|
|
230
|
+
or `@job`. To show a roster on a page, publish it to a [View](./views.md) from a
|
|
231
|
+
derive and `get` that view in the route.
|
|
232
|
+
- **`list` returns a bounded page.** A large set may have more members than your
|
|
233
|
+
`limit`. Do not assume the returned array is the entire set.
|
|
234
|
+
- **Sets are unordered.** Do not rely on the order `list` returns members in.
|
|
235
|
+
- **Use `Membership` for growth or concurrency.** A small, stable set is fine as
|
|
236
|
+
an array in a Document. Reach for `Membership` when the set grows, is edited
|
|
237
|
+
concurrently, or you frequently ask "is X a member?".
|
|
238
|
+
|
|
239
|
+
## Related
|
|
240
|
+
|
|
241
|
+
- [Documents](./documents.md): when a small array field is enough (and when it is
|
|
242
|
+
not).
|
|
243
|
+
- [Views](./views.md): where a roster/list belongs so routes can read it cheaply.
|
|
244
|
+
- [`@derive`](../background/derive.md): runs `list` off the request path.
|
|
245
|
+
- [Data types (`@data`)](../concepts/types.md): how set keys and members are stored.
|
|
246
|
+
- [Decorators](../concepts/decorators.md): which handler kinds may add/remove vs scan.
|
|
@@ -0,0 +1,155 @@
|
|
|
1
|
+
# Declaring a database
|
|
2
|
+
|
|
3
|
+
You set up ToilDB by **declaring** it in code: a `@database` class listing your collections, plus the `@data` key and value types those collections use. There is no schema file, no migration command, and no connection string.
|
|
4
|
+
|
|
5
|
+
## What and why
|
|
6
|
+
|
|
7
|
+
A **database declaration** tells toiljs three things at compile time: which collections exist, which family each one is, and what key and value types it stores. The compiler bakes that list into your server's WebAssembly, and both the dev server and the production edge read it back to set up your collections automatically. You declare; the platform provisions.
|
|
8
|
+
|
|
9
|
+
Reach for this on day one: before you can read or write any data, you need a `@database` with at least one `@collection`.
|
|
10
|
+
|
|
11
|
+
## How: the three pieces
|
|
12
|
+
|
|
13
|
+
A working database is always these three pieces together.
|
|
14
|
+
|
|
15
|
+
1. **`@data` key and value types.** Plain classes, tagged `@data`, that describe what you store.
|
|
16
|
+
2. **A `@database` class** whose static fields are `@collection`s, each typed by a family.
|
|
17
|
+
3. **A handler** (a `@rest` route or an RPC method) that reads and writes those collections.
|
|
18
|
+
|
|
19
|
+
Here is a complete, minimal example: a store of users you look up by id.
|
|
20
|
+
|
|
21
|
+
```ts
|
|
22
|
+
// A @data key: how you address one user.
|
|
23
|
+
@data
|
|
24
|
+
class UserId {
|
|
25
|
+
id: string = '';
|
|
26
|
+
constructor(id: string = '') { this.id = id; }
|
|
27
|
+
}
|
|
28
|
+
|
|
29
|
+
// A @data value: what you store for each user.
|
|
30
|
+
@data
|
|
31
|
+
class User {
|
|
32
|
+
id: string = '';
|
|
33
|
+
name: string = '';
|
|
34
|
+
score: u64 = 0;
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
// The database declaration. Each @collection is one collection,
|
|
38
|
+
// typed by its family (here, Documents) with <Key, Value>.
|
|
39
|
+
@database
|
|
40
|
+
class AppDb {
|
|
41
|
+
@collection static users: Documents<UserId, User>;
|
|
42
|
+
}
|
|
43
|
+
|
|
44
|
+
// A route that reads and writes the collection.
|
|
45
|
+
@rest('users')
|
|
46
|
+
class Users {
|
|
47
|
+
@get('/:id')
|
|
48
|
+
public getUser(ctx: RouteContext): User {
|
|
49
|
+
const user = AppDb.users.get(new UserId(ctx.param('id')));
|
|
50
|
+
return user == null ? new User() : user;
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
@post('/')
|
|
54
|
+
public createUser(input: User): User {
|
|
55
|
+
AppDb.users.create(new UserId(input.id), input);
|
|
56
|
+
return input;
|
|
57
|
+
}
|
|
58
|
+
}
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
That is the whole setup. Run `toiljs dev` and `AppDb.users` works immediately.
|
|
62
|
+
|
|
63
|
+
### The `@data` types
|
|
64
|
+
|
|
65
|
+
Both the key and the value are `@data` classes. `@data` is what makes a class storable: the compiler synthesizes a binary codec (pack to bytes, unpack from bytes) so ToilDB can persist it. The rules that matter here:
|
|
66
|
+
|
|
67
|
+
- **Give every field a default** (`= ''`, `= 0`, and so on). The decoder builds an empty instance and fills it, so it needs defaults.
|
|
68
|
+
- **A value type must be default-constructible** (creatable with `new User()` and no arguments). Keys often add a convenience constructor, as `UserId` does above, so you can write `new UserId('abc')`.
|
|
69
|
+
- Fields may be numbers (`u8`..`u256`, `i8`..`i256`, `f32`, `f64`), `bool`, `string`, another `@data` class, or an array of any of these.
|
|
70
|
+
|
|
71
|
+
The full reference, including how the same type becomes a typed client type, is on the [data types page](../backend/data.md).
|
|
72
|
+
|
|
73
|
+
### The `@database` class and `@collection` fields
|
|
74
|
+
|
|
75
|
+
```ts
|
|
76
|
+
@database
|
|
77
|
+
class AppDb {
|
|
78
|
+
@collection static users: Documents<UserId, User>;
|
|
79
|
+
@collection static likes: Counter<UserId>;
|
|
80
|
+
}
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
- `@database` marks the class as a database. You can have more than one `@database` class; each is a separate namespace of collections.
|
|
84
|
+
- Each `@collection` is one collection. Declare it as a **`static`** field with **no initializer**: you write the type, and the compiler wires up the actual handle for you.
|
|
85
|
+
- The field's **type is its family**: `Documents<K, V>`, `Counter<K>`, `Events<K, V>`, `Unique<K, V>`, `Membership<K, M>`, `Capacity<K>`, or `View<K, V>`. The compiler reads the family straight from this type, so getting it right here is how you pick a family (see [Choosing a family](./README.md#choosing-a-family-the-decision-guide)).
|
|
86
|
+
|
|
87
|
+
You reach a collection through the class, statically: `AppDb.users.get(...)`, `AppDb.likes.add(...)`. There is nothing to instantiate.
|
|
88
|
+
|
|
89
|
+
### Reaching collections from a handler
|
|
90
|
+
|
|
91
|
+
Any backend function can read and write collections by referencing them on the database class. What that function is allowed to do depends on its **kind**, covered next.
|
|
92
|
+
|
|
93
|
+
## How access is gated: `@query`, `@action`, and friends
|
|
94
|
+
|
|
95
|
+
ToilDB will not let every function do everything. Each backend function runs as one **function kind**, and each kind is allowed a different slice of database operations. This is a safety rail: a read-only endpoint physically cannot write, and an expensive scan cannot run on the hot request path.
|
|
96
|
+
|
|
97
|
+
You rarely write these decorators by hand, because routes get a sensible kind automatically:
|
|
98
|
+
|
|
99
|
+
- A **`@get`** route (a safe, read-only HTTP method) runs as a **Query**.
|
|
100
|
+
- A **`@post`** route (a mutating method) runs as an **Action**.
|
|
101
|
+
- A plain RPC **`@remote`** method defaults to a **Query** (read-only) because it has no HTTP method to infer from. Tag it `@action` if it writes.
|
|
102
|
+
|
|
103
|
+
You can override the default with `@query` or `@action` on the method when the automatic choice is wrong (for example, a `@get` that genuinely needs to write, though that is unusual).
|
|
104
|
+
|
|
105
|
+
What each kind may do:
|
|
106
|
+
|
|
107
|
+
| Kind | Set by | May do | May **not** do |
|
|
108
|
+
| --- | --- | --- | --- |
|
|
109
|
+
| **Query** | `@get`, plain `@remote`, or `@query` | Point reads: `get`, `getMany`, `exists`, `lookup`, `contains`, counter `get`, view `get`, capacity `available`. | Any write. Any scan. |
|
|
110
|
+
| **Action** | `@post` or `@action` | Everything a Query can, plus bounded writes: `create`, `patch`, `delete`, `getDelete`, `enqueue`, `append`, `appendOnce`, counter `add`, membership `add`/`remove`, unique `claim`/`release`, capacity `reserve`/`confirm`/`cancel`. | Scans. Publishing a View. |
|
|
111
|
+
| **Derive / Job** | `@derive` / `@job` (background work) | Reads including **scans** (`latest`, membership `list`), plus `publish` a View. | (Run off the request path; see below.) |
|
|
112
|
+
|
|
113
|
+
Two rules trip people up, so they are worth stating plainly:
|
|
114
|
+
|
|
115
|
+
- **Scans are barred from request handlers.** Reading "the newest N events" (`events.latest`) or "the members of this set" (`membership.list`) can fan out across many rows, so a `@get` or `@post` cannot call them. Do the scan in a `@derive` (a small function that recomputes a snapshot off the request path) and have the request read the snapshot. See [Views](./views.md) and [@derive](../background/derive.md).
|
|
116
|
+
- **Only a `@derive` or `@job` may `publish` a View.** Requests read views; background work writes them.
|
|
117
|
+
|
|
118
|
+
Both gates are enforced twice: the compiler rejects an illegal call at build time, and the edge rejects it again at runtime, so a hand-edited module cannot sneak past. For the decorator catalog, see [Decorators](../concepts/decorators.md).
|
|
119
|
+
|
|
120
|
+
## No manual provisioning: how it actually gets set up
|
|
121
|
+
|
|
122
|
+
You never create a table or run a migration. Here is the machinery, so the "it just works" is not a mystery.
|
|
123
|
+
|
|
124
|
+
When toilscript compiles your backend, it scans every `@database` class and writes a small catalog into the `.wasm` file: for each collection, its name, its family, its key and value type names, and the value's schema version. This catalog rides *inside* the compiled module.
|
|
125
|
+
|
|
126
|
+
Then, wherever your backend runs, the host reads that catalog once at startup and builds your collections to match:
|
|
127
|
+
|
|
128
|
+
- Under **`toiljs dev`**, the host is an in-process, in-memory emulator. It reads the catalog and stands up all seven families in memory. This is a development store: single process, single tenant, and cleared when you restart. It exists so you can build and test against real ToilDB behavior with no services to run.
|
|
129
|
+
- On the **Dacely edge**, the host is the real ToilDB, backed by a globally distributed ScyllaDB cluster. It reads the *same* catalog and serves the *same* operations, now durable and worldwide.
|
|
130
|
+
|
|
131
|
+
Because both sides read the same catalog, **the same code runs unchanged in dev and in production.** There is no connection string to swap and no provisioning step to run.
|
|
132
|
+
|
|
133
|
+
```mermaid
|
|
134
|
+
flowchart TD
|
|
135
|
+
SRC["@database AppDb<br/>@collection static users: Documents...<br/>@data UserId / User"]
|
|
136
|
+
SRC -->|toilscript compile| WASM["server.wasm<br/>(with an embedded<br/>toildb catalog section)"]
|
|
137
|
+
WASM -->|toiljs dev reads the catalog| DEV["In-memory dev store<br/>(one process, cleared on restart)"]
|
|
138
|
+
WASM -->|edge reads the same catalog| EDGE[("ToilDB on the Dacely edge<br/>(ScyllaDB, worldwide, durable)")]
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
## Gotchas
|
|
142
|
+
|
|
143
|
+
- **Declare collections as `static` fields with no initializer.** Do not try to `new` a collection or assign a handle yourself; the compiler owns that.
|
|
144
|
+
- **Every `@data` field needs a default, and value types must be default-constructible.** A missing default fails to compile.
|
|
145
|
+
- **Dev data is not durable.** The `toiljs dev` store lives in memory and resets on restart. Do not rely on it to persist across dev runs; that is what the edge is for.
|
|
146
|
+
- **Changing a `@data` type is a format change.** Reordering fields or changing a field's type changes the stored layout. Add new fields at the end, and use a migration when you evolve a stored type. See [data types](../backend/data.md).
|
|
147
|
+
- **Pick the family at the type.** The family is read from the collection's declared type, so `Counter<K>` versus `Documents<K, V>` is a real, load-bearing choice, not a hint. Revisit [Choosing a family](./README.md#choosing-a-family-the-decision-guide) if unsure.
|
|
148
|
+
|
|
149
|
+
## Related
|
|
150
|
+
|
|
151
|
+
- [ToilDB overview](./README.md): the seven families and how to choose.
|
|
152
|
+
- [Documents](./documents.md): the general-purpose record family (a good first collection).
|
|
153
|
+
- [Data types (`@data`)](../backend/data.md): keys, values, and the codec.
|
|
154
|
+
- [Decorators](../concepts/decorators.md): `@database`, `@collection`, `@query`, `@action`, `@derive`.
|
|
155
|
+
- [@derive](../background/derive.md): recompute snapshots and run scans off the request path.
|
|
@@ -0,0 +1,216 @@
|
|
|
1
|
+
# Unique
|
|
2
|
+
|
|
3
|
+
The **Unique** family enforces that a value is claimed by only one owner across the entire world. It is how you make usernames, email addresses, and URL slugs one-of-a-kind, safely, even when two people try to grab the same one at the same instant.
|
|
4
|
+
|
|
5
|
+
## What and why
|
|
6
|
+
|
|
7
|
+
A **Unique collection** maps a `@data` **claim key** (the thing that must be unique, like a username) to a `@data` **owner value** (who or what claimed it, like a user id). At any moment a claim key is either **unclaimed** or **owned by exactly one owner**. The family gives you three operations: look up who owns a key, claim a key, and release it.
|
|
8
|
+
|
|
9
|
+
Reach for Unique whenever a value must be globally singular:
|
|
10
|
+
|
|
11
|
+
- usernames or handles
|
|
12
|
+
- email addresses
|
|
13
|
+
- URL slugs or workspace names
|
|
14
|
+
- any "reserve this name for me" scenario
|
|
15
|
+
|
|
16
|
+
Why not just check a Documents record first? Because a check-then-write has a race: two requests can both check "is `alice` free?", both see yes, and both create it. Unique closes that gap. The claim is decided at the key's single **home** (see [consistency](./README.md#eventual-consistency-in-plain-words)), where claims are processed one at a time, so exactly one of the two racers wins.
|
|
17
|
+
|
|
18
|
+
Declare one by typing a `@collection` as `Unique<ClaimKey, OwnerValue>`:
|
|
19
|
+
|
|
20
|
+
```ts
|
|
21
|
+
@data
|
|
22
|
+
class Username {
|
|
23
|
+
name: string = '';
|
|
24
|
+
constructor(name: string = '') { this.name = name; }
|
|
25
|
+
}
|
|
26
|
+
|
|
27
|
+
@data
|
|
28
|
+
class OwnerId {
|
|
29
|
+
userId: string = '';
|
|
30
|
+
constructor(userId: string = '') { this.userId = userId; }
|
|
31
|
+
}
|
|
32
|
+
|
|
33
|
+
@database
|
|
34
|
+
class AppDb {
|
|
35
|
+
@collection static usernames: Unique<Username, OwnerId>;
|
|
36
|
+
}
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
## The operations
|
|
40
|
+
|
|
41
|
+
`K` is the claim-key type, `V` the owner value type.
|
|
42
|
+
|
|
43
|
+
| Operation | Signature | Returns | Use it to |
|
|
44
|
+
| --- | --- | --- | --- |
|
|
45
|
+
| `lookup` | `lookup(key: K): V \| null` | the current owner, or `null` if unclaimed | find out who owns a name |
|
|
46
|
+
| `claim` | `claim(key: K, value: V): ClaimResult<V>` | a `ClaimResult` (see below) | try to take a name for an owner |
|
|
47
|
+
| `release` | `release(key: K, value: V): void` | nothing; **traps** if you are not the owner | give a name back |
|
|
48
|
+
|
|
49
|
+
`lookup` is a read, so it works in any function. `claim` and `release` are writes, so they need an **Action** (a `@post` route or an `@action`); see [Setup](./setup.md#how-access-is-gated-query-action-and-friends).
|
|
50
|
+
|
|
51
|
+
### `ClaimResult`
|
|
52
|
+
|
|
53
|
+
`claim` returns a small object that tells you what happened:
|
|
54
|
+
|
|
55
|
+
```ts
|
|
56
|
+
class ClaimResult<V> {
|
|
57
|
+
claimed: bool; // true if YOU own the key now
|
|
58
|
+
owner: V | null; // when claimed is false, who owns it instead
|
|
59
|
+
}
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
- **`claimed == true`**: you own the key. This covers both a fresh claim and an **idempotent re-claim**: if you claim a key you already own (same owner value), you still get `true`, so retrying a claim is safe. `owner` is `null` in this case.
|
|
63
|
+
- **`claimed == false`**: someone else got there first. `owner` is their value, so you can tell the user "that name is taken."
|
|
64
|
+
|
|
65
|
+
### `lookup`
|
|
66
|
+
|
|
67
|
+
`lookup` just reads the current owner without changing anything:
|
|
68
|
+
|
|
69
|
+
```ts
|
|
70
|
+
const owner = AppDb.usernames.lookup(new Username('alice'));
|
|
71
|
+
if (owner == null) {
|
|
72
|
+
// 'alice' is free
|
|
73
|
+
} else {
|
|
74
|
+
// owner.userId currently holds 'alice'
|
|
75
|
+
}
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
Note that `lookup` is subject to [eventual consistency](./README.md#eventual-consistency-in-plain-words): a claim made moments ago in another region may not show up in a far-away `lookup` yet. Do not use `lookup` as your uniqueness guard. `lookup` is for display ("this name is taken by ..."); the real guarantee comes from `claim`, which is decided at the home and cannot race.
|
|
79
|
+
|
|
80
|
+
### `claim`
|
|
81
|
+
|
|
82
|
+
`claim` is the operation that actually enforces uniqueness. You pass the key and the owner value:
|
|
83
|
+
|
|
84
|
+
```ts
|
|
85
|
+
const result = AppDb.usernames.claim(new Username('alice'), new OwnerId('u_123'));
|
|
86
|
+
if (result.claimed) {
|
|
87
|
+
// 'alice' is now yours
|
|
88
|
+
} else {
|
|
89
|
+
// taken; result.owner is the current owner
|
|
90
|
+
}
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
Because claims are serialized at the key's home, this is race-safe. If two requests anywhere in the world call `claim('alice', ...)` at the same moment, the home applies them in order: the first gets `claimed: true`, the second gets `claimed: false` with the first as `owner`. There is no window where both win.
|
|
94
|
+
|
|
95
|
+
### `release`
|
|
96
|
+
|
|
97
|
+
`release` gives a claim back so the key becomes available again. Only the **current owner** may release: you pass both the key and the owner value, and if that value is not the current owner, `release` **traps** (aborts the request). This prevents one user from releasing another user's name.
|
|
98
|
+
|
|
99
|
+
```ts
|
|
100
|
+
AppDb.usernames.release(new Username('alice'), new OwnerId('u_123'));
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Release when a name is being changed or an account is deleted, so the name returns to the pool.
|
|
104
|
+
|
|
105
|
+
## The claim / release lifecycle
|
|
106
|
+
|
|
107
|
+
A claim is a small state machine: unclaimed, owned, and back to unclaimed.
|
|
108
|
+
|
|
109
|
+
```mermaid
|
|
110
|
+
stateDiagram-v2
|
|
111
|
+
[*] --> Unclaimed
|
|
112
|
+
Unclaimed --> Owned: claim(key, me) -> claimed: true
|
|
113
|
+
Owned --> Owned: claim(key, me) again -> claimed: true (idempotent)
|
|
114
|
+
Owned --> Owned: claim(key, someoneElse) -> claimed: false, owner: me
|
|
115
|
+
Owned --> Unclaimed: release(key, me)
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
The key insight: **claiming is the guard, releasing is cleanup.** You claim to reserve, you release to free. A claim that is never released stays owned forever (there is no automatic expiry; if you want time-limited holds, that is what the [Capacity](./capacity.md) family's TTL reservations are for).
|
|
119
|
+
|
|
120
|
+
## Worked example: reserving a username on signup
|
|
121
|
+
|
|
122
|
+
The safe signup pattern is: claim the username first, then create the account, and if creating the account fails, release the claim so the name is not stranded.
|
|
123
|
+
|
|
124
|
+
```ts
|
|
125
|
+
import { Response } from 'toiljs/server/runtime';
|
|
126
|
+
|
|
127
|
+
@data
|
|
128
|
+
class Username {
|
|
129
|
+
name: string = '';
|
|
130
|
+
constructor(name: string = '') { this.name = name; }
|
|
131
|
+
}
|
|
132
|
+
|
|
133
|
+
@data
|
|
134
|
+
class OwnerId {
|
|
135
|
+
userId: string = '';
|
|
136
|
+
constructor(userId: string = '') { this.userId = userId; }
|
|
137
|
+
}
|
|
138
|
+
|
|
139
|
+
@data
|
|
140
|
+
class UserId {
|
|
141
|
+
id: string = '';
|
|
142
|
+
constructor(id: string = '') { this.id = id; }
|
|
143
|
+
}
|
|
144
|
+
|
|
145
|
+
@data
|
|
146
|
+
class User {
|
|
147
|
+
id: string = '';
|
|
148
|
+
username: string = '';
|
|
149
|
+
}
|
|
150
|
+
|
|
151
|
+
@data
|
|
152
|
+
class SignupInput {
|
|
153
|
+
username: string = '';
|
|
154
|
+
userId: string = '';
|
|
155
|
+
}
|
|
156
|
+
|
|
157
|
+
@database
|
|
158
|
+
class AppDb {
|
|
159
|
+
@collection static usernames: Unique<Username, OwnerId>;
|
|
160
|
+
@collection static users: Documents<UserId, User>;
|
|
161
|
+
}
|
|
162
|
+
|
|
163
|
+
@rest('signup')
|
|
164
|
+
class Signup {
|
|
165
|
+
// POST /signup (Action: may claim and write)
|
|
166
|
+
@post('/')
|
|
167
|
+
public signup(input: SignupInput): Response {
|
|
168
|
+
const owner = new OwnerId(input.userId);
|
|
169
|
+
|
|
170
|
+
// 1. Claim the username. This is the uniqueness guard.
|
|
171
|
+
const claim = AppDb.usernames.claim(new Username(input.username), owner);
|
|
172
|
+
if (!claim.claimed) {
|
|
173
|
+
return Response.text('username taken', 409);
|
|
174
|
+
}
|
|
175
|
+
|
|
176
|
+
// 2. Create the account. If this fails, undo the claim so the name is free.
|
|
177
|
+
const user = new User();
|
|
178
|
+
user.id = input.userId;
|
|
179
|
+
user.username = input.username;
|
|
180
|
+
if (!AppDb.users.create(new UserId(input.userId), user)) {
|
|
181
|
+
AppDb.usernames.release(new Username(input.username), owner);
|
|
182
|
+
return Response.text('could not create account', 409);
|
|
183
|
+
}
|
|
184
|
+
|
|
185
|
+
return Response.json(user.toJSON().toString());
|
|
186
|
+
}
|
|
187
|
+
}
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
Two things make this correct:
|
|
191
|
+
|
|
192
|
+
- **The claim happens before the account write**, so uniqueness is decided up front.
|
|
193
|
+
- **The claim is released if the follow-up write fails**, so a failed signup does not permanently burn a username.
|
|
194
|
+
|
|
195
|
+
Because `claim` is idempotent for the same owner, a client that retries the whole request after a network hiccup does not get a false "taken" error: the second `claim('alice', u_123)` returns `claimed: true` again.
|
|
196
|
+
|
|
197
|
+
## Consistency notes
|
|
198
|
+
|
|
199
|
+
- **`claim` and `release` are strongly consistent at the key's home.** Two callers can never both own the same key; the home serializes claims. This is the whole point of the family.
|
|
200
|
+
- **`lookup` is eventually consistent.** It reads a possibly-nearby copy, so a very recent claim from elsewhere may not appear yet. Never gate uniqueness on `lookup`; gate it on the result of `claim`.
|
|
201
|
+
- **Claims do not expire.** A claim stays until someone releases it. Use [Capacity](./capacity.md) if you need holds that auto-release after a timeout.
|
|
202
|
+
|
|
203
|
+
## Gotchas
|
|
204
|
+
|
|
205
|
+
- **Do not use `lookup` as the uniqueness check.** Its answer can be stale. Call `claim` and trust its `claimed` flag.
|
|
206
|
+
- **Release on failure and on account deletion.** A claim you forget to release stays owned forever, quietly blocking that name.
|
|
207
|
+
- **`release` traps for a non-owner.** Pass the correct owner value, or you abort the request. If you are not sure you own it, `lookup` first (for display) but expect `release` to enforce ownership regardless.
|
|
208
|
+
- **The owner value is data you choose.** Store enough in it (a user id, say) to know who holds the claim, so you can display and release it later.
|
|
209
|
+
|
|
210
|
+
## Related
|
|
211
|
+
|
|
212
|
+
- [ToilDB overview](./README.md): the seven families and how to choose.
|
|
213
|
+
- [Setup](./setup.md): declaring the collection and which function kinds may claim.
|
|
214
|
+
- [Documents](./documents.md): the account record you create alongside a claim.
|
|
215
|
+
- [Capacity](./capacity.md): time-limited holds that auto-release (a different kind of "reserve").
|
|
216
|
+
- [Data types (`@data`)](../backend/data.md): the claim key and owner value.
|