toiljs 0.0.85 → 0.0.87
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +5 -0
- package/README.md +2 -2
- package/build/cli/.tsbuildinfo +1 -1
- package/build/cli/index.js +303 -293
- package/build/compiler/.tsbuildinfo +1 -1
- package/build/compiler/config.d.ts +2 -0
- package/build/compiler/config.js +1 -0
- package/build/compiler/docs.js +8 -24
- package/build/compiler/generate.js +4 -2
- package/build/compiler/index.d.ts +1 -1
- package/build/compiler/index.js +69 -6
- package/build/compiler/toil-docs.generated.js +64 -22
- package/build/devserver/.tsbuildinfo +1 -1
- package/build/devserver/analytics/index.js +7 -3
- package/build/devserver/db/database.d.ts +4 -0
- package/build/devserver/db/database.js +43 -1
- package/build/devserver/db/index.d.ts +1 -1
- package/build/devserver/db/index.js +1 -1
- package/build/devserver/db/types.d.ts +3 -0
- package/build/devserver/db/types.js +2 -0
- package/build/devserver/runtime/module.js +4 -1
- package/docs/README.md +104 -65
- package/docs/auth/README.md +102 -0
- package/docs/auth/configuration.md +94 -0
- package/docs/auth/extending.md +202 -0
- package/docs/auth/how-it-works.md +138 -0
- package/docs/auth/usage.md +188 -0
- package/docs/backend/README.md +143 -0
- package/docs/backend/data.md +351 -0
- package/docs/backend/rest.md +402 -0
- package/docs/backend/rpc.md +226 -0
- package/docs/background/README.md +114 -0
- package/docs/background/daemons.md +230 -0
- package/docs/background/derive.md +179 -0
- package/docs/cli/README.md +377 -0
- package/docs/concepts/config.md +416 -0
- package/docs/concepts/decorators.md +127 -0
- package/docs/concepts/security.md +108 -0
- package/docs/concepts/tiers.md +166 -0
- package/docs/concepts/types.md +216 -0
- package/docs/database/README.md +143 -0
- package/docs/database/capacity.md +350 -0
- package/docs/database/counters.md +174 -0
- package/docs/database/documents.md +255 -0
- package/docs/database/events.md +307 -0
- package/docs/database/membership.md +246 -0
- package/docs/database/setup.md +155 -0
- package/docs/database/unique.md +216 -0
- package/docs/database/views.md +246 -0
- package/docs/frontend/README.md +101 -0
- package/docs/frontend/data-fetching.md +243 -0
- package/docs/frontend/images.md +148 -0
- package/docs/frontend/metadata.md +344 -0
- package/docs/frontend/rendering.md +236 -0
- package/docs/frontend/routing.md +344 -0
- package/docs/frontend/scripts.md +118 -0
- package/docs/frontend/search.md +191 -0
- package/docs/frontend/styling.md +147 -0
- package/docs/getting-started/README.md +81 -0
- package/docs/getting-started/create-project.md +131 -0
- package/docs/getting-started/deploy.md +101 -0
- package/docs/getting-started/first-app.md +215 -0
- package/docs/getting-started/installation.md +106 -0
- package/docs/getting-started/migrating.md +125 -0
- package/docs/getting-started/project-structure.md +163 -0
- package/docs/introduction/README.md +41 -0
- package/docs/introduction/design-principles.md +55 -0
- package/docs/introduction/distributed.md +74 -0
- package/docs/introduction/how-it-works.md +74 -0
- package/docs/introduction/hyperscale.md +51 -0
- package/docs/introduction/modern-stack.md +36 -0
- package/docs/introduction/vs-other-frameworks.md +48 -0
- package/docs/introduction/why-toil.md +93 -0
- package/docs/llms.txt +90 -0
- package/docs/realtime/README.md +102 -0
- package/docs/realtime/channels.md +211 -0
- package/docs/realtime/streams.md +369 -0
- package/docs/services/README.md +60 -0
- package/docs/services/analytics.md +268 -0
- package/docs/services/caching.md +175 -0
- package/docs/services/cookies.md +235 -0
- package/docs/services/crypto.md +209 -0
- package/docs/services/email.md +289 -0
- package/docs/services/environment.md +141 -0
- package/docs/services/ratelimit.md +174 -0
- package/docs/services/time.md +85 -0
- package/examples/basic/client/routes/analytics.tsx +13 -12
- package/examples/basic/server/models/SiteAnalytics.ts +29 -17
- package/examples/basic/server/routes/Analytics.ts +15 -17
- package/examples/basic/server/routes/UserId.ts +22 -0
- package/package.json +15 -11
- package/scripts/gen-toil-docs.mjs +24 -35
- package/server/auth/AuthController.ts +336 -0
- package/server/auth/AuthUser.ts +23 -0
- package/server/auth/index.ts +16 -0
- package/server/globals/auth.ts +31 -0
- package/server/globals/userid.ts +107 -0
- package/src/compiler/config.ts +13 -0
- package/src/compiler/docs.ts +16 -33
- package/src/compiler/generate.ts +6 -2
- package/src/compiler/index.ts +114 -6
- package/src/compiler/toil-docs.generated.ts +64 -22
- package/src/devserver/analytics/index.ts +10 -4
- package/src/devserver/db/database.ts +67 -1
- package/src/devserver/db/index.ts +1 -0
- package/src/devserver/db/types.ts +13 -0
- package/src/devserver/runtime/module.ts +7 -0
- package/test/analytics-dev.test.ts +2 -1
- package/test/devserver-database.test.ts +113 -0
- package/docs/auth-todo.md +0 -149
- package/docs/auth.md +0 -322
- package/docs/caching.md +0 -115
- package/docs/cli.md +0 -17
- package/docs/client.md +0 -39
- package/docs/cookies.md +0 -457
- package/docs/crypto.md +0 -130
- package/docs/daemon.md +0 -123
- package/docs/data.md +0 -131
- package/docs/derive.md +0 -159
- package/docs/email.md +0 -326
- package/docs/environment.md +0 -97
- package/docs/getting-started.md +0 -128
- package/docs/index.md +0 -30
- package/docs/ratelimit.md +0 -95
- package/docs/routing.md +0 -259
- package/docs/rpc.md +0 -149
- package/docs/server.md +0 -61
- package/docs/ssr.md +0 -632
- package/docs/streams.md +0 -178
- package/docs/styling.md +0 -22
- package/docs/tiers.md +0 -133
- package/docs/time.md +0 -43
|
@@ -0,0 +1,255 @@
|
|
|
1
|
+
# Documents
|
|
2
|
+
|
|
3
|
+
The **Documents** family is ToilDB's general-purpose record store: you keep a value under a key and look it up, update it, or delete it by that key. It is the family you will use most, and the one to reach for whenever the other six do not obviously fit.
|
|
4
|
+
|
|
5
|
+
## What and why
|
|
6
|
+
|
|
7
|
+
A **Documents collection** maps a `@data` key to a `@data` value: one value per key, which you can read, replace, and remove. Think users by user id, posts by post id, orders by order number. If you are storing "a thing with fields that I look up and update by its id," this is the family.
|
|
8
|
+
|
|
9
|
+
Declare one by typing a `@collection` as `Documents<Key, Value>`:
|
|
10
|
+
|
|
11
|
+
```ts
|
|
12
|
+
@data
|
|
13
|
+
class UserId {
|
|
14
|
+
id: string = '';
|
|
15
|
+
constructor(id: string = '') { this.id = id; }
|
|
16
|
+
}
|
|
17
|
+
|
|
18
|
+
@data
|
|
19
|
+
class User {
|
|
20
|
+
id: string = '';
|
|
21
|
+
name: string = '';
|
|
22
|
+
email: string = '';
|
|
23
|
+
score: u64 = 0;
|
|
24
|
+
}
|
|
25
|
+
|
|
26
|
+
@database
|
|
27
|
+
class AppDb {
|
|
28
|
+
@collection static users: Documents<UserId, User>;
|
|
29
|
+
}
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
## The operations
|
|
33
|
+
|
|
34
|
+
Here is every operation, its shape, and what it gives back. `K` is your key type, `V` your value type.
|
|
35
|
+
|
|
36
|
+
| Operation | Signature | Returns | Use it to |
|
|
37
|
+
| --- | --- | --- | --- |
|
|
38
|
+
| `get` | `get(key: K): V \| null` | the value, or `null` if absent | read one record |
|
|
39
|
+
| `require` | `require(key: K): V` | the value; **traps** if absent | read a record you are sure exists |
|
|
40
|
+
| `getMany` | `getMany(keys: K[]): Array<V \| null>` | one entry per key, in order, each value or `null` | read several records in one call |
|
|
41
|
+
| `exists` | `exists(key: K): bool` | `true` if the record is present | check presence without reading the value |
|
|
42
|
+
| `create` | `create(key: K, value: V): bool` | `true` if inserted, `false` if the key was already taken | add a **new** record without overwriting |
|
|
43
|
+
| `patch` | `patch(key: K, value: V): V` | the newly stored value; **traps** if the record is absent | replace an **existing** record's value |
|
|
44
|
+
| `enqueue` | `enqueue(key: K, value: V): bool` | `true` if applied, `false` if a concurrent write won first or the record is absent | a version-checked (compare-and-swap) overwrite of an existing record |
|
|
45
|
+
| `delete` | `delete(key: K): void` | nothing (idempotent) | remove a record |
|
|
46
|
+
| `getDelete` | `getDelete(key: K): V \| null` | the value that was there, or `null`; removes it atomically | consume a record exactly once |
|
|
47
|
+
|
|
48
|
+
Which kind of function may call which operation is covered in [Setup](./setup.md#how-access-is-gated-query-action-and-friends). In short: reads (`get`, `getMany`, `exists`) work anywhere; writes (`create`, `patch`, `enqueue`, `delete`, `getDelete`) need an **Action** (a `@post` route or an `@action`).
|
|
49
|
+
|
|
50
|
+
### Reading: `get`, `require`, `exists`, `getMany`
|
|
51
|
+
|
|
52
|
+
`get` is the everyday read. It returns the value or `null`, so you handle "not found" explicitly:
|
|
53
|
+
|
|
54
|
+
```ts
|
|
55
|
+
const user = AppDb.users.get(new UserId('u_123'));
|
|
56
|
+
if (user == null) {
|
|
57
|
+
return Response.notFound();
|
|
58
|
+
}
|
|
59
|
+
// user is a fully typed User here
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
`require` is `get` for the case where absence is a bug, not a normal outcome: it returns the value directly and traps (aborts the request) if the record is missing. Use it only when you have already guaranteed the record exists.
|
|
63
|
+
|
|
64
|
+
`exists` answers "is there a record here?" without paying to decode the value. It is handy as a cheap precondition:
|
|
65
|
+
|
|
66
|
+
```ts
|
|
67
|
+
if (AppDb.users.exists(new UserId(name))) {
|
|
68
|
+
// username already registered, do not overwrite
|
|
69
|
+
}
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
`getMany` reads several keys in a single operation. You hand it an array of keys; you get back an array the same length and in the same order, each entry either the value or `null` for a key that was absent. Reach for it instead of a loop of `get` calls when you already know the handful of keys you need.
|
|
73
|
+
|
|
74
|
+
```ts
|
|
75
|
+
const ids = [new UserId('a'), new UserId('b'), new UserId('c')];
|
|
76
|
+
const found: Array<User | null> = AppDb.users.getMany(ids);
|
|
77
|
+
// found[0] lines up with ids[0], and so on; each is a User or null
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
`getMany` is a **bounded batch of point reads**, not a scan: the number of keys you may pass is capped by the request budget, and it never walks the whole collection. There is no "get all records" operation on a request path, by design (an unbounded scan could fan out across a huge collection). If you need "the latest N of something," model it as [Events](./events.md) or precompute a [View](./views.md).
|
|
81
|
+
|
|
82
|
+
### Writing: `create` vs `patch` vs `enqueue`
|
|
83
|
+
|
|
84
|
+
These three all put a value under a key, but they differ in one important way each. Choosing correctly is the heart of using this family.
|
|
85
|
+
|
|
86
|
+
```mermaid
|
|
87
|
+
flowchart TD
|
|
88
|
+
START["I want to write a record"] --> Q1{"Is this a brand-new<br/>record that must not<br/>clobber an existing one?"}
|
|
89
|
+
Q1 -->|Yes| CREATE["create<br/>(insert only; false if the key exists)"]
|
|
90
|
+
Q1 -->|"No, it already exists"| Q2{"Must the write fail if another<br/>request changed the record<br/>since I read it?"}
|
|
91
|
+
Q2 -->|"Yes, guard against a lost update"| ENQUEUE["enqueue<br/>(version-checked CAS; false if<br/>someone wrote first, then retry)"]
|
|
92
|
+
Q2 -->|"No, last write wins"| PATCH["patch<br/>(unconditional overwrite;<br/>returns the new value)"]
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
**`create` inserts a new record.** It only writes if the key is free. If the key already has a record, `create` does nothing and returns `false`. This is your tool for "sign up a new user" or "claim this order id," where accidentally overwriting an existing record would be a bug. Because every key is serialized at its home (see [eventual consistency](./README.md#eventual-consistency-in-plain-words)), `create` is race-safe: if two requests create the same key at the same instant, exactly one gets `true` and the other gets `false`.
|
|
96
|
+
|
|
97
|
+
```ts
|
|
98
|
+
const ok = AppDb.users.create(new UserId(input.id), input);
|
|
99
|
+
if (!ok) {
|
|
100
|
+
return Response.text('that id is taken', 409);
|
|
101
|
+
}
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
**`patch` overwrites an existing record** and returns the value now stored. The record **must already exist**: `patch` on a missing key traps (aborts the request), so create it first. Despite the name, `patch` replaces the whole value; there is no field-level partial update, so read the current value, change the fields you want, and patch the whole thing back:
|
|
105
|
+
|
|
106
|
+
```ts
|
|
107
|
+
const current = AppDb.users.get(new UserId('u_123'));
|
|
108
|
+
if (current == null) return Response.notFound();
|
|
109
|
+
current.score = current.score + 10;
|
|
110
|
+
const saved: User = AppDb.users.patch(new UserId('u_123'), current);
|
|
111
|
+
// saved is what is now stored
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
**`enqueue` is a version-checked overwrite (a compare-and-swap).** Like `patch`, it replaces the whole value of an **existing** record, but it does so *only if the record has not changed since you read it*. A **compare-and-swap** (CAS) is exactly that: "write my new value, but only if the current value is still the one I saw." It returns a `bool`: `true` means your write was applied; `false` means either a concurrent write changed the record first (someone else beat you to it) or the record is absent. A `false` is **not an error**; it is the signal to **re-read and try again**. This approach is called **optimistic concurrency**: rather than locking the record, you assume nobody else will touch it, and you simply re-run the update on the rare occasion someone did.
|
|
115
|
+
|
|
116
|
+
Reach for `enqueue` when several requests may update the *same* record at once and you must not silently lose any of their changes. A plain `patch` cannot promise that: two overlapping patches clobber each other (the last writer wins and the earlier update just vanishes). The intended pattern for `enqueue` is always a read-modify-CAS **retry loop**:
|
|
117
|
+
|
|
118
|
+
```ts
|
|
119
|
+
const key = new UserId('u_123');
|
|
120
|
+
for (let attempt = 0; attempt < 5; attempt++) {
|
|
121
|
+
const current = AppDb.users.get(key); // 1. read the current value
|
|
122
|
+
if (current == null) return Response.notFound();
|
|
123
|
+
current.score = current.score + 10; // 2. modify your copy
|
|
124
|
+
if (AppDb.users.enqueue(key, current)) { // 3. try to commit it
|
|
125
|
+
return Response.text('ok'); // true: applied, we are done
|
|
126
|
+
}
|
|
127
|
+
// false: someone wrote between our get and our enqueue.
|
|
128
|
+
// Loop: re-read the now-newer value and reapply the change on top of it.
|
|
129
|
+
}
|
|
130
|
+
return Response.text('too much contention, try again later', 409);
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
Because every retry re-reads the latest value, the two updates **compose** (both `+10`s land) instead of one silently overwriting the other. If you do not need this guard (only one writer touches the key, or last-write-wins is genuinely fine), a plain `patch` is simpler and also hands you the stored value back directly.
|
|
134
|
+
|
|
135
|
+
> There is no single "create or overwrite" (upsert) call. To get that behavior, try `create` first and fall back to `patch` if the key was taken:
|
|
136
|
+
>
|
|
137
|
+
> ```ts
|
|
138
|
+
> const key = new UserId(input.id);
|
|
139
|
+
> if (!AppDb.users.create(key, input)) {
|
|
140
|
+
> AppDb.users.patch(key, input);
|
|
141
|
+
> }
|
|
142
|
+
> ```
|
|
143
|
+
|
|
144
|
+
### Removing: `delete` and `getDelete`
|
|
145
|
+
|
|
146
|
+
`delete` removes a record. It is **idempotent**: deleting a key that is already gone is not an error, it just does nothing. So you can call it without first checking that the record exists.
|
|
147
|
+
|
|
148
|
+
```ts
|
|
149
|
+
AppDb.users.delete(new UserId('u_123'));
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
`getDelete` is the atomic **fetch-and-remove**: in one indivisible step it reads the current value and deletes it, returning what it removed (or `null` if there was nothing). "Atomic" here means no other request can slip in between the read and the delete, so exactly one caller can ever receive a given value. That makes it the right tool for **consume-once** data: one-time login challenges, single-use invite codes, password-reset tokens. The PQ-auth demo uses it to consume a login challenge exactly once, so a challenge cannot be replayed:
|
|
153
|
+
|
|
154
|
+
```ts
|
|
155
|
+
const challenge = AppDb.challenges.getDelete(new ChallengeId(cid));
|
|
156
|
+
if (challenge == null) return fail(); // unknown, already used, or expired
|
|
157
|
+
// ...verify against challenge...
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
If two requests race to `getDelete` the same key, only one gets the value; the other gets `null`. That is the guarantee a plain `get` then `delete` cannot give you, because two racers could both `get` the value before either `delete`s it.
|
|
161
|
+
|
|
162
|
+
## A full worked example: a small CRUD entity
|
|
163
|
+
|
|
164
|
+
Putting it together, here is a complete `notes` resource: create, read, update, and delete, backed by a Documents collection.
|
|
165
|
+
|
|
166
|
+
```ts
|
|
167
|
+
import { Response, RouteContext } from 'toiljs/server/runtime';
|
|
168
|
+
|
|
169
|
+
// ---- key + value ----
|
|
170
|
+
@data
|
|
171
|
+
class NoteId {
|
|
172
|
+
id: string = '';
|
|
173
|
+
constructor(id: string = '') { this.id = id; }
|
|
174
|
+
}
|
|
175
|
+
|
|
176
|
+
@data
|
|
177
|
+
class Note {
|
|
178
|
+
id: string = '';
|
|
179
|
+
title: string = '';
|
|
180
|
+
body: string = '';
|
|
181
|
+
updatedAt: u64 = 0;
|
|
182
|
+
}
|
|
183
|
+
|
|
184
|
+
// ---- database ----
|
|
185
|
+
@database
|
|
186
|
+
class NotesDb {
|
|
187
|
+
@collection static notes: Documents<NoteId, Note>;
|
|
188
|
+
}
|
|
189
|
+
|
|
190
|
+
// ---- routes ----
|
|
191
|
+
@rest('notes')
|
|
192
|
+
class Notes {
|
|
193
|
+
// GET /notes/:id -> read one (Query: read-only)
|
|
194
|
+
@get('/:id')
|
|
195
|
+
public read(ctx: RouteContext): Response {
|
|
196
|
+
const note = NotesDb.notes.get(new NoteId(ctx.param('id')));
|
|
197
|
+
if (note == null) return Response.notFound();
|
|
198
|
+
return Response.json(note.toJSON().toString());
|
|
199
|
+
}
|
|
200
|
+
|
|
201
|
+
// POST /notes -> create a new one, refusing a duplicate id (Action: may write)
|
|
202
|
+
@post('/')
|
|
203
|
+
public create(input: Note): Response {
|
|
204
|
+
input.updatedAt = <u64>(Date.now() / 1000);
|
|
205
|
+
if (!NotesDb.notes.create(new NoteId(input.id), input)) {
|
|
206
|
+
return Response.text('id already exists', 409);
|
|
207
|
+
}
|
|
208
|
+
return Response.json(input.toJSON().toString());
|
|
209
|
+
}
|
|
210
|
+
|
|
211
|
+
// POST /notes/:id -> overwrite an existing note (Action)
|
|
212
|
+
@post('/:id')
|
|
213
|
+
public update(input: Note, ctx: RouteContext): Response {
|
|
214
|
+
const key = new NoteId(ctx.param('id'));
|
|
215
|
+
if (!NotesDb.notes.exists(key)) return Response.notFound();
|
|
216
|
+
input.id = ctx.param('id');
|
|
217
|
+
input.updatedAt = <u64>(Date.now() / 1000);
|
|
218
|
+
const saved = NotesDb.notes.patch(key, input);
|
|
219
|
+
return Response.json(saved.toJSON().toString());
|
|
220
|
+
}
|
|
221
|
+
|
|
222
|
+
// POST /notes/:id/delete -> remove one (Action)
|
|
223
|
+
@post('/:id/delete')
|
|
224
|
+
public remove(ctx: RouteContext): Response {
|
|
225
|
+
NotesDb.notes.delete(new NoteId(ctx.param('id')));
|
|
226
|
+
return Response.text('deleted');
|
|
227
|
+
}
|
|
228
|
+
}
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
That is a complete persistent CRUD entity. Run it under `toiljs dev` and the notes survive across requests; deploy it and the same code stores them worldwide on the edge.
|
|
232
|
+
|
|
233
|
+
## Consistency notes
|
|
234
|
+
|
|
235
|
+
Documents follows ToilDB's general model (see [the overview](./README.md#eventual-consistency-in-plain-words)):
|
|
236
|
+
|
|
237
|
+
- **Writes to one key are serialized at that key's home**, so `create` is race-safe and `patch`/`enqueue`/`getDelete` never corrupt a record under concurrency.
|
|
238
|
+
- **Reads are eventually consistent across regions.** Right after a write, a read from a far-away region may briefly still see the old value (or, for a just-created record, not see it yet). The copies converge within moments.
|
|
239
|
+
- Because `patch` replaces the whole value, two updates to *different* fields of the same record can clobber each other if they overlap (read-modify-write races). `enqueue`'s version check is exactly the guard against that: use the read-modify-CAS retry loop shown above so a lost update turns into a retry instead of silent data loss. If you find yourself contending on one hot record a lot, a counter or a set is often a better fit than a Documents value. See [Counters](./counters.md) and [Membership](./membership.md).
|
|
240
|
+
|
|
241
|
+
## Gotchas
|
|
242
|
+
|
|
243
|
+
- **`patch` requires an existing record.** Calling it on a missing key traps the request. Use `create` for new records, or the `create`-then-`patch` upsert pattern above.
|
|
244
|
+
- **`patch` replaces the whole value.** There is no field-level merge; read, modify, and write back the full value.
|
|
245
|
+
- **`enqueue` returning `false` is not a failure.** It means a concurrent write beat you to the record (or the record is absent), so re-read and retry in a loop; never ignore the return value or treat `false` as a hard error. `enqueue` also does not hand back the stored value; use `patch` when you want the value returned and last-write-wins is acceptable.
|
|
246
|
+
- **No "get all."** There is no scan on the request path. Use `getMany` for known keys, and [Events](./events.md) or a [View](./views.md) for "the latest N."
|
|
247
|
+
- **`getDelete`, not `get` + `delete`, for consume-once.** Only `getDelete` guarantees exactly one caller receives the value.
|
|
248
|
+
|
|
249
|
+
## Related
|
|
250
|
+
|
|
251
|
+
- [ToilDB overview](./README.md): the seven families and how to choose.
|
|
252
|
+
- [Setup](./setup.md): declaring the collection and which function kinds may write.
|
|
253
|
+
- [Data types (`@data`)](../backend/data.md): keys and values.
|
|
254
|
+
- [Counters](./counters.md): when you are really just counting.
|
|
255
|
+
- [Events](./events.md) and [Views](./views.md): for "the latest N" and precomputed reads.
|
|
@@ -0,0 +1,307 @@
|
|
|
1
|
+
# Events (append-only logs)
|
|
2
|
+
|
|
3
|
+
An `Events` collection is an **append-only log**: a growing list of immutable
|
|
4
|
+
facts under a key. You add events to the end and read the newest ones back. You
|
|
5
|
+
never edit or delete an event once it is written.
|
|
6
|
+
|
|
7
|
+
## What and why
|
|
8
|
+
|
|
9
|
+
An "append-only log" means exactly two things happen to it:
|
|
10
|
+
|
|
11
|
+
1. You **append** a new event (it goes on the end).
|
|
12
|
+
2. You **read** events back (newest first).
|
|
13
|
+
|
|
14
|
+
That is it. There is no "update event 5" and no "delete event 3". Each event is a
|
|
15
|
+
permanent record of something that happened: "Ada signed the guestbook", "order
|
|
16
|
+
1234 was refunded", "user logged in from a new device".
|
|
17
|
+
|
|
18
|
+
Use an `Events` log when you care about the **history**, not just the latest
|
|
19
|
+
state:
|
|
20
|
+
|
|
21
|
+
- **Activity feeds:** "what happened in this room, newest first".
|
|
22
|
+
- **Audit trails:** a tamper-resistant record of every important action.
|
|
23
|
+
- **Event sourcing:** treat the log as the source of truth and compute other
|
|
24
|
+
values (totals, summaries, leaderboards) from it.
|
|
25
|
+
|
|
26
|
+
If you only ever need the *current* value of something (a user's profile, a
|
|
27
|
+
shopping cart), use [Documents](./documents.md) instead. If you only need a
|
|
28
|
+
running total (likes, page views), use a [Counter](./counters.md). Events are
|
|
29
|
+
for keeping the full list of what happened.
|
|
30
|
+
|
|
31
|
+
```mermaid
|
|
32
|
+
flowchart LR
|
|
33
|
+
A["append(user, event)"] --> L[("Event log for user<br/>(oldest ... newest)")]
|
|
34
|
+
B["append(user, event)"] --> L
|
|
35
|
+
L --> R["latest(user, 20)<br/>returns newest 20"]
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
## The type
|
|
39
|
+
|
|
40
|
+
An `Events` collection has two type parameters: the **key** (which log) and the
|
|
41
|
+
**event** (what each entry stores).
|
|
42
|
+
|
|
43
|
+
```ts
|
|
44
|
+
Events<K, V>
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
- `K` is the key type: it picks *which* log you are appending to. One key, one
|
|
48
|
+
log. For a per-user activity feed, the key is the user id.
|
|
49
|
+
- `V` is the event type: the shape of a single entry. Both `K` and `V` are
|
|
50
|
+
[`@data`](../concepts/types.md) classes so they can be stored as bytes.
|
|
51
|
+
|
|
52
|
+
You declare it as a `@collection` field inside a `@database` class:
|
|
53
|
+
|
|
54
|
+
```ts
|
|
55
|
+
@data
|
|
56
|
+
class UserKey {
|
|
57
|
+
id: string = '';
|
|
58
|
+
constructor(id: string = '') { this.id = id; }
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
@data
|
|
62
|
+
class Activity {
|
|
63
|
+
action: string = ''; // 'login', 'post.create', 'comment', ...
|
|
64
|
+
detail: string = '';
|
|
65
|
+
at: u64 = 0; // unix seconds
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
@database
|
|
69
|
+
class FeedDb {
|
|
70
|
+
@collection static activity: Events<UserKey, Activity>;
|
|
71
|
+
}
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
The field name (`activity`) names the collection; the class name (`FeedDb`) names
|
|
75
|
+
the database. You reach the log through `FeedDb.activity`.
|
|
76
|
+
|
|
77
|
+
## Operations
|
|
78
|
+
|
|
79
|
+
There are three operations, exactly matching the [toilscript](../concepts/decorators.md)
|
|
80
|
+
API. Their exact signatures:
|
|
81
|
+
|
|
82
|
+
| Operation | Signature | What it does |
|
|
83
|
+
| --- | --- | --- |
|
|
84
|
+
| `append` | `append(key: K, event: V): void` | Add one event to the end of the log. |
|
|
85
|
+
| `appendOnce` | `appendOnce(key: K, eventId: string, event: V): bool` | Add one event, but only if that `eventId` was never seen before. |
|
|
86
|
+
| `latest` | `latest(key: K, limit: i32): V[]` | Read up to `limit` events, newest first. |
|
|
87
|
+
| `since` | `since(key: K, limit: i32): V[]` | Read the NEXT batch of events past a `@derive`'s checkpoint, oldest first. For folding a growing log incrementally. |
|
|
88
|
+
|
|
89
|
+
### `append`
|
|
90
|
+
|
|
91
|
+
The everyday operation. It adds one event and returns nothing.
|
|
92
|
+
|
|
93
|
+
```ts
|
|
94
|
+
FeedDb.activity.append(
|
|
95
|
+
new UserKey('ada'),
|
|
96
|
+
new Activity('login', 'from Paris', now),
|
|
97
|
+
);
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
`append` is a **write**, so you may call it from an action handler (`@post`,
|
|
101
|
+
`@put`, `@patch`, `@del`) but not from a read-only `@get`. See
|
|
102
|
+
[decorators](../concepts/decorators.md) for what each handler kind may do.
|
|
103
|
+
|
|
104
|
+
### `appendOnce` (idempotent append)
|
|
105
|
+
|
|
106
|
+
`appendOnce` adds an event **at most once** for a given id. "Idempotent" means
|
|
107
|
+
running it more than once has the same effect as running it once: the extra calls
|
|
108
|
+
change nothing.
|
|
109
|
+
|
|
110
|
+
Why you want this: networks retry. A client (or another service) may send the
|
|
111
|
+
same request twice because the first reply got lost. With plain `append`, that
|
|
112
|
+
double-delivery writes the event twice. With `appendOnce`, you pass a stable
|
|
113
|
+
`eventId` (a string you choose that is unique for that logical event), and the
|
|
114
|
+
log records it only once.
|
|
115
|
+
|
|
116
|
+
```ts
|
|
117
|
+
// `orderId` uniquely identifies this order event. If the request is retried,
|
|
118
|
+
// the second call is a no-op and the log still has exactly one entry.
|
|
119
|
+
const first = FeedDb.activity.appendOnce(
|
|
120
|
+
new UserKey('ada'),
|
|
121
|
+
`order-refunded:${orderId}`, // the idempotency id
|
|
122
|
+
new Activity('order.refund', orderId, now),
|
|
123
|
+
);
|
|
124
|
+
// first === true -> this call appended the event
|
|
125
|
+
// first === false -> a previous call already appended it; nothing happened now
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
The return value tells you which happened:
|
|
129
|
+
|
|
130
|
+
- `true`: this call appended the event (it was the first with that id).
|
|
131
|
+
- `false`: a call with the same id already appended it, so this was a no-op.
|
|
132
|
+
|
|
133
|
+
Pick an `eventId` that is truly unique per event: an order id, a payment id, a
|
|
134
|
+
message uuid. Do not reuse one id for two different events, or the second will be
|
|
135
|
+
silently dropped as a "duplicate".
|
|
136
|
+
|
|
137
|
+
### `latest`
|
|
138
|
+
|
|
139
|
+
Reads the newest events back, up to a limit, newest first.
|
|
140
|
+
|
|
141
|
+
```ts
|
|
142
|
+
const recent = FeedDb.activity.latest(new UserKey('ada'), 20);
|
|
143
|
+
// recent[0] is the most recent event, recent[1] the next, ...
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
`latest` is a **scan**: it can walk many rows. Scans are barred on the request
|
|
147
|
+
path, so you **cannot** call `latest` from a `@get` or a `@post`. This is a
|
|
148
|
+
deliberate guardrail: a log can grow without bound, and a per-request scan would
|
|
149
|
+
get slower and more expensive as the log grows.
|
|
150
|
+
|
|
151
|
+
Instead you scan the log **off** the request path, in a
|
|
152
|
+
[`@derive`](../background/derive.md), and publish a small precomputed
|
|
153
|
+
[View](./views.md) that your routes read cheaply. The next section shows the full
|
|
154
|
+
pattern.
|
|
155
|
+
|
|
156
|
+
### `since` (incremental read for a `@derive`)
|
|
157
|
+
|
|
158
|
+
`latest` rescans the tail every time. For a log that grows without bound, that gets slower and slower. `since`
|
|
159
|
+
reads only the events you have **not folded yet**: the next batch past a saved cursor, oldest first, up to a
|
|
160
|
+
limit. It is how a [`@derive`](../background/derive.md) folds a huge log incrementally instead of rescanning
|
|
161
|
+
it from the start on every change.
|
|
162
|
+
|
|
163
|
+
You do not pass or manage the cursor. The host owns it: it seeds `since` from a durable checkpoint, advances
|
|
164
|
+
it as it hands you events, and saves it after your fold's view publish lands. So you just loop until the
|
|
165
|
+
batch is empty:
|
|
166
|
+
|
|
167
|
+
```ts
|
|
168
|
+
@derive
|
|
169
|
+
rollup(): void {
|
|
170
|
+
const key = new StatsKey('global');
|
|
171
|
+
const view = StatsDb.summary.get(key) ?? new Summary(); // the running view
|
|
172
|
+
let batch = StatsDb.events.since(key, 500);
|
|
173
|
+
while (batch.length > 0) { // drain in bounded batches
|
|
174
|
+
for (let i = 0; i < batch.length; i++) view.apply(batch[i]);
|
|
175
|
+
batch = StatsDb.events.since(key, 500);
|
|
176
|
+
}
|
|
177
|
+
StatsDb.summary.publish(key, view);
|
|
178
|
+
}
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
Two rules:
|
|
182
|
+
|
|
183
|
+
- **`@derive` or `@job` only.** Like `latest`, `since` is a scan, so it is barred on the request path (`@get`
|
|
184
|
+
/ `@post`). Calling it there is rejected.
|
|
185
|
+
- **Your fold must be idempotent per event.** A rare crash-recovery case (a "healed" event that arrives at an
|
|
186
|
+
older position) makes the host re-read the log from the start that one run, so applying an event twice must
|
|
187
|
+
not change the result. Use set-style updates (`view.byId[e.id] = e.value`), not blind accumulation
|
|
188
|
+
(`view.count += 1`). If your fold cannot be idempotent, use `latest` and recompute instead.
|
|
189
|
+
|
|
190
|
+
`since` is the efficient path; `latest` (recompute) is the simple path. See [`@derive`](../background/derive.md)
|
|
191
|
+
for the full picture.
|
|
192
|
+
|
|
193
|
+
## Worked example: an activity feed
|
|
194
|
+
|
|
195
|
+
Here is the whole loop: an action appends an event, a `@derive` folds the log
|
|
196
|
+
into a view, and a `@get` serves the view.
|
|
197
|
+
|
|
198
|
+
```ts
|
|
199
|
+
import { Activity } from '../models/Activity';
|
|
200
|
+
import { UserKey } from '../models/UserKey';
|
|
201
|
+
import { FeedView } from '../models/FeedView';
|
|
202
|
+
import { NewActivity } from '../models/NewActivity';
|
|
203
|
+
|
|
204
|
+
@database
|
|
205
|
+
class FeedDb {
|
|
206
|
+
// The source of truth: every activity, appended forever.
|
|
207
|
+
@collection static activity: Events<UserKey, Activity>;
|
|
208
|
+
// The precomputed snapshot the GET serves: the newest 20 activities.
|
|
209
|
+
@collection static feed: View<UserKey, FeedView>;
|
|
210
|
+
|
|
211
|
+
// A @derive runs OFF the request path, so it is allowed to scan the log
|
|
212
|
+
// (`latest`) and publish the view. The runtime runs it right after an append,
|
|
213
|
+
// so the view reflects the new event on the next read.
|
|
214
|
+
@derive
|
|
215
|
+
rebuild(): void {
|
|
216
|
+
const key = new UserKey('ada'); // (real code would loop per active user)
|
|
217
|
+
const view = new FeedView();
|
|
218
|
+
view.items = FeedDb.activity.latest(key, 20);
|
|
219
|
+
FeedDb.feed.publish(key, view);
|
|
220
|
+
}
|
|
221
|
+
}
|
|
222
|
+
|
|
223
|
+
@rest('feed')
|
|
224
|
+
class Feed {
|
|
225
|
+
// GET reads the precomputed view: a single keyed read, NOT a scan.
|
|
226
|
+
@get('/')
|
|
227
|
+
public list(): FeedView {
|
|
228
|
+
const view = FeedDb.feed.get(new UserKey('ada'));
|
|
229
|
+
return view == null ? new FeedView() : view;
|
|
230
|
+
}
|
|
231
|
+
|
|
232
|
+
// POST appends one activity. The @derive republishes `feed` right after.
|
|
233
|
+
@post('/')
|
|
234
|
+
public record(input: NewActivity): FeedView {
|
|
235
|
+
const at = <u64>(Date.now() / 1000);
|
|
236
|
+
FeedDb.activity.append(new UserKey('ada'), new Activity(input.action, input.detail, at));
|
|
237
|
+
// We do NOT scan here (that would be barred). We just acknowledge; the GET
|
|
238
|
+
// serves the updated list from the view the derive rebuilds.
|
|
239
|
+
return new FeedView();
|
|
240
|
+
}
|
|
241
|
+
}
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
The two `@data` models:
|
|
245
|
+
|
|
246
|
+
```ts
|
|
247
|
+
@data
|
|
248
|
+
export class Activity {
|
|
249
|
+
action: string = '';
|
|
250
|
+
detail: string = '';
|
|
251
|
+
at: u64 = 0;
|
|
252
|
+
}
|
|
253
|
+
|
|
254
|
+
@data
|
|
255
|
+
export class FeedView {
|
|
256
|
+
items: Activity[] = [];
|
|
257
|
+
}
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
The key idea: the **log** is the durable history, and the **view** is a small,
|
|
261
|
+
read-cheap snapshot derived from it. Reads hit the view; the log grows quietly in
|
|
262
|
+
the background.
|
|
263
|
+
|
|
264
|
+
## Ordering and consistency
|
|
265
|
+
|
|
266
|
+
ToilDB is a **worldwide** database: your data lives in many regions at once. A
|
|
267
|
+
few honest details about what that means for events:
|
|
268
|
+
|
|
269
|
+
- **Ordering within a log is stable.** Each key (each log) has one "home" location
|
|
270
|
+
where appends are serialized and stamped with an increasing sequence number. So
|
|
271
|
+
`latest` always returns a consistent newest-first order for that key.
|
|
272
|
+
- **`latest` is newest first.** Index `0` is the most recent event.
|
|
273
|
+
- **Reads can lag slightly across regions.** An append is applied at the log's
|
|
274
|
+
home first, then copied out to other regions in the background (this is called
|
|
275
|
+
*asynchronous replication*, and the result is *eventual consistency*: given a
|
|
276
|
+
little time, every region converges to the same log). A read served from a far
|
|
277
|
+
region may briefly miss the very newest event. A `@derive` runs right after the
|
|
278
|
+
append that triggered it, so the view it publishes reflects that write.
|
|
279
|
+
- **`appendOnce` dedups by your id.** The at-most-once guarantee is anchored to
|
|
280
|
+
the `eventId` you pass, so retries are safe even across a flaky network.
|
|
281
|
+
- **A log grows without bound.** Appending never shrinks it. Keep events small,
|
|
282
|
+
and read them through a bounded `latest(key, N)` in a derive rather than trying
|
|
283
|
+
to hold the whole log in memory.
|
|
284
|
+
|
|
285
|
+
## Gotchas
|
|
286
|
+
|
|
287
|
+
- **You cannot read the log from a route.** `latest` is a scan and is only legal
|
|
288
|
+
in a `@derive` or a `@job`. If you try to call it from a `@get`/`@post`, the
|
|
289
|
+
compiler rejects it. Read a [View](./views.md) from your route instead.
|
|
290
|
+
- **Events are immutable.** There is no "edit" or "delete an event". If a fact
|
|
291
|
+
changes, append a new event that records the change (for example, an
|
|
292
|
+
`order.refund` event, not an edit to the original `order.create`).
|
|
293
|
+
- **Choose `eventId` carefully for `appendOnce`.** It must be unique per logical
|
|
294
|
+
event. Reusing an id drops the new event as a duplicate.
|
|
295
|
+
- **Do not treat a counter as a log.** A [Counter](./counters.md) gives you a
|
|
296
|
+
running total but forgets the individual events. If you need the list, use
|
|
297
|
+
`Events`.
|
|
298
|
+
|
|
299
|
+
## Related
|
|
300
|
+
|
|
301
|
+
- [Documents](./documents.md): mutable, one value per key (edit in place).
|
|
302
|
+
- [Counters](./counters.md): a single running total per key.
|
|
303
|
+
- [Views](./views.md): the precomputed snapshot your routes read.
|
|
304
|
+
- [`@derive`](../background/derive.md): folds an event log into a View, off the
|
|
305
|
+
request path.
|
|
306
|
+
- [Data types (`@data`)](../concepts/types.md): how keys and events are stored.
|
|
307
|
+
- [Decorators](../concepts/decorators.md): which handler kinds may append vs scan.
|