letopis 1.4.1 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENT-CHEATSHEET.md +329 -278
- package/CHANGELOG.md +558 -1
- package/README.md +606 -1037
- package/dist/acl.d.ts +78 -1
- package/dist/acl.js +264 -13
- package/dist/admin.d.ts +10 -141
- package/dist/admin.js +24 -175
- package/dist/auth.d.ts +2 -2
- package/dist/auth.js +5 -15
- package/dist/cache.d.ts +8 -1
- package/dist/cache.js +33 -10
- package/dist/chain.d.ts +19 -41
- package/dist/chain.js +169 -201
- package/dist/cli.js +51 -88
- package/dist/demo/booking.d.ts +352 -36
- package/dist/describe.d.ts +47 -0
- package/dist/describe.js +195 -0
- package/dist/errors.d.ts +59 -13
- package/dist/errors.js +190 -29
- package/dist/index.d.ts +52 -71
- package/dist/index.js +245 -130
- package/dist/internal.d.ts +12 -0
- package/dist/internal.js +5 -0
- package/dist/jsonschema.js +18 -17
- package/dist/load.d.ts +20 -12
- package/dist/load.js +225 -186
- package/dist/mcp.d.ts +5 -7
- package/dist/mcp.js +171 -215
- package/dist/messages/access.d.ts +90 -0
- package/dist/messages/access.js +113 -0
- package/dist/messages/api.d.ts +171 -0
- package/dist/messages/api.js +215 -0
- package/dist/messages/boot.d.ts +201 -0
- package/dist/messages/boot.js +256 -0
- package/dist/messages/codes.d.ts +172 -0
- package/dist/messages/codes.js +173 -0
- package/dist/messages/index.d.ts +95 -0
- package/dist/messages/index.js +118 -0
- package/dist/messages/mcp.d.ts +306 -0
- package/dist/messages/mcp.js +387 -0
- package/dist/messages/model.d.ts +233 -0
- package/dist/messages/model.js +283 -0
- package/dist/messages/query.d.ts +131 -0
- package/dist/messages/query.js +158 -0
- package/dist/messages/sql-access.d.ts +70 -0
- package/dist/messages/sql-access.js +87 -0
- package/dist/messages/sql-core.d.ts +150 -0
- package/dist/messages/sql-core.js +181 -0
- package/dist/messages/sql-time.d.ts +71 -0
- package/dist/messages/sql-time.js +88 -0
- package/dist/messages/sql-write.d.ts +246 -0
- package/dist/messages/sql-write.js +270 -0
- package/dist/messages/time.d.ts +23 -0
- package/dist/messages/time.js +29 -0
- package/dist/messages/types.d.ts +27 -0
- package/dist/messages/types.js +5 -0
- package/dist/messages/write.d.ts +405 -0
- package/dist/messages/write.js +509 -0
- package/dist/model.d.ts +73 -60
- package/dist/model.js +26 -29
- package/dist/ops.d.ts +35 -0
- package/dist/ops.js +101 -0
- package/dist/pglite.d.ts +0 -9
- package/dist/pglite.js +3 -1
- package/dist/registry.d.ts +3 -19
- package/dist/registry.js +3 -16
- package/dist/signals.d.ts +5 -9
- package/dist/signals.js +5 -9
- package/dist/sql.d.ts +13 -6
- package/dist/sql.js +176 -83
- package/dist/sync.d.ts +9 -3
- package/dist/sync.js +51 -21
- package/dist/tx.d.ts +26 -1
- package/dist/tx.js +44 -14
- package/dist/typed.d.ts +240 -45
- package/dist/types.d.ts +23 -91
- package/dist/types.js +12 -14
- package/dist/up.d.ts +53 -32
- package/dist/up.js +184 -73
- package/dist/uuid.js +5 -5
- package/dist/validate.d.ts +13 -1
- package/dist/validate.js +61 -29
- package/dist/watch.d.ts +11 -10
- package/dist/watch.js +33 -70
- package/dist/write.d.ts +9 -16
- package/dist/write.js +118 -187
- package/llms.txt +22 -25
- package/package.json +8 -11
- package/sql/10-core.sql +7 -50
- package/sql/15-errors.sql +64 -10
- package/sql/20-context.sql +7 -6
- package/sql/30-validate.sql +35 -27
- package/sql/40-class.sql +20 -114
- package/sql/50-acl.sql +8 -13
- package/sql/60-write.sql +220 -667
- package/sql/70-read.sql +20 -16
- package/sql/80-auth.sql +24 -27
- package/sql/90-time.sql +70 -859
- package/sql/95-seed.system.sql +7 -14
- package/sql/99-revision.sql +2 -1
- package/sql/README.md +32 -24
- package/sql/migrate/README.md +14 -0
- package/AGENT-CHEATSHEET.en.md +0 -485
- package/MIGRATION.md +0 -190
- package/README.en.md +0 -2859
- package/dist/import.d.ts +0 -185
- package/dist/import.js +0 -820
- package/dist/project.d.ts +0 -350
- package/dist/project.js +0 -1224
- package/sql/75-history.sql +0 -358
- package/sql/88-derived.sql +0 -1235
- package/sql/migrate/0006-changed-paths.sql +0 -3
- package/sql/migrate/0007-secret-account-fk.sql +0 -12
package/README.en.md
DELETED
|
@@ -1,2859 +0,0 @@
|
|
|
1
|
-
# letopis 1.4
|
|
2
|
-
|
|
3
|
-
**letopis** is a Node.js library: a store of versioned entities on PostgreSQL 18. Every change is saved as a version
|
|
4
|
-
in a journal. So every object has a history, a state as of any date and a change subscription. The database itself
|
|
5
|
-
checks data integrity, tenant isolation and permissions, with triggers and RLS policies. The rules work the same for
|
|
6
|
-
the library and for raw SQL. Data is read and written with chains:
|
|
7
|
-
`db.Shop(shop).Product({ price: lt(1000) }).rows()`.
|
|
8
|
-
|
|
9
|
-
[Русская версия](README.md) · [Cheatsheet for AI agents](AGENT-CHEATSHEET.en.md) · [Migration from 0.21 (in Russian)](MIGRATION.md) · [Changelog (in Russian)](CHANGELOG.md) · [llms.txt](llms.txt)
|
|
10
|
-
|
|
11
|
-
What letopis can do:
|
|
12
|
-
|
|
13
|
-
- **Zod models** are stored in the database as classes: hubs (objects) and links (relations with their own data) — [section 4](#4-data-model).
|
|
14
|
-
- **Writing with chains**: `create`, `update` with merge, `upsert`, deletion with preview and cascade, the `inc` increment
|
|
15
|
-
for money and counters — [sections 5](#5-writing), [7](#7-deleting-and-restoring) and [8](#8-transactions-batches-and-money).
|
|
16
|
-
- **Reading with chains**: hops along links, filters, trees, aggregates, pagination — [section 6](#6-reading).
|
|
17
|
-
- **History**: versions, point-in-time reads, what changed and who changed it (`diff`, `blame`), restoring deleted
|
|
18
|
-
data, retention policy, hash chain verification — [section 11](#11-history-and-point-in-time-reads).
|
|
19
|
-
- **Tenants, sign-in and permissions** in the database itself: passwords, API keys, one-time codes, TOTP, sessions,
|
|
20
|
-
impersonation, access rules — [section 10](#10-users-sign-in-and-permissions).
|
|
21
|
-
- **Change subscription** by cursor without losses, and a **result cache** — [section 12](#12-change-subscription-and-cache).
|
|
22
|
-
- **Derived data**: aggregates in the database, sums and counts that the database itself maintains, and projectors
|
|
23
|
-
that build summaries from the journal — [sections 4](#aggregates-in-the-database) and [12](#projectors).
|
|
24
|
-
- **Agents and MCP**: an MCP server gives an AI agent the model, reads, history and, if allowed, writes, with the
|
|
25
|
-
rights of one user — [section 13](#agents-and-mcp).
|
|
26
|
-
- **Bulk loading** of millions of rows and **import from 0.21** — [sections 9](#9-bulk-loading) and [14](#14-migrating-from-021).
|
|
27
|
-
|
|
28
|
-
Requirements:
|
|
29
|
-
|
|
30
|
-
- PostgreSQL 18 or newer with the pgcrypto extension (it is part of the standard distribution), and a database in UTF8
|
|
31
|
-
encoding.
|
|
32
|
-
- Node.js 22 or newer; the package is an ES module.
|
|
33
|
-
- One runtime dependency: `postgres`. Zod 4 is an optional peer dependency: only the `letopis/model` subpath needs it,
|
|
34
|
-
and `import { connect } from 'letopis'` works without it. For development without PostgreSQL there are the optional
|
|
35
|
-
`@electric-sql/pglite` and `@electric-sql/pglite-socket`. The MCP server (`letopis/mcp` and `letopis mcp`) needs
|
|
36
|
-
the optional `@modelcontextprotocol/server`, version 2.4 or newer in the 2.x line.
|
|
37
|
-
|
|
38
|
-
## Contents
|
|
39
|
-
|
|
40
|
-
1. [Quick start](#1-quick-start)
|
|
41
|
-
2. [How letopis works](#2-how-letopis-works)
|
|
42
|
-
3. [Installation and connection](#3-installation-and-connection)
|
|
43
|
-
4. [Data model](#4-data-model)
|
|
44
|
-
5. [Writing](#5-writing)
|
|
45
|
-
6. [Reading](#6-reading)
|
|
46
|
-
7. [Deleting and restoring](#7-deleting-and-restoring)
|
|
47
|
-
8. [Transactions, batches and money](#8-transactions-batches-and-money)
|
|
48
|
-
9. [Bulk loading](#9-bulk-loading)
|
|
49
|
-
10. [Users, sign-in and permissions](#10-users-sign-in-and-permissions)
|
|
50
|
-
11. [History and point-in-time reads](#11-history-and-point-in-time-reads)
|
|
51
|
-
12. [Change subscription and cache](#12-change-subscription-and-cache)
|
|
52
|
-
13. [Operations](#13-operations)
|
|
53
|
-
14. [Migrating from 0.21](#14-migrating-from-021)
|
|
54
|
-
15. [API reference](#15-api-reference)
|
|
55
|
-
16. [Errors](#16-errors)
|
|
56
|
-
17. [Performance](#17-performance)
|
|
57
|
-
18. [Limitations](#18-limitations)
|
|
58
|
-
19. [Tests](#19-tests)
|
|
59
|
-
|
|
60
|
-
Sections 3–13 are built on one example, a shop: installation, model, writing, reading and so on. The code blocks of
|
|
61
|
-
these sections go in order and together form one program. The `docs` test runs this program, like the quick start
|
|
62
|
-
example, against a real database, so the examples do not drift away from the code.
|
|
63
|
-
|
|
64
|
-
## 1. Quick start
|
|
65
|
-
|
|
66
|
-
You need PostgreSQL 18 and an admin role: a superuser or a role with the `CREATEROLE` privilege. Install the package:
|
|
67
|
-
|
|
68
|
-
```sh
|
|
69
|
-
npm install letopis zod postgres # postgres: for createTenant and your own SQL queries
|
|
70
|
-
```
|
|
71
|
-
|
|
72
|
-
The application connects to the database with a separate role that is a member of `letopis_app`. The installation
|
|
73
|
-
creates `letopis_app`. So after the first installation, create the application role once:
|
|
74
|
-
|
|
75
|
-
```sql
|
|
76
|
-
create role shop_app login password 'app-secret';
|
|
77
|
-
grant letopis_app to shop_app;
|
|
78
|
-
```
|
|
79
|
-
|
|
80
|
-
The example installs the schema, creates a shop with its first user, describes two classes and works with them:
|
|
81
|
-
|
|
82
|
-
<!-- run: quickstart -->
|
|
83
|
-
```ts
|
|
84
|
-
import { up, connect, syncModels, inc, gte } from 'letopis'
|
|
85
|
-
import { hub } from 'letopis/model'
|
|
86
|
-
import { z } from 'zod'
|
|
87
|
-
|
|
88
|
-
const ADMIN = 'postgres://postgres:secret@localhost:5432/shop' // admin: installs the schema
|
|
89
|
-
const APP = 'postgres://shop_app:app-secret@localhost:5432/shop' // application: a member of letopis_app
|
|
90
|
-
|
|
91
|
-
// 1. Install the v2.shop schema. tenant is the first tenant (the shop) and its user Anna.
|
|
92
|
-
const r = await up({ dsn: ADMIN, schema: 'shop',
|
|
93
|
-
tenant: { name: 'Chamomile', user: { name: 'Anna', login: 'anna@shop.ru', password: 'Chamomile-2026!' } } })
|
|
94
|
-
// r.serviceKey is the application's service key: it is issued on the first installation, so save it
|
|
95
|
-
|
|
96
|
-
// 2. The application service signs in with the key, checks Anna's password and works on her behalf in the shop.
|
|
97
|
-
const svc = await connect({ dsn: APP, schema: r.schema, apiKey: r.serviceKey! })
|
|
98
|
-
const login = await svc.auth.verifyPassword({ identifier: 'anna@shop.ru', password: 'Chamomile-2026!' })
|
|
99
|
-
const db = await svc.as(login!.account, { tenant: r.tenant!.tenant })
|
|
100
|
-
|
|
101
|
-
// 3. Model: a shop and a product. syncModels writes the class descriptions to the database.
|
|
102
|
-
const Shop = hub('Shop', { name: z.string() }, { key: ['name'] })
|
|
103
|
-
const Product = hub('Product', { sku: z.string(), title: z.string(), price: z.number().int().min(0) },
|
|
104
|
-
{ ends: { Shop }, key: ['Shop', 'sku'] })
|
|
105
|
-
await syncModels(db, [Shop, Product], { apply: true })
|
|
106
|
-
|
|
107
|
-
// 4. Writing and reading with chains.
|
|
108
|
-
const [shop] = await db.Shop().create({ name: 'Chamomile' }).rows()
|
|
109
|
-
const [kettle] = await db.Shop(shop).Product().create({ sku: 'K-1', title: 'Tea kettle', price: 2500 }).rows()
|
|
110
|
-
await db.Product(kettle).update({ price: inc(-500) }).rows() // discount: the database subtracts inside the statement
|
|
111
|
-
const expensive = await db.Shop(shop).Product({ price: gte(1000) }).rows()
|
|
112
|
-
const history = await db.Product(kettle).versions() // two versions: creation and edit
|
|
113
|
-
|
|
114
|
-
await svc.close()
|
|
115
|
-
```
|
|
116
|
-
|
|
117
|
-
What happened here:
|
|
118
|
-
|
|
119
|
-
- `up()` installed the `v2.shop` schema in the database: three tables, triggers, RLS policies, functions and the
|
|
120
|
-
system seed. The `tenant` option created the tenant "Chamomile" with the rule "authenticated users may do
|
|
121
|
-
everything", and the user Anna with the `owner` role in it.
|
|
122
|
-
- `connect({ apiKey })` exchanged the service key for a session. `verifyPassword` returned `{ account, token }` (or
|
|
123
|
-
`null` if the password is wrong). `svc.as(account, { tenant })` is the same connection pool, but acting as Anna in
|
|
124
|
-
the shop; the database checks that the service has the right to do this.
|
|
125
|
-
- `hub('Product', fields, options)` described an object class. `ends: { Shop }` is a required reference to the shop.
|
|
126
|
-
`key: ['Shop', 'sku']` is the key: the product id is computed from the shop and the SKU, and the database does not
|
|
127
|
-
accept a second product with the same key.
|
|
128
|
-
- `db.Shop(shop).Product().create(…)` created a product and took the reference to the shop from the chain path.
|
|
129
|
-
`inc(-500)` lowered the price inside the `UPDATE` statement, without reading it first. `versions()` read the history
|
|
130
|
-
from the journal.
|
|
131
|
-
|
|
132
|
-
Calling `up()` again is safe: it re-applies the same revision and does not touch the data, and it does not issue the
|
|
133
|
-
service key again while the installation still has at least one API key that is not revoked. Next, section 2 explains the design, and sections
|
|
134
|
-
3–13 go through the tasks in order.
|
|
135
|
-
|
|
136
|
-
## 2. How letopis works
|
|
137
|
-
|
|
138
|
-
### Core concepts
|
|
139
|
-
|
|
140
|
-
| Concept | What it is |
|
|
141
|
-
|---|---|
|
|
142
|
-
| installation | a PostgreSQL schema `v2.<name>` with the letopis tables and functions; one database can hold several |
|
|
143
|
-
| class | a data type: name, fields (JSON Schema), ends and key. A class description is also a row in the database (class `Class`) |
|
|
144
|
-
| hub | an object class: a shop, a product, a customer |
|
|
145
|
-
| link | a relation class between objects, with its own data: an order line, a specialist's skill |
|
|
146
|
-
| end (role) | a reference from one row to another row. An end has a name, the role: `Shop`, `Parent`. Ends are in `row.links`, fields are in `row.data` |
|
|
147
|
-
| key | the fields and ends that identify an object within a class and a tenant. The id is computed from them, so uniqueness is the key |
|
|
148
|
-
| id | a version 5 UUID: a hash of the tenant, the class and the key values; for a class without a key, a hash of a random part |
|
|
149
|
-
| account | a user, a service or an organization: a row of class `Account` |
|
|
150
|
-
| tenant | the account whose data is being worked with. Data of different tenants is isolated |
|
|
151
|
-
| membership | a `member` link "user — tenant" with roles; it allows work in another tenant |
|
|
152
|
-
| session | a token by which the database identifies the account and the tenant. A service gets it with an API key, a user by signing in |
|
|
153
|
-
| version, journal | each change of a row is a version in the `log` journal, with author, time and hash |
|
|
154
|
-
| tombstone | a "deleted" version with the last snapshot of the object |
|
|
155
|
-
| chain | a query like `db.Shop(shop).Product().rows()`: steps, modifiers, write verbs and a terminal |
|
|
156
|
-
| terminal | a method that executes the chain: `rows`, `first`, `ids`, `count`, `paths`, aggregates |
|
|
157
|
-
|
|
158
|
-
### What is stored in the database
|
|
159
|
-
|
|
160
|
-
There are three tables. `entity` is the current state: one row per live entity, including the rows of system classes
|
|
161
|
-
(`Class`, `Account`, `Resource`, `member`, `rule`). `log` is the journal: a full snapshot of every version with author,
|
|
162
|
-
time, kind of operation and hash. `secret` holds credentials, sessions and projector checkpoints: the application sees
|
|
163
|
-
none of its rows; only the sign-in and projector functions work with it.
|
|
164
|
-
|
|
165
|
-
Classes are entities too. A class description is stored in a `Class` row; the database also keeps the expanded
|
|
166
|
-
description (with ancestors applied) and the compiled data check there. The version hash is a generated column: sha256 of the
|
|
167
|
-
previous version's hash and the row snapshot. So the next version exposes a tampered version, and for the last version
|
|
168
|
-
an anchor outside the database does this ([section 11](#11-history-and-point-in-time-reads)).
|
|
169
|
-
|
|
170
|
-
### Who checks the data
|
|
171
|
-
|
|
172
|
-
There are two roles. `letopis_owner` owns the schema, tables and functions. `letopis_app` is the application role: the
|
|
173
|
-
application's LOGIN role is a member of this role only. Everything the application runs, both the library and raw SQL,
|
|
174
|
-
goes through triggers and RLS.
|
|
175
|
-
|
|
176
|
-
- **Triggers.** On insert, a trigger computes the id, fills in defaults and checks the data against the class JSON
|
|
177
|
-
Schema (the schema is compiled to jsonpath). It checks the ends and their targets, fills in the service columns and
|
|
178
|
-
writes a version to the journal. An update merges the patch inside the statement (`merge`, `inc`). After a delete,
|
|
179
|
-
the trigger applies the `restrict`, `cascade` and `unset` rules and writes tombstones.
|
|
180
|
-
- **RLS and permissions.** Policies isolate tenants and apply access rules. At the start of a query, the database
|
|
181
|
-
builds the actor's permission plan once (`acl_plan()`); each row then needs only the cheap `acl_ok` check.
|
|
182
|
-
- **Query context** is the transaction settings: `letopis.token` (the session), `letopis.tenant` and `letopis.account`
|
|
183
|
-
(the tenant and impersonation), `letopis.reason` (the reason for the change) and `letopis.acl` (the permission pass,
|
|
184
|
-
see [section 10](#10-users-sign-in-and-permissions)). The database finds the session by the sha256 of the token, so
|
|
185
|
-
without a real token the settings give nothing.
|
|
186
|
-
- **Bypassing the checks** is possible only for code that runs as the owner: owner functions (`security definer` with
|
|
187
|
-
a fixed `search_path`) and the loader on an admin connection. `connect()` refuses a role that bypasses RLS (a
|
|
188
|
-
superuser, `BYPASSRLS`, a member of `letopis_owner`) unless `allowBypassRls: true` is set.
|
|
189
|
-
|
|
190
|
-
### How reads work
|
|
191
|
-
|
|
192
|
-
Under RLS, the indexes on `data` and on back references are not directly available to the policy. So the ids for chain
|
|
193
|
-
steps are found by the owner functions `find_ids()`, `find_hop()` and `find_deep()`. They apply the same tenant and
|
|
194
|
-
permission conditions as the policy. The rows themselves are then read by primary key under RLS. A point-in-time read
|
|
195
|
-
is built from the journal.
|
|
196
|
-
|
|
197
|
-
### Signals
|
|
198
|
-
|
|
199
|
-
A write reports changes through `NOTIFY`. The channel is the full name of the installation schema, version included,
|
|
200
|
-
for example `v2.shop`; because of the dot, SQL takes the name in double quotes: `LISTEN "v2.shop"`. A raw SQL write on
|
|
201
|
-
its own connection sends the signal from a trigger, and PostgreSQL delivers it on commit. Library transactions,
|
|
202
|
-
`db.sql` and `db.transaction()` included, defer their signals: the process notifier sends them every 100 ms
|
|
203
|
-
(`notifyIntervalMs`) and retries the ones that failed to send. What a write signal carries (its payload) is set by the
|
|
204
|
-
installation mode `up({ signals })`:
|
|
205
|
-
|
|
206
|
-
| Mode | Write signal payload | What any role that can connect to the database sees |
|
|
207
|
-
|---|---|---|
|
|
208
|
-
| `class` (default) | the class name: `Booking` | class names and write times |
|
|
209
|
-
| `tenant` | `Class@tenant token`: `Booking@3f9c…` | also tenant pseudonyms |
|
|
210
|
-
| `opaque` | `class token@tenant token`: `9a1e…@3f9c…` | only write times and frequency per pseudonym |
|
|
211
|
-
| `none` | no signals | nothing |
|
|
212
|
-
|
|
213
|
-
A token is 24 hexadecimal characters of a keyed HMAC: only a process with the signal key can match a pseudonym to a
|
|
214
|
-
name, and the key is given to services (the `letopis.signals` permission). Besides write signals, there are service
|
|
215
|
-
ones: a revoked session sends `session:<16 hex>` (the first 16 hexadecimal characters of the sha256 of its token), a
|
|
216
|
-
session reset sends `session:*`, and a reset of a tenant or of the whole installation sends `*` (a reset of the whole
|
|
217
|
-
installation also sends `session:*`). In the `none` mode there are none of them either. Signals carry no ids and no
|
|
218
|
-
data. On signals, the class registry is re-read, the result cache is cleared and subscriptions wake up; what the mode
|
|
219
|
-
changes for the cache and subscriptions is in [section 12](#12-change-subscription-and-cache).
|
|
220
|
-
|
|
221
|
-
## 3. Installation and connection
|
|
222
|
-
|
|
223
|
-
### Installation: `up()`
|
|
224
|
-
|
|
225
|
-
`up()` waits for the database and creates it if it does not exist. It checks for PostgreSQL 18 and UTF8 encoding,
|
|
226
|
-
installs the roles `letopis_owner` and `letopis_app`, the pgcrypto extension and the `v2.<name>` schema with all
|
|
227
|
-
functions and the system seed. On the first installation, it issues a service key. Call it from an admin role.
|
|
228
|
-
|
|
229
|
-
| Option | What it does |
|
|
230
|
-
|---|---|
|
|
231
|
-
| `dsn` or `sql` | an admin connection string or a ready postgres.js pool |
|
|
232
|
-
| `schema` | the installation name: `'shop'` gives the schema `v2.shop`; you can also pass `'v2.shop'` directly |
|
|
233
|
-
| `tenant` | the first tenant and its user (see below) |
|
|
234
|
-
| `seeds` | seeds after the core: `'booking'` is the salon demo domain; or paths to your own `.sql` files |
|
|
235
|
-
| `history` | the history retention policy for classes without their own ([section 11](#11-history-and-point-in-time-reads)) |
|
|
236
|
-
| `serviceKey` | your own service key instead of a random one: `lts_` and 32–128 hexadecimal characters |
|
|
237
|
-
| `allowReset` | allow `db.reset()` ([section 13](#13-operations)); off by default |
|
|
238
|
-
| `signals` | the signal mode: `'class'` (default), `'tenant'`, `'opaque'` or `'none'` (["Signals"](#signals)); the mode is stored in the schema — `up()` without the option keeps it |
|
|
239
|
-
| `fresh` | drop the schema and install it again — **the schema data is lost** |
|
|
240
|
-
| `upgrade` | upgrade a schema of an older revision: migrations and a re-apply |
|
|
241
|
-
| `pglite` | a database for development without PostgreSQL (see below) |
|
|
242
|
-
| `quiet`, `waitTimeoutMs` | no console messages; how long to wait for the database (60,000 ms by default) |
|
|
243
|
-
| `owner`, `app` | the names of the owner and application roles (`letopis_owner`, `letopis_app` by default) |
|
|
244
|
-
|
|
245
|
-
Result: `schema` (the full name), `serviceKey` (on the first installation, and on a re-run when the installation has no
|
|
246
|
-
API key left except revoked ones: an expired or disabled key counts too; otherwise `null`), `created`, `upgraded`, `revision`, `migrations`, `analyzed`, `owner`, `app`,
|
|
247
|
-
`signals` (the signal mode of the installation), `warnings` (without `quiet` they are printed), and also `tenant` and
|
|
248
|
-
`pglite` if they were requested. On PostgreSQL below 18.2, `up()` warns: such a server could truncate CLOG that unread
|
|
249
|
-
signals still need — upgrade it.
|
|
250
|
-
|
|
251
|
-
Several instances of a service may call `up()` for the same schema at the same time: the installation runs under a lock
|
|
252
|
-
on the schema. The first `up()` installs or upgrades it, the others wait and re-apply the ready schema
|
|
253
|
-
(`created: false`, `upgraded: false`); one of them creates the first `tenant`, the others find it by name. `up()` calls
|
|
254
|
-
for different schemas of one database do not wait for each other.
|
|
255
|
-
|
|
256
|
-
To hide class names from signals, use the `opaque` mode (or `tenant` if the names are not secret but you want the
|
|
257
|
-
cache to be precise per tenant); after changing the mode, restart the application processes — they read the mode on
|
|
258
|
-
connect:
|
|
259
|
-
|
|
260
|
-
```ts
|
|
261
|
-
await up({ dsn: ADMIN, schema: 'shop', signals: 'opaque' }) // later up({ dsn, schema }) keeps the mode
|
|
262
|
-
```
|
|
263
|
-
|
|
264
|
-
<!-- run: guide -->
|
|
265
|
-
```ts
|
|
266
|
-
import { up, connect } from 'letopis'
|
|
267
|
-
|
|
268
|
-
const ADMIN = 'postgres://postgres:secret@localhost:5432/shop'
|
|
269
|
-
const APP = 'postgres://shop_app:app-secret@localhost:5432/shop'
|
|
270
|
-
|
|
271
|
-
const r = await up({ dsn: ADMIN, schema: 'shop',
|
|
272
|
-
tenant: { name: 'Chamomile', user: { name: 'Anna', login: 'anna@shop.ru', password: 'Chamomile-2026!' } } })
|
|
273
|
-
const SHOP = r.tenant!.tenant // the tenant: the Chamomile shop
|
|
274
|
-
const ANNA = r.tenant!.user! // its first user
|
|
275
|
-
```
|
|
276
|
-
|
|
277
|
-
### Application role
|
|
278
|
-
|
|
279
|
-
The application works as a LOGIN role that is a member of `letopis_app` only:
|
|
280
|
-
`create role shop_app login password '…'; grant letopis_app to shop_app;`. Then RLS and triggers apply to it, as to
|
|
281
|
-
any raw SQL from this role. `connect()` rejects a role that bypasses RLS with the error `bypass_rls`. The admin
|
|
282
|
-
connection (`allowBypassRls: true`) is needed only for the loader, projectors, aggregate repair (`rebuildAggregates`),
|
|
283
|
-
verification and maintenance; keep it in a separate process, not in the web application.
|
|
284
|
-
|
|
285
|
-
### First tenant and user
|
|
286
|
-
|
|
287
|
-
Permissions in letopis are always on and deny everything by default. So a new tenant needs rules. The
|
|
288
|
-
`up({ tenant })` option and the `createTenant(sql, schema, opts)` function do this in one transaction:
|
|
289
|
-
|
|
290
|
-
- the tenant: an account in System with an id from the name (`tenant:<name>`);
|
|
291
|
-
- the rule "authenticated users may do everything" in this tenant (`grantAll`, on by default, weight 10);
|
|
292
|
-
- a user (id from the login) with a membership in the tenant with the roles `user.roles` (`['owner']` by default), a
|
|
293
|
-
password and a session in the tenant.
|
|
294
|
-
|
|
295
|
-
A repeat with the same name and login duplicates nothing and does not change the password; it only issues a new
|
|
296
|
-
session. The login goes into the id as is: `Anna@shop.ru` and `anna@shop.ru` would give two accounts, and the second one
|
|
297
|
-
would have no password, so write logins in one case. The `created` field in the result tells whether this call created
|
|
298
|
-
the tenant: for an existing tenant with a new user it is `false`. The roles are set by `user.roles`, a non-empty list
|
|
299
|
-
of names (not masks), otherwise `invalid_data`. The password kind `user.kind` (`PASSWORD` by default) follows the rule
|
|
300
|
-
for a credential kind, as in `auth.setPassword` ([API reference](#dbauth)), otherwise it is `invalid_data` too. A
|
|
301
|
-
repeat does not change the roles of an existing membership: they are
|
|
302
|
-
changed by editing the membership ([Granting a permission](#granting-a-permission)). A repeat does not bring back
|
|
303
|
-
what was deleted: if the tenant, the user account or the membership is deleted (the row is gone but its history is in
|
|
304
|
-
the journal), `createTenant` answers `invalid_data` and writes nothing. The error text names what was deleted and
|
|
305
|
-
advises bringing it back with `restore()`: an account with `db.Account(id).restore()` on behalf of the system
|
|
306
|
-
administrator, a membership with `db.member(id).restore()` in that tenant; the other way out is a different name or
|
|
307
|
-
login. The session from `createTenant` has no sign-in method, so the user cannot change their own password with it:
|
|
308
|
-
for that the user signs in with the password. `createTenant` needs an admin postgres.js pool:
|
|
309
|
-
|
|
310
|
-
<!-- run: guide -->
|
|
311
|
-
```ts
|
|
312
|
-
import postgres from 'postgres'
|
|
313
|
-
import { createTenant } from 'letopis'
|
|
314
|
-
|
|
315
|
-
const pg = postgres(ADMIN)
|
|
316
|
-
const lutik = await createTenant(pg, r.schema, {
|
|
317
|
-
name: 'Buttercup', user: { name: 'Boris', login: 'boris@lutik.ru', password: 'Buttercup-2026!' } })
|
|
318
|
-
await pg.end()
|
|
319
|
-
const BORIS = lutik.user! // lutik: { tenant, user, token, created: true }
|
|
320
|
-
```
|
|
321
|
-
|
|
322
|
-
### Connecting: `connect()`
|
|
323
|
-
|
|
324
|
-
A connection gets a session in one of two ways: with an API key (`apiKey`) or with a token (`token`). An application
|
|
325
|
-
service usually keeps one connection by key and acts on behalf of users through `as()`:
|
|
326
|
-
|
|
327
|
-
<!-- run: guide -->
|
|
328
|
-
```ts
|
|
329
|
-
// service: the API key is exchanged for a service session
|
|
330
|
-
const svc = await connect({ dsn: APP, schema: r.schema, apiKey: r.serviceKey! })
|
|
331
|
-
// sign-in: the service checks the password and acts on behalf of the user in the tenant
|
|
332
|
-
const login = await svc.auth.verifyPassword({ identifier: 'anna@shop.ru', password: 'Chamomile-2026!' })
|
|
333
|
-
const db = await svc.as(login!.account, { tenant: SHOP })
|
|
334
|
-
// a separate connection by session token, for example in another process
|
|
335
|
-
const anna = await connect({ dsn: APP, schema: r.schema, token: r.tenant!.token! })
|
|
336
|
-
```
|
|
337
|
-
|
|
338
|
-
| `connect()` option | What it does |
|
|
339
|
-
|---|---|
|
|
340
|
-
| `dsn` or `sql` | a connection string or a ready postgres.js pool |
|
|
341
|
-
| `schema` | the full installation name: `'v2.shop'` (it is handy to take `r.schema`) |
|
|
342
|
-
| `token`, `apiKey` | a session token or an API key (`service` is a synonym of `apiKey`) |
|
|
343
|
-
| `models` | `letopis/model` models, only for chain types ([section 4](#4-data-model)) |
|
|
344
|
-
| `max` | pool size (10 by default) |
|
|
345
|
-
| `listen` | listen to database signals (yes by default; `false` for PgBouncer in transaction mode) |
|
|
346
|
-
| `cache` | result cache: `{ ttl, max, ttlOnly }` or `false` ([section 12](#12-change-subscription-and-cache)) |
|
|
347
|
-
| `maintain` | run `maintain()` on a schedule: `'1h'`, `'15m'` or milliseconds |
|
|
348
|
-
| `onQuery`, `slowMs` | a hook on every chain query and the slow query threshold |
|
|
349
|
-
| `refreshSession` | extend the session once per minute of activity (yes by default) |
|
|
350
|
-
| `notifyIntervalMs` | the interval for sending deferred signals (100 ms) |
|
|
351
|
-
| `allowBypassRls`, `owner` | allow a role that bypasses RLS (admin connection); the name of the owner role |
|
|
352
|
-
|
|
353
|
-
A connection has `db.schema`, `db.token`, `db.tenant` (the session tenant) and `db.close()`. Connections obtained
|
|
354
|
-
through `as()` share the pool with the original one: close the original one.
|
|
355
|
-
|
|
356
|
-
### Admin connection
|
|
357
|
-
|
|
358
|
-
The loader, projectors and `rebuildAggregates` need a connection of a role that can become the owner, with
|
|
359
|
-
`allowBypassRls: true`. `verify()` and
|
|
360
|
-
`maintain()` work on it too, and also for a system administrator on a regular connection. The admin connection prints
|
|
361
|
-
the warning "RLS не действует" ("RLS does not apply"); this is expected. The session sets the tenant it works with. A
|
|
362
|
-
separate loading process is better off with a service key:
|
|
363
|
-
`connect({ dsn: ADMIN, schema, apiKey, allowBypassRls: true })`, and the tenant of the rows is set by the
|
|
364
|
-
`load(…, { tenant })` option. The example uses the session from `up()`:
|
|
365
|
-
|
|
366
|
-
<!-- run: guide -->
|
|
367
|
-
```ts
|
|
368
|
-
const adm = await connect({ dsn: ADMIN, schema: r.schema, token: r.tenant!.token!, allowBypassRls: true })
|
|
369
|
-
```
|
|
370
|
-
|
|
371
|
-
### Upgrading the package and schema
|
|
372
|
-
|
|
373
|
-
The schema has an engine revision (currently 7). If a new package version brings a new revision, `connect()` only warns
|
|
374
|
-
and names the command. It is `up()` that refuses: a lagging schema without `upgrade: true`, and a schema newer than the
|
|
375
|
-
package. To a schema newer than the package (a newer package has already upgraded it, and old application instances
|
|
376
|
-
are still running) `connect()` connects anyway and warns that the letopis package should be updated. So a startup script that calls `up()` on every start will stop after a package update until it gets
|
|
377
|
-
`upgrade: true`. `up({ dsn, schema, upgrade: true })` applies the migrations from `sql/migrate/` and re-applies the
|
|
378
|
-
functions; the data is kept. Revision 7 (1.4.1) adds the foreign key `secret.account`: migration `0007` first erases
|
|
379
|
-
the `secret` rows whose accounts are already gone (the credentials of accounts deleted earlier were kept with the
|
|
380
|
-
`deleted_at` mark), and then sets the key.
|
|
381
|
-
|
|
382
|
-
The revision does not change with every version: 1.1–1.4.0 share one (6), while their functions differ. So `up()` also writes
|
|
383
|
-
an engine stamp into the schema comment: the package version and a hash of its SQL files. If the stamp does not match
|
|
384
|
-
the package files, `connect()` warns. When the schema was applied by an older version (or by a version before 1.4,
|
|
385
|
-
which has no stamp), it suggests `up({ schema })`. When a newer version has already upgraded the schema, it suggests
|
|
386
|
-
updating the package: `up()` from the older version would bring the old functions back. A new package version without
|
|
387
|
-
SQL changes keeps the stamp, and there is no warning.
|
|
388
|
-
|
|
389
|
-
The first `up()` after moving to 1.2 builds the journal index `log_class_tenant_at (class, tenant, at)` instead of
|
|
390
|
-
`log_class_at`: an `asOf` and `withDeleted` slice by class no longer reads the journal of other tenants with the same
|
|
391
|
-
class name. While the index is being built, writes to the journal wait. On a large installation, build it in advance
|
|
392
|
-
without stopping writes: `create index concurrently log_class_tenant_at on "v2.<schema>".log (class, tenant, at)` as
|
|
393
|
-
the owner role — then `up()` only drops the old one.
|
|
394
|
-
|
|
395
|
-
The `upgrade` test checks upgrading 1.1.0 and 1.0.1 schemas. The old package installs the schema and the old engine
|
|
396
|
-
writes the data, then the current version's `up()` runs. After it, the versions and hashes written by the old engine
|
|
397
|
-
are in place, `verify({ data: true })` is clean, and the schema catalog matches a clean installation. Both schemas are
|
|
398
|
-
upgraded with `up({ upgrade: true })`: 1.1.0 (revision 6) with migration `0007`, 1.0.1 (revision 5) with migrations
|
|
399
|
-
`0006` and `0007`.
|
|
400
|
-
|
|
401
|
-
### Development without PostgreSQL: PGlite
|
|
402
|
-
|
|
403
|
-
`up({ pglite: true })` starts PGlite (PostgreSQL 18 on WebAssembly) with pgcrypto and a socket on a free port. It needs
|
|
404
|
-
the `@electric-sql/pglite` and `@electric-sql/pglite-socket` packages. There is only one connection, so:
|
|
405
|
-
|
|
406
|
-
```ts
|
|
407
|
-
const dev = await up({ schema: 'dev', pglite: true })
|
|
408
|
-
const local = await connect({ dsn: dev.pglite!.dsn, schema: dev.schema, apiKey: dev.serviceKey!,
|
|
409
|
-
listen: false, max: 1, allowBypassRls: true })
|
|
410
|
-
// … work …
|
|
411
|
-
await local.close()
|
|
412
|
-
await dev.pglite!.close()
|
|
413
|
-
```
|
|
414
|
-
|
|
415
|
-
The connection to PGlite is a superuser, so it needs `allowBypassRls: true`, and RLS does not apply on it: tenant
|
|
416
|
-
isolation and permissions cannot be checked on PGlite. It is not fit for race tests and CI either: there is one
|
|
417
|
-
connection and no `LISTEN`.
|
|
418
|
-
|
|
419
|
-
## 4. Data model
|
|
420
|
-
|
|
421
|
-
Classes are described with Zod models from the `letopis/model` subpath and written to the database. The database
|
|
422
|
-
stores a normalized JSON Schema and checks the data itself on every write.
|
|
423
|
-
|
|
424
|
-
### Hubs and links
|
|
425
|
-
|
|
426
|
-
`hub()` describes an object, `link()` describes a link with its own data. Both have two forms:
|
|
427
|
-
|
|
428
|
-
- short: `hub(name, fields, options)`: fields as an object of Zod types, options as the third argument;
|
|
429
|
-
- full: `hub(name, { fields, ends, key, extends, … })`: it is needed, for example, for inheritance (`extends`), so
|
|
430
|
-
that TypeScript infers the data types of the descendant.
|
|
431
|
-
|
|
432
|
-
The shop model that the following examples are built on:
|
|
433
|
-
|
|
434
|
-
<!-- run: guide -->
|
|
435
|
-
```ts
|
|
436
|
-
import { z } from 'zod'
|
|
437
|
-
import { hub, link, one, many } from 'letopis/model'
|
|
438
|
-
import { syncModels } from 'letopis'
|
|
439
|
-
|
|
440
|
-
const Shop = hub('Shop', { name: z.string(), city: z.string().optional() }, { key: ['name'] })
|
|
441
|
-
const Category = hub('Category', { name: z.string() }, {
|
|
442
|
-
key: ['name'],
|
|
443
|
-
ends: { Parent: one('Category').optional().onDelete('unset') }, // tree: a reference to the parent
|
|
444
|
-
})
|
|
445
|
-
const Product = hub('Product', {
|
|
446
|
-
sku: z.string(), title: z.string(), price: z.number().int().min(0), views: z.number().int().optional(),
|
|
447
|
-
}, {
|
|
448
|
-
ends: { Shop, Category: many(Category, { max: 5, onDelete: 'unset' }) },
|
|
449
|
-
key: ['Shop', 'sku'],
|
|
450
|
-
meta: { title: 'Product', icon: 'mdi:package-variant' },
|
|
451
|
-
})
|
|
452
|
-
const Customer = hub('Customer', { name: z.string(), email: z.string() }, {}) // no key: the database assigns the id
|
|
453
|
-
const Vip = hub('Vip', { extends: Customer, fields: { discount: z.number().int().min(0).max(50) } })
|
|
454
|
-
const Order = hub('Order', {
|
|
455
|
-
number: z.string(), status: z.enum(['new', 'paid', 'cancelled']).default('new'),
|
|
456
|
-
}, { ends: { Customer, Shop }, key: ['Shop', 'number'] })
|
|
457
|
-
const line = link('line', { qty: z.number().int().min(1), price: z.number().int().min(0) }, {
|
|
458
|
-
ends: { Order: one(Order).onDelete('cascade'), Product },
|
|
459
|
-
key: ['Order', 'Product'],
|
|
460
|
-
})
|
|
461
|
-
const Wallet = hub('Wallet', { number: z.string(), balance: z.number().int().min(0).default(0) },
|
|
462
|
-
{ ends: { Customer }, key: ['number'] })
|
|
463
|
-
const Movement = link('Movement', {
|
|
464
|
-
opId: z.string(), kind: z.enum(['topup', 'debit', 'adjust']), amount: z.number().int(), target: z.number().int().optional(),
|
|
465
|
-
}, { ends: { Wallet }, key: ['Wallet', 'opId'] })
|
|
466
|
-
|
|
467
|
-
const models = [Shop, Category, Product, Customer, Vip, Order, line, Wallet, Movement]
|
|
468
|
-
const plan = await syncModels(db, models) // dry run: what will be created and changed
|
|
469
|
-
const applied = await syncModels(db, models, { apply: true }) // { applied: true, created: ['Shop', …], … }
|
|
470
|
-
```
|
|
471
|
-
|
|
472
|
-
### Fields
|
|
473
|
-
|
|
474
|
-
Fields are Zod types: strings, numbers (`int`, `min`, `max`), booleans, enums (`z.enum`), dates and times
|
|
475
|
-
(`z.iso.date()`, `z.iso.datetime()`), nested objects of any depth, arrays and `z.record`. `.optional()` makes a field
|
|
476
|
-
optional. `.default(…)` sets a default that the database fills in on insert. The database stores the `.meta()`
|
|
477
|
-
annotations of a field (`title`, `description`, your own keys), but they do not affect data checks.
|
|
478
|
-
|
|
479
|
-
The `refine`, `superRefine` and `transform` rules cannot be expressed in JSON Schema, and the database does not run
|
|
480
|
-
them. `syncModels` returns them in `lost`: check such rules in the application. A description with a JSON Schema
|
|
481
|
-
keyword that the database does not know is rejected, so no rule is lost silently. `pattern` regexes are a limited
|
|
482
|
-
subset of regular expressions shared by Zod, the validator and the database.
|
|
483
|
-
|
|
484
|
-
### Ends
|
|
485
|
-
|
|
486
|
-
- `ends: { Shop }` is the same as `one(Shop)`: a single required end with the role `Shop`.
|
|
487
|
-
- `one(A, B)` is a union: the end accepts rows of several classes. `one('Category')` is a target by name: this is how
|
|
488
|
-
a class refers to itself.
|
|
489
|
-
- `.optional()` makes the end optional.
|
|
490
|
-
- `many(X, { min, max, onDelete })` is a multiple end: a list of targets in one end.
|
|
491
|
-
- `.onDelete(…)` sets what happens when a target is deleted: `restrict` (the default: deleting the target is
|
|
492
|
-
forbidden), `cascade` (delete this row too), `unset` (remove the reference; only for an optional single end and for a
|
|
493
|
-
multiple end without `min`).
|
|
494
|
-
|
|
495
|
-
The database checks the targets: the target exists, belongs to the same tenant or to System, and is of the declared
|
|
496
|
-
class or its descendant. A role name points to one class across the whole schema. So the roles `User` and `Tenant`
|
|
497
|
-
(pointing to `Account`), `Group` and `Object` (pointing to `Resource`) are taken by system classes: your own end with
|
|
498
|
-
such a name is possible only with the same target. In a result row, a single end is the target id (`row.links.Shop`),
|
|
499
|
-
and a multiple end is a list of ids.
|
|
500
|
-
|
|
501
|
-
### Key and id
|
|
502
|
-
|
|
503
|
-
A key is made of required string fields and required single ends. Numbers, booleans, objects, optional fields and
|
|
504
|
-
multiple ends cannot be part of a key. The id of a class with a key is a version 5 UUID of the tenant, the class and
|
|
505
|
-
the key values. The values are taken as they are, without normalization: an email in a different case gives a
|
|
506
|
-
different id. Set the required format of a key field in the model, and the database will check it.
|
|
507
|
-
|
|
508
|
-
- You do not pass the id: the database computes it. A different id gives `id_mismatch`; an id for a class without a
|
|
509
|
-
key gives `id_from_db`.
|
|
510
|
-
- `db.idOf(class, key)` computes the id in advance, for example `db.idOf('Shop', { name: 'Chamomile' })`.
|
|
511
|
-
- A class without a key (`Customer` above) gets an id from a random part; you can learn it from the write result.
|
|
512
|
-
- A key field cannot be changed by a regular update (`immutable_key`): a new key means a new id, and that is `rekey`
|
|
513
|
-
([section 5](#5-writing)).
|
|
514
|
-
|
|
515
|
-
If something other than the key must be unique, make it a separate class. For example, the email is the key of an
|
|
516
|
-
`Email` hub, and the link from a person to it has the key `['Email']`. It is better not to make personal data the key
|
|
517
|
-
of the person itself: an id from an email can be found by brute force, and it stays in references and in the journal.
|
|
518
|
-
|
|
519
|
-
### Inheritance and aliases
|
|
520
|
-
|
|
521
|
-
`extends`: the descendant gets the fields, ends and key of the ancestor. Its own field with the same name replaces the
|
|
522
|
-
ancestor's field entirely. `abstract: true`: the class never has its own rows. A chain step by class also returns the
|
|
523
|
-
rows of descendants: `db.Customer()` also finds `Vip`, and `.exact()` keeps only the class itself. `alias` is a second
|
|
524
|
-
name of the class for a chain step, for example a name in another language.
|
|
525
|
-
|
|
526
|
-
### Class settings
|
|
527
|
-
|
|
528
|
-
- `history: false`: no journal is kept for the class: no versions, no point-in-time reads, no restore and no
|
|
529
|
-
subscription events. A retention policy is `history: { all: '1 day', daily: '3 months', … }`
|
|
530
|
-
([section 11](#11-history-and-point-in-time-reads)). Only someone with the `READ` and `DELETE` permissions without
|
|
531
|
-
conditions on all rows of the class and its descendants may change `history` of an existing class: `maintain()` thins
|
|
532
|
-
the journal by the policy, and `false` turns it off.
|
|
533
|
-
- `ownerDefault: 'actor'`: the owner of new rows is the user, not the tenant (the default is `'tenant'`). This is
|
|
534
|
-
useful for the rule "everyone sees their own records".
|
|
535
|
-
- `cache: { ttl }`: reads of this class alone are cached without `.cache()` ([section 12](#12-change-subscription-and-cache)).
|
|
536
|
-
- `meta`: free-form metadata for the application: a title, an icon, a color, an order in a menu. Up to 64 KB,
|
|
537
|
-
inherited key by key (`null` removes an inherited value), visible to anyone who has a session. For an end, use
|
|
538
|
-
`.meta()`.
|
|
539
|
-
- `derivedBy: 'projector-name'`: a derived class: only this projector writes its rows ([Projectors](#projectors)).
|
|
540
|
-
The name is lowercase Latin letters, digits and `_.:-` (the first character is a letter or a digit), up to 64
|
|
541
|
-
characters. The option is inherited, and a descendant does not change it. The ends of a derived class are only
|
|
542
|
-
`onDelete: 'cascade'` or `'unset'`: a derived row does not prevent deleting its target. Nothing refers to the rows
|
|
543
|
-
of a derived class. The application, raw SQL and the loader do not write to such a class: writing, editing and
|
|
544
|
-
deleting its rows gives `derived_class`. The derived class itself (own or inherited) is created, edited and deleted
|
|
545
|
-
only by an administrator — an admin connection or an account with the System permission `letopis.projector`; for an
|
|
546
|
-
application session it is `acl_denied` ([Projectors](#projectors)).
|
|
547
|
-
- `aggregate: { field: { sum | count, of, via, where } }`: an aggregate in the database: the database itself keeps
|
|
548
|
-
the sum or count of source rows ([Aggregates in the database](#aggregates-in-the-database)).
|
|
549
|
-
|
|
550
|
-
### Aggregates in the database
|
|
551
|
-
|
|
552
|
-
An aggregate is a field that the database itself maintains: the sum of a field or the number of live rows of a
|
|
553
|
-
source class whose end leads to the row. It is needed where the total must match the source even for raw SQL: a
|
|
554
|
-
wallet balance, the number of order lines.
|
|
555
|
-
|
|
556
|
-
```ts
|
|
557
|
-
const Wallet = hub('Wallet', { number: z.string(), balance: z.number().int().min(0), deposits: z.number().int().optional() }, {
|
|
558
|
-
key: ['number'],
|
|
559
|
-
aggregate: {
|
|
560
|
-
balance: { sum: 'amount', of: 'Movement', via: 'Wallet' }, // sum of movements
|
|
561
|
-
deposits: { count: true, of: 'Movement', via: 'Wallet', where: { properties: { amount: { minimum: 1 } } } }, // number of top-ups
|
|
562
|
-
},
|
|
563
|
-
})
|
|
564
|
-
const Movement = link('Movement', { amount: z.number().int() }, { ends: { Wallet } })
|
|
565
|
-
```
|
|
566
|
-
|
|
567
|
-
- `sum: 'field'` is the sum of a numeric field of the source, `count: true` is the number of rows. `of` is the source
|
|
568
|
-
class (a name or a model) together with its descendants, `via` is its single end that leads to the target row.
|
|
569
|
-
`where` is a JSON Schema over the source data: rows that do not pass it are not counted.
|
|
570
|
-
- Only live rows of the same tenant as the target are counted. A reference to a System row is not counted: a shared
|
|
571
|
-
directory does not count the data of all tenants. An aggregate of a tenant class, including one inherited from a
|
|
572
|
-
System class, counts only classes of its own tenant: a System class named `of` is not counted, while the tenant's
|
|
573
|
-
own descendant of a System class is. An aggregate of a System class counts System classes and the classes of the
|
|
574
|
-
target row's tenant.
|
|
575
|
-
- A trigger recalculates the field after every statement: writes by the library, raw SQL, a delete cascade,
|
|
576
|
-
`restore`, `reclass`, `rekey`, the loader, account deletion. The operations of one statement add up to one increment
|
|
577
|
-
of the target, and the class rule (`min(0)`) is checked on the total: an operation that would take the balance below
|
|
578
|
-
zero is rejected with `invalid_data`. Every change of the target is its new version with the author and reason of
|
|
579
|
-
the operation.
|
|
580
|
-
- Only the database writes the field. A value in `create`, `update`, `upsert`, `rekey`, `reclass` or raw SQL is an
|
|
581
|
-
`aggregate_field` error. A new row gets the total of already existing sources, usually 0. Replacing the data as a
|
|
582
|
-
whole without the field keeps the previous value. The loader does not write a passed value. No permission to write
|
|
583
|
-
the target is needed: the permission to write the source is enough.
|
|
584
|
-
- The database checks the description (`invalid_class`): the field is in the schema, it is numeric and not part of the
|
|
585
|
-
key; the `via` end is single and accepts the target class; the `sum` field is a number; aggregates do not form a
|
|
586
|
-
cycle; the target and the source are not system or derived classes. A source class is described before the
|
|
587
|
-
aggregate: an aggregate over a class that does not exist yet is refused. For a tenant class, the source is looked up
|
|
588
|
-
among the tenant's classes. `syncModels` describes sources before targets itself, and models in a circle (an
|
|
589
|
-
aggregate over the class's own descendant, a cycle of aggregates) it refuses before writing (`invalid_class` naming
|
|
590
|
-
the circle). The `of` class is not deleted while an aggregate of a class of the same tenant names it: first the
|
|
591
|
-
aggregate is removed from the target's description.
|
|
592
|
-
- Rows get into an aggregate only with the knowledge of someone who reads them all: the field also counts rows that
|
|
593
|
-
the author cannot see. Only someone who reads all rows of the source and its descendants without conditions can
|
|
594
|
-
describe or change an aggregate. Only someone who reads all rows of a class and its descendants without conditions,
|
|
595
|
-
both before the change and after it, can make the class a source (a new descendant of a source, a `parent` change).
|
|
596
|
-
Otherwise `acl_denied`.
|
|
597
|
-
- A new source is also checked against whoever described the aggregate: they too must read all rows of the class
|
|
598
|
-
without conditions. The describer is the author of the version of the target's `Class` row from which the aggregate
|
|
599
|
-
description became the current one. If they cannot read those rows, the class does not become a source
|
|
600
|
-
(`acl_denied`) until someone who can read them describes the aggregate again. For an aggregate of a System class, the
|
|
601
|
-
describer is checked against System classes, as at description time. A tenant class under a System aggregate is
|
|
602
|
-
checked only for whoever creates it.
|
|
603
|
-
- The option is inherited, and a descendant replaces it as a whole. Adding or changing an aggregate is a tightening:
|
|
604
|
-
writes to the target and to the sources wait while the field is recalculated for all rows.
|
|
605
|
-
- `db.rebuildAggregates(class)` recalculates the field from the current rows, for repair. A version is written only
|
|
606
|
-
where the value has changed. `db.verify({ data: true })` shows discrepancies. It needs an admin connection.
|
|
607
|
-
- A raw `INSERT … ON CONFLICT DO UPDATE SET data = excluded.data` for a target is rejected: the insert trigger has
|
|
608
|
-
already set the field in `excluded.data`. Write `excluded.data - array['balance']`, and the field stays as it was;
|
|
609
|
-
the library's `upsert` does this itself.
|
|
610
|
-
|
|
611
|
-
### Writing the model to the database
|
|
612
|
-
|
|
613
|
-
`syncModels(db, models)` without `apply` is a dry run; with `{ apply: true }` it applies the changes. The result is
|
|
614
|
-
`{ applied, created, updated, unchanged, skipped, lost }`; `skipped` lists System classes, which only the installation
|
|
615
|
-
changes. `syncModels` touches only the classes passed to it and deletes nothing. Safe changes (a new optional field or
|
|
616
|
-
end, metadata, an alias) are applied at once. Anything else, including any change to the `anyOf` and `oneOf` conditions, is a
|
|
617
|
-
tightening: it checks all rows of the family; if some rows conflict with it, you get
|
|
618
|
-
`tightening_conflict` with a list. Classes are written to the session tenant, and other tenants do not see them.
|
|
619
|
-
Classes shared by everyone are in System: the installation seed or the import puts them there.
|
|
620
|
-
|
|
621
|
-
The same from the command line:
|
|
622
|
-
|
|
623
|
-
```sh
|
|
624
|
-
npx letopis sync ./models.js --dsn "$APP" --schema v2.shop --token "$TOKEN" # dry run
|
|
625
|
-
npx letopis sync ./models.js --dsn "$APP" --schema v2.shop --token "$TOKEN" --apply # apply
|
|
626
|
-
npx letopis types --dsn "$APP" --schema v2.shop --token "$TOKEN" --out types.ts # class types from the database
|
|
627
|
-
```
|
|
628
|
-
|
|
629
|
-
### Chain types
|
|
630
|
-
|
|
631
|
-
`connect({ …, models })` gives types of steps, rows and filters from the models. The library still takes the class
|
|
632
|
-
descriptions from the database; only TypeScript needs the models. The row data type is `Infer<typeof Product>`, the
|
|
633
|
-
ends type is `InferLinks<typeof Product>`. Your own `meta` shape is typed by extending the `ClassMeta` interface.
|
|
634
|
-
|
|
635
|
-
### Class names
|
|
636
|
-
|
|
637
|
-
A chain step is named after a class, its alias or a role name. Where a chain or root method exists, the library parses
|
|
638
|
-
its name (`count`, `sort`, `set`, `sql`, `auth` and others) before classes: a class `count` is reachable as the first
|
|
639
|
-
step `db.count(…)`, but not after another step. The database does not accept a link with such a name. Do not give
|
|
640
|
-
classes these names; the full list is in the
|
|
641
|
-
["Reserved names"](#reserved-names) section.
|
|
642
|
-
|
|
643
|
-
## 5. Writing
|
|
644
|
-
|
|
645
|
-
The write verbs — `create`, `update`, `upsert`, `delete`, `anonymize`, `reclass`, `rekey`, `restore`, `purge` — are
|
|
646
|
-
parts of a plan. A verb does nothing by itself: a terminal (`rows`, `first`, `ids`, `count`) executes the plan, and the
|
|
647
|
-
whole plan is one transaction.
|
|
648
|
-
|
|
649
|
-
### Create: `create`
|
|
650
|
-
|
|
651
|
-
<!-- run: guide -->
|
|
652
|
-
```ts
|
|
653
|
-
import { LetopisError } from 'letopis'
|
|
654
|
-
|
|
655
|
-
const [shop] = await db.Shop().create({ name: 'Chamomile', city: 'Kazan' }).rows()
|
|
656
|
-
const [tea] = await db.Category().create({ name: 'Tea' }).rows()
|
|
657
|
-
const [green] = await db.Category().create({ name: 'Green tea' }).Parent.set(tea).rows()
|
|
658
|
-
const [kettle] = await db.Shop(shop).Product().create({ sku: 'K-1', title: 'Tea kettle', price: 2500 }).rows()
|
|
659
|
-
const [sencha] = await db.Shop(shop).Product()
|
|
660
|
-
.create({ sku: 'T-7', title: 'Sencha', price: 900 }).Category.add(green).rows()
|
|
661
|
-
const [olga] = await db.Customer().create({ name: 'Olga', email: 'olga@mail.ru' }).rows()
|
|
662
|
-
|
|
663
|
-
try {
|
|
664
|
-
await db.Shop().create({ name: 'Chamomile' }).rows()
|
|
665
|
-
} catch (e) {
|
|
666
|
-
if (!(e instanceof LetopisError) || e.code !== 'exists') throw e // e.detail.id === shop.id: this shop already exists
|
|
667
|
-
}
|
|
668
|
-
```
|
|
669
|
-
|
|
670
|
-
- `create` only creates. A taken key gives the error `exists` with `detail.id`: the library computes the id itself. So
|
|
671
|
-
for a class with a key, a retry after a network failure shows at once that the first attempt went through. A class
|
|
672
|
-
without a key has no such protection: a retry creates a second object.
|
|
673
|
-
- The tags of a new row are set by `.tags([…])` before the verb, the owner by `.owner(id)`; an explicit owner must be
|
|
674
|
-
allowed by the permissions.
|
|
675
|
-
- A deleted object can be created again with the same key: its history continues.
|
|
676
|
-
- The result is the rows of the created objects with all columns ([section 15](#15-api-reference), "Result row").
|
|
677
|
-
|
|
678
|
-
### Update: `update`
|
|
679
|
-
|
|
680
|
-
<!-- run: guide -->
|
|
681
|
-
```ts
|
|
682
|
-
import { gte } from 'letopis'
|
|
683
|
-
|
|
684
|
-
await db.Product(kettle).update({ title: 'Glass tea kettle' }).rows()
|
|
685
|
-
await db.Shop(shop).Product({ price: gte(2000) }).update({ title: 'Glass tea kettle, 1 L' }).rows()
|
|
686
|
-
```
|
|
687
|
-
|
|
688
|
-
`update` edits every target of the step and never creates. The patch is merged with the data inside the statement:
|
|
689
|
-
nested objects are merged recursively, while arrays, scalars and `null` are replaced. So concurrent edits of different
|
|
690
|
-
fields are not lost. If the step has no filter of its own, the target is taken from `data.id` of the patch, and without
|
|
691
|
-
`data.id` all entities of the step are edited: `db.Product().update(…)` edits all products of the tenant. If the step
|
|
692
|
-
found nothing, the result is empty and there is no error, so check the length of the result. If `update` affected fewer
|
|
693
|
-
rows than the step found, the whole plan is rolled back: a row deleted in the meantime gives `target_not_found`, and a
|
|
694
|
-
row that is visible but may not be edited gives `acl_denied`. The `.tags(…)` modifier before `update` selects the rows;
|
|
695
|
-
a string or a list also replaces their tags entirely, while an operator (`has`, `hasAny`, `hasAll`) only selects — the
|
|
696
|
-
tags stay as they are.
|
|
697
|
-
|
|
698
|
-
### Create or replace: `upsert`
|
|
699
|
-
|
|
700
|
-
<!-- run: guide -->
|
|
701
|
-
```ts
|
|
702
|
-
const [kept] = await db.Shop().upsert({ name: 'Chamomile', city: 'Kazan' }).rows() // kept.$upsert: 'unchanged'
|
|
703
|
-
const [moved] = await db.Shop().upsert({ name: 'Chamomile', city: 'Ufa' }).rows() // moved.$upsert: 'updated'
|
|
704
|
-
```
|
|
705
|
-
|
|
706
|
-
`upsert` creates an object, and if an object with this key already exists, it replaces its data, ends and tags
|
|
707
|
-
entirely. This is one `INSERT … ON CONFLICT` statement, so the race "checked that it does not exist, then created it"
|
|
708
|
-
cannot happen. Rules:
|
|
709
|
-
|
|
710
|
-
- only for a class with a key, otherwise `no_key` without a call to the database;
|
|
711
|
-
- the object is passed whole: fields that are not in the passed object disappear;
|
|
712
|
-
- the result carries `$upsert`: `created`, `updated` or `unchanged`;
|
|
713
|
-
- `upsert` cannot increment and overwrites concurrent increments, so it is not suitable for money;
|
|
714
|
-
- to sync large reference tables, the loader with `existing: 'update'` is cheaper ([section 9](#9-bulk-loading)).
|
|
715
|
-
|
|
716
|
-
### Ends: from the path and slots
|
|
717
|
-
|
|
718
|
-
The ends of a created row are taken from the chain path: `db.Shop(shop).Product().create(…)` references `shop`. If
|
|
719
|
-
several roles fit, the role whose slot is free is chosen, then the role named after the step class; otherwise you get
|
|
720
|
-
`invalid_query`. A slot after the verb sets an end explicitly: `.RoleName.set(value)` and `.RoleName.unset()`, and for
|
|
721
|
-
a multiple end also `.add()` and `.remove()`. The value is an id, a list of ids, a result row or a nested chain. Slots
|
|
722
|
-
change `links` atomically, without reading the list:
|
|
723
|
-
|
|
724
|
-
<!-- run: guide -->
|
|
725
|
-
```ts
|
|
726
|
-
await db.Product(sencha).update({}).Category.add(tea).rows() // one more category
|
|
727
|
-
await db.Product(sencha).update({}).Category.remove(tea).rows()
|
|
728
|
-
await db.Category(green).update({}).Parent.unset().rows() // the category is now a root
|
|
729
|
-
await db.Category(green).update({}).Parent.set(tea).rows() // and inside "Tea" again
|
|
730
|
-
```
|
|
731
|
-
|
|
732
|
-
### Numbers and counters: `inc`
|
|
733
|
-
|
|
734
|
-
<!-- run: guide -->
|
|
735
|
-
```ts
|
|
736
|
-
import { inc } from 'letopis'
|
|
737
|
-
|
|
738
|
-
await db.Product(kettle).update({ price: inc(-500) }).rows() // the price went down by 500
|
|
739
|
-
await db.Product(kettle).update({ views: inc(1, { start: 0 }) }).rows() // the field does not exist yet: start from 0
|
|
740
|
-
```
|
|
741
|
-
|
|
742
|
-
`inc(n)` adds the amount inside the `UPDATE` statement, over the fresh locked row. So concurrent increments are not
|
|
743
|
-
lost, and a class rule like `min(0)` is checked against the result: going below zero is rejected with `invalid_data`,
|
|
744
|
-
without reads or locks in the application. Rules:
|
|
745
|
-
|
|
746
|
-
- only in `update`; a marker in `create`, `upsert`, the loader or inside an array gives `invalid_data` before sending;
|
|
747
|
-
- the field must already be a number; if it is missing, you get `invalid_data`, or `start` is used. `null` and a
|
|
748
|
-
non-number are an error even with `start`;
|
|
749
|
-
- the path goes only through existing objects;
|
|
750
|
-
- an `inc` marker cannot come from JSON: `{ "$inc": 5 }` that came from the network stays data;
|
|
751
|
-
- store money as an integer number of minor units (cents), no greater than 2^53 − 1.
|
|
752
|
-
|
|
753
|
-
A counter that must match the source even for raw SQL is easier to keep as an aggregate
|
|
754
|
-
([Aggregates in the database](#aggregates-in-the-database)): the database recalculates it itself, and you cannot forget
|
|
755
|
-
an `inc`. An aggregate and `inc` cost the same: operations on one object go one after another under the lock of its
|
|
756
|
-
row, and each writes its version. On the benchmark machine, one object took 150–190 operations per second this way, no
|
|
757
|
-
matter how many writers were waiting for it ([section 17](#17-performance)). For hotter counters, use a projector that
|
|
758
|
-
folds a batch ([Projectors](#projectors)): the total lags behind the source, but writers do not wait for each other.
|
|
759
|
-
|
|
760
|
-
### Protection against concurrent edits: `{ rev }`
|
|
761
|
-
|
|
762
|
-
<!-- run: guide -->
|
|
763
|
-
```ts
|
|
764
|
-
const cur = await db.Product(kettle).first()
|
|
765
|
-
await db.Product(kettle).update({ price: 1900 }, { rev: cur!.rev }).rows() // the version matched: written
|
|
766
|
-
try {
|
|
767
|
-
await db.Product(kettle).update({ price: 1800 }, { rev: cur!.rev }).rows() // the version is already different
|
|
768
|
-
} catch (e) {
|
|
769
|
-
if (!(e instanceof LetopisError) || e.code !== 'conflict') throw e // re-read and decide again
|
|
770
|
-
}
|
|
771
|
-
```
|
|
772
|
-
|
|
773
|
-
The strict mode is needed when the new value is computed from what was read: if the row was changed in the meantime,
|
|
774
|
-
you get `conflict`, and nothing is written. The mode works when the step has one target (otherwise `rev_ambiguous`).
|
|
775
|
-
The library does not retry `conflict` by itself: the application decides what to do with the changed data. Another way
|
|
776
|
-
is a read with a `forUpdate()` lock ([section 8](#8-transactions-batches-and-money)).
|
|
777
|
-
|
|
778
|
-
### Changing the key or class
|
|
779
|
-
|
|
780
|
-
<!-- run: guide -->
|
|
781
|
-
```ts
|
|
782
|
-
const [mug] = await db.Shop(shop).Product().create({ sku: 'M-1', title: 'Mug', price: 400 }).rows()
|
|
783
|
-
const [mug2] = await db.Product(mug).rekey({ sku: 'M-100' }).rows() // new key, new id; references moved
|
|
784
|
-
const [vip] = await db.Customer(olga).reclass('Vip', { name: 'Olga', email: 'olga@mail.ru', discount: 10 }).rows()
|
|
785
|
-
```
|
|
786
|
-
|
|
787
|
-
- `rekey(patch)` sets a new key value: a row with a new id, all references moved, the old row deleted. If a referencing
|
|
788
|
-
row includes this end in its own key, its id changes too, and the move goes on. The journal links the old and the new
|
|
789
|
-
id (`moved`).
|
|
790
|
-
- `db.rekeyClass(class, key)` sets a new key for all rows of the family, in one transaction. For a derived class and
|
|
791
|
-
for a class with a derived descendant — `derived_class`: only its projector writes the rows of a derived class.
|
|
792
|
-
- `reclass(class, data)` changes the class; the data is optional, and if given, it is given whole. The row leaves its
|
|
793
|
-
previous class, so it needs the `DELETE` permission on it, and `WRITE` on the new class and on every row whose
|
|
794
|
-
references the move rewrites, at any depth. If the new class has no key, the id is kept (as for `vip` above); if
|
|
795
|
-
it has one, the id is computed anew and the references are moved.
|
|
796
|
-
The roles of referencing rows must accept the new class, otherwise `reclass_denied`. A raw `UPDATE … SET class`
|
|
797
|
-
passes only if neither the old nor the new class has a key (otherwise `use_reclass`).
|
|
798
|
-
- Deleting, changing the class and changing the key work only in `read committed`: library transactions always open at
|
|
799
|
-
this level.
|
|
800
|
-
|
|
801
|
-
### Anonymize: `anonymize`
|
|
802
|
-
|
|
803
|
-
<!-- run: guide -->
|
|
804
|
-
```ts
|
|
805
|
-
await db.Customer(olga).anonymize(['email']).rows() // email → '[erased]', tag anonymized
|
|
806
|
-
```
|
|
807
|
-
|
|
808
|
-
`anonymize(fields)` overwrites string fields with a new version and marks the row with the `anonymized` tag. The
|
|
809
|
-
earlier versions stay in the journal. For complete erasure, the next steps are history trimming
|
|
810
|
-
([section 11](#11-history-and-point-in-time-reads)), then deletion and `purge` ([section 7](#7-deleting-and-restoring)).
|
|
811
|
-
|
|
812
|
-
### How a plan runs
|
|
813
|
-
|
|
814
|
-
One plan can create several related objects. The chain after a verb continues from the verb's result: after `create`
|
|
815
|
-
and `upsert`, from the created row; after other verbs, from all rows.
|
|
816
|
-
|
|
817
|
-
<!-- run: guide -->
|
|
818
|
-
```ts
|
|
819
|
-
const [l1] = await db.Customer(olga).Order()
|
|
820
|
-
.create({ number: 'A-1' }).Shop.set(shop) // order: Customer from the path, Shop via a slot
|
|
821
|
-
.line().create({ qty: 2, price: 2000 }).Product.set(kettle) // order line: Order from the path
|
|
822
|
-
.rows() // rows of the last step: the order line
|
|
823
|
-
const order = l1.links.Order as string
|
|
824
|
-
await db.Order(order).line().create({ qty: 1, price: 900 }).Product.set(sencha).rows()
|
|
825
|
-
```
|
|
826
|
-
|
|
827
|
-
- The whole plan is one `read committed` transaction. On a deadlock (`40P01`), a serialization failure (`40001`) and a
|
|
828
|
-
journal conflict, the library retries the whole plan, up to three times.
|
|
829
|
-
- An error in any part rolls back the whole plan. `exists` and `conflict` are not retried: they are answers for the
|
|
830
|
-
application.
|
|
831
|
-
- Modifiers before a verb (`limit`, `offset`, `sort`) narrow its targets.
|
|
832
|
-
- A modifier after a verb applies to what follows it: a class step, `versions()`, `diff()`, `blame()` and aggregates,
|
|
833
|
-
or the targets of the next verb. If a `rows`, `first`, `ids`, `count` or `paths` terminal follows right away, there is
|
|
834
|
-
nothing left to narrow — the verb has gone through all targets of the step — so
|
|
835
|
-
`db.Product().update(…).limit(1).rows()` gives `invalid_query` and writes nothing. In a batch, `run()` refuses the
|
|
836
|
-
same way.
|
|
837
|
-
- A verb right after a verb writes to the same entities.
|
|
838
|
-
- The result rows of a verb come in the order of its targets: by `sort()` before the verb, otherwise by id; `first()`
|
|
839
|
-
after a verb is the first one in that order.
|
|
840
|
-
|
|
841
|
-
## 6. Reading
|
|
842
|
-
|
|
843
|
-
### Steps and hops
|
|
844
|
-
|
|
845
|
-
A chain starts with a class: `db.Shop()` is all shops, `db.Shop(shop)` is one shop. The next step is a class, an alias
|
|
846
|
-
or an end role. A hop goes along ends: a role step goes along its own end; other steps go along the ends of all classes
|
|
847
|
-
of both families. From an object to the rows that reference it, the hop goes backward; to the target of a reference,
|
|
848
|
-
forward; to the same class, to the children.
|
|
849
|
-
|
|
850
|
-
<!-- run: guide -->
|
|
851
|
-
```ts
|
|
852
|
-
const products = await db.Shop(shop).Product().rows() // products of the shop: backward along the Shop end
|
|
853
|
-
const shopOf = await db.Product(kettle).Shop().first() // the shop of the product: forward
|
|
854
|
-
const inOrder = await db.Order(order).line().Product().rows() // products of the order: hub → link → hub
|
|
855
|
-
const parent = await db.Category(green).Parent().first() // role step: the parent category
|
|
856
|
-
const allCustomers = await db.Customer().rows() // together with descendants (Vip)
|
|
857
|
-
const onlyCustomer = await db.Customer().exact().rows() // only the class itself
|
|
858
|
-
```
|
|
859
|
-
|
|
860
|
-
### Filters
|
|
861
|
-
|
|
862
|
-
A step filter is an id, a list of ids, a result row, an object of fields or `or(…)`. An object of fields can have any
|
|
863
|
-
depth. A plain value means equality through containment in `data`, without type casting: `{ price: 900 }` will not find
|
|
864
|
-
a record with the price `"900"`. A list as a field value means "the array contains all the elements". Operators cast
|
|
865
|
-
their value to the field type from the JSON Schema. An end role in a filter takes a target id or `null`.
|
|
866
|
-
|
|
867
|
-
<!-- run: guide -->
|
|
868
|
-
```ts
|
|
869
|
-
import { lt, ilike, between, or, not } from 'letopis'
|
|
870
|
-
|
|
871
|
-
const cheap = await db.Product({ price: lt(1000) }).rows()
|
|
872
|
-
const teaCount = await db.Product({ title: ilike('%tea%'), price: between(500, 3000) }).count()
|
|
873
|
-
const twoIds = await db.Product(or({ sku: 'K-1' }, { sku: 'T-7' })).ids()
|
|
874
|
-
const active = await db.Order({ status: not('cancelled') }).rows()
|
|
875
|
-
const ofShop = await db.Product({ Shop: shop.id }).rows()
|
|
876
|
-
const roots = await db.Category({ Parent: null }).rows() // root categories
|
|
877
|
-
```
|
|
878
|
-
|
|
879
|
-
| Operator | Condition |
|
|
880
|
-
|---|---|
|
|
881
|
-
| `ne(v)`, `gt(v)`, `gte(v)`, `lt(v)`, `lte(v)` | ≠ (taking `null` into account), >, ≥, <, ≤ |
|
|
882
|
-
| `between(a, b)`, `inList([…])` | a ≤ x ≤ b; a value from the list |
|
|
883
|
-
| `like(s)`, `ilike(s)`, `starts(s)`, `ends(s)` | a pattern with `%` and `_`; case-insensitive; starts with; ends with |
|
|
884
|
-
| `has(v)`, `hasAny([…])`, `hasAll([…])` | an array field contains the value; at least one; all (`hasAll([])` matches every row, `hasAny([])` matches none) |
|
|
885
|
-
| `exists(true \| false)`, `isNull()` | the field is present or not; the field is `null` or missing |
|
|
886
|
-
| `not(v)` | negation of an operator; for a scalar, "not equal" |
|
|
887
|
-
| `or(f1, f2, …)` | any of the step filters |
|
|
888
|
-
|
|
889
|
-
Tags and the row owner are filtered with the `.tags(…)` and `.owner(…)` modifiers.
|
|
890
|
-
|
|
891
|
-
### Terminals and the result row
|
|
892
|
-
|
|
893
|
-
| Terminal | What it returns |
|
|
894
|
-
|---|---|
|
|
895
|
-
| `rows()` | the rows of the entities of the last step, without duplicates |
|
|
896
|
-
| `first()` | the first row or `null` |
|
|
897
|
-
| `ids()` | the ids of the entities of the last step |
|
|
898
|
-
| `count()` | the number of entities of the last step; `count({ paths: true })` is the number of paths |
|
|
899
|
-
| `paths()` | paths: `[{ Step: Row, … }]`, a node for each step |
|
|
900
|
-
| `versions()` | all versions of the entities of the last step ([section 11](#11-history-and-point-in-time-reads)) |
|
|
901
|
-
| `sum`, `avg`, `min`, `max`, `countBy` | aggregates over the field `'data.<path>'`; on an empty set, `null`, and `{}` for `countBy` |
|
|
902
|
-
|
|
903
|
-
Result row: `id`, `class`, `rev` (version number), `tenant`, `owner`, `links` (ends), `data` (fields), `tags`, `at`
|
|
904
|
-
(version time), `author`, `op` (kind of operation) and, if present, `agent`, `reason`, `moved`. Service fields start
|
|
905
|
-
with `$`: `$deleted`, `$depth`, `$upsert`, `$action`, `$versions`, `$purged`.
|
|
906
|
-
|
|
907
|
-
### Sorting and pagination
|
|
908
|
-
|
|
909
|
-
<!-- run: guide -->
|
|
910
|
-
```ts
|
|
911
|
-
import { cursorOf } from 'letopis'
|
|
912
|
-
|
|
913
|
-
const page1 = await db.Product().sort('data.price', 'desc').limit(2).rows()
|
|
914
|
-
const page2 = await db.Product().sort('data.price', 'desc')
|
|
915
|
-
.after(cursorOf(page1.at(-1)!, 'data.price')).limit(2).rows() // strictly after the last row of the page
|
|
916
|
-
```
|
|
917
|
-
|
|
918
|
-
`sort(field, 'asc' | 'desc')` sorts by `'at'`, `'rev'`, `'id'` or `'data.<path>'`. `after(cursorOf(row, field))` is
|
|
919
|
-
pagination "after this row"; it requires `sort()`. Such pagination withstands inserts and deletes between pages, but a
|
|
920
|
-
row whose sort field has changed may drop out or repeat. A cursor is an object `{ v, id }` (the field value and the row
|
|
921
|
-
id): it survives JSON, so you can hand it to a client. The field in `cursorOf` must match the `sort` field. The cursor
|
|
922
|
-
value is checked against the type of the `sort` field before the query, and a wrong one is `invalid_query`: a numeric
|
|
923
|
-
field needs a finite number, `rev` an integer, a boolean field `true` or `false`, a text field a string, uuid fields
|
|
924
|
-
and `id` a uuid, time fields and `at` an ISO string with a time zone, a date field `YYYY-MM-DD`, a field without a
|
|
925
|
-
description a string, a number or `true`/`false`. `null` fits only data fields, and the cursor's `id` is always a uuid.
|
|
926
|
-
The check cannot tell apart a cursor of the same type taken from another field. `limit` and `offset` are available
|
|
927
|
-
too.
|
|
928
|
-
|
|
929
|
-
A data field may be `null` or missing. `sort` places such rows the way `order by` in PostgreSQL does: last when
|
|
930
|
-
ascending, first when descending, and `after()` treats `null` as a value after all others (ascending) or before them
|
|
931
|
-
(descending). So pages go through such rows without gaps or repeats, and a cursor with the value `null` gives the next
|
|
932
|
-
page.
|
|
933
|
-
|
|
934
|
-
### Trees: `deep`
|
|
935
|
-
|
|
936
|
-
<!-- run: guide -->
|
|
937
|
-
```ts
|
|
938
|
-
const subtree = await db.Category(tea).Category().deep().rows() // subcategories of any depth; $depth is the level
|
|
939
|
-
```
|
|
940
|
-
|
|
941
|
-
`deep(max)` repeats the hop to the same class up to the depth `max` (32 levels by default). The filter, tags and owner
|
|
942
|
-
of the step select nodes at any level but do not cut the traversal. Only a node invisible by permissions cuts it: such a
|
|
943
|
-
node also hides its subtree.
|
|
944
|
-
|
|
945
|
-
### Aggregates
|
|
946
|
-
|
|
947
|
-
<!-- run: guide -->
|
|
948
|
-
```ts
|
|
949
|
-
const total = await db.Order(order).line().sum('data.price') // sum of the order lines
|
|
950
|
-
const avgPrice = await db.Shop(shop).Product().avg('data.price')
|
|
951
|
-
const byStatus = await db.Order().countBy('data.status') // { new: 1 }
|
|
952
|
-
```
|
|
953
|
-
|
|
954
|
-
The database computes aggregates over the entities of the last step. `limit(n)` and `offset(n)` before `countBy()`
|
|
955
|
-
apply to the groups: `db.Order().limit(3).countBy('data.status')` gives the three most frequent values. For a boolean
|
|
956
|
-
field, `min()` is `false` if there is at least one `false` (whether all values are `true`), and `max()` is `true` if
|
|
957
|
-
there is at least one `true`; `sum()` and `avg()` do not work on a boolean field — PostgreSQL returns an error.
|
|
958
|
-
|
|
959
|
-
### Paths
|
|
960
|
-
|
|
961
|
-
<!-- run: guide -->
|
|
962
|
-
```ts
|
|
963
|
-
const orderPaths = await db.Order(order).line().Product().paths() // [{ Order: Row, line: Row, Product: Row }, …]
|
|
964
|
-
```
|
|
965
|
-
|
|
966
|
-
`paths()` returns the nodes of every variant of the path; the key of a node is the step name or the name from
|
|
967
|
-
`.alias(name)`. Repeating a link in the chain returns to its node: this way you can go from one link in several
|
|
968
|
-
directions. `db.entity(row | chain)` inserts a ready node into the path.
|
|
969
|
-
|
|
970
|
-
## 7. Deleting and restoring
|
|
971
|
-
|
|
972
|
-
### Preview and deletion
|
|
973
|
-
|
|
974
|
-
<!-- run: guide -->
|
|
975
|
-
```ts
|
|
976
|
-
const preview = await db.Order(order).delete().rows() // deletes nothing: the order and its lines, $action of each
|
|
977
|
-
await db.Order(order).delete({ confirm: true }).rows() // deleted: tombstones with $deleted: true
|
|
978
|
-
|
|
979
|
-
const blockers = (await db.Shop(shop).delete().rows()).filter((x) => x.$action === 'restrict')
|
|
980
|
-
try {
|
|
981
|
-
await db.Shop(shop).delete({ confirm: true }).rows() // products block the shop: the Shop end is restrict
|
|
982
|
-
} catch (e) {
|
|
983
|
-
if (!(e instanceof LetopisError) || e.code !== 'delete_restricted') throw e
|
|
984
|
-
}
|
|
985
|
-
```
|
|
986
|
-
|
|
987
|
-
`delete()` without `{ confirm: true }` is only a preview: the whole cascade closure. Every preview row has `$action`
|
|
988
|
-
(`delete`, `unset` or `restrict`) and `$depth`, the cascade level (0 is the target itself; `unset` and `restrict` rows
|
|
989
|
-
do not have it). With the confirmation, all targets of the step are deleted in one statement, and the cascade applies
|
|
990
|
-
the rules of the ends. The result is the tombstones level by level, by id within a level (the journal has them in the
|
|
991
|
-
same order); targets of classes without a journal (`history: false`) come last. The preview does not take permissions
|
|
992
|
-
into account: a row with `$action: 'delete'` may fail to be deleted for a user without the permission (`acl_denied`),
|
|
993
|
-
so a user interface should ask `acl.checkData(class, 'DELETE')`. The rules of the ends:
|
|
994
|
-
|
|
995
|
-
- `restrict`: deletion is forbidden while the target is referenced (`delete_restricted` with a list of the visible rows
|
|
996
|
-
and the number of the others);
|
|
997
|
-
- `cascade`: the referencing rows are deleted together with the target (the order lines above);
|
|
998
|
-
- `unset`: the reference is removed.
|
|
999
|
-
|
|
1000
|
-
The cascade requires delete permission on every deleted row, and removing a reference requires write permission. A raw
|
|
1001
|
-
`DELETE … WHERE id IN (…)` of any set of related rows works the same way.
|
|
1002
|
-
|
|
1003
|
-
### Restore: `restore`
|
|
1004
|
-
|
|
1005
|
-
<!-- run: guide -->
|
|
1006
|
-
```ts
|
|
1007
|
-
await db.Order(order).restore().rows() // the order, from its last snapshot, with the same id
|
|
1008
|
-
await db.Order(order).line().withDeleted().restore().rows() // and its lines deleted by the cascade
|
|
1009
|
-
```
|
|
1010
|
-
|
|
1011
|
-
`restore()` brings back the last snapshot of a deleted object as a new version and checks it again against the current
|
|
1012
|
-
class description. Only the targets of the step are restored, not the whole cascade. A live object gives `not_deleted`.
|
|
1013
|
-
|
|
1014
|
-
### Erase history: `purge`
|
|
1015
|
-
|
|
1016
|
-
<!-- run: guide -->
|
|
1017
|
-
```ts
|
|
1018
|
-
const [temp] = await db.Customer().create({ name: 'Test', email: 'test@example.com' }).rows()
|
|
1019
|
-
await db.Customer(temp).delete({ confirm: true }).rows()
|
|
1020
|
-
const before = await db.Customer(temp).purge().rows() // preview: $versions is how many versions the journal has
|
|
1021
|
-
await db.Customer(temp).purge({ confirm: true }).rows() // history erased; the journal has a purge record
|
|
1022
|
-
```
|
|
1023
|
-
|
|
1024
|
-
`purge` works only for deleted objects (a live one gives `not_deleted`) and erases their whole history; in place of
|
|
1025
|
-
version 1, a `purge` record stays with the number of erased versions. The history of system classes is not erased.
|
|
1026
|
-
While a projector whose `from` sources include the class of the object ([“Projectors”](#projectors)) has not read
|
|
1027
|
-
the deletion, `purge` refuses (`projector_pending`): the projector would never see an erased tombstone, and the
|
|
1028
|
-
object's contribution would stay in the totals. So a deletion and a `purge` in one transaction do not go through in
|
|
1029
|
-
such a class. Likewise, `db.trimHistory` refuses while the projector has not read the first remaining version: it
|
|
1030
|
-
would no longer find the previous version for `before`. To delete an account, use `db.auth.purgeAccount` ([section 10](#10-users-sign-in-and-permissions)).
|
|
1031
|
-
|
|
1032
|
-
## 8. Transactions, batches and money
|
|
1033
|
-
|
|
1034
|
-
### Every operation is a transaction
|
|
1035
|
-
|
|
1036
|
-
Every chain terminal is its own `read committed` transaction, whatever the default level in the database is. At this
|
|
1037
|
-
level, increments and the cascade see fresh rows. The library retries deadlocks, serialization failures and journal
|
|
1038
|
-
conflicts by itself. If the connection drops during `COMMIT`, it does not retry and throws `commit_unknown`: the
|
|
1039
|
-
transaction may have committed. A repeat of the same operation with the same key finds out the outcome: `exists` means
|
|
1040
|
-
"already written".
|
|
1041
|
-
|
|
1042
|
-
### Explicit transaction: `db.begin()`
|
|
1043
|
-
|
|
1044
|
-
<!-- run: guide -->
|
|
1045
|
-
```ts
|
|
1046
|
-
const tr = await db.begin()
|
|
1047
|
-
try {
|
|
1048
|
-
const [o2] = await tr.Customer(olga).Order().create({ number: 'A-2' }).Shop.set(shop).rows()
|
|
1049
|
-
await tr.Order(o2).line().create({ qty: 1, price: 2000 }).Product.set(kettle).rows()
|
|
1050
|
-
await tr.commit()
|
|
1051
|
-
} catch (e) {
|
|
1052
|
-
await tr.rollback()
|
|
1053
|
-
throw e
|
|
1054
|
-
}
|
|
1055
|
-
```
|
|
1056
|
-
|
|
1057
|
-
- Inside, you have the same chains, `db.sql`, `forUpdate()`, `lock()`, `commit()` and `rollback()`.
|
|
1058
|
-
- An error in any query breaks the transaction: every next call and `commit()` throw the original error, and `commit()`
|
|
1059
|
-
rolls back. `rollback()` goes through. If the error was caught inside `tr.transaction(fn)` and the library does not
|
|
1060
|
-
know about it, PostgreSQL silently answers `ROLLBACK` to `COMMIT`, and the library turns this into the
|
|
1061
|
-
`tx_rolled_back` error.
|
|
1062
|
-
- There are no retries inside `db.begin()`: on `40P01`, repeat the whole block.
|
|
1063
|
-
- `lock(...keys)` is an advisory lock until the end of the transaction. It protects only if everyone who edits this data
|
|
1064
|
-
takes it.
|
|
1065
|
-
|
|
1066
|
-
### Batch
|
|
1067
|
-
|
|
1068
|
-
<!-- run: guide -->
|
|
1069
|
-
```ts
|
|
1070
|
-
const b = db.batch('storefront')
|
|
1071
|
-
b.Shop(shop).Product().create({ sku: 'C-1', title: 'Cup', price: 300 })
|
|
1072
|
-
b.Shop(shop).Product().create({ sku: 'C-2', title: 'Saucer', price: 200 })
|
|
1073
|
-
const results = await b.run() // Row[][]: an array of rows for each plan, in one transaction
|
|
1074
|
-
```
|
|
1075
|
-
|
|
1076
|
-
A batch collects plans, and `run()` executes them in one transaction. Consecutive simple `create` and `upsert` calls of
|
|
1077
|
-
a class with a key are merged into one multi-row statement. `b.size()` is the number of queued plans, `b.discard()`
|
|
1078
|
-
clears the queue. A terminal on a queued plan gives `invalid_query`: only `run()` executes. The name is the key of a
|
|
1079
|
-
queue of the facade: the root connection, each `db.as()` and each `db.begin()` have their own queues. A plan runs on
|
|
1080
|
-
behalf of the facade that queued it (actor, tenant, owner, transaction), and `run()` never executes plans of other
|
|
1081
|
-
facades. Collect plans through one and the same facade object.
|
|
1082
|
-
|
|
1083
|
-
### Recipe: wallet
|
|
1084
|
-
|
|
1085
|
-
The balance is the `balance` field of the `Wallet` hub (integer cents, `min(0)`); operations are the `Movement` link
|
|
1086
|
-
with the key `[Wallet, opId]` ([section 4](#4-data-model)). An operation is never edited or deleted; it is cancelled by
|
|
1087
|
-
a reverse operation with a new `opId`. The key makes an operation idempotent: a repeat after a lost response gets
|
|
1088
|
-
`exists`, and the recipe checks whether it is the same request.
|
|
1089
|
-
|
|
1090
|
-
<!-- run: guide -->
|
|
1091
|
-
```ts
|
|
1092
|
-
import { inc, type Db } from 'letopis'
|
|
1093
|
-
|
|
1094
|
-
type Kind = 'topup' | 'debit' | 'adjust'
|
|
1095
|
-
|
|
1096
|
-
/** Operation: write a Movement and increment the balance — one plan, one transaction. */
|
|
1097
|
-
async function operate(db: Db, wallet: string, opId: string, kind: 'topup' | 'debit', amount: number) {
|
|
1098
|
-
try {
|
|
1099
|
-
await db.Wallet(wallet).Movement().create({ opId, kind, amount })
|
|
1100
|
-
.Wallet().update({ balance: inc(kind === 'topup' ? amount : -amount) }).rows()
|
|
1101
|
-
return 'done'
|
|
1102
|
-
} catch (e) {
|
|
1103
|
-
if (e instanceof LetopisError && e.code === 'exists') return same(db, wallet, opId, kind, amount) // repeat
|
|
1104
|
-
throw e // invalid_data: not enough funds (min(0))
|
|
1105
|
-
}
|
|
1106
|
-
}
|
|
1107
|
-
|
|
1108
|
-
/** Repeat fingerprint: is the request already written under this opId the same request? */
|
|
1109
|
-
async function same(db: Db, wallet: string, opId: string, kind: Kind, value: number) {
|
|
1110
|
-
const m = await db.Movement(db.idOf('Movement', { Wallet: wallet, opId })).first()
|
|
1111
|
-
if (!m || m.data.kind !== kind || (kind === 'adjust' ? m.data.target : m.data.amount) !== value)
|
|
1112
|
-
throw new Error(`operation key ${opId} is already used by another request`)
|
|
1113
|
-
return 'repeat'
|
|
1114
|
-
}
|
|
1115
|
-
|
|
1116
|
-
/** Adjustment "balance = target": an adjust operation for the difference, the total under { rev }. */
|
|
1117
|
-
async function adjust(db: Db, wallet: string, opId: string, target: number) {
|
|
1118
|
-
for (;;) {
|
|
1119
|
-
if (await db.Movement(db.idOf('Movement', { Wallet: wallet, opId })).first()) return same(db, wallet, opId, 'adjust', target)
|
|
1120
|
-
const cur = (await db.Wallet(wallet).first())!
|
|
1121
|
-
try {
|
|
1122
|
-
await db.Wallet(wallet).Movement().create({ opId, kind: 'adjust', amount: target - (cur.data.balance as number), target })
|
|
1123
|
-
.Wallet().update({ balance: target }, { rev: cur.rev }).rows()
|
|
1124
|
-
return 'done'
|
|
1125
|
-
} catch (e) {
|
|
1126
|
-
if (e instanceof LetopisError && e.code === 'conflict') continue // the wallet changed: re-read
|
|
1127
|
-
if (e instanceof LetopisError && e.code === 'exists') return same(db, wallet, opId, 'adjust', target)
|
|
1128
|
-
throw e
|
|
1129
|
-
}
|
|
1130
|
-
}
|
|
1131
|
-
}
|
|
1132
|
-
|
|
1133
|
-
const [wallet] = await db.Customer(olga).Wallet().create({ number: 'W-1' }).rows()
|
|
1134
|
-
await operate(db, wallet.id, 'op-1', 'topup', 10_000) // 'done': balance 10,000
|
|
1135
|
-
await operate(db, wallet.id, 'op-1', 'topup', 10_000) // 'repeat': a repeat after a lost response, the balance is the same
|
|
1136
|
-
await operate(db, wallet.id, 'op-2', 'debit', 3_000) // balance 7,000
|
|
1137
|
-
await adjust(db, wallet.id, 'op-3', 5_000) // adjustment: adjust by −2,000, balance 5,000
|
|
1138
|
-
```
|
|
1139
|
-
|
|
1140
|
-
The same result can be written under a locking read, without retries on `conflict`:
|
|
1141
|
-
|
|
1142
|
-
<!-- run: guide -->
|
|
1143
|
-
```ts
|
|
1144
|
-
const [wallet2] = await db.Customer(olga).Wallet().create({ number: 'W-2' }).rows()
|
|
1145
|
-
const tx = await db.begin()
|
|
1146
|
-
try {
|
|
1147
|
-
const w = (await tx.Wallet(wallet2).forUpdate().first())! // the wallet row is locked until commit
|
|
1148
|
-
await tx.Wallet(wallet2).Movement()
|
|
1149
|
-
.create({ opId: 'op-1', kind: 'adjust', amount: 6_000 - (w.data.balance as number), target: 6_000 })
|
|
1150
|
-
.Wallet().update({ balance: 6_000 }).rows()
|
|
1151
|
-
await tx.commit()
|
|
1152
|
-
} catch (e) {
|
|
1153
|
-
await tx.rollback()
|
|
1154
|
-
throw e
|
|
1155
|
-
}
|
|
1156
|
-
```
|
|
1157
|
-
|
|
1158
|
-
- `inc` adds inside the statement, over the locked row: concurrent debits are not lost, and `min(0)` rejects a debit
|
|
1159
|
-
beyond the balance.
|
|
1160
|
-
- A plain `update({ balance: 700 })` writes the value; it does not add. A total computed from what was read, without
|
|
1161
|
-
`{ rev }` or `forUpdate()`, loses concurrent debits. An advisory `lock()` does not help if not all writers take it.
|
|
1162
|
-
`upsert` overwrites increments.
|
|
1163
|
-
- `forUpdate()` works only inside `db.begin()` (otherwise `tx_required`) and only with `rows`, `first`, `ids`
|
|
1164
|
-
(otherwise `lock_unsupported`). Rows are locked in id order; a row that no longer matches the conditions of the step
|
|
1165
|
-
is not included in the result.
|
|
1166
|
-
- Choose one path for one wallet: under `{ rev }` and under `forUpdate()`, the wallet and the operation key are locked
|
|
1167
|
-
in a different order, and mixing the two in concurrent requests gives `40P01`. Inside a block with `forUpdate()`, do
|
|
1168
|
-
not wait for external calls: all writers wait for the wallet (`lock_timeout` and
|
|
1169
|
-
`idle_in_transaction_session_timeout` help here).
|
|
1170
|
-
- A transfer is two operations in one `db.begin()`, with the wallets edited in id order: then there are no deadlocks.
|
|
1171
|
-
- Insufficient funds are told apart from other data errors by `e.issues`: the field `balance`, the rule `minimum`.
|
|
1172
|
-
|
|
1173
|
-
### Recipe: balance as an aggregate
|
|
1174
|
-
|
|
1175
|
-
The same wallet, but the database itself keeps the balance: `balance` is an aggregate, the sum of `amount` over the
|
|
1176
|
-
live operations ([Aggregates in the database](#aggregates-in-the-database)). An operation is one `Movement` record;
|
|
1177
|
-
there is no `inc` increment in the code, so it cannot be forgotten. The balance stays correct even when raw SQL or the
|
|
1178
|
-
loader writes the operations and a cascade deletes them.
|
|
1179
|
-
|
|
1180
|
-
```ts
|
|
1181
|
-
const Wallet = hub('Wallet', { number: z.string(), balance: z.number().int().min(0) }, {
|
|
1182
|
-
ends: { Customer }, key: ['number'],
|
|
1183
|
-
aggregate: { balance: { sum: 'amount', of: 'Movement', via: 'Wallet' } }, // the balance is the sum of live operations
|
|
1184
|
-
})
|
|
1185
|
-
const Movement = link('Movement', { opId: z.string(), amount: z.number().int() },
|
|
1186
|
-
{ ends: { Wallet }, key: ['Wallet', 'opId'] }) // the operation key gives idempotency
|
|
1187
|
-
|
|
1188
|
-
/** Operation: an amount above zero tops up, below zero debits; the database recalculates the balance in the same transaction. */
|
|
1189
|
-
async function operate(db: Db, wallet: string, opId: string, amount: number) {
|
|
1190
|
-
try {
|
|
1191
|
-
await db.Wallet(wallet).Movement().create({ opId, amount }).rows()
|
|
1192
|
-
return 'done'
|
|
1193
|
-
} catch (e) {
|
|
1194
|
-
if (e instanceof LetopisError && e.code === 'exists') return 'repeat' // a repeat after a lost response
|
|
1195
|
-
throw e // invalid_data: the balance would go below zero (min(0))
|
|
1196
|
-
}
|
|
1197
|
-
}
|
|
1198
|
-
|
|
1199
|
-
/** Adjustment "balance = target": an operation for the difference, with the wallet locked until commit. */
|
|
1200
|
-
async function adjust(db: Db, wallet: string, opId: string, target: number) {
|
|
1201
|
-
const tx = await db.begin()
|
|
1202
|
-
try {
|
|
1203
|
-
const w = (await tx.Wallet(wallet).forUpdate().first())!
|
|
1204
|
-
await tx.Wallet(wallet).Movement().create({ opId, amount: target - (w.data.balance as number) }).rows()
|
|
1205
|
-
await tx.commit()
|
|
1206
|
-
return 'done'
|
|
1207
|
-
} catch (e) {
|
|
1208
|
-
await tx.rollback()
|
|
1209
|
-
if (e instanceof LetopisError && e.code === 'exists') return 'repeat' // an adjustment with this opId is already written
|
|
1210
|
-
throw e
|
|
1211
|
-
}
|
|
1212
|
-
}
|
|
1213
|
-
|
|
1214
|
-
const [wallet] = await db.Customer(olga).Wallet().create({ number: 'W-1' }).rows() // balance is 0: the database sets it
|
|
1215
|
-
await operate(db, wallet.id, 'op-1', 10_000) // balance 10,000
|
|
1216
|
-
await operate(db, wallet.id, 'op-2', -3_000) // balance 7,000
|
|
1217
|
-
await adjust(db, wallet.id, 'op-3', 5_000) // an operation of −2,000, balance 5,000
|
|
1218
|
-
```
|
|
1219
|
-
|
|
1220
|
-
- The model here is its own, and the block is not part of the example program: there the balance grows through `inc`,
|
|
1221
|
-
and an aggregate field does not change that way.
|
|
1222
|
-
- `balance` is passed neither to `create` nor to `update` (`aggregate_field`), so an adjustment is an operation too,
|
|
1223
|
-
not an `update` under `{ rev }`. Writing an operation needs no permission to write the wallet: the permission to
|
|
1224
|
-
write `Movement` is enough.
|
|
1225
|
-
- Insufficient funds are `invalid_data`, as with `inc`: `e.issues` holds the field `balance` and the rule `minimum`.
|
|
1226
|
-
Check a repeat as `same()` in the recipe above does: `exists` only says that an operation with this `opId` already
|
|
1227
|
-
exists.
|
|
1228
|
-
- Editing and deleting an operation change the balance too: the database subtracts the previous value. It is still
|
|
1229
|
-
better to cancel an operation with a reverse one, so that the history of the wallet stays complete.
|
|
1230
|
-
- Every operation locks the wallet row and writes its version, so the operations of one wallet go one after another,
|
|
1231
|
-
as with `inc`: about 150 per second on the benchmark machine ([section 17](#17-performance)). If one object gets
|
|
1232
|
-
more, compute the total with a projector instead ([Recipe: daily revenue](#recipe-daily-revenue)).
|
|
1233
|
-
|
|
1234
|
-
## 9. Bulk loading
|
|
1235
|
-
|
|
1236
|
-
### When you need the loader
|
|
1237
|
-
|
|
1238
|
-
The loader is the path for import, seeding and syncing large reference tables. It checks rows in the library code
|
|
1239
|
-
with the same validator as the database, and writes them with the `COPY` command as the owner role, without triggers
|
|
1240
|
-
and RLS. In the benchmark, this is about 6,400 rows per second against 130 for row-by-row `create`
|
|
1241
|
-
([section 17](#17-performance)). It works only on an admin connection (`allowBypassRls` and the right to become the
|
|
1242
|
-
owner), otherwise you get `admin_required`.
|
|
1243
|
-
|
|
1244
|
-
### `db.load()`
|
|
1245
|
-
|
|
1246
|
-
<!-- run: guide -->
|
|
1247
|
-
```ts
|
|
1248
|
-
const rep = await adm.load({
|
|
1249
|
-
Shop: [{ name: 'Cornflower', city: 'Perm' }],
|
|
1250
|
-
Product: [
|
|
1251
|
-
{ sku: 'V-1', title: 'Mug', price: 450, Shop: { $key: ['Cornflower'] } },
|
|
1252
|
-
{ sku: 'V-2', title: 'Spoon', price: 150, Shop: { $key: ['Cornflower'] }, $tags: ['new'] },
|
|
1253
|
-
],
|
|
1254
|
-
Customer: [{ name: 'Petr', email: 'petr@mail.ru', $ext: 'crm:1001' }],
|
|
1255
|
-
})
|
|
1256
|
-
// rep: { rows: 4, loaded: 4, updated: 0, skipped: 0, duplicates: 0, byClass: { Shop: 1, Product: 2, Customer: 1 }, … }
|
|
1257
|
-
```
|
|
1258
|
-
|
|
1259
|
-
- The input is an object "class → rows", an array, or a stream of rows with a `$class` field.
|
|
1260
|
-
- A row has its fields and end roles at the top level. Service keys: `$class` is the class of the row, `$ext` is an
|
|
1261
|
-
external key of a class without a key (loading the same file again gives the same ids), `$tags` are the tags,
|
|
1262
|
-
`$owner` is the owner.
|
|
1263
|
-
- A reference is a target id, a key `{ $key: [...] }` or an external key `{ $ext: '…' }`; for a polymorphic end, add
|
|
1264
|
-
the `$class` of the target. The order of rows and classes does not matter, and circular references are allowed.
|
|
1265
|
-
- Keys that start with `$` inside the data are ordinary data.
|
|
1266
|
-
- Rows go to the session tenant (or to `tenant`). A load is one transaction: an error in any row cancels everything.
|
|
1267
|
-
Subscriptions see the rows after the commit.
|
|
1268
|
-
|
|
1269
|
-
### Repeats and existing objects
|
|
1270
|
-
|
|
1271
|
-
<!-- run: guide -->
|
|
1272
|
-
```ts
|
|
1273
|
-
const again = await adm.load({ Shop: [{ name: 'Cornflower', city: 'Perm' }] }, { existing: 'skip' }) // skipped: 1
|
|
1274
|
-
const changed = await adm.load({ Shop: [{ name: 'Cornflower', city: 'Samara' }] }, { existing: 'update' }) // updated: 1
|
|
1275
|
-
```
|
|
1276
|
-
|
|
1277
|
-
| Option | Values |
|
|
1278
|
-
|---|---|
|
|
1279
|
-
| `duplicates`: a repeated id within the load | `'error'` (default): cancel; `'first'` or `'last'`: keep the first or the last row |
|
|
1280
|
-
| `existing`: the object already exists in the database | `'error'` (default): cancel; `'skip'`: skip, do not resurrect deleted objects; `'update'`: a new version if the content differs (unchanged objects are counted in `skipped`), deleted objects are created again |
|
|
1281
|
-
|
|
1282
|
-
### Portions, history and verification
|
|
1283
|
-
|
|
1284
|
-
- `commitEvery: N` commits in portions of about N rows. Portions go by class in dependency order; the classes of one
|
|
1285
|
-
cycle go in one portion. An error cancels only the current portion, and a repeat with `existing: 'skip'` writes the
|
|
1286
|
-
rest.
|
|
1287
|
-
- `withHistory: true`: the rows are versions with `$rev`, `$at` and `$op`; this is how the import carries history over.
|
|
1288
|
-
- `verify: true`: at the end, check the loaded rows again with the database check.
|
|
1289
|
-
- `batch` (rows per `COPY` batch, 10,000), `class` (the class of rows without `$class`), `tenant`, `agent` (the agent of
|
|
1290
|
-
the versions).
|
|
1291
|
-
|
|
1292
|
-
### Loading on behalf of a user
|
|
1293
|
-
|
|
1294
|
-
<!-- run: guide -->
|
|
1295
|
-
```ts
|
|
1296
|
-
await adm.load({ Category: [{ name: 'Tableware' }] }, { as: ANNA, tenant: SHOP })
|
|
1297
|
-
```
|
|
1298
|
-
|
|
1299
|
-
With `as`, the user becomes the author of the versions, and the loader checks the same things as the database does on a
|
|
1300
|
-
regular write: the tenant is available to the user, write permission on each class, the permission conditions on rows,
|
|
1301
|
-
the visibility of reference targets, the owner per `ownerDefault`. Without `tenant`, the rows go to the user's own
|
|
1302
|
-
tenant. This is how a loading process can accept files sent by users.
|
|
1303
|
-
|
|
1304
|
-
### Initial load
|
|
1305
|
-
|
|
1306
|
-
```ts
|
|
1307
|
-
const r = await adm.load(rows, { initial: true }) // empty schema: one transaction, indexes and triggers at the end
|
|
1308
|
-
```
|
|
1309
|
-
|
|
1310
|
-
`db.load(rows, { initial: true })` (`letopis load --initial`) loads data into an empty schema faster than the regular
|
|
1311
|
-
loader — about 2.2 times on a million objects: everything goes in one transaction, the tables are closed for reading and
|
|
1312
|
-
writing during the load, indexes, constraints and triggers are dropped and rebuilt at the end, and rows are written with
|
|
1313
|
-
`COPY … FREEZE`. Empty means that the schema has nothing besides the rows of system classes (class descriptions,
|
|
1314
|
-
accounts, memberships, resources, rules) and `reset` records; otherwise you get `not_empty` before any change. The mode
|
|
1315
|
-
is incompatible with `commitEvery`, `existing` (other than `'error'`) and `{ as }`; an error in any row rolls
|
|
1316
|
-
everything back, indexes and triggers included. `letopis import --initial` runs the whole import this way: in one
|
|
1317
|
-
transaction, without `--update` and `--commit-every`. WAL is written here too (it is skipped only with
|
|
1318
|
-
`wal_level = minimal`), but the indexes are built once, so there is almost three times less of it; after `FREEZE`, the
|
|
1319
|
-
first read and the first vacuum do not rewrite the table.
|
|
1320
|
-
|
|
1321
|
-
### Command line
|
|
1322
|
-
|
|
1323
|
-
```sh
|
|
1324
|
-
npx letopis load shops.ndjson products.ndjson --dsn "$ADMIN" --schema v2.shop --token "$TOKEN" \
|
|
1325
|
-
[--class Product] [--duplicates first] [--existing skip] [--commit-every 100000] [--verify]
|
|
1326
|
-
# shops.ndjson: {"$class":"Shop","name":"South"}
|
|
1327
|
-
# products.ndjson: {"$class":"Product","sku":"Y-1","title":"Vase","price":800,"Shop":{"$key":["South"]}}
|
|
1328
|
-
```
|
|
1329
|
-
|
|
1330
|
-
NDJSON files have one object per line; all files form one load.
|
|
1331
|
-
|
|
1332
|
-
### Report and errors
|
|
1333
|
-
|
|
1334
|
-
Report: `rows`, `loaded`, `updated`, `skipped`, `duplicates`, `byClass`, `external` (how many targets outside the load
|
|
1335
|
-
were checked by a query at the end), `portions`, `ms`. An error is a `LetopisError` with the code of the first
|
|
1336
|
-
problem. Its `detail`: `rows` is the first 100 violations with the row number, code and details (`line`, `code`,
|
|
1337
|
-
`message`, `id`, `class`, `issues`), `total` is how many violations were found, `truncated` means the list is cut,
|
|
1338
|
-
`limit` is the list limit (100), `complete` means the checks that found violations went through all rows. Row codes:
|
|
1339
|
-
the codes of the database checks, and also `duplicate` (a repeated id), `exists` (the object already exists), `deleted`
|
|
1340
|
-
(an object with this id existed before) and `class_not_loadable` (the loader does not accept `Class` rows: classes are
|
|
1341
|
-
set by `syncModels`). The loader rejects rows of derived classes and references to them with the code
|
|
1342
|
-
`derived_class`: only a projector writes such rows.
|
|
1343
|
-
|
|
1344
|
-
Violations are searched for in stages: parsing the rows (classes, ids, repeats), permissions on classes
|
|
1345
|
-
(`load({ as })`), data and ends in batches, and at the end of a portion the targets outside the load, owners and the
|
|
1346
|
-
tenant. The load is cancelled after the first stage that found violations, and `total` counts the violations of that
|
|
1347
|
-
stage. A target outside the load is one violation per target (the row is the first one that references it).
|
|
1348
|
-
"(и ещё N)" ("and N more") in the error text is the exact number of the remaining violations. If not all rows were
|
|
1349
|
-
checked, `complete` is `false`, and the text says "(и ещё не меньше N: причины)" ("at least N more: reasons") or, if
|
|
1350
|
-
one violation was found, "(возможно, не единственная: причины)" ("possibly not the only one: reasons"). The reasons:
|
|
1351
|
-
the following `commitEvery` portions were not checked; the ACL conditions of the following batches were not checked
|
|
1352
|
-
(`load({ as })`); the database check (`verify: true`) names at most 100 rows.
|
|
1353
|
-
|
|
1354
|
-
## 10. Users, sign-in and permissions
|
|
1355
|
-
|
|
1356
|
-
### Accounts, tenants and memberships
|
|
1357
|
-
|
|
1358
|
-
An account is a row of class `Account` in System: a name, categories (`User`, `Service` and your own) and an `enabled`
|
|
1359
|
-
flag. Every account is its own tenant. A `member` membership allows work in another tenant: it is a link from a user
|
|
1360
|
-
(`User`) to a tenant (`Tenant`) with roles in its data. Accounts and their categories are edited by the system
|
|
1361
|
-
administrator, a member of System with the `admin` role. There is no separate method to appoint one: a membership in
|
|
1362
|
-
System with the `admin` role is put in place by your own installation seed or by the loader on an admin connection (a
|
|
1363
|
-
`member` row with `tenant: <System id>`). New users of a tenant are easy to create with `createTenant`
|
|
1364
|
-
([section 3](#3-installation-and-connection)).
|
|
1365
|
-
|
|
1366
|
-
### Service and API keys
|
|
1367
|
-
|
|
1368
|
-
A service is an account of category `Service` with an API key. `up()` issues the installation key. The service checks
|
|
1369
|
-
passwords and codes, issues sessions and acts on behalf of users. It issues and revokes the keys of regular users
|
|
1370
|
-
itself (the `auth.credential` permission), as for Boris below. The service also issues and revokes its own API keys
|
|
1371
|
-
and "key + secret" pairs itself, with a session opened by its own key (`connect({ apiKey })`, `verifyApiKey`,
|
|
1372
|
-
`verifyKeySecret`). Its other credentials and the keys of other services, of System and of system administrators are
|
|
1373
|
-
issued and revoked only by a system administrator; a new service key is also registered by `up({ serviceKey })`.
|
|
1374
|
-
|
|
1375
|
-
There can be several keys. A session opened by a key remembers it, and revoking a key revokes only its sessions:
|
|
1376
|
-
connections by other keys keep working. So rotation goes without downtime: the service issues itself a new key
|
|
1377
|
-
(`issueApiKey({ account: <service id> })`), moves its processes to it and revokes the old one (`revokeCredential(id)`).
|
|
1378
|
-
The revocation keeps the current session; `keepCurrent: false` revokes it too. The last active key of a service
|
|
1379
|
-
cannot be revoked: `acl_denied` "последний действующий ключ сервиса не отзывается: сначала выпустите новый" ("the
|
|
1380
|
-
last active key of a service is not revoked: issue a new one first"). To disable a service, set `enabled: false` on
|
|
1381
|
-
its account.
|
|
1382
|
-
|
|
1383
|
-
<!-- run: guide -->
|
|
1384
|
-
```ts
|
|
1385
|
-
const { key, id: keyId } = await svc.auth.issueApiKey({ account: BORIS, name: 'integration' }) // the key is returned only once
|
|
1386
|
-
const viaKey = await svc.auth.verifyApiKey(key) // { account, token } | null
|
|
1387
|
-
await svc.auth.revokeCredential(keyId)
|
|
1388
|
-
```
|
|
1389
|
-
|
|
1390
|
-
A "key + secret" pair: `issueKeySecret({ account, name })` and `verifyKeySecret(key, secret)`.
|
|
1391
|
-
|
|
1392
|
-
### Passwords
|
|
1393
|
-
|
|
1394
|
-
<!-- run: guide -->
|
|
1395
|
-
```ts
|
|
1396
|
-
await svc.auth.setPassword({ account: BORIS, identifier: 'boris@lutik.ru', password: 'New-password-2026', confirmed: true })
|
|
1397
|
-
const failed = await svc.auth.verifyPassword({ identifier: 'boris@lutik.ru', password: 'Buttercup-2026!' }) // null: the password was changed
|
|
1398
|
-
```
|
|
1399
|
-
|
|
1400
|
-
- A password is stored as a scrypt hash; the service checks it, and the response time for a nonexistent login is
|
|
1401
|
-
equalized.
|
|
1402
|
-
- The identifier is compared case-insensitively: `Anna@Shop.ru` and `anna@shop.ru` are one login.
|
|
1403
|
-
- A password change revokes the other sessions of the account (`keepCurrent` keeps the current one).
|
|
1404
|
-
- By default, sign-in works only by a confirmed credential, otherwise `null`; `requireConfirmed: false` also lets in by
|
|
1405
|
-
an unconfirmed one. The service confirms a credential: `confirmed: true` in `setPassword` or `setFlags`.
|
|
1406
|
-
- The second factor is the `totp` option: `verifyPassword({ identifier, password, totp })` checks the account's TOTP
|
|
1407
|
-
code after a correct password and issues a session with the methods `[kind, 'TOTP']`, `['PASSWORD', 'TOTP']` by
|
|
1408
|
-
default (see below). Without `totp`, sign-in does not check the code: the service decides whether to require the
|
|
1409
|
-
second factor.
|
|
1410
|
-
|
|
1411
|
-
### One-time codes and TOTP
|
|
1412
|
-
|
|
1413
|
-
<!-- run: guide -->
|
|
1414
|
-
```ts
|
|
1415
|
-
import { totpCode } from 'letopis'
|
|
1416
|
-
|
|
1417
|
-
const { code } = await svc.auth.issueOtp({ account: BORIS, identifier: 'boris@lutik.ru', ttl: 300 }) // the service sends the code by email
|
|
1418
|
-
const byCode = await svc.auth.verifyOtp({ identifier: 'boris@lutik.ru', code }) // { account, token } | null
|
|
1419
|
-
const { secret, uri } = await svc.auth.enrollTotp({ account: BORIS, issuer: 'Buttercup' }) // uri is for a QR code
|
|
1420
|
-
const totpOk = await svc.auth.verifyTotp({ account: BORIS, code: totpCode(secret) }) // the first check enables the factor
|
|
1421
|
-
```
|
|
1422
|
-
|
|
1423
|
-
A code is single-use: a successful check and brute force (5 attempts by default) burn it. The attempt counters are
|
|
1424
|
-
rolled back together with the caller's transaction, so the service limits the rate of attempts. A repeat of the same
|
|
1425
|
-
TOTP step is rejected. `totpCode(secret)` is a helper for tests and services.
|
|
1426
|
-
|
|
1427
|
-
Sign-in with the second factor is `verifyPassword({ …, totp })` and `lookup({ …, totp })`. The code is checked only
|
|
1428
|
-
after a correct password or a found identity, by the same check as `verifyTotp`: the `auth.totp` permission is needed,
|
|
1429
|
-
the window is one step each way, and the failure counter is shared (five in a row close the check for 5 minutes). A
|
|
1430
|
-
correct code gives a session with the methods `[kind, 'TOTP']` and enables a not yet confirmed factor; a wrong or
|
|
1431
|
-
repeated code, like a code for an account without TOTP, gives `null`. With such a session, a user with TOTP changes
|
|
1432
|
-
their own credentials for 10 minutes.
|
|
1433
|
-
|
|
1434
|
-
### External identities
|
|
1435
|
-
|
|
1436
|
-
`link({ account, kind, identifier, confirmed })` links an external identity (Telegram, Google, SSO): the service checks
|
|
1437
|
-
its token, and the database stores the binding. `lookup({ kind, identifier })` signs in with it (with the `totp`
|
|
1438
|
-
option, also by a TOTP code, like `verifyPassword`):
|
|
1439
|
-
|
|
1440
|
-
<!-- run: guide -->
|
|
1441
|
-
```ts
|
|
1442
|
-
await svc.auth.link({ account: BORIS, kind: 'TELEGRAM', identifier: '100500', confirmed: true })
|
|
1443
|
-
const viaTelegram = await svc.auth.lookup({ kind: 'TELEGRAM', identifier: '100500' }) // { account, token } | null
|
|
1444
|
-
```
|
|
1445
|
-
|
|
1446
|
-
A user can link their own credentials only after a recent sign-in (10 minutes; an API key, a "key + secret" pair and a
|
|
1447
|
-
session from a service do not count), and cannot confirm them. With TOTP enabled, the database requires a session that
|
|
1448
|
-
has passed TOTP: `verifyPassword` and `lookup` with the `totp` option issue one. If the identifier is taken, linking
|
|
1449
|
-
answers with the same `acl_denied` as without a recent sign-in. But a user with a recent sign-in can tell from such a
|
|
1450
|
-
denial that the identifier is taken ([section 18](#18-limitations)).
|
|
1451
|
-
|
|
1452
|
-
### Sessions
|
|
1453
|
-
|
|
1454
|
-
<!-- run: guide -->
|
|
1455
|
-
```ts
|
|
1456
|
-
const shortToken = await svc.auth.sessionFor(BORIS, { ttl: 600 }) // a session for 10 minutes of idle time for another process
|
|
1457
|
-
const whose = await svc.auth.session(shortToken) // { account, tenant, member, expiresAt, loginAt, methods }
|
|
1458
|
-
const boris = await connect({ dsn: APP, schema: r.schema, token: shortToken })
|
|
1459
|
-
await boris.auth.revoke() // sign-out: the session is revoked
|
|
1460
|
-
const ended = await svc.auth.session(shortToken) // null
|
|
1461
|
-
await boris.close()
|
|
1462
|
-
```
|
|
1463
|
-
|
|
1464
|
-
A token is 256 random bits; the database stores only its sha256. The default lifetime is 7 days. `refresh()` extends
|
|
1465
|
-
the current session: the new lifetime is counted from the current moment, but it is not longer than the lifetime issued
|
|
1466
|
-
at sign-in; it returns the new expiration time. `connect()` extends the session itself once per minute of work
|
|
1467
|
-
(`refreshSession`), so a session in use does not expire: the lifetime is the idle time. `revoke()` signs out,
|
|
1468
|
-
`revokeAll()` signs out everywhere. Sessions are revoked when the account is disabled, the password is changed, a credential is deleted or the
|
|
1469
|
-
account is deleted. Deleting a credential revokes the other sessions of the account, while revoking an API key or a
|
|
1470
|
-
"key + secret" pair revokes only the sessions opened by that key (and old sessions of the same sign-in method without
|
|
1471
|
-
a key mark); the current session stays unless you pass `keepCurrent: false`.
|
|
1472
|
-
|
|
1473
|
-
A service that holds a user's token (for example, in a cookie) learns about the session with `svc.auth.session(token)`:
|
|
1474
|
-
no connection is opened with that token, and the session is not extended. The answer tells whose session it is
|
|
1475
|
-
(`account`), which tenant it works in (`tenant`: the one chosen by `auth.switch()` or the account's own tenant),
|
|
1476
|
-
whether the membership there is intact (`member`; always `true` in the own tenant), when it expires (`expiresAt`), and
|
|
1477
|
-
when and how the sign-in happened (`loginAt`, `methods`: `PASSWORD`, `APIKEY`, `SERVICE`, `TOTP` …). A wrong, revoked
|
|
1478
|
-
or expired token and a disabled account give `null`; the reason is not told apart. This is how a web service checks the
|
|
1479
|
-
session at the start of every request: reading under a revoked session gives no error, the answers are simply empty.
|
|
1480
|
-
The method is called by the service or the system administrator (the `auth.session` permission); a user gets
|
|
1481
|
-
`acl_denied`.
|
|
1482
|
-
|
|
1483
|
-
### Working in another tenant
|
|
1484
|
-
|
|
1485
|
-
A service acts on behalf of a user through `svc.as(account, { tenant, owner })`. The database checks the service's
|
|
1486
|
-
`auth.impersonate` permission (by System rules), that the user exists and is enabled, and that the tenant is available
|
|
1487
|
-
to the user, either their own or through membership. You cannot act on behalf of System, services, system
|
|
1488
|
-
administrators or disabled accounts. The user's visibility under the access rules is not checked. In the journal,
|
|
1489
|
-
`author` is the user and `agent` is the service. A user's connection by their own token switches between tenants like
|
|
1490
|
-
this:
|
|
1491
|
-
|
|
1492
|
-
<!-- run: guide -->
|
|
1493
|
-
```ts
|
|
1494
|
-
const mine = await anna.auth.tenants() // [{ tenant, roles }]: her own tenant and the shop via membership
|
|
1495
|
-
await anna.auth.switch(SHOP) // the session tenant is now the shop
|
|
1496
|
-
```
|
|
1497
|
-
|
|
1498
|
-
### Permissions
|
|
1499
|
-
|
|
1500
|
-
Permissions are always on and deny by default. A rule connects a group with an object:
|
|
1501
|
-
|
|
1502
|
-
- **group**: a `Resource` row of category `ACCOUNT`: accounts by category (`{ categories: '{User}' }`) or members by
|
|
1503
|
-
membership role (`{ roles: '{staff}' }`);
|
|
1504
|
-
- **object**: a `Resource` row of category `API` (an endpoint by mask, `{ endpoint: 'shop.reports.*' }`), or of
|
|
1505
|
-
category `READ`, `WRITE`, `DELETE` (the rows of a class by mask, with descendants, `{ class: 'Product' }`, and a
|
|
1506
|
-
condition on the columns `id`, `owner`, `tenant`, `tags`, `data`, `links` with the values `$actor` and `$tenant`);
|
|
1507
|
-
- **rule**: a `rule` link group → object: `permission` (`allow` or `deny`) and `weight`.
|
|
1508
|
-
|
|
1509
|
-
Exactly one rule decides: the highest weight; at equal weight, deny beats allow. A tenant rule weight above 999 is
|
|
1510
|
-
cut down to 999, so System rules with a weight of 1000 or more are an administrative layer that a tenant cannot
|
|
1511
|
-
outweigh. Tenant rules apply to the tenant's rows, System rules apply everywhere. Permissions apply to raw SQL too: rows
|
|
1512
|
-
without permission are invisible, and an edit changes nothing or fails with `acl_denied`.
|
|
1513
|
-
|
|
1514
|
-
A class mask in a `READ`, `WRITE` or `DELETE` object matches the descendants of the class too, by the names of the
|
|
1515
|
-
ancestors and their aliases. So an edit of a class that changes `parent`, `alias` or `derivedBy` needs the `READ`,
|
|
1516
|
-
`WRITE` and `DELETE` permissions without conditions on all rows of the class and of each of its descendants,
|
|
1517
|
-
otherwise `acl_denied`: the `WRITE` permission on `Class` alone is not enough to put the rows of a class under other
|
|
1518
|
-
rules. An alias does not match the name of another class (including `Class` and the system ones) or another class's
|
|
1519
|
-
alias in the tenant and System, and the name of a new class does not match another class's alias: otherwise a rule
|
|
1520
|
-
written for one class would act on the rows of another too. A violation gives `invalid_class`.
|
|
1521
|
-
|
|
1522
|
-
By the same names, the rules also find the old versions of rows in the journal. So a class is deleted only when it has
|
|
1523
|
-
no rows, no descendants and no history of rows: `purge()` erases the history of deleted rows first. For a System class,
|
|
1524
|
-
the rows, descendants and history of a tenant that has its own class with the same name do not count: they belong to
|
|
1525
|
-
that tenant's class. A class deleted with history before is created again only with the same `parent`, `alias`,
|
|
1526
|
-
`derivedBy` and `history` — otherwise `invalid_class`: the old versions of its rows would fall under other rules or a
|
|
1527
|
-
different thinning. The rules also find old versions by the aliases of ancestors. So while deleted descendants of a
|
|
1528
|
-
class keep row history, the class does not change its alias (`invalid_class`), and once deleted it is created again
|
|
1529
|
-
only with the same description. If the journal holds rows of a class with this name but not its tombstone (for
|
|
1530
|
-
example, rows of a System class deleted in 1.1), a class with this name is not created until `purge()` erases the
|
|
1531
|
-
history.
|
|
1532
|
-
|
|
1533
|
-
A derived class (`derivedBy`) is created, edited and deleted only by an administrator: an admin connection or an
|
|
1534
|
-
account with the System permission `letopis.projector` (the system administrator has it through the `letopis.*`
|
|
1535
|
-
mask). The projector writes totals over all source rows of the tenant into such a class, without read permissions,
|
|
1536
|
-
so a user who created it on their own would read totals over rows they may not read. A class that has derived
|
|
1537
|
-
descendants is also edited only by an administrator: editing an ancestor changes the description of the descendant
|
|
1538
|
-
(fields, key, `history`). The database itself sets the `aggregatedBy` mark (the class became an aggregate source) on a
|
|
1539
|
-
derived class too: it does not change its fields or key.
|
|
1540
|
-
|
|
1541
|
-
### Granting a permission
|
|
1542
|
-
|
|
1543
|
-
Example: Oleg is a shop employee with the `staff` role, and employees may not delete products.
|
|
1544
|
-
|
|
1545
|
-
<!-- run: guide -->
|
|
1546
|
-
```ts
|
|
1547
|
-
const pg2 = postgres(ADMIN)
|
|
1548
|
-
const OLEG = (await createTenant(pg2, r.schema, { // the same shop: only the user is created
|
|
1549
|
-
name: 'Chamomile', user: { name: 'Oleg', login: 'oleg@shop.ru', password: 'Oleg-2026!' } })).user!
|
|
1550
|
-
await pg2.end()
|
|
1551
|
-
await db.member(db.idOf('member', { User: OLEG, Tenant: SHOP })).update({ roles: ['staff'] }).rows()
|
|
1552
|
-
|
|
1553
|
-
const [staff] = await db.Resource().create({ alias: 'staff:ACCOUNT', category: 'ACCOUNT', pattern: { roles: '{staff}' } }).rows()
|
|
1554
|
-
const [delProduct] = await db.Resource().create({ alias: 'product:DELETE', category: 'DELETE', pattern: { class: 'Product' } }).rows()
|
|
1555
|
-
await db.rule().create({ permission: 'deny', weight: 50 }).Group.set(staff).Object.set(delProduct).rows() // stronger than "may do everything" (10)
|
|
1556
|
-
const [reports] = await db.Resource().create({ alias: 'reports:API', category: 'API', pattern: { endpoint: 'shop.reports.*' } }).rows()
|
|
1557
|
-
await db.rule().create({ permission: 'allow', weight: 50 }).Group.set(staff).Object.set(reports).rows()
|
|
1558
|
-
|
|
1559
|
-
const asOleg = await svc.as(OLEG, { tenant: SHOP })
|
|
1560
|
-
const canDelete = await asOleg.acl.checkData('Product', 'DELETE') // { allow: false }
|
|
1561
|
-
const canRead = await asOleg.acl.checkData('Product', 'READ') // { allow: true }
|
|
1562
|
-
const canReport = await asOleg.acl.check('shop.reports.daily') // { allow: true, rule: { weight: 50, … } }
|
|
1563
|
-
```
|
|
1564
|
-
|
|
1565
|
-
- `acl.check(endpoint)` → `{ allow, rule, code, message }`: the `code` and `message` fields of the denying rule set the
|
|
1566
|
-
endpoint's own denial.
|
|
1567
|
-
- `acl.checkData(class, 'READ' | 'WRITE' | 'DELETE')` → `{ allow, filter }`: `filter` is the condition on rows.
|
|
1568
|
-
- The database recomputes permissions by itself. To remove a permission, delete the rule:
|
|
1569
|
-
`db.rule(id).delete({ confirm: true })`.
|
|
1570
|
-
- The library takes the actor's signed permission plan — the permission pass — from the database once and passes it
|
|
1571
|
-
in every transaction, so a permission check costs tens of microseconds. Raw SQL without the pass pays 0.3–1.3 ms per
|
|
1572
|
-
query.
|
|
1573
|
-
- The question "who had access at a given date" is answered by an `asOf` read over the classes `rule`, `member` and
|
|
1574
|
-
`Resource`.
|
|
1575
|
-
|
|
1576
|
-
Fields of the system permission classes:
|
|
1577
|
-
|
|
1578
|
-
- `member`: the ends `User` and `Tenant`, the field `roles` (a list of roles);
|
|
1579
|
-
- `Resource`: `alias` (the unique name of the resource and its key; this is not a class alias), `category` (`ACCOUNT`,
|
|
1580
|
-
`API`, `READ`, `WRITE`, `DELETE`) and `pattern`, the mask. A group's mask is `categories` or `roles`, and only a
|
|
1581
|
-
string: `'{staff}'`, `'{a,b}'`, negation `'!{a}'`. An object's mask is `endpoint` or `class` (also strings only) plus
|
|
1582
|
-
conditions on the columns. **The database rejects a non-string mask (for example, the array `['staff']`, a number or
|
|
1583
|
-
`null`)** with `invalid_data`, whether written by the library, raw SQL or the loader. Old rows with such a mask
|
|
1584
|
-
(written before this check or around it) grant no permissions, while a denial by them applies to everyone: by a
|
|
1585
|
-
group, to all accounts, by a `class` mask, to all classes (a non-string `endpoint` mask, as before, matches no
|
|
1586
|
-
endpoint). The write check comes with the description of the `Resource` class: it arrives with a fresh installation
|
|
1587
|
-
and after `reset({ all })`; in an existing schema, the corrected matching works after `up()` is run again;
|
|
1588
|
-
- `rule`: the ends `Group` and `Object` (`Resource` rows), the fields `permission` (`allow` or `deny`), `weight`, and,
|
|
1589
|
-
for the endpoint's own denial, `code` and `message`.
|
|
1590
|
-
|
|
1591
|
-
### Deleting an account
|
|
1592
|
-
|
|
1593
|
-
`db.auth.purgeAccount(account)` deletes an account and the data of its tenant. The system administrator or the owner of
|
|
1594
|
-
its tenant can call it; you cannot delete yourself or the last owner of a tenant. Ownership of rows in other tenants
|
|
1595
|
-
passes to those tenants, sessions are revoked, credentials are deleted. An owner can also delete the tenant they are
|
|
1596
|
-
working in right now: their session returns to their personal tenant.
|
|
1597
|
-
|
|
1598
|
-
The system administrator can also delete the account row itself — `db.Account(id).delete({ confirm: true })` in the
|
|
1599
|
-
System tenant; the data of the account's tenant stays then. Either way, the account's rows in `secret` are erased at
|
|
1600
|
-
once: sessions, password and key hashes, TOTP secrets, external accounts (the foreign key `secret.account` with a
|
|
1601
|
-
cascade, revision 7). Before that, the sessions are revoked with a signal, and processes drop the cached answers of
|
|
1602
|
-
these sessions. The System account cannot be deleted (`acl_denied`): the service rows of the installation in `secret`
|
|
1603
|
-
belong to it.
|
|
1604
|
-
|
|
1605
|
-
## 11. History and point-in-time reads
|
|
1606
|
-
|
|
1607
|
-
### Versions
|
|
1608
|
-
|
|
1609
|
-
<!-- run: guide -->
|
|
1610
|
-
```ts
|
|
1611
|
-
const versions = await db.Product(kettle).versions() // all versions: op, rev, author, at, data
|
|
1612
|
-
const followed = await db.Product(mug2).versions({ follow: true }) // also across id changes (rekey, reclass)
|
|
1613
|
-
```
|
|
1614
|
-
|
|
1615
|
-
Every version has an author, an agent, a time, a number, a kind of operation (`create`, `update`, `delete`, `reclass`,
|
|
1616
|
-
`rekey`, `restore`, `purge`, `trim`, `reset`) and a reason (`reason`). The reason comes from the `reason` option of a verb
|
|
1617
|
-
(`update(patch, { reason })`, `create(data, { reason })`, `delete({ confirm, reason })` and others); for every
|
|
1618
|
-
operation of a block, from `db.begin({ reason })` and `db.batch(name, { reason })`; in raw SQL, from the
|
|
1619
|
-
`letopis.reason` setting of the transaction. A deletion version has `$deleted`.
|
|
1620
|
-
|
|
1621
|
-
Without `sort()`, versions go by id and version number (with `follow`, by time). `sort()` sets the order of versions
|
|
1622
|
-
too, and `limit()` limits them in the query itself: `db.Product(kettle).sort('at', 'desc').limit(5).versions()` gives
|
|
1623
|
-
the five latest versions, newest first.
|
|
1624
|
-
|
|
1625
|
-
### Point-in-time read
|
|
1626
|
-
|
|
1627
|
-
<!-- run: guide -->
|
|
1628
|
-
```ts
|
|
1629
|
-
const createdAt = versions[0].at
|
|
1630
|
-
const atCreation = await db.Product(kettle).asOf(createdAt).first() // the price at creation time: 2500
|
|
1631
|
-
const assortment = await db.Shop(shop).Product().asOf(createdAt).rows()
|
|
1632
|
-
```
|
|
1633
|
-
|
|
1634
|
-
`asOf(moment)` is the state "as it was", from the journal; the moment is an ISO string or a `Date`. A point-in-time
|
|
1635
|
-
read works at any step of a chain, including with `deep()`. Only the current state is cheap: a point-in-time read is
|
|
1636
|
-
rebuilt from the journal. The version time `at` stores microseconds, while a `Date` holds only milliseconds: pass the
|
|
1637
|
-
`at` string to `asOf` as is, otherwise a read at the moment of a version may miss it.
|
|
1638
|
-
|
|
1639
|
-
### Deleted rows
|
|
1640
|
-
|
|
1641
|
-
<!-- run: guide -->
|
|
1642
|
-
```ts
|
|
1643
|
-
const withGone = await db.Customer().withDeleted().rows() // live and deleted; deleted rows have $deleted: true
|
|
1644
|
-
```
|
|
1645
|
-
|
|
1646
|
-
### Retention policy
|
|
1647
|
-
|
|
1648
|
-
Without a policy, the journal keeps every version. A policy sets how long to keep old versions and with what
|
|
1649
|
-
granularity:
|
|
1650
|
-
|
|
1651
|
-
```ts
|
|
1652
|
-
history: { all: '1 day', daily: '1 week', weekly: '1 month', monthly: '1 year', yearly: 'forever', tz: 'Europe/Moscow' }
|
|
1653
|
-
```
|
|
1654
|
-
|
|
1655
|
-
This means: all versions for the last day; for the last week, one per day; for the last month, one per week; for the
|
|
1656
|
-
last year, one per month; older, one per year, forever. Each period keeps its last version. A week starts on Monday;
|
|
1657
|
-
period boundaries are in the `tz` time zone (UTC by default).
|
|
1658
|
-
|
|
1659
|
-
- A policy is set on a class (descendants inherit it) or for all classes without their own policy: `up({ history })`.
|
|
1660
|
-
- If the last tier has a term instead of `'forever'`, the history older than the term is deleted: `{ all: '30 days' }`.
|
|
1661
|
-
- While the object's history is alive, the current version, version 1 and versions of special kinds (`create`,
|
|
1662
|
-
`delete`, `reclass`, `rekey`, `restore`, `purge`, `reset`, `trim`) are not deleted. If the tombstone of a deleted
|
|
1663
|
-
object is older than the last tier with a term, its history is erased entirely. The history of system classes is not
|
|
1664
|
-
thinned.
|
|
1665
|
-
- `maintain()` does the thinning in batches of 10,000 versions ([section 13](#13-operations)).
|
|
1666
|
-
- In the thinned part, a point-in-time read shows the last kept version before the moment, so its precision equals the
|
|
1667
|
-
period of the tier. A subscriber that lags behind by more than the `all` tier gets `cursor_expired`.
|
|
1668
|
-
|
|
1669
|
-
### What changed and who: `diff` and `blame`
|
|
1670
|
-
|
|
1671
|
-
```ts
|
|
1672
|
-
const [d] = await db.Order(id).diff() // previous version → latest
|
|
1673
|
-
await db.Order(id).diff({ from: 3, to: 7 }) // by version numbers
|
|
1674
|
-
await db.Order(id).diff({ from: '2026-10-01' }) // state at a moment → latest
|
|
1675
|
-
await db.Order(id).diff({ follow: true }) // across id changes (reclass, rekey)
|
|
1676
|
-
const [b] = await db.Order(id).blame() // every field of the latest version
|
|
1677
|
-
await db.Order(id).blame({ at: '2026-10-01', limit: 200 }) // the version at a moment; scan 200 versions
|
|
1678
|
-
```
|
|
1679
|
-
|
|
1680
|
-
Both terminals return an array, one item per object of the step. This is what the answers look like for a kettle
|
|
1681
|
-
whose price was lowered with a reason:
|
|
1682
|
-
|
|
1683
|
-
```ts
|
|
1684
|
-
import type { BlameResult, DiffResult } from 'letopis'
|
|
1685
|
-
|
|
1686
|
-
// db.Product(kettle).blame(): version 2 changed the price, the other leaves date from the creation (version 1)
|
|
1687
|
-
const blamed: BlameResult = {
|
|
1688
|
-
id: '6f1c…', class: 'Product', rev: 2, complete: true,
|
|
1689
|
-
fields: [ // in path order
|
|
1690
|
-
{ path: ['class'], rev: 1, at: '2026-10-27T09:15:02.118204+00:00', author: '<Anna id>', agent: '<service id>',
|
|
1691
|
-
op: 'create', exact: true },
|
|
1692
|
-
{ path: ['data', 'price'], rev: 2, at: '2026-10-27T09:20:41.502617+00:00', author: '<Anna id>', agent: '<service id>',
|
|
1693
|
-
op: 'update', reason: 'promotion discount', exact: true },
|
|
1694
|
-
// ['data', 'sku'], ['data', 'title'], ['links', 'Shop'], ['owner'] — also version 1, like ['class']
|
|
1695
|
-
],
|
|
1696
|
-
}
|
|
1697
|
-
// db.Product(kettle).diff(): previous version → latest
|
|
1698
|
-
const diffed: DiffResult = {
|
|
1699
|
-
id: '6f1c…', class: 'Product',
|
|
1700
|
-
from: { rev: 1, at: '2026-10-27T09:15:02.118204+00:00' },
|
|
1701
|
-
to: { rev: 2, at: '2026-10-27T09:20:41.502617+00:00' },
|
|
1702
|
-
changes: [{ path: ['data', 'price'], kind: 'changed', before: 2500, after: 2000 }],
|
|
1703
|
-
}
|
|
1704
|
-
```
|
|
1705
|
-
|
|
1706
|
-
`agent` here is the service that writes on behalf of Anna (`db = await svc.as(…)`); a write without impersonation has
|
|
1707
|
-
none.
|
|
1708
|
-
|
|
1709
|
-
- `diff()` returns an array of `DiffResult`: `{ id, class, from, to, changes }`. `from` and `to` are the compared
|
|
1710
|
-
versions `{ rev, at }`. They have `id` if the version belongs to another id (`follow: true` across `reclass` or
|
|
1711
|
-
`rekey`), and `to` has `deleted: true` if the object was deleted at that moment; then `changes` is empty. `from` is
|
|
1712
|
-
`null` if the object did not exist yet: every path is `added`. If there is no version at all, `class` and `to` are
|
|
1713
|
-
`null` too.
|
|
1714
|
-
- A change, `DiffChange`, is `{ path, kind, before, after }`. `path` is an array of segments, not a dotted string;
|
|
1715
|
-
the first segment is the section of the row: `data`, `links`, `tags`, `owner`, `class`. `kind` is `added` (there
|
|
1716
|
-
was no value), `removed` (the value is gone) or `changed`. A missing value is omitted: `added` has no `before`,
|
|
1717
|
-
`removed` has no `after`. Data objects are compared by keys, arrays as a whole. A multi-valued end and tags are
|
|
1718
|
-
sets: one change per id or tag, the path is `['links', role]` or `['tags']`, and the id or tag itself is in
|
|
1719
|
-
`before` or `after`.
|
|
1720
|
-
- `blame()` returns an array of `BlameResult`: `{ id, class, rev, complete, fields }`, plus `deleted: true` if that
|
|
1721
|
-
version is a tombstone. An object with no versions at that moment gives no item. `fields` is an array of
|
|
1722
|
-
`BlameField`, one item per leaf of the version, ordered by path: `{ path, rev, at, author, op, exact }` and, when
|
|
1723
|
-
present, `agent`, `reason`, `id`. A leaf is a scalar or an array in the data, an empty object, the role of a
|
|
1724
|
-
single-valued end, every id of a multi-valued end (`['links', role, id]`), every tag (`['tags', tag]`), `owner` and
|
|
1725
|
-
`class`. `rev`, `at`, `author`, `agent`, `op` and `reason` come from the version that changed the leaf last. Empty
|
|
1726
|
-
`agent` and `reason` are omitted; `id` is present only if it is a version of the former id (the history went
|
|
1727
|
-
through `reclass` or `rekey`).
|
|
1728
|
-
- `exact: false` means there are gaps between the current version and the found one (the history is thinned,
|
|
1729
|
-
trimmed or partly invisible): a missing version may have changed the leaf too. `complete: true` means the whole
|
|
1730
|
-
history was scanned without gaps; by default at most the 1000 latest versions are scanned.
|
|
1731
|
-
- The database computes both with the caller's rights: raw SQL gets the same, `select "v2.shop".blame('<id>')`,
|
|
1732
|
-
`select "v2.shop".diff_versions('<id>')`.
|
|
1733
|
-
- The journal column `changed` (the paths changed by a version, which make `blame` fast) is not granted to the
|
|
1734
|
-
application: the paths describe the difference from the previous version, and the reader may not see that one.
|
|
1735
|
-
So `select * from log` does not work for the application — list the columns; `blame` gets the paths through a
|
|
1736
|
-
database function only when both versions are visible.
|
|
1737
|
-
|
|
1738
|
-
### Trimming history
|
|
1739
|
-
|
|
1740
|
-
<!-- run: guide -->
|
|
1741
|
-
```ts
|
|
1742
|
-
const trimmed = await db.trimHistory(kettle, new Date()) // erase the versions of the kettle before this moment
|
|
1743
|
-
```
|
|
1744
|
-
|
|
1745
|
-
`db.trimHistory(target, moment)` erases the history before the moment right away. The target is a class with its
|
|
1746
|
-
descendants (by name), an id, a list of ids or rows. This is the middle step of the "right to be forgotten":
|
|
1747
|
-
`anonymize` → history trimming → deletion and `purge`. In place of version 1, a `trim` record stays with its time, so a
|
|
1748
|
-
point-in-time read knows when the object appeared.
|
|
1749
|
-
|
|
1750
|
-
### Integrity check
|
|
1751
|
-
|
|
1752
|
-
<!-- run: guide -->
|
|
1753
|
-
```ts
|
|
1754
|
-
const audit = await adm.verify({ data: true }) // { ok, issues, anchor }
|
|
1755
|
-
```
|
|
1756
|
-
|
|
1757
|
-
`verify()` recomputes the hash chains of all versions and returns an anchor: a hash of all chain heads. Keep the anchor
|
|
1758
|
-
outside the database: a mismatch with it exposes tampering with the latest versions. `{ data: true }` also checks again
|
|
1759
|
-
the data, ends, tenants, owners and ids of all rows. The system administrator or an admin connection can call it; from
|
|
1760
|
-
the command line, use `letopis verify [--data]` (exit code 1 if something is found). Legitimate gaps (`trim` and
|
|
1761
|
-
`purge` records, thinning and `reset`) are not counted as errors.
|
|
1762
|
-
|
|
1763
|
-
## 12. Change subscription and cache
|
|
1764
|
-
|
|
1765
|
-
### Subscription
|
|
1766
|
-
|
|
1767
|
-
<!-- run: guide -->
|
|
1768
|
-
```ts
|
|
1769
|
-
const sub = db.watch({ from: 'start', classes: ['Product'] }) // the whole journal of the class, then new events
|
|
1770
|
-
await db.Product(kettle).update({ price: 2100 }).rows()
|
|
1771
|
-
let cursor = ''
|
|
1772
|
-
for await (const ev of sub) { // ev: { cursor, row, id, class, op, rev, at }
|
|
1773
|
-
cursor = sub.cursor! // the subscription cursor: watch({ from: cursor }) continues after this event
|
|
1774
|
-
if (ev.id === kettle.id && ev.row.data.price === 2100) break // break (or sub.close()) stops the subscription
|
|
1775
|
-
}
|
|
1776
|
-
```
|
|
1777
|
-
|
|
1778
|
-
- `from` is `'now'` (the default: only changes after subscribing), `'start'` (the whole journal) or a saved cursor.
|
|
1779
|
-
`db.watch()` returns at once, and the subscription takes the "now" moment with its first query to the database: a
|
|
1780
|
-
write made right after `db.watch()` may commit earlier and miss the stream. `await sub.ready()` waits for that
|
|
1781
|
-
moment: a write after it comes to the stream. The example above reads from `'start'`.
|
|
1782
|
-
- Calls of `sub.next()` go in turn: parallel calls get different events. A call abandoned by a timeout
|
|
1783
|
-
(`Promise.race`) still gets its event — keep its promise and wait for it again.
|
|
1784
|
-
- `classes`: only these classes with their descendants; `pollMs`: fallback polling (1000 ms). A row that left these
|
|
1785
|
-
classes through a class change without an id change arrives as a deletion event in the previous class (`op:
|
|
1786
|
-
'delete'`, `row` is the previous version with `$deleted`), even if the subscriber cannot see the new class. The
|
|
1787
|
-
event comes if the subscriber could see the previous version.
|
|
1788
|
-
- Events come without losses or repeats, under the reader's permissions, in the order of transaction numbers, and
|
|
1789
|
-
within a transaction in the order of writing. This is not the commit order: compare versions of one object by `rev`.
|
|
1790
|
-
- A long write transaction in any database of the cluster delays the stream: `await sub.lag()` shows the lag in ms.
|
|
1791
|
-
- A cursor older than the `all` tier of the retention policy gives `cursor_expired`: read the current state again and
|
|
1792
|
-
subscribe with `'now'`. The time in the subscription cursor `sub.cursor` is the time of the last event, and once the
|
|
1793
|
-
subscription has caught up with the journal, the time of the poll, so a subscription that got no events for a day
|
|
1794
|
-
does not expire. The event cursor `ev.cursor` keeps the time of the event itself.
|
|
1795
|
-
- Without `LISTEN` (`connect({ listen: false })`), the subscription polls the journal.
|
|
1796
|
-
- A subscription without losses is a saved cursor: after processing an event, store `sub.cursor` (the position after
|
|
1797
|
-
the last delivered event) and resume from it after a restart. `ev.cursor` also continues the stream, but after a
|
|
1798
|
-
quiet period longer than the `all` tier such a cursor is older than the tier, and resuming from it gives
|
|
1799
|
-
`cursor_expired`.
|
|
1800
|
-
|
|
1801
|
-
### Projectors
|
|
1802
|
-
|
|
1803
|
-
A projector builds rows of derived classes from the journal: summaries, denormalized cards and anything else that may
|
|
1804
|
-
be computed with a lag behind the writes. The application writes the source classes (`from`), and only the projector
|
|
1805
|
-
writes the rows of the `into` classes. Each `into` class is marked with the option `derivedBy` holding the projector
|
|
1806
|
-
name ([Class settings](#class-settings)). Only an administrator creates derived classes: sync them with an admin
|
|
1807
|
-
connection holding a tenant session — `adm` from section 3 (`connect({ dsn: ADMIN, schema, token, allowBypassRls:
|
|
1808
|
-
true })`) or `as()` of a service on such a connection; in the CLI, `letopis sync` with the `--dsn` of the admin role.
|
|
1809
|
-
For an application session it is `acl_denied`. In the example below, the projector counts product sales from the
|
|
1810
|
-
order lines `line`.
|
|
1811
|
-
|
|
1812
|
-
```ts
|
|
1813
|
-
const ProductSales = hub('ProductSales', { qty: z.number().int(), revenue: z.number().int() }, {
|
|
1814
|
-
ends: { Product: one(Product).onDelete('cascade') }, key: ['Product'], derivedBy: 'product-sales', history: false,
|
|
1815
|
-
}) // the derived class is in the model; syncModels — on an admin connection
|
|
1816
|
-
|
|
1817
|
-
const sales = adm.projector('product-sales', { // adm is an admin connection
|
|
1818
|
-
from: ['line'], // source classes with descendants; they must keep a journal
|
|
1819
|
-
into: ['ProductSales'], // the derived classes of this projector
|
|
1820
|
-
version: 1, // the handle logic changed: a new version and a rebuild
|
|
1821
|
-
withBefore: true, // c.before is the previous version of the object or null
|
|
1822
|
-
handle(changes, out) {
|
|
1823
|
-
for (const c of changes) {
|
|
1824
|
-
if (c.op === 'purge' || c.op === 'trim' || c.op === 'reset') continue // service changes, not versions
|
|
1825
|
-
const was = c.before // the previous version of the order line: subtract it
|
|
1826
|
-
const now = c.row.$deleted ? null : c.row // the new version: add it; a deletion has none
|
|
1827
|
-
if (was) out.inc('ProductSales', { Product: was.links.Product },
|
|
1828
|
-
{ qty: -Number(was.data.qty), revenue: -Number(was.data.qty) * Number(was.data.price) }, { tenant: c.tenant })
|
|
1829
|
-
if (now) out.inc('ProductSales', { Product: now.links.Product },
|
|
1830
|
-
{ qty: Number(now.data.qty), revenue: Number(now.data.qty) * Number(now.data.price) }, { tenant: c.tenant })
|
|
1831
|
-
}
|
|
1832
|
-
},
|
|
1833
|
-
})
|
|
1834
|
-
const report = await sales.run({ once: true }) // catch up with the journal: { batches, changes, created, updated, … }
|
|
1835
|
-
const state = await sales.status() // { name, state, version, cursor, at, lag, failed, rebuildingTenants }
|
|
1836
|
-
await sales.rebuild() // rebuild from the current state; { chunked: true } works in batches
|
|
1837
|
-
await sales.rebuild({ tenant: tenantId }) // rebuild one tenant, for example a dropped one (failed)
|
|
1838
|
-
await sales.rebuild({ tenant: tenantId, chunked: true }) // the same in batches: meanwhile the journal tail goes on for all
|
|
1839
|
-
sales.run().catch((err) => console.error('the projector stopped', err)) // follow the journal until stop()
|
|
1840
|
-
sales.close() // stop() after the current batch and unsubscribe from signals
|
|
1841
|
-
```
|
|
1842
|
-
|
|
1843
|
-
- A projector needs an admin connection: `connect({ allowBypassRls: true })` with a role that may become the owner of
|
|
1844
|
-
the schema. Otherwise you get `admin_required`, and inside `db.begin()` you get `invalid_query`. The projector reads
|
|
1845
|
-
the journal of all tenants.
|
|
1846
|
-
- The first `run()` rebuilds the projection from the current state: the journal is thinned by the retention policy
|
|
1847
|
-
and cut by `purge`, so the projection cannot be assembled from the start of the journal.
|
|
1848
|
-
- Each batch is one transaction: the journal changes, `handle`, the written results and the cursor move are committed
|
|
1849
|
-
together. A break in the middle of a batch rolls the whole batch back, and the retry starts from the same cursor, so
|
|
1850
|
-
every change is applied exactly once. A projector has one writer: a second process of the same projector skips the
|
|
1851
|
-
batch (the `skipped` counter of the report), and `rebuild()` started while another process writes a batch gives
|
|
1852
|
-
`projector_busy`.
|
|
1853
|
-
- A batch is split by tenant: `handle` gets the changes of one tenant, and its results are written under a savepoint
|
|
1854
|
-
of its own. If the database rejects writing the results of a tenant (the data do not pass the description of its
|
|
1855
|
-
derived class, the target of an end is not found and so on) or `out` rejects a row or a delta built from its data
|
|
1856
|
-
(a delta that is not a number), only its changes are rolled back: the tenant drops out, and the others are written
|
|
1857
|
-
in the same batch. A dropped tenant gets into `status().failed` — `{ [tenant id]: { cursor, code, message } }`, and
|
|
1858
|
-
`message` names the tenant and the row id — and its changes are skipped until it is rebuilt. The derived rows of a
|
|
1859
|
-
dropped tenant no longer change: a tenant dropped in `run()` keeps them as they were before the failed batch, and a
|
|
1860
|
-
tenant dropped in the middle of a rebuild is left without them — the partly built rows are deleted. An exception
|
|
1861
|
-
from `handle`, a breach of the `out` limits and a failure of the environment (a deadlock, a cancelled query, a
|
|
1862
|
-
broken connection) roll the whole batch back, and `run()` is rejected. So catch `run()` without `once`: after an
|
|
1863
|
-
error it stops.
|
|
1864
|
-
- A change (`ProjectorChange`) is `cursor`, `id`, `class`, `tenant`, `op`, `rev`, `at`, `author`, `agent`, `reason`,
|
|
1865
|
-
`moved` and `row`, the version from the journal (for a deletion, a tombstone with `$deleted`). During a rebuild, the
|
|
1866
|
-
changes are rows of the current state: `rebuild: true`, `op` is `create`, `cursor` is empty.
|
|
1867
|
-
- A class change that keeps the id (classes without a key: a raw `UPDATE … SET class` and `reclass()`) is written to
|
|
1868
|
-
the journal as a version of the new class. If it takes the row out of the `from` family, the projector gets a
|
|
1869
|
-
deletion: `op` is `delete`, `row` is the last version of the row in the family with `$deleted`, and `rev`, `at` and
|
|
1870
|
-
the author are those of the class change. A row that enters the family this way comes as a `reclass` change with
|
|
1871
|
-
`before` = `null`.
|
|
1872
|
-
- With `withBefore`, a change has `before`, the previous version of the same id. `before` is `null` if there is no
|
|
1873
|
-
previous version (a creation and the first version of a new id after `reclass` and `rekey`), if the previous
|
|
1874
|
-
version is a deletion (this happens after `restore`) or lies outside the `from` family, and for every change of a
|
|
1875
|
-
rebuild. With `reclass` and `rekey`, the former data are subtracted through the tombstone of the former id: it comes
|
|
1876
|
-
as a change with `$deleted`, its `before` is the last version of the former id, and `moved` of both changes points
|
|
1877
|
-
to the other id. So the fold of the example gives the same sum after `rekey`. The service changes `purge`, `trim`
|
|
1878
|
-
and `reset` have no `before` field at all. So subtract `before` only after checking it for `null`, as in the example
|
|
1879
|
-
above.
|
|
1880
|
-
- The service journal records `purge`, `trim` and `reset` also come as changes, and their `rev` is 1. So check `op`
|
|
1881
|
-
first, and only then compare `rev`.
|
|
1882
|
-
- `out.upsert(class, row, { tenant })` creates a row or replaces it entirely by the class key.
|
|
1883
|
-
`out.inc(class, row, deltas, { tenant })` creates a row or completes it and adds the numbers without reading the row
|
|
1884
|
-
(missing fields count from 0). `out.delete(class, key or id, { tenant })` deletes a row and does nothing if there is
|
|
1885
|
-
none. `await out.get(class, key or id, { tenant })` reads the current row together with the writes of the batch.
|
|
1886
|
-
`delete` and `get` find a row of a class with a key by the key fields (`{ day: '2026-10-29' }`), and by id (a string
|
|
1887
|
-
or `{ $id }`) only a row of a class without a key; an id alone for a class with a key is `invalid_data`, and the
|
|
1888
|
-
batch is rolled back.
|
|
1889
|
-
`out` writes only to the `into` classes and only to the tenant of the changes passed to `handle`; otherwise you get
|
|
1890
|
-
`derived_class` or `tenant_denied`, and the batch is rolled back.
|
|
1891
|
-
- `out` adds up the increments of a row given by key fields only (or by `$id` alone) before sending them to the
|
|
1892
|
-
database, as long as there is no `upsert`, `delete` or `get` between them: one edit of a row per batch, and the class
|
|
1893
|
-
rule is checked on the total of its increments. `inc` does not fold a row with other fields (data, `$tags`,
|
|
1894
|
-
`$owner`): its fields edit the row, and the order of such edits is kept.
|
|
1895
|
-
- A row for `out` is in the loader format: fields and end roles at the top level, the service keys `$tags`, `$owner`
|
|
1896
|
-
and `$id` (`$id` is needed only by a class without a key). The value of a role is the id of the target. An edit that
|
|
1897
|
-
changes nothing gives no version. Versions of derived rows are written by System with the agent
|
|
1898
|
-
`letopis projector <name>`; the journal is kept according to the `history` of the class, and the change signal is
|
|
1899
|
-
always sent.
|
|
1900
|
-
- `rebuild()` rebuilds the projection atomically, in one transaction: until it commits, readers see the previous rows,
|
|
1901
|
-
but the subscription horizon stands still. `rebuild({ chunked: true })` works in batches: the derived rows are
|
|
1902
|
-
deleted at once, the data stays incomplete until the rebuild ends, and `status().state` is `'rebuilding'`. The next
|
|
1903
|
-
start continues an interrupted batch rebuild. `run()` itself, on the first start of a new projection and with
|
|
1904
|
-
`onExpired: 'rebuild'`, rebuilds in batches: the subscription horizon of the cluster stands still no longer than a
|
|
1905
|
-
batch. Meanwhile a second process of the same projector waits or continues the same rebuild instead of starting it
|
|
1906
|
-
anew. A rebuild deletes the rows of the participants only, and only of the classes of their `into` families, and
|
|
1907
|
-
starts all participants anew: the marks of dropped tenants are removed, and a tenant whose data fail again drops
|
|
1908
|
-
out again.
|
|
1909
|
-
- `rebuild({ tenant: id })` rebuilds one tenant in one transaction: it deletes its derived rows, builds them from its
|
|
1910
|
-
current source rows and removes its mark in `status().failed`; the projector does not apply again the changes the
|
|
1911
|
-
rebuild has already counted. Fix the data or the class description of the tenant, then rebuild it. If the database
|
|
1912
|
-
rejects the write again, the rebuild is rolled back with this error, and the tenant stays dropped. No projection yet
|
|
1913
|
-
or a batch rebuild in progress gives `rebuild_required`; a tenant that is not a participant gives `tenant_denied`.
|
|
1914
|
-
With `chunked: true` — in batches, each in a transaction of its own with its own snapshot, while the journal tail goes
|
|
1915
|
-
on for all: the tail skips the tenant's changes above the last batch (a batch will read them) and, in a passed batch,
|
|
1916
|
-
those whose transaction is visible in its snapshot. The tenant's derived rows are deleted at once and are incomplete
|
|
1917
|
-
until the end of the rebuild (`status().rebuildingTenants`); the rebuild waits out a lock held by `run()` of another
|
|
1918
|
-
process. An interrupted rebuild (the process stopped) is continued by `run()` of any process, with or without
|
|
1919
|
-
`onExpired`.
|
|
1920
|
-
- `status().state` describes the projection in the database, not a process: `'new'` means there is no projection yet,
|
|
1921
|
-
and the first `run()` or `rebuild()` builds it; `'running'` means the projection is built and kept up with the
|
|
1922
|
-
journal (it does not mean that some process is executing `run()` right now); `'rebuilding'` means a batch rebuild
|
|
1923
|
-
is in progress or was interrupted.
|
|
1924
|
-
- `version` is the version of the `handle` logic. If the projection was built by another version, `run()` throws
|
|
1925
|
-
`rebuild_required`, and a rebuild with `rebuild()` is needed.
|
|
1926
|
-
- If the cursor is older than the `all` tier of the retention policy of the sources and there are changes after it,
|
|
1927
|
-
some changes may have been thinned out. Each tenant has a tier of its own: it is the description of its classes. If
|
|
1928
|
-
the cursor is older than the tier of every participant, `run()` throws `cursor_expired`, and with
|
|
1929
|
-
`onExpired: 'rebuild'` the projector rebuilds entirely, in batches. If it is older than the tier of only some of
|
|
1930
|
-
them, they drop out (`status().failed`, code `cursor_expired`) and the others go on, and with
|
|
1931
|
-
`onExpired: 'rebuild'` the dropped tenants rebuild themselves, in batches: the rebuild state lies in the checkpoint,
|
|
1932
|
-
and `run()` of any process continues it; a lock held by another process is only a retry on the next pass, and a data
|
|
1933
|
-
error drops the tenant with its code, without retries. The tier of a rebuilt tenant counts from the start of its
|
|
1934
|
-
rebuild until the cursor catches up with it: a lagging projector does not drop it again. A projector without new
|
|
1935
|
-
changes does not expire. `status().lag` is the age of the oldest unprocessed change in ms.
|
|
1936
|
-
- A projector works with the tenants in which every `into` class is marked with its `derivedBy`; it skips the changes
|
|
1937
|
-
of other tenants, so a class with the same name and no mark in another tenant does not get in its way. No such tenant
|
|
1938
|
-
at all — `derived_class`. Only an administrator may mark the classes of a tenant, so it is the administrator who
|
|
1939
|
-
chooses the participants: a tenant user cannot join a projector on their own. The option `tenants: [id, …]`
|
|
1940
|
-
is an explicit list of participants: the projector works only with these tenants (their `into` classes must be
|
|
1941
|
-
marked too) and ignores the marks of the others. The sources are the `from` families of these tenants; an `into`
|
|
1942
|
-
class is never a source, even if it falls into `from`: a projector does not read its own results. A projector does
|
|
1943
|
-
not see the changes of a class with `history: false`: it has no journal. So a tenant with such a class among the
|
|
1944
|
-
sources drops out with the code `invalid_class`, and the projector does not build its derived rows. Turn the journal
|
|
1945
|
-
of the class on and rebuild the tenant with `rebuild({ tenant })`.
|
|
1946
|
-
- `batch` is how many changes go into a batch (500), `pollMs` is fallback polling (1000 ms): source signals wake the
|
|
1947
|
-
projector earlier.
|
|
1948
|
-
- `lockTimeout` is how long a projector transaction waits for another transaction's row lock (5000 ms; 0 — no limit).
|
|
1949
|
-
Otherwise a transaction of any tenant holding a total row or an end target would hold the batch of all tenants.
|
|
1950
|
-
Longer than that, the batch rolls back, the cursor stays, and `run()` does not reject but skips the pass
|
|
1951
|
-
(`skipped`) and repeats it. The report names the tenant whose row stopped the batch (`locked`).
|
|
1952
|
-
`rebuild({ tenant, chunked: true })` waits such a lock out.
|
|
1953
|
-
- `lockRetries` is how many passes in a row the writes of one tenant may run into `lockTimeout` before it drops out
|
|
1954
|
-
(0 by default — it never drops out). The dropped tenant gets the code `55P03`, and the batch goes on for the others;
|
|
1955
|
-
its changes are skipped until `rebuild({ tenant })`. The process keeps the count, and a successful write of the
|
|
1956
|
-
tenant resets it. Note: a long exclusive lock on the family of a System class (a tightening of its description) will
|
|
1957
|
-
hit every tenant in turn.
|
|
1958
|
-
- Every edit that `out` sends to the database is a separate edit of a row, and the speed of a projector is set by the
|
|
1959
|
-
number of edits, not by the number of changes. The folding of `inc` brings the edits of a row down to one per batch,
|
|
1960
|
-
so the handle from the example above (`out.inc` for every change) caught up with 6 thousand changes per second with
|
|
1961
|
-
`batch` 500 and 33 thousand with `batch` 2000 on the benchmark machine ([section 17](#17-performance)): the larger
|
|
1962
|
-
the batch, the more increments of one row fold into one edit.
|
|
1963
|
-
|
|
1964
|
-
### Recipe: daily revenue
|
|
1965
|
-
|
|
1966
|
-
An aggregate in the database and a projector solve similar tasks in different ways:
|
|
1967
|
-
|
|
1968
|
-
| | Aggregate in the database | Projector |
|
|
1969
|
-
|---|---|---|
|
|
1970
|
-
| when it is computed | in the same transaction as the write of the source | after the commit, with a lag (`status().lag`) |
|
|
1971
|
-
| what it computes | the sum of a field or the number of source rows for one target row | anything: summaries by day and by group, denormalized cards |
|
|
1972
|
-
| matches the source | always, raw SQL included | when the projector has caught up with the journal |
|
|
1973
|
-
| cost of a write | every operation locks the target row and writes its version, so the writers of one target queue up | writers do not wait for each other: the projector applies changes in batches after the commit |
|
|
1974
|
-
| rule over the total | `min(0)` rejects an operation that would take the total beyond the rule | the total does not block writes: a data error drops the tenant out of the projection |
|
|
1975
|
-
| what to run | nothing: the database keeps the field | a process on an admin connection |
|
|
1976
|
-
|
|
1977
|
-
A wallet balance that must not go below zero is an aggregate
|
|
1978
|
-
([Recipe: balance as an aggregate](#recipe-balance-as-an-aggregate)). The revenue of a shop by day is a total over many
|
|
1979
|
-
sales that may lag a little, which is a job for a projector:
|
|
1980
|
-
|
|
1981
|
-
```ts
|
|
1982
|
-
const Sale = hub('Sale', { day: z.iso.date(), amount: z.number().int().min(0) }, { ends: { Shop } }) // a sale: day and amount
|
|
1983
|
-
const DayRevenue = hub('DayRevenue', { day: z.iso.date(), revenue: z.number().int(), sales: z.number().int() }, {
|
|
1984
|
-
ends: { Shop: one(Shop).onDelete('cascade') }, key: ['Shop', 'day'], derivedBy: 'day-revenue', history: false,
|
|
1985
|
-
})
|
|
1986
|
-
await syncModels(db, [Sale], { apply: true })
|
|
1987
|
-
await syncModels(adm, [DayRevenue], { apply: true }) // the derived class is created by an administrator: adm holds a tenant session
|
|
1988
|
-
|
|
1989
|
-
const revenue = adm.projector('day-revenue', {
|
|
1990
|
-
from: ['Sale'], into: ['DayRevenue'], version: 1, withBefore: true,
|
|
1991
|
-
handle(changes, out) {
|
|
1992
|
-
for (const c of changes) {
|
|
1993
|
-
if (c.op === 'purge' || c.op === 'trim' || c.op === 'reset') continue // service changes
|
|
1994
|
-
const was = c.before // the previous version of the sale: subtract it from its day
|
|
1995
|
-
const now = c.row.$deleted ? null : c.row // the new one: add it to its day; a deletion has none
|
|
1996
|
-
if (was) out.inc('DayRevenue', { Shop: was.links.Shop, day: was.data.day },
|
|
1997
|
-
{ revenue: -Number(was.data.amount), sales: -1 }, { tenant: c.tenant })
|
|
1998
|
-
if (now) out.inc('DayRevenue', { Shop: now.links.Shop, day: now.data.day },
|
|
1999
|
-
{ revenue: Number(now.data.amount), sales: 1 }, { tenant: c.tenant })
|
|
2000
|
-
}
|
|
2001
|
-
},
|
|
2002
|
-
})
|
|
2003
|
-
revenue.run().catch((err) => console.error('the revenue projector stopped', err)) // follows the journal until stop()
|
|
2004
|
-
const week = await db.Shop(shop).DayRevenue().sort('data.day', 'desc').limit(7).rows() // the last seven days
|
|
2005
|
-
```
|
|
2006
|
-
|
|
2007
|
-
- The day is a field of the sale, not the version time `at`: with the version time, an edit, `restore` or `rekey`
|
|
2008
|
-
would move the amount of the sale to its own day.
|
|
2009
|
-
- `out.inc` adds without reading the row, so the total does not depend on the order of changes: the journal gives
|
|
2010
|
-
transactions in the order of their first write, not of their commit. Editing a sale subtracts the previous amount
|
|
2011
|
-
from the previous day and adds the new one to the new day; a deletion only subtracts. A day with no sales left stays
|
|
2012
|
-
as a row of zeros. `out` folds the increments of one day in a batch into one edit.
|
|
2013
|
-
- `DayRevenue` has `history: false`: the derived rows do not double the journal, and a rebuild assembles them without
|
|
2014
|
-
history anyway. `Sale` needs a journal: a projector does not see the changes of a class with `history: false`.
|
|
2015
|
-
- The application reads `DayRevenue` rows like any class, and only the projector writes them: `create` or `update` of
|
|
2016
|
-
such a row gives `derived_class`. The total lags behind the sales until the next batch: write signals wake the
|
|
2017
|
-
projector, and without them it polls every `pollMs`.
|
|
2018
|
-
|
|
2019
|
-
### Result cache
|
|
2020
|
-
|
|
2021
|
-
<!-- run: guide -->
|
|
2022
|
-
```ts
|
|
2023
|
-
const cached = await db.Product().cache({ ttl: 30 }).rows() // the result is kept in process memory for 30 s
|
|
2024
|
-
const stats = db.cache.stats() // { size, hits, misses, stores, raced, … }
|
|
2025
|
-
```
|
|
2026
|
-
|
|
2027
|
-
The cache is enabled only explicitly: `.cache()` in a chain or `cache: { ttl }` on a class in the model. This is how it
|
|
2028
|
-
stays correct:
|
|
2029
|
-
|
|
2030
|
-
- the result key includes the query, the parameters, the session, the actor and the tenant, so one user never gets the
|
|
2031
|
-
result of another user;
|
|
2032
|
-
- your own writes clear the results of the changed classes right after commit (including the cascade classes); other
|
|
2033
|
-
writers' changes clear them on a database signal. In the `tenant` and `opaque` modes, a process with the signal key
|
|
2034
|
-
clears a class only in the tenant of the write (System rows — in all tenants); a process without the key clears it
|
|
2035
|
-
in all tenants, and in the `opaque` mode any signal clears its whole cache;
|
|
2036
|
-
- changes of permissions and classes, writes through `db.sql`, a revoked session and a listener reconnect clear the
|
|
2037
|
-
whole cache or the results of the session;
|
|
2038
|
-
- the default term is 60 seconds; the cache does not work inside `db.begin()`;
|
|
2039
|
-
- without `LISTEN` and in the `none` mode the cache is off, and `connect({ cache: { ttlOnly: true } })` enables it: an
|
|
2040
|
-
answer lives until its term ends unless a write of this connection resets it;
|
|
2041
|
-
- `db.load()`, the projector and `db.rebuildAggregates()` reset the cache, even of this connection, only by a database
|
|
2042
|
-
signal.
|
|
2043
|
-
|
|
2044
|
-
A process re-reads the class registry right away on a class change signal (one re-read at a time), and the registry
|
|
2045
|
-
and the signal key on `*`, on an unknown token and on a payload of another mode (and without the key in the `opaque`
|
|
2046
|
-
mode — on any signal) not more often than once in 5 seconds: a stream of signals, fake ones included, does not turn
|
|
2047
|
-
into a stream of queries. In the `none` mode, the registry checks the class
|
|
2048
|
-
fingerprint every 5 seconds, and subscriptions poll the log. A subscription wakes up only for its own classes and its
|
|
2049
|
-
own tenant, and without the key — for any signal; it always reads the events from the log, so nothing is lost.
|
|
2050
|
-
|
|
2051
|
-
Between processes, the cache goes stale for the signal sending interval (`notifyIntervalMs`, 100 ms) plus delivery;
|
|
2052
|
-
`db.flushSignals()` sends the accumulated signals at once. `db.cache.clear()` clears the cache.
|
|
2053
|
-
|
|
2054
|
-
## 13. Operations
|
|
2055
|
-
|
|
2056
|
-
### Maintenance: `maintain()`
|
|
2057
|
-
|
|
2058
|
-
<!-- run: guide -->
|
|
2059
|
-
```ts
|
|
2060
|
-
const report = await adm.maintain() // { sessions, codes, thinned, batches, more, queue, skipped }
|
|
2061
|
-
```
|
|
2062
|
-
|
|
2063
|
-
`maintain()` deletes sessions and one-time codes of all kinds (`OTP` and custom `OTP_<NAME>` from `issueOtp({ kind })`)
|
|
2064
|
-
from the `secret` table — expired ones, as well as
|
|
2065
|
-
revoked sessions and codes that were used, replaced by a new one or ran out of attempts, — thins the journal by the
|
|
2066
|
-
retention policy and checks the PostgreSQL signal queue: if it is more than half full, you get a warning. You do not
|
|
2067
|
-
need `maintain()` for the rows of a deleted account: the deletion itself erases them ([Deleting an account](#deleting-an-account)).
|
|
2068
|
-
The system administrator, a role with the `letopis.maintain` permission or an admin connection can call it. An external scheduler
|
|
2069
|
-
runs it (`letopis maintain`), or the library itself does: `connect({ maintain: '1h' })`. The interval is a number of
|
|
2070
|
-
milliseconds or a string (`'500ms'`, `'30s'`, `'15m'`, `'1h'`, `'1d'`) from 1 ms to 2³¹−1 ms (about 24 days):
|
|
2071
|
-
`connect()` rejects any other value, including zero, a negative number or `NaN`, with `invalid_data` without opening
|
|
2072
|
-
connections. `close()` waits for a scheduled run in progress: it finishes the current batch and does not start the
|
|
2073
|
-
next one. Of several processes, only one works: each batch is its own
|
|
2074
|
-
transaction, and the first thing it does is take the maintenance lock — a row of the `secret` table that neither the
|
|
2075
|
-
application role nor any other database role can take; the lock is released when the batch commits. If another
|
|
2076
|
-
process holds the lock, the call answers `skipped: true` before the first batch, and between batches it stops (that
|
|
2077
|
-
process continues) and reports what it has done with `more: true`. There is no session-level lock, so `maintain()`
|
|
2078
|
-
works behind any pool, including PgBouncer in transaction mode.
|
|
2079
|
-
|
|
2080
|
-
### Reset: `reset()`
|
|
2081
|
-
|
|
2082
|
-
```ts
|
|
2083
|
-
await adm.reset({ level: 'tenant', confirm: 'v2.shop', tenant: SHOP }) // sessions | tenant | all
|
|
2084
|
-
```
|
|
2085
|
-
|
|
2086
|
-
Reset is meant for tests and test environments. It works only if the installation was set up with
|
|
2087
|
-
`up({ allowReset: true })` (otherwise `reset_disabled`, even for System), the caller has the `letopis.reset`
|
|
2088
|
-
permission, and `confirm` matches the schema name (`confirm_mismatch`). Levels: `sessions` resets all sessions
|
|
2089
|
-
except the current one; `tenant` the tenant's data; `all` everything, with the system seed laid down again and a new
|
|
2090
|
-
service key. The first journal record after a reset is `reset`.
|
|
2091
|
-
|
|
2092
|
-
### Model description for people and AI
|
|
2093
|
-
|
|
2094
|
-
<!-- run: guide -->
|
|
2095
|
-
```ts
|
|
2096
|
-
import { describeText } from 'letopis'
|
|
2097
|
-
|
|
2098
|
-
const model = db.describe() // { schema, tenant, classes: [...], methods }
|
|
2099
|
-
const modelText = describeText(model) // the same as text: for the application's llms.txt or a prompt for an AI agent
|
|
2100
|
-
```
|
|
2101
|
-
|
|
2102
|
-
`db.describe()` returns a description of the classes available to the session: fields, ends, keys, policies and
|
|
2103
|
-
metadata, plus the multi-field conditions `anyOf` and `oneOf` with inheritance applied, the derived class mark
|
|
2104
|
-
`derivedBy` and the aggregates `aggregate`, when a class has them. From the command line:
|
|
2105
|
-
`letopis describe --format text --out llms.txt`.
|
|
2106
|
-
|
|
2107
|
-
### Agents and MCP
|
|
2108
|
-
|
|
2109
|
-
An agent — Claude, Cursor or your own — works with the data through MCP tools: it learns the model, reads with
|
|
2110
|
-
chains, looks at the history and, if allowed, writes. The MCP server acts on behalf of one user and only with that
|
|
2111
|
-
user's rights: every call is a chain under the user's session, and the database checks rights and tenants. The agent
|
|
2112
|
-
has no raw SQL.
|
|
2113
|
-
|
|
2114
|
-
| Tool | What it does |
|
|
2115
|
-
|---|---|
|
|
2116
|
-
| `letopis_model` | the model: classes, fields (JSON Schema), the `anyOf` and `oneOf` conditions, ends, keys, aliases, metadata; with `class`, one class in detail |
|
|
2117
|
-
| `letopis_query` | reading with a JSON chain: the start is `class` and `filter`, `id` or `ids`; `steps` are hops by roles and classes; `asOf`, `withDeleted`, `sort`, `limit`, `cursor`; the terminal is `rows`, `first`, `ids`, `count`, `sum`, `avg`, `min`, `max` or `countBy` |
|
|
2118
|
-
| `letopis_history` | `versions`, `diff` or `blame` of one object |
|
|
2119
|
-
| `letopis_write` | `create`, `update` (only with `rev`), `upsert`, `delete`, only with `write: true` (`letopis mcp --write`) |
|
|
2120
|
-
|
|
2121
|
-
A `letopis_query` request is the same chain as `db.Product({ price: lt(1000) }).Shop().sort('data.name').rows()`, written as JSON:
|
|
2122
|
-
|
|
2123
|
-
```json
|
|
2124
|
-
{ "class": "Product", "filter": { "price": { "$lt": 1000 } }, "steps": [{ "step": "Shop" }], "sort": { "field": "data.name" } }
|
|
2125
|
-
```
|
|
2126
|
-
|
|
2127
|
-
A filter value means equality, a nested object addresses the fields of nested objects, `{ "Role": id }` is an end.
|
|
2128
|
-
Operators: `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$between`, `$in`, `$like`, `$ilike`, `$starts`, `$ends`, `$has`,
|
|
2129
|
-
`$hasAny`, `$hasAll`, `$exists`, `$isNull`, `$not`; OR is `{ "$or": [filter, …] }`. An unknown class, role, parameter
|
|
2130
|
-
or operator gives an `isError` answer with a hint of similar names. There are at most 16 `steps` and at most 64
|
|
2131
|
-
conditions in the `$or` of one filter; otherwise `invalid_query`. These limits do not bound the time of a query: set
|
|
2132
|
-
`statement_timeout` for the database role the MCP server connects as (`alter role … set statement_timeout = '5s'`).
|
|
2133
|
-
|
|
2134
|
-
The answer is `structuredContent` and its text copy. `structuredContent` is `{ notice, data }`: `notice` is a constant
|
|
2135
|
-
text of the package saying that `data` holds database data, not instructions, and `data` is the answer itself. The
|
|
2136
|
-
shape of `data` of `letopis_query` depends on the terminal:
|
|
2137
|
-
|
|
2138
|
-
| Terminal | `structuredContent.data` |
|
|
2139
|
-
|---|---|
|
|
2140
|
-
| `rows` (default) | `{ rows, count, more, cursor? }` |
|
|
2141
|
-
| `ids` | `{ ids, count, more, cursor? }` |
|
|
2142
|
-
| `first` | `{ row }`, a row or `null` |
|
|
2143
|
-
| `count` | `{ count }` |
|
|
2144
|
-
| `countBy` | `{ counts, truncated? }` |
|
|
2145
|
-
| `sum`, `avg`, `min`, `max` | `{ value }` |
|
|
2146
|
-
|
|
2147
|
-
For `rows` and `ids`, `count` is the number of rows on this page, not the total. A page holds at most `limit`,
|
|
2148
|
-
`maxRows` rows (100) and `maxBytes` bytes of row JSON (50,000); a row that is itself larger than `maxBytes` comes
|
|
2149
|
-
alone on its page. `more: true` means there are more rows, and then the same request with `cursor` returns the next
|
|
2150
|
-
page. `counts` is `{ value: number }`, the most frequent values; if the groups did not fit into the limits,
|
|
2151
|
-
`truncated: true` comes. `countBy` reads at most `maxRows` + 1 groups from the database. The
|
|
2152
|
-
field of an aggregate and of `countBy` is `field: 'data.<path>'`. Sorting is `sort: { field, dir }`: `field` is `at`,
|
|
2153
|
-
`rev`, `id` or `data.<path>`, `dir` is `'asc'` (default) or `'desc'`; without `sort`, rows go by `id`. `sort`, `limit`
|
|
2154
|
-
and `cursor` work only with `rows`, `first` and `ids`.
|
|
2155
|
-
|
|
2156
|
-
`letopis_model` without `class` answers (in `data`) `{ schema, tenant, classes }`, the same classes as in `db.describe()`. If such
|
|
2157
|
-
a model is larger than `maxBytes`, a brief one comes: `brief: true`, and the classes have only the names of their
|
|
2158
|
-
fields and ends; `letopis_model` with `class` returns each class in detail. `letopis_history` takes `class`, `id` and
|
|
2159
|
-
`kind`. `versions` answers `{ versions, more }`, the latest versions within the answer limits, from the oldest to the
|
|
2160
|
-
newest; `diff` with `from` and `to` (a version number or a moment) answers `{ diff }`; `blame` with `at` and `limit`
|
|
2161
|
-
answers `{ blame }`. If the changes or the leaves did not fit, the answer is cut, and it has `truncated: true` and
|
|
2162
|
-
`total`. `follow` works with `versions` and `diff`.
|
|
2163
|
-
|
|
2164
|
-
A tool refusal is `isError: true` with a `structuredContent` of the form `{ notice, error }`, where `error` is `{ code, message }`, plus `similar` (similar names),
|
|
2165
|
-
`detail` (details) and `sqlstate` (the PostgreSQL code of an error not from letopis) when present. A parameter that
|
|
2166
|
-
does not match the tool input schema (a wrong type, a value outside the list, a missing required one) is rejected by
|
|
2167
|
-
the SDK before letopis: then it is `isError: true` with text only, without `structuredContent`.
|
|
2168
|
-
|
|
2169
|
-
In the text of the answer, everything that came from the database (rows, names and descriptions of classes, error
|
|
2170
|
-
texts, hints) sits inside `<untrusted-data-…>` with a random suffix and a note that this is data, not instructions.
|
|
2171
|
-
Tool descriptions and the server's `instructions` are fixed text of the package: tenants write the `meta` of classes,
|
|
2172
|
-
so it only appears in results.
|
|
2173
|
-
|
|
2174
|
-
Writing. Without `write`, the `letopis_write` tool is not in the tool list, and a call to it is an SDK protocol error
|
|
2175
|
-
(`Tool letopis_write not found`), not a tool answer with `isError`. The ends of a write are set by
|
|
2176
|
-
`links: { Role: id | [id] | null }`, and `create`, `update` and `upsert` answer `{ rows }`. `update` takes only the
|
|
2177
|
-
`rev` of the version that was read: someone else's version gives `conflict`, and no update is lost. `delete` without
|
|
2178
|
-
`confirm` returns a preview of the cascade and the operation code: `{ preview, operation }`. With `confirm: true`, a
|
|
2179
|
-
human confirms the deletion in the client through elicitation (in protocol revision 2025 as a request to the client,
|
|
2180
|
-
in 2026-07-28 as an `input_required` answer), the confirmation is bound to the hash of the operation, and the answer
|
|
2181
|
-
is `{ deleted, operation }`. In revision 2026-07-28, the `requestState` signature with which the client repeats the
|
|
2182
|
-
call is also bound to the user of the server and is valid for 10 minutes: a signature of another user's server and an
|
|
2183
|
-
expired one give the refusal `confirmation_required`. The confirmation window shows the canonical class name and the ids of the targets (the
|
|
2184
|
-
first 10, then «и ещё K», that is "and K more"); a tenant writes the alias and the `meta` of a class, so they do not
|
|
2185
|
-
get into the window. An operation code sent by the agent is not a confirmation; a client without elicitation gets the
|
|
2186
|
-
refusal `confirmation_required`. Every write gets the reason `mcp: letopis_write`, and `blame()` shows it. By default,
|
|
2187
|
-
`letopis_write` does not create, edit or delete the permission classes, `rule`, `Resource`, `member`, `Account`,
|
|
2188
|
-
`Class` and their descendants (rules, memberships, class descriptions): the refusal is `acl_denied`. An agent under an
|
|
2189
|
-
injection in the data would grant a stranger a membership or a rule without a human confirmation. Only the option
|
|
2190
|
-
`allowAclWrite: true` allows it.
|
|
2191
|
-
|
|
2192
|
-
**stdio.** `letopis mcp` is a stdio server for MCP clients. It needs the SDK package: `npm i @modelcontextprotocol/server`
|
|
2193
|
-
(an optional peer dependency). The client passes the connection string, the schema and the token as flags or as the
|
|
2194
|
-
variables `LETOPIS_DSN`, `LETOPIS_SCHEMA`, `LETOPIS_TOKEN`; `--write` turns writing on. Only a user session token is
|
|
2195
|
-
accepted, and the application issues it: `svc.auth.sessionFor(user, { ttl })`, while `auth.switch(tenant)` sets the
|
|
2196
|
-
working tenant of the session. The command rejects the service key (`--api-key`): it grants all `auth.*` rights and
|
|
2197
|
-
would lie on the agent's machine. Without a token and with `--api-key`, the command exits with code 2 and writes the
|
|
2198
|
-
reason to stderr. The command also rejects the token of a session in the System tenant (this is how a system
|
|
2199
|
-
administrator works): it has no `allowSystemAdmin` and `allowAclWrite` options. Client configuration (Claude Desktop,
|
|
2200
|
-
Claude Code, Cursor):
|
|
2201
|
-
|
|
2202
|
-
```json
|
|
2203
|
-
{
|
|
2204
|
-
"mcpServers": {
|
|
2205
|
-
"letopis": {
|
|
2206
|
-
"command": "npx",
|
|
2207
|
-
"args": ["letopis", "mcp"],
|
|
2208
|
-
"env": {
|
|
2209
|
-
"LETOPIS_DSN": "postgres://shop_app:app-secret@localhost:5432/shop",
|
|
2210
|
-
"LETOPIS_SCHEMA": "v2.shop",
|
|
2211
|
-
"LETOPIS_TOKEN": "<user session token>"
|
|
2212
|
-
}
|
|
2213
|
-
}
|
|
2214
|
-
}
|
|
2215
|
-
}
|
|
2216
|
-
```
|
|
2217
|
-
|
|
2218
|
-
**HTTP.** There is no built-in HTTP server. `createMcpServer(db, { write, maxRows, maxBytes })` from `letopis/mcp`
|
|
2219
|
-
returns an MCP server, and the application plugs it into its own `createMcpHandler` handler from the SDK with its own
|
|
2220
|
-
OAuth 2.1 authorization: the application checks the request token and finds the letopis user by it. The connection
|
|
2221
|
-
is a user session (`connect({ token })`) or the facade `svc.as(user)`: the service key stays in the application
|
|
2222
|
-
process. A connection that bypasses RLS is refused with `bypass_rls`; the service root without `as()` and a connection
|
|
2223
|
-
without a session, with `user_session_required`; the facade of an explicit transaction `db.begin()`, with
|
|
2224
|
-
`invalid_query`. The server also refuses a session in the System tenant with `user_session_required` unless the option
|
|
2225
|
-
`allowSystemAdmin: true` is passed: this is how a system administrator works, and their `admin` role grants rights to
|
|
2226
|
-
the accounts, memberships and rules of the installation.
|
|
2227
|
-
|
|
2228
|
-
<!-- run: mcp -->
|
|
2229
|
-
```ts
|
|
2230
|
-
import { up, connect } from 'letopis'
|
|
2231
|
-
import { createMcpServer } from 'letopis/mcp'
|
|
2232
|
-
import { createMcpHandler } from '@modelcontextprotocol/server'
|
|
2233
|
-
|
|
2234
|
-
const ADMIN = 'postgres://postgres:secret@localhost:5432/shop'
|
|
2235
|
-
const APP = 'postgres://shop_app:app-secret@localhost:5432/shop'
|
|
2236
|
-
const r = await up({ dsn: ADMIN, schema: 'shop', quiet: true,
|
|
2237
|
-
tenant: { name: 'Chamomile', user: { name: 'Anna', login: 'anna@shop.ru', password: 'Chamomile-2026!' } } })
|
|
2238
|
-
const svc = await connect({ dsn: APP, schema: r.schema, apiKey: r.serviceKey! })
|
|
2239
|
-
|
|
2240
|
-
// authInfo is the result of the application's check of the request token; it puts the letopis account id into extra
|
|
2241
|
-
const mcp = createMcpHandler(async ({ authInfo }) =>
|
|
2242
|
-
createMcpServer(await svc.as(String(authInfo?.extra?.account), { tenant: r.tenant!.tenant })))
|
|
2243
|
-
// in the application's HTTP handler, after checking the token: return mcp.fetch(request, { authInfo })
|
|
2244
|
-
```
|
|
2245
|
-
|
|
2246
|
-
The handler calls the factory on every HTTP request, so `createMcpServer` checks the connection every time (one query
|
|
2247
|
-
to the database).
|
|
2248
|
-
|
|
2249
|
-
The handler can be checked without a network: the SDK client from the `@modelcontextprotocol/client` package reaches
|
|
2250
|
-
it through the application's `fetch`: `new StreamableHTTPClientTransport(new URL('http://localhost/mcp'), { fetch:
|
|
2251
|
-
(url, init) => mcp.fetch(new Request(url, init), { authInfo }) })`. This is how the `docs` test checks the example
|
|
2252
|
-
above (the `mcp` scenario in `lib/test/docs.test.ts` of the repository): without `write` the client sees three tools,
|
|
2253
|
-
and `letopis_model` answers on behalf of Anna in the shop.
|
|
2254
|
-
|
|
2255
|
-
### Raw SQL
|
|
2256
|
-
|
|
2257
|
-
<!-- run: guide -->
|
|
2258
|
-
```ts
|
|
2259
|
-
const [{ n }] = await db.sql`select count(*)::int as n from "v2.shop".entity where class = 'Product'`
|
|
2260
|
-
```
|
|
2261
|
-
|
|
2262
|
-
`` db.sql`…` `` runs a query in one transaction under the session of the connection and under RLS: it sees the same
|
|
2263
|
-
rows as chains do. `db.transaction(fn)` gives a postgres.js connection for the whole transaction. On a deadlock or a
|
|
2264
|
-
serialization failure, the query and `fn` are retried up to three times, so `fn` must withstand a retry. You can also
|
|
2265
|
-
write with raw SQL: the same triggers check the data, and the database functions `merge` and `inc` are available to the
|
|
2266
|
-
application.
|
|
2267
|
-
|
|
2268
|
-
### Command line
|
|
2269
|
-
|
|
2270
|
-
```sh
|
|
2271
|
-
letopis sync <module> --dsn <dsn> --schema <v2.name> (--token <token> | --api-key <key>) [--apply]
|
|
2272
|
-
letopis types --dsn <dsn> --schema <v2.name> (--token <token> | --api-key <key>) [--out <file>]
|
|
2273
|
-
letopis load <files…> --dsn <dsn> --schema <v2.name> (--token <token> | --api-key <key>) [--class <Class>]
|
|
2274
|
-
[--duplicates error|first|last] [--existing error|skip|update] [--commit-every <N>] [--verify] [--initial]
|
|
2275
|
-
letopis verify --dsn <dsn> --schema <v2.name> [--token <token> | --api-key <key>] [--data]
|
|
2276
|
-
letopis maintain --dsn <dsn> --schema <v2.name> [--token <token> | --api-key <key>] [--batches <N>]
|
|
2277
|
-
letopis describe --dsn <dsn> --schema <v2.name> (--token <token> | --api-key <key>) [--format json|text] [--out <file>]
|
|
2278
|
-
letopis import --from <0.21 dsn> --from-schema <v1.name> --dsn <dsn> --schema <v2.name> [--tenant <name>] [--keyless <Class>]…
|
|
2279
|
-
[--rename <Old>=<New>]… [--update] [--commit-every <N>] [--partition <name>] [--initial]
|
|
2280
|
-
letopis mcp --dsn <dsn> --schema <v2.name> --token <user token> [--write]
|
|
2281
|
-
(or LETOPIS_DSN, LETOPIS_SCHEMA, LETOPIS_TOKEN; the service key --api-key is not accepted)
|
|
2282
|
-
```
|
|
2283
|
-
|
|
2284
|
-
`letopis types` and `letopis describe` build their output from the session's classes, so without a valid session (the
|
|
2285
|
-
token expired, was revoked, is wrong or not given, or the API key is wrong) they exit with code 1 and the `no_session`
|
|
2286
|
-
error and leave the `--out` file untouched.
|
|
2287
|
-
|
|
2288
|
-
### Monitoring
|
|
2289
|
-
|
|
2290
|
-
- `connect({ onQuery, slowMs })`: for every chain query, the hook gets the mode, the classes, the duration, the row
|
|
2291
|
-
count and a slow query flag; without a hook, a slow query prints a warning.
|
|
2292
|
-
- `sub.lag()` is the lag of a subscription; the `maintain()` report shows how full the signal queue is.
|
|
2293
|
-
- Hot reference targets (an organization that almost everything references) accumulate `for key share` locks and
|
|
2294
|
-
MultiXact, as with regular foreign keys. This is worth monitoring.
|
|
2295
|
-
|
|
2296
|
-
### Connection pools and PgBouncer
|
|
2297
|
-
|
|
2298
|
-
With PgBouncer in transaction mode, `LISTEN` is not available: use `connect({ listen: false })`. Then the registry is
|
|
2299
|
-
re-read with `db.refresh()`, the subscription polls the journal, and the result cache is off or works by term only.
|
|
2300
|
-
Prepared statements work with PgBouncer 1.21 and newer; with older versions, pass your own postgres.js pool with
|
|
2301
|
-
`prepare: false` in `connect({ sql })`. `maintain()` takes a transaction-level lock in each batch and works behind
|
|
2302
|
-
such a pool.
|
|
2303
|
-
|
|
2304
|
-
<!-- run: guide -->
|
|
2305
|
-
```ts
|
|
2306
|
-
await adm.close()
|
|
2307
|
-
await anna.close()
|
|
2308
|
-
await svc.close() // and the connections obtained through svc.as()
|
|
2309
|
-
```
|
|
2310
|
-
|
|
2311
|
-
## 14. Migrating from 0.21
|
|
2312
|
-
|
|
2313
|
-
1.0 is a new implementation, not a continuation of 0.21: a 0.21 schema is not upgraded in place. 1.0 is installed into
|
|
2314
|
-
a new schema, the `letopis import` command moves the data, and the code is ported with the replacement table in
|
|
2315
|
-
[MIGRATION.md](MIGRATION.md) (in Russian). The main differences in behavior: `create` does not overwrite (a taken key
|
|
2316
|
-
gives `exists`), `count()` counts entities, paths are `paths()`, permissions and tenant isolation are always on.
|
|
2317
|
-
|
|
2318
|
-
```sh
|
|
2319
|
-
npx letopis import --from postgres://…/old --from-schema v1.salon --dsn postgres://…/new --schema v2.salon \
|
|
2320
|
-
--tenant salon [--keyless booking] [--rename link=link_] [--update] [--commit-every 100000]
|
|
2321
|
-
```
|
|
2322
|
-
|
|
2323
|
-
- The 1.0 schema is installed in advance (`up()`); both connections are admin connections. The source is the 0.21
|
|
2324
|
-
tables (`Schema`, `Entity`, `Account`, `Credential`, `Resource`, `Rule`) in any database: PostgreSQL 17 with
|
|
2325
|
-
TimescaleDB or a dump of plain tables. The loader (history mode, `existing: 'skip'`, agent `letopis import`) and the
|
|
2326
|
-
owner functions do the writing.
|
|
2327
|
-
- Classes become `Class` rows in System. fastest-validator rules are translated to JSON Schema (`optional` lets `null`
|
|
2328
|
-
through, `$$strict` → `additionalProperties: false`, `record` → `propertyNames`). Ends become roles named after the
|
|
2329
|
-
target class; a union of all descendants of a class becomes a polymorphic end. The `Entity` and `link` roots are
|
|
2330
|
-
dropped; `description`, `appearance` and `order` move to `meta`. For names taken by 1.0 methods, use `--rename` (by
|
|
2331
|
-
default, the name with `_`).
|
|
2332
|
-
- Ids: for a class with a key (the v5 rule from 0.21), v5 of the key; for a class without a key, v5 of the old id.
|
|
2333
|
-
System data moves to the `--tenant` tenant, accounts to `Account` rows, credentials to `secret` as they are (scrypt,
|
|
2334
|
-
keys); one-time codes are not carried over. Resources and rules go to System.
|
|
2335
|
-
- The import stops before the first write if two 0.21 objects give one 1.0 id (the data violates the key; use
|
|
2336
|
-
`--keyless <Class>`), if a key changed between versions, or if it references a missing object. A dangling reference
|
|
2337
|
-
in a live row gives `target_not_found` with the 0.21 object.
|
|
2338
|
-
- If the loader rejects the data, the error starts with `import: арендатор …`, and its `detail` is the loader's
|
|
2339
|
-
([Report and errors](#report-and-errors)), except that rows are named by 0.21 objects: `rows` are the violations
|
|
2340
|
-
with the `class`, `id` and `rev` of the 0.21 object, `code` and `message`; `total`, `truncated`, `limit` and
|
|
2341
|
-
`complete` are as in the loader's report; `notes` are the untranslated rules.
|
|
2342
|
-
- A repeat adds what is missing without duplicates; `--update` also adds new versions of changed objects. The report is
|
|
2343
|
-
JSON at the end of the output: objects and versions by class in 0.21 and in 1.0, `verify()`, untranslated rules,
|
|
2344
|
-
classes without a key, and uuids in `data` that match old ids (the import does not rewrite them). The exit code is 1
|
|
2345
|
-
if `verify()` found discrepancies.
|
|
2346
|
-
|
|
2347
|
-
## 15. API reference
|
|
2348
|
-
|
|
2349
|
-
The full machine-readable list of exports, methods and error texts is `docs/api-contract.json` in the repository.
|
|
2350
|
-
|
|
2351
|
-
### Package exports
|
|
2352
|
-
|
|
2353
|
-
| Entry point | What it exports |
|
|
2354
|
-
|---|---|
|
|
2355
|
-
| `letopis` | `connect`, `up`, `createTenant`; operators `ne`, `gt`, `gte`, `lt`, `lte`, `between`, `inList`, `like`, `ilike`, `starts`, `ends`, `has`, `hasAny`, `hasAll`, `exists`, `isNull`, `not`, `or`, `isOp`; `inc`, `isInc`, `cursorOf`; `idOf`, `uuidv5`, `v5Name`, `v5Id`; `LetopisError`, `ValidationError`, `fromDbError`; `syncModels`, `generateTypes`, `schemaToTs`; `compileValidator`, `applyDefaults`, `Registry`; `totpCode`, `hashPassword`, `verifyHash`; `describeText`; constants `DDL_REVISION`, `RESERVED_CLASS_NAMES`, `CHAIN_METHODS`, `DB_METHODS`; types `Db`, `DbTx`, `TypedDb`, `Chain`, `Row`, `Filter`, `Cursor`, `ConnectOptions`, `UpOptions`, `LoadOptions`, `LoadReport`, `WatchOptions`, `WatchEvent`, `AuthApi`, `AclApi`, `DiffResult`, `DiffChange`, `BlameResult`, `BlameField`, `Projector`, `ProjectorOptions`, `ProjectorChange`, `ProjectorOut`, `ProjectorRow`, `ProjectorReport`, `ProjectorStatus`, `ProjectorFailure` ([Projector](#projector)), `AggregateInfo` and others |
|
|
2356
|
-
| `letopis/model` | `hub`, `link`, `one`, `many`, `End`, types `Infer`, `InferLinks`, `Model`, `ClassDef`, `ClassMeta`, `HistoryPolicy`, `AggregateDef`; `normalizeSchema`, `translatePattern`, type `JsonSchema` |
|
|
2357
|
-
| `letopis/mcp` | `createMcpServer(db, options)`: an MCP SDK server with the letopis tools; the type `McpOptions`: `write` adds the `letopis_write` tool (absent by default), `maxRows` is the rows in an answer (100, from 1 to 10,000), `maxBytes` is the bytes in an answer (50,000, from 1,000 to 50,000,000); `allowAclWrite` lets `letopis_write` write the permission classes too (by default `acl_denied`); `allowSystemAdmin` accepts a session in the System tenant (by default `user_session_required`); a value out of range gives `invalid_data`. Needs the `@modelcontextprotocol/server` package ([section 13](#agents-and-mcp)) |
|
|
2358
|
-
| CLI `letopis` | `sync`, `types`, `load`, `verify`, `maintain`, `describe`, `import`, `mcp` ([section 13](#13-operations)); `mcp` takes `--dsn`, `--schema`, `--token` (or `LETOPIS_DSN`, `LETOPIS_SCHEMA`, `LETOPIS_TOKEN`) and `--write` |
|
|
2359
|
-
|
|
2360
|
-
Removed 0.21 names (`uuidv7`, `LETOPIS_NS` and the methods listed below) throw `removed` with the replacement.
|
|
2361
|
-
|
|
2362
|
-
### The `db` root
|
|
2363
|
-
|
|
2364
|
-
| Member | What it does |
|
|
2365
|
-
|---|---|
|
|
2366
|
-
| `db.ClassName(filter)` | the start of a chain |
|
|
2367
|
-
| `schema`, `token`, `tenant`, `registry` | the installation name, the token, the session tenant, the class registry (`resolve`, `find`, `has`, `all`, `family`) |
|
|
2368
|
-
| `auth`, `acl` | sign-in, credentials and sessions; permission decisions ([section 10](#10-users-sign-in-and-permissions)) |
|
|
2369
|
-
| `as(account, { tenant, owner })` | the same pool on behalf of an account (async) |
|
|
2370
|
-
| `begin()`, `commit(tr)`, `rollback(tr)`, `lock(…keys)` | an explicit transaction ([section 8](#8-transactions-batches-and-money)) |
|
|
2371
|
-
| `batch(name)` | a queue of plans: `run()`, `discard()`, `size()` |
|
|
2372
|
-
| `load(input, options)` | the loader ([section 9](#9-bulk-loading)) |
|
|
2373
|
-
| `projector(name, options)` | a projector from the journal: `run()`, `rebuild({ chunked, tenant })`, `status()`, `stop()`, `close()` ([Projectors](#projectors)) |
|
|
2374
|
-
| `rebuildAggregates(class)` | repair of the aggregates of a class and its descendants in all tenants: the field anew from live sources, a version only for corrected rows; → their number ([Aggregates in the database](#aggregates-in-the-database)) |
|
|
2375
|
-
| `watch(options)`, `cache` | subscription; result cache (`clear()`, `stats()`) |
|
|
2376
|
-
| `` sql`…` ``, `transaction(fn)` | raw SQL under the session and RLS |
|
|
2377
|
-
| `idOf(class, key)` | the id of an object of a class with a key, in the session tenant |
|
|
2378
|
-
| `entity(row \| chain)` | start a path from a ready node |
|
|
2379
|
-
| `describe()` | a machine-readable description of the model |
|
|
2380
|
-
| `verify(options)`, `maintain(options)`, `reset(options)` | verification, maintenance, reset ([sections 11](#11-history-and-point-in-time-reads) and [13](#13-operations)) |
|
|
2381
|
-
| `trimHistory(target, moment)`, `rekeyClass(class, key)` | history trimming; a new class key |
|
|
2382
|
-
| `refresh()`, `flushSignals()`, `close()` | re-read the classes; send the signals now; close |
|
|
2383
|
-
|
|
2384
|
-
### Chain
|
|
2385
|
-
|
|
2386
|
-
| Group | Methods |
|
|
2387
|
-
|---|---|
|
|
2388
|
-
| steps | `.ClassName(filter)`, `.AliasName(filter)`, `.RoleName(filter)` |
|
|
2389
|
-
| modifiers | `limit(n)`, `offset(n)`, `sort(field, 'asc' \| 'desc')`, `after(cursor)`, `asOf(moment)`, `withDeleted()`, `deep(max)`, `exact()`, `alias(name)`, `tags(…)`, `owner(…)`, `entity(…)`, `forUpdate()`, `cache({ ttl })` |
|
|
2390
|
-
| verbs | `create(data)`, `update(patch, { rev })`, `upsert(data)`, `delete({ confirm })`, `anonymize(fields)`, `reclass(class, data)`, `rekey(patch)`, `restore()`, `purge({ confirm })` |
|
|
2391
|
-
| slots | after a verb: `.RoleName.set(value)`, `.RoleName.unset()`, `.RoleName.add(value)`, `.RoleName.remove(value)` |
|
|
2392
|
-
| terminals | `rows()`, `first()`, `ids()`, `count({ paths })`, `paths()`, `versions({ follow })`, `diff({ from, to, follow })`, `blame({ at, limit })`, `sum(field)`, `avg(field)`, `min(field)`, `max(field)`, `countBy(field)` |
|
|
2393
|
-
|
|
2394
|
-
### `db.auth`
|
|
2395
|
-
|
|
2396
|
-
| Method | Parameters → result |
|
|
2397
|
-
|---|---|
|
|
2398
|
-
| `setPassword` | `{ account, identifier, password, kind?, confirmed?, keepCurrent? }` → credential id |
|
|
2399
|
-
| `verifyPassword` | `{ identifier, password, kind?, requireConfirmed?, ttl?, totp? }` → `{ account, token }` or `null` |
|
|
2400
|
-
| `issueApiKey`, `verifyApiKey` | `{ account, name?, ttl? }` → `{ key, id }`; `(key)` → sign-in or `null` |
|
|
2401
|
-
| `issueKeySecret`, `verifyKeySecret` | `{ account, name? }` → `{ key, secret, id }`; `(key, secret)` → sign-in or `null` |
|
|
2402
|
-
| `enrollTotp`, `verifyTotp`, `totpEnabled` | `{ account, issuer?, label? }` → `{ secret, uri }`; `{ account, code, window? }` → `boolean`; `(account)` → `boolean` |
|
|
2403
|
-
| `issueOtp`, `verifyOtp` | `{ account, identifier, kind?, ttl?, digits?, maxAttempts? }` → `{ code }`; `{ identifier, code, kind? }` → sign-in or `null` |
|
|
2404
|
-
| `link`, `lookup` | `{ account, kind, identifier, meta?, confirmed? }` → id; `{ kind, identifier, requireConfirmed?, ttl?, totp? }` → sign-in or `null` |
|
|
2405
|
-
| `credentials`, `revokeCredential`, `setFlags` | `(account)` → credentials without secrets; `(id, { keepCurrent? })`; `(id, { confirmed?, enabled? })` |
|
|
2406
|
-
| `sessionFor`, `refresh`, `revoke`, `revokeAll` | `(account, { ttl? })` → token; `({ ttl? })` → the new expiration time (the token stays the same); `()`; `(account?)` → number of sessions |
|
|
2407
|
-
| `session` | `(token)` → `{ account, tenant, member, expiresAt, loginAt, methods }` or `null` |
|
|
2408
|
-
| `tenants`, `switch` | `()` → `[{ tenant, roles }]`; `(tenant)` |
|
|
2409
|
-
| `purgeAccount` | `(account)` → `{ account, rows, versions, memberships, owned }` |
|
|
2410
|
-
|
|
2411
|
-
Password, code and TOTP checks, session issuing, session details by token and credential confirmation are called by the
|
|
2412
|
-
service (`auth.*` permissions); a regular user gets `acl_denied`. The credentials of System, services and system
|
|
2413
|
-
administrators can be changed only by the account itself after a recent interactive sign-in, or by the system
|
|
2414
|
-
administrator. The exception is the service's keys: the service issues and revokes its own API keys and "key + secret"
|
|
2415
|
-
pairs itself with a session of its own key, but not its other credentials; nobody can revoke the last active key of a
|
|
2416
|
-
service (`acl_denied`). With `totp`, `verifyPassword` and `lookup` also check the TOTP code (the `auth.totp` permission)
|
|
2417
|
-
and issue a session with the `TOTP` method. The credential kind `kind` is `^[A-Z][A-Z0-9_]{1,31}$` (`PASSWORD`, `EMAIL`,
|
|
2418
|
-
`TELEGRAM` …); `SESSION`, `APIKEY`, `KEYSECRET`, `OTP`, `TOTP`, `KEY`, `PROJECTOR` and `LOCK` are taken. The same rule
|
|
2419
|
-
applies to the password kind `user.kind` in `createTenant` and `up({ tenant })`.
|
|
2420
|
-
|
|
2421
|
-
### Projector
|
|
2422
|
-
|
|
2423
|
-
`db.projector(name, options)` works only on an admin connection ([Projectors](#projectors)). The name is lowercase
|
|
2424
|
-
Latin letters, digits and `_.:-`, up to 64 characters; it is also the `derivedBy` of the derived classes. Wrong options
|
|
2425
|
-
give `invalid_data`.
|
|
2426
|
-
|
|
2427
|
-
| Member | What it does |
|
|
2428
|
-
|---|---|
|
|
2429
|
-
| options `ProjectorOptions` | `from`, `into`: classes with descendants; `version`: an integer from 1; `handle(changes, out)`; `withBefore`; `onExpired: 'rebuild' \| 'error'` (`'rebuild'` rebuilds in batches); `batch`: changes in a batch (500, from 1 to 10,000); `pollMs`: fallback polling (1000 ms); `lockTimeout`: waiting for another transaction's row lock (5000 ms, 0 — no limit); `lockRetries`: passes in a row before a tenant with another transaction's lock drops out (0 — never); `tenants`: ids of the participant tenants |
|
|
2430
|
-
| `run({ once })` | catch up with the journal (`once`) or follow it until `stop()` → `ProjectorReport` |
|
|
2431
|
-
| `rebuild({ chunked, tenant })` | rebuild from the current state: atomically, in batches or for one tenant → `ProjectorReport` |
|
|
2432
|
-
| `status()` | → `ProjectorStatus`: `{ name, state, version, cursor, at, lag, failed, rebuildingTenants }`; `failed` is `{ [tenant id]: ProjectorFailure }`, where `ProjectorFailure` is `{ cursor, code, message }` |
|
|
2433
|
-
| `stop()`, `close()` | stop `run()` after the current batch; the same plus unsubscribing from signals |
|
|
2434
|
-
| `ProjectorReport` | `{ batches, changes, created, updated, deleted, unchanged, skipped, rebuilds, locked }` |
|
|
2435
|
-
| `ProjectorChange` | `{ cursor, id, class, tenant, op, rev, at, author, row }` and, when present, `agent`, `reason`, `moved`, `before`, `rebuild` |
|
|
2436
|
-
| `ProjectorOut` | `upsert(class, row, { tenant })`, `inc(class, row, deltas, { tenant })`, `delete(class, key or id, { tenant })`, `get(class, key or id, { tenant })` → `Row \| null`; a row is a `ProjectorRow` in the loader format; for a class with a key `delete` and `get` go by the key, an id only for a class without a key |
|
|
2437
|
-
|
|
2438
|
-
### Result row
|
|
2439
|
-
|
|
2440
|
-
| Field | Meaning |
|
|
2441
|
-
|---|---|
|
|
2442
|
-
| `id`, `class`, `rev` | id, class, version number (for `update(patch, { rev })`) |
|
|
2443
|
-
| `tenant`, `owner` | the tenant and the owner of the row |
|
|
2444
|
-
| `links`, `data`, `tags` | ends, fields, tags |
|
|
2445
|
-
| `at`, `author`, `agent`, `op`, `reason` | version time (ISO, UTC, microseconds), author, agent (the service under impersonation), kind of operation, reason |
|
|
2446
|
-
| `moved` | the reference to the old or the new id after `reclass` and `rekey` |
|
|
2447
|
-
| `$deleted` | tombstone: a deleted entity or a deletion version |
|
|
2448
|
-
| `$depth` | the depth of a node in `deep()`; the cascade level in a deletion preview |
|
|
2449
|
-
| `$upsert`, `$action`, `$versions`, `$purged` | what `upsert` did; what the deletion will do; the number of versions in `purge`; the history is erased |
|
|
2450
|
-
|
|
2451
|
-
### Reserved names
|
|
2452
|
-
|
|
2453
|
-
A method name hides a class only where that method exists: after a step — chain methods, at the root `db` — its
|
|
2454
|
-
methods, in a batch — its own. So a hub named like a chain-only method (for example, `count`) is reachable as the first
|
|
2455
|
-
step `db.count(…)` but not after another step, and a hub named like a root method cannot be the first step. The
|
|
2456
|
-
database does not accept a link with any of these names (`reserved_names()`; the installer fills in the list from
|
|
2457
|
-
`RESERVED_CLASS_NAMES`). Do not give classes these names.
|
|
2458
|
-
|
|
2459
|
-
- chain methods: `then`, `entity`, `paths`, `rows`, `first`, `ids`, `count`, `limit`, `offset`, `sort`, `asOf`,
|
|
2460
|
-
`withDeleted`, `deep`, `exact`, `sum`, `avg`, `min`, `max`, `countBy`, `after`, `versions`, `create`, `update`,
|
|
2461
|
-
`upsert`, `set`, `unset`, `add`, `remove`, `delete`, `anonymize`, `reclass`, `restore`, `purge`, `alias`, `tags`,
|
|
2462
|
-
`owner`, `forUpdate`, `cache`, `rekey`, `diff`, `blame`; guards of the removed API: `run`, `execute`, `account`;
|
|
2463
|
-
- the `db` root: `as`, `begin`, `commit`, `rollback`, `lock`, `batch`, `watch`, `close`, `registry`, `sql`, `load`,
|
|
2464
|
-
`verify`, `maintain`, `reset`, `describe`, `auth`, `acl`, `idOf`, `refresh`, `transaction`, `flushSignals`,
|
|
2465
|
-
`schema`, `token`, `tenant`, `trimHistory`, `rekeyClass`, `cache`, `projector`, `rebuildAggregates`; guards: `reloadSchema`,
|
|
2466
|
-
`accounts`, `credentials`, `resources`, `rules`, `compact`;
|
|
2467
|
-
- batch: `discard`, `size`.
|
|
2468
|
-
|
|
2469
|
-
## 16. Errors
|
|
2470
|
-
|
|
2471
|
-
The text of an error is `letopis: <code>: <message>`. The code is in `err.code`, the details are in `err.detail`.
|
|
2472
|
-
Errors are `LetopisError` objects; data errors are `ValidationError` with `err.issues: [{ path, keyword, message }]`.
|
|
2473
|
-
|
|
2474
|
-
Database codes. Raw SQL gets them too: SQLSTATE class `LT`, the details are JSON in `DETAIL`.
|
|
2475
|
-
|
|
2476
|
-
| Code | SQLSTATE | When |
|
|
2477
|
-
|---|---|---|
|
|
2478
|
-
| `no_session` | LT001 | there is no valid session |
|
|
2479
|
-
| `target_not_found` | LT002 | the target is not found; the same for a nonexistent, a foreign and an invisible target |
|
|
2480
|
-
| `invalid_data` | LT003 | the data does not match the class description; `err.issues` gives the path, keyword and message |
|
|
2481
|
-
| `undeclared_end` | LT004 | the end is not declared in the class |
|
|
2482
|
-
| `tenant_denied` | LT005 | a foreign tenant or no membership |
|
|
2483
|
-
| `acl_denied` | LT006 | denied by the access rules |
|
|
2484
|
-
| `id_mismatch`, `id_from_db` | LT007, LT008 | the id is not v5 of the key; the id of a class without a key is assigned by the database |
|
|
2485
|
-
| `immutable_key` | LT009 | a key field, the id, the tenant and the owner do not change (a new key is `rekey`) |
|
|
2486
|
-
| `invalid_class` | LT010 | the class description is rejected, the class is not found or is abstract; a class with rows, descendants or a history of rows is not deleted |
|
|
2487
|
-
| `use_reclass`, `reclass_denied` | LT011, LT012 | changing the class with `update` when the old or the new class has a key: the id changes, so use `reclass()`; the roles of referencing rows do not accept the new class |
|
|
2488
|
-
| `tightening_conflict` | LT013 | rows prevent tightening the description |
|
|
2489
|
-
| `isolation_level` | LT014 | deleting, changing the class and changing the key work only in `read committed` |
|
|
2490
|
-
| `recreated_in_statement` | LT015 | an id was deleted and created again in one statement |
|
|
2491
|
-
| `cursor_expired` | LT016 | the cursor of a subscription or a projector is older than the `all` tier of the retention policy |
|
|
2492
|
-
| `delete_restricted` | LT017 | referencing rows prevent the deletion (`onDelete: 'restrict'`) |
|
|
2493
|
-
| `not_deleted` | LT018 | `restore` and `purge` work only for deleted objects |
|
|
2494
|
-
| `reset_disabled` | LT019 | reset is disabled in the installation (`allowReset`) |
|
|
2495
|
-
| `confirm_mismatch` | LT020 | the reset confirmation does not match the schema name |
|
|
2496
|
-
| `derived_class` | LT021 | a row of a derived class (`derivedBy`) is written, edited or deleted by someone other than its projector; an end leads to such a row or to a derived class; a projector writes to a class outside its `into` or has found no tenant in which every `into` class is marked with its `derivedBy` |
|
|
2497
|
-
| `aggregate_field` | LT022 | the application passed or changed an aggregate field (`aggregate`): the database calculates it |
|
|
2498
|
-
| `projector_pending` | LT023 | `purge` of a deleted object whose deletion a projector of its class has not read yet, or `trimHistory` whose first remaining version it has not read yet: run the projector first |
|
|
2499
|
-
|
|
2500
|
-
Library codes:
|
|
2501
|
-
|
|
2502
|
-
| Code | When |
|
|
2503
|
-
|---|---|
|
|
2504
|
-
| `exists` | the object already exists: `create` (`detail.id` is its id), `rekey` (the new id is taken) or the loader |
|
|
2505
|
-
| `duplicate`, `deleted` | the loader: a repeated id in the load; an object with this id existed before. The loader throws the code of the first problem row, the first 100 are in `detail.rows`, the total is `detail.total` |
|
|
2506
|
-
| `conflict` | `update(patch, { rev })`: the row was changed in the meantime (`detail`: `id`, `rev`, `expected`) |
|
|
2507
|
-
| `rev_ambiguous` | `{ rev }` on a step that does not have exactly one target |
|
|
2508
|
-
| `tx_required` | `forUpdate()`, `lock()`, `commit()` or `rollback()` outside an active `db.begin()` |
|
|
2509
|
-
| `lock_unsupported` | `forUpdate()` with an aggregate, `asOf`, `versions`, `withDeleted`, `deep` or a verb |
|
|
2510
|
-
| `commit_unknown` | the connection dropped during `COMMIT`: the outcome is unknown |
|
|
2511
|
-
| `tx_rolled_back` | PostgreSQL answered `ROLLBACK` to `COMMIT`: the error was caught inside `tr.transaction(fn)` |
|
|
2512
|
-
| `no_key` | `upsert`, `rekey` or `idOf` on a class without a key |
|
|
2513
|
-
| `no_path` | there is no hop between the classes of neighboring steps |
|
|
2514
|
-
| `invalid_query` | the chain or the API is used incorrectly (a step modifier `deep`, `exact`, `alias`, `tags` or `owner` after a verb, a selection modifier between a verb and a `rows`, `first`, `ids`, `count` or `paths` terminal, a slot not after a verb, an `after()` cursor with a value of a different type than the `sort` field, without `v` or with an `id` that is not a uuid, and so on) |
|
|
2515
|
-
| `admin_required` | the loader, a projector and `rebuildAggregates` need an admin connection |
|
|
2516
|
-
| `rebuild_required` | the projection was built by another version of the projector (`version`): a rebuild with `rebuild()` is needed; `rebuild({ tenant })` while there is no projection or a batch rebuild is in progress |
|
|
2517
|
-
| `projector_busy` | a rebuild cannot start: the projector is running in another process right now |
|
|
2518
|
-
| `not_empty` | the initial load (`initial`) goes only into an empty schema |
|
|
2519
|
-
| `class_not_loadable` | the loader does not accept `Class` rows |
|
|
2520
|
-
| `bypass_rls` | `connect()` with a role that bypasses RLS, without `allowBypassRls`; `createMcpServer` with a connection that bypasses RLS |
|
|
2521
|
-
| `user_session_required` | `createMcpServer` and `letopis mcp`: a user session is needed — the connection has no session, it is the service root without `as()`, a session in the System tenant without `allowSystemAdmin`, or the command line got the service key or no token |
|
|
2522
|
-
| `confirmation_required` | `letopis_write` (MCP): a deletion without a human confirmation — the client has no elicitation, the human declined, the answer belongs to another operation, or the data changed after the preview |
|
|
2523
|
-
| `removed` | a removed 0.21 API; the text gives the replacement |
|
|
2524
|
-
|
|
2525
|
-
A PostgreSQL permission denial (`42501`) comes, for example, on an `upsert` over someone else's row that you may not
|
|
2526
|
-
edit.
|
|
2527
|
-
|
|
2528
|
-
## 17. Performance
|
|
2529
|
-
|
|
2530
|
-
Benchmark of 1.0 against 0.21 on the same datasets (`cd lib && npx tsx bench/compare.bench.mjs`, 2026-10-09,
|
|
2531
|
-
PostgreSQL 18, and 0.21 on PostgreSQL 17 with TimescaleDB, one machine): salon has ~980 thousand rows (in 0.21 they
|
|
2532
|
-
belong to System without isolation, in 1.0 they are in a tenant under RLS and permissions); tenants has 500 thousand
|
|
2533
|
-
customers with 2 versions each, across ten tenants. The numbers depend on the machine; what matters is the ratio.
|
|
2534
|
-
|
|
2535
|
-
| Measurement | 1.0 | 0.21 | Unit | Note |
|
|
2536
|
-
|---|---|---|---|---|
|
|
2537
|
-
| loading the salon dataset | 6359 | 3609 | rows/s | 1.0: the loader in history mode with checks; 0.21: raw INSERTs into Entity |
|
|
2538
|
-
| loading the tenants dataset | 8566 | 35186 | rows/s | 0.21: INSERT … SELECT with triggers disabled |
|
|
2539
|
-
| row-by-row create | 128 | 156 | rows/s | |
|
|
2540
|
-
| read by key | 2.2 | 3.81 | ms (p50) | |
|
|
2541
|
-
| three-step chain (count) | 222.79 | 1233.83 | ms (p50) | organization → specialists → bookings |
|
|
2542
|
-
| point-in-time query (asOf) | 202.77 | 327.98 | ms (p50) | a specialist's bookings as of mid-year |
|
|
2543
|
-
| 20 concurrent writers | 982 | 522 | rows/s | |
|
|
2544
|
-
| class change (reclass) | 9.5 | — | ms (p50) | 0.21 has no such verb |
|
|
2545
|
-
| count() of bookings: a tenant under RLS and permissions | 272.25 | 890.73 | ms (p50) | 0.21: without isolation and permissions (enforceAccount: false); 1.0 without RLS (admin): 18.59 ms |
|
|
2546
|
-
| count() of customers of tenant 1 (50.0%) | 114.09 | 667.19 | ms (p50) | |
|
|
2547
|
-
| count() of customers of tenant 3 (10.0%) | 23.8 | 229.94 | ms (p50) | |
|
|
2548
|
-
| count() of customers of tenant 8 (1.0%) | 5.3 | 88.02 | ms (p50) | |
|
|
2549
|
-
| count() of customers of tenant 10 (0.3%) | 3.54 | 26.08 | ms (p50) | |
|
|
2550
|
-
|
|
2551
|
-
A full import of the salon dataset from 0.21 (`npm run test:import-salon`): 539,580 objects and 980,180 versions in
|
|
2552
|
-
193 s (5,084 rows/s); the per-class counters matched, and `verify()` is clean. Checkpoint B tuning (revision 5): the
|
|
2553
|
-
journal index on reference targets, `log_ends`, made an `asOf` read across a hop 16 times faster (3.37 s → 0.20 s) and
|
|
2554
|
-
costs the loader 15–17% of its speed. Reading `letopis.changed` and `COMMIT` in one pipeline removed a round trip to the
|
|
2555
|
-
database from every write (row-by-row writes: 119 → 128–147 rows/s in different runs). Row-by-row `create` is still
|
|
2556
|
-
12–18% slower than 0.21 (the measurements vary): most of the time goes to the check triggers in the database and to the
|
|
2557
|
-
commit with a disk write.
|
|
2558
|
-
|
|
2559
|
-
Initial load (`cd lib && npx tsx bench/initial-load.mjs`, 2026-10-11, median of three rounds): the regular loader
|
|
2560
|
-
writes 1.07 million objects of the salon dataset without history in 169 s (6,351 rows/s, 4.0 GB of WAL), the initial
|
|
2561
|
-
load in 77 s (13,902 rows/s, 1.4 GB of WAL). Most of the difference comes from the GIN indexes: maintaining them during
|
|
2562
|
-
the load costs 36 s, building them at the end about 5 s. Preparing rows in the loader itself (checks, ends, ids) runs
|
|
2563
|
-
alongside the `COPY` of the previous batch and now sets the pace: about 60 s of the 77.
|
|
2564
|
-
|
|
2565
|
-
Derived data in 1.2 (`cd lib && npx tsx bench/derived.bench.mjs`, 2026-10-27, PostgreSQL 18.6, one machine; for
|
|
2566
|
-
aggregates, the median of three rounds; the projector rows are a repeat of series 2 and 3 after the folding of `inc` in
|
|
2567
|
-
`out`, `BENCH_SERIES=2,3`):
|
|
2568
|
-
|
|
2569
|
-
| Measurement | 1.2 | For comparison |
|
|
2570
|
-
|---|---|---|
|
|
2571
|
-
| an operation with an aggregate, one writer | 94 operations/s, p50 10.7 ms | without an aggregate 137 and 6.9 ms; the `inc` recipe 69 |
|
|
2572
|
-
| operations on one wallet with an aggregate, 20 writers | 152 operations/s, p95 0.47 s | without an aggregate 1,987; the `inc` recipe 190 |
|
|
2573
|
-
| operations on 20 wallets with an aggregate, 20 writers | 1,379 operations/s | without an aggregate 2,023 |
|
|
2574
|
-
| projector catch-up, `out.inc` for every change | 6,076 and 33,395 changes/s (`batch` 500 and 2000) | before the folding of `inc` in `out`: 992 and 1,203 |
|
|
2575
|
-
| rebuild of 75 thousand rows | atomic 13.3–13.5 s, in batches 13.0–13.2 s | before the folding of `inc` in `out`: 101 s and 58–63 s |
|
|
2576
|
-
| delay of a subscription event during a rebuild | atomic up to 13.4 s, in batches up to 0.21 s | without a rebuild up to 0.12 s |
|
|
2577
|
-
|
|
2578
|
-
The aggregate triggers fire on every write statement, but for classes without aggregates the measurement does not see
|
|
2579
|
-
their cost: the p50 difference is from −0.28 to +0.21 ms. The benchmark against 1.0.1 (`bench/vs-101.bench.mjs`, four
|
|
2580
|
-
runs) showed no write or read regressions against 1.1: two outliers did not repeat in separate checks.
|
|
2581
|
-
|
|
2582
|
-
## 18. Limitations
|
|
2583
|
-
|
|
2584
|
-
This section lists what letopis 1.4 deliberately does not do or does with caveats, and where you need to watch the
|
|
2585
|
-
behavior.
|
|
2586
|
-
|
|
2587
|
-
**Platform**
|
|
2588
|
-
|
|
2589
|
-
- PostgreSQL 18 or newer is required; there are no workarounds for 15–17. AWS RDS, Google Cloud SQL, Azure Flexible
|
|
2590
|
-
Server and Neon support 18, while Supabase, as of October 2026, offers only 15 and 17, so 1.0 will not work there.
|
|
2591
|
-
- The database must be in UTF8 encoding (`casefold()` and the `pg_unicode_fast` collation). If a new major PostgreSQL
|
|
2592
|
-
version updates the Unicode tables, the unique index of credential identifiers is rebuilt with the `reindex` command.
|
|
2593
|
-
- PGlite has one superuser connection: no concurrent writers and no `LISTEN`, and RLS does not apply
|
|
2594
|
-
(`allowBypassRls: true`); it does not replace `postgres:18` for permissions, race tests and CI.
|
|
2595
|
-
- A pool in transaction mode (PgBouncer): `LISTEN` is not available, so use `connect({ listen: false })`; the
|
|
2596
|
-
subscription polls the journal, and the result cache is off. Prepared statements work with PgBouncer 1.21; with older
|
|
2597
|
-
versions, pass a postgres.js pool with `prepare: false` in `connect({ sql })`.
|
|
2598
|
-
- The NUL character (`\u0000`) is impossible in data strings: PostgreSQL does not store it in `jsonb`; such data is
|
|
2599
|
-
rejected already when the JSON is parsed (in the official JSON-Schema-Test-Suite, the "nul characters" groups are
|
|
2600
|
-
skipped).
|
|
2601
|
-
- On PostgreSQL before 18.6, a parallel GIN index build could write a wrong row count into the table statistics
|
|
2602
|
-
(`pg_class.reltuples`), after which autovacuum stopped processing the table. That is why the initial load always ends
|
|
2603
|
-
with `ANALYZE`, but it is better to upgrade the server to 18.6 or newer.
|
|
2604
|
-
|
|
2605
|
-
**Writing and integrity**
|
|
2606
|
-
|
|
2607
|
-
- Row-by-row writing costs more than a plain insert: triggers check every row (data, ends, id, journal, hash, RLS).
|
|
2608
|
-
Row-by-row `create` is 12–18% slower than 0.21 ([section 17](#17-performance)); for bulk writing there is the loader.
|
|
2609
|
-
- Deleting, changing the class and changing the key work only in `read committed` (`isolation_level`); library
|
|
2610
|
-
transactions always open at this level.
|
|
2611
|
-
- PostgreSQL does not run `COPY` into tables under RLS; bulk loading works only through the loader on an admin
|
|
2612
|
-
connection.
|
|
2613
|
-
- The initial load closes the schema's tables for the whole load, for writing and for reading: it is meant for an
|
|
2614
|
-
empty schema (seeding, migration, import), not for a running system. Its gain is in the database (indexes and
|
|
2615
|
-
constraints are built once); it does not speed up the row preparation in the loader, only runs it alongside `COPY`,
|
|
2616
|
-
so on a million objects it is about 2.2 times faster than the regular load.
|
|
2617
|
-
- The database trusts the loader: the data, ends and ids of loaded rows are checked by the library code (the TypeScript
|
|
2618
|
-
validator), not by the triggers; the database checks only the targets outside the load, the tenants and the owners,
|
|
2619
|
-
with one query at the end. The safeguards: the loader is available only to an admin, differential tests of the
|
|
2620
|
-
validator against the database, `load({ verify: true })` and `db.verify({ data: true })`. Keep the admin connection
|
|
2621
|
-
in a separate loading process, not in the web application: an SQL injection there would get the owner's rights.
|
|
2622
|
-
- The schema owner and a superuser can disable the triggers and change anything. letopis does not prevent this; it
|
|
2623
|
-
detects it: `verify()` recomputes the hash chains, `verify({ data: true })` checks the rows again, and the anchor (a
|
|
2624
|
-
hash of all chain heads) should be kept outside the database.
|
|
2625
|
-
- The database checks a subset of JSON Schema: Zod `refine`, `superRefine` and `transform` work only in the library,
|
|
2626
|
-
and the database does not accept a description with an unknown keyword; `pattern` is a limited subset of regular
|
|
2627
|
-
expressions shared by Zod, the validator and the database. The output of `z.toJSONSchema()` changes between Zod
|
|
2628
|
-
versions: the database stores an already normalized schema, and a `syncModels` dry run shows the difference before it
|
|
2629
|
-
is applied.
|
|
2630
|
-
- The key is the only way to get uniqueness: the id is computed from the key values without normalization, so the same
|
|
2631
|
-
value written in a different way (an email in a different case, a time with and without milliseconds) gives different
|
|
2632
|
-
ids. The model sets the format of a key field, and the database checks it.
|
|
2633
|
-
- Frequent changes of a key value (email, phone) require `rekey`. Values that change often and personal values are
|
|
2634
|
-
better moved to a separate hub with a link: an id from an email can be found by brute force and stays in references and in the journal
|
|
2635
|
-
even after `anonymize`, while a separate hub is deleted together with the link through `purge`.
|
|
2636
|
-
- `create` with a taken key answers `exists` with the id, and `upsert` without permission on the existing row answers
|
|
2637
|
-
with a `42501` denial: this reveals that the key is taken, but only within your own tenant, because in other tenants
|
|
2638
|
-
the same keys give other ids.
|
|
2639
|
-
- Changing the class and the key rewrites all incoming references (and, recursively, the links whose key includes this
|
|
2640
|
-
end), so its cost is proportional to the number of references. Tightening a class description and `rekeyClass` pause
|
|
2641
|
-
writes to the family while they check or move rows.
|
|
2642
|
-
- `rekeyClass` moves rows in one transaction under an exclusive lock of the family, not in batches: between batches,
|
|
2643
|
-
rows with old ids would break the rule "the id is v5 of the key".
|
|
2644
|
-
- v5 ids land in the primary key index in scattered places; on very large tables with heavy inserts this is more
|
|
2645
|
-
noticeable than with sequential ids.
|
|
2646
|
-
- Hot reference targets (an organization that almost everything references) accumulate `for key share` locks and
|
|
2647
|
-
MultiXact, as with regular foreign keys; this needs monitoring. Operations on one wallet go one after another under a
|
|
2648
|
-
lock of its row ([section 8](#8-transactions-batches-and-money), "Recipe: wallet").
|
|
2649
|
-
- A link cannot have a reserved name, and a hub with such a name is not reachable everywhere: for example, `count` —
|
|
2650
|
-
only as the first step (["Reserved names"](#reserved-names)).
|
|
2651
|
-
|
|
2652
|
-
**Access**
|
|
2653
|
-
|
|
2654
|
-
- The service key is the root of trust: its compromise allows acting on behalf of any enabled user of the
|
|
2655
|
-
installation except System, services and system administrators; the access rules do not narrow this scope. The key
|
|
2656
|
-
is reissued (`db.auth.issueApiKey`) and revoked (`db.auth.revokeCredential`) by the service itself with a session of
|
|
2657
|
-
its own key, or by the system administrator; with its key the service also issues itself new keys, so after a leak
|
|
2658
|
-
revoke all its keys except the new one (`credentials` lists them). Nobody can revoke the last active key of a
|
|
2659
|
-
service: to cut a service off, set `enabled: false` on its account. A database password alone, without a token,
|
|
2660
|
-
gives nothing.
|
|
2661
|
-
- The permission pass reveals its own plan to the client (the classes and rule conditions that apply to it).
|
|
2662
|
-
- Raw SQL without the permission pass pays for the permission plan on every query: 0.3–1.3 ms depending on the number
|
|
2663
|
-
of rules.
|
|
2664
|
-
- The end names `User`, `Tenant` (membership), `Group` and `Object` (rule) are taken by system classes across the
|
|
2665
|
-
whole schema.
|
|
2666
|
-
- OTP and TOTP attempt counters are rolled back together with the caller's transaction: the service limits the rate of
|
|
2667
|
-
attempts.
|
|
2668
|
-
- When users link a taken identifier themselves, the answer is `acl_denied`, so a user with a recent sign-in learns
|
|
2669
|
-
this way that the identifier is taken. Permission changes in all tenants are serialized through one counter row.
|
|
2670
|
-
- Refusal texts name the class, role and id of a referencing row only if the actor can see it: a cascade delete,
|
|
2671
|
-
unsetting an end, rewriting references on a class or key change, and a `parent` change conflict
|
|
2672
|
-
(`tightening_conflict`). The number of invisible rows that block a deletion or a
|
|
2673
|
-
class change (`reclass_denied`), and the existence of an account by id (as a reference target), are not hidden.
|
|
2674
|
-
- Tightening a class description and `rekeyClass` check all rows of the family, so they need the `READ` permission
|
|
2675
|
-
without conditions on all rows of the class and its descendants: otherwise a trial tightening (`maximum`, `enum`,
|
|
2676
|
-
`pattern`) would answer with the number of invisible rows that do not match it. A system administrator who tightens
|
|
2677
|
-
a System class gets the number of violating rows of other tenants (`others`): this is the installation level.
|
|
2678
|
-
- Class metadata (`meta`) is visible to anyone who has a session: anyone can read `Class` rows. Secrets do not belong
|
|
2679
|
-
there.
|
|
2680
|
-
- An application role can slow the installation down for everyone: `LOCK TABLE entity`, advisory locks with the keys
|
|
2681
|
-
of the application's `lock()`, fake `NOTIFY` signals (they only clear the cache), frequent permission changes (one
|
|
2682
|
-
counter row per installation). The global journal number reveals the overall write activity. Internal locks — class
|
|
2683
|
-
families, projectors and maintenance — are rows of the `secret` table, not advisory locks: neither the application
|
|
2684
|
-
role nor any other database role can take them. A family writer passes the tightening gate in a subtransaction — one
|
|
2685
|
-
per transaction for the first write to the family.
|
|
2686
|
-
|
|
2687
|
-
**History, signals, cache**
|
|
2688
|
-
|
|
2689
|
-
- An `asOf` read is rebuilt from the journal; only the current state is cheap. In the thinned part of the journal, a
|
|
2690
|
-
point-in-time read shows the last kept version before the moment: its precision there equals the period of the tier,
|
|
2691
|
-
a day, a week, a month or a year. A history import into a class with a policy is thinned right away, and a subscriber
|
|
2692
|
-
may not see the old versions.
|
|
2693
|
-
- The thinned part of the journal proves neither that the history is complete nor that old versions are authentic: for
|
|
2694
|
-
a class with a policy, `verify()` cannot tell the tampering of a version older than the `all` tier from thinning.
|
|
2695
|
-
- The journal has no partitions: thinning deletes versions in batches, and the table file does not shrink by itself
|
|
2696
|
-
(new versions take the space after autovacuum); to return the space to the system, use `VACUUM FULL` or `pg_repack`
|
|
2697
|
-
in a maintenance window.
|
|
2698
|
-
- Signals reveal activity: PostgreSQL lets any role listen to a channel. In the `class` mode one sees when and to which
|
|
2699
|
-
classes writes happen; `tenant` adds tenant pseudonyms; `opaque` hides what was written but not when: the write
|
|
2700
|
-
time and frequency per pseudonym stay visible, and a deterministic token is the basis of frequency attacks. Write
|
|
2701
|
-
times are visible without signals too: `pg_stat_all_tables` and `pg_locks` are open to PUBLIC. The `none` mode
|
|
2702
|
-
removes signals, but then a revoked session leaves the cache of other processes only by term, and a new class
|
|
2703
|
-
appears in their registry within 5 seconds. Real protection from foreign roles is a separate database. A fake
|
|
2704
|
-
signal only clears the cache one extra time: on `*` and unknown tokens the registry is re-read not more often than
|
|
2705
|
-
once in 5 seconds, and a fake class change signal re-reads it like a real one, but with at most one re-read at a
|
|
2706
|
-
time.
|
|
2707
|
-
- The signal mode is changed by `up({ signals })`, and the processes have to be restarted after that: a process reads
|
|
2708
|
-
the mode and the signal key on connect (it treats a payload of another mode as foreign and clears the whole cache).
|
|
2709
|
-
- PostgreSQL below 18.2 could truncate CLOG that unread signals still need: `up()` warns about it.
|
|
2710
|
-
- The `NOTIFY` queue (8 GB) is shared by the whole cluster: a stuck listener fills it up, and then transactions with
|
|
2711
|
-
`NOTIFY` fail on commit. The library reads signals right away, and `maintain()` warns when the queue is more than
|
|
2712
|
-
half full.
|
|
2713
|
-
- PostgreSQL commits transactions with `NOTIFY` one at a time: a commit holds the cluster-wide lock of the signal queue
|
|
2714
|
-
together with the disk write. A raw SQL write on its own connection sends the signal from a trigger in the same
|
|
2715
|
-
transaction, so concurrent raw SQL writers commit one after another: on the benchmark machine, 20 such writers did
|
|
2716
|
-
about 520 inserts per second. Library transactions, `db.sql` and `db.transaction()` included, commit the data
|
|
2717
|
-
without `NOTIFY` and send the signals later, separately, so their writers do not wait in this queue: 20 library
|
|
2718
|
-
writers did about 2,000 operations per second.
|
|
2719
|
-
- The subscription horizon is shared by the whole cluster: a long write transaction delays the events of all
|
|
2720
|
-
subscribers (`sub.lag()`); long operations are better done in batches.
|
|
2721
|
-
- Between processes, the result cache goes stale for the signal sending interval (100 ms) plus delivery, and for the
|
|
2722
|
-
time of a listener reconnect: postgres.js does not report a broken listener connection, so the cache works until the
|
|
2723
|
-
reconnect and is cleared completely after it. The `ttl` term is a safeguard.
|
|
2724
|
-
- A "from now" subscription cursor carries a snapshot of the subscription moment until it passes that boundary: such a
|
|
2725
|
-
cursor is longer than a regular one.
|
|
2726
|
-
|
|
2727
|
-
**Projectors**
|
|
2728
|
-
|
|
2729
|
-
- A projector is an admin process: it works only on an admin connection, reads the journal of all tenants bypassing
|
|
2730
|
-
RLS and writes its results as the owner role. Keep it in a separate process, like the loader. A participant of a
|
|
2731
|
-
projector is a tenant in which all `into` classes are marked with its `derivedBy`; only an administrator may mark a
|
|
2732
|
-
class (the `letopis.projector` permission).
|
|
2733
|
-
- The totals of a derived class are seen by everyone whom the tenant rules give `READ` on it — even by those who do not
|
|
2734
|
-
read the source rows: decide who reads it with rules on the derived class itself. With `history: true`, its journal
|
|
2735
|
-
shows the change of every batch, that is, the contribution of single source rows too.
|
|
2736
|
-
- Only their projector writes the rows of derived classes: the application, raw SQL and the loader get
|
|
2737
|
-
`derived_class`, and ends may not refer to such rows.
|
|
2738
|
-
- An atomic rebuild is one long write transaction: while it runs, the subscription horizon stands still, and events
|
|
2739
|
-
are delayed for the subscribers and projectors of all processes. On the benchmark machine, a rebuild of 75 thousand
|
|
2740
|
-
rows took 13 s, and the subscriber got no event until it ended. That is why `run()` itself rebuilds in batches, and
|
|
2741
|
-
only an explicit `rebuild()` runs an atomic rebuild.
|
|
2742
|
-
- A batch rebuild holds the horizon only for the time of a batch (in the same measurements, events were delayed by up
|
|
2743
|
-
to 0.2 s), but the derived data is incomplete until it ends: the rows are deleted at once and reappear as the
|
|
2744
|
-
rebuild goes on.
|
|
2745
|
-
- A projector sees only the changes from the journal, so a class with `history: false` cannot be a source: a tenant
|
|
2746
|
-
with such a source drops out of the projector (`invalid_class`) until the journal of the class is turned on and the
|
|
2747
|
-
tenant is rebuilt.
|
|
2748
|
-
- Thinning the journal older than the `all` tier leads to a rebuild: a projector that lags behind by more than that
|
|
2749
|
-
tier gets `cursor_expired` or, with `onExpired: 'rebuild'`, rebuilds itself. The tier is per tenant: if it has
|
|
2750
|
-
expired only for some participants, only they drop out and rebuild.
|
|
2751
|
-
- A projector works only on a schema with the engine revision the library expects; on a schema of another revision it
|
|
2752
|
-
answers `invalid_data`.
|
|
2753
|
-
- `purge()` waits for projectors: while a projector whose sources include the class of the object, and which has a
|
|
2754
|
-
derived class in the object's tenant, has not read the deletion, `purge` answers `projector_pending`. A projector
|
|
2755
|
-
that is no longer run holds `purge` of its sources while the tenant has its derived classes: run it or delete them.
|
|
2756
|
-
`trimHistory` waits the same way: while the projector has not read the first remaining version, trimming answers
|
|
2757
|
-
`projector_pending`.
|
|
2758
|
-
|
|
2759
|
-
**Aggregates in the database**
|
|
2760
|
-
|
|
2761
|
-
- An aggregate queues the writers of one object: every operation locks the target row and writes its version. On the
|
|
2762
|
-
benchmark machine, 20 writers of one wallet with an aggregate did 152 operations per second (p95 is 0.47 s), without
|
|
2763
|
-
an aggregate 1,987. For hot counters, use a projector that folds a batch.
|
|
2764
|
-
- `reclass` and `rekey` accumulate contributions until the end of the move, while a delete cascade and the loader apply
|
|
2765
|
-
them per statement: the class rule is checked on the total of each cascade statement and of each loader batch. So a
|
|
2766
|
-
load in which debits come in a batch before top-ups gets `invalid_data`, although the total of the whole load is
|
|
2767
|
-
valid. Every batch that touches a target writes its version.
|
|
2768
|
-
- A tenant aggregate does not count rows of a System class, and the field of a System row counts only System sources.
|
|
2769
|
-
|
|
2770
|
-
**Agents and MCP**
|
|
2771
|
-
|
|
2772
|
-
- The `<untrusted-data-…>` wrapper is a mitigation, not a protection: the model may follow the text inside it. The
|
|
2773
|
-
protection is the rights in the database: the agent reads and writes only what its user is allowed to. In
|
|
2774
|
-
`structuredContent`, the database data lies in `data` and `error` next to the `notice`, without marks: a client that
|
|
2775
|
-
gives the model that instead of the text passes the data with the note but without a random mark.
|
|
2776
|
-
- By default the server only reads; writing is `write: true` (`--write`); deletion is only with a human confirmation.
|
|
2777
|
-
The server writes the permission classes only with `allowAclWrite` and accepts a session in the System tenant only
|
|
2778
|
-
with `allowSystemAdmin`; `create`, `update` and `upsert` of the other classes need no human confirmation.
|
|
2779
|
-
- The HTTP handler serves revision 2025 statelessly: the per-request instance does not know the client's
|
|
2780
|
-
capabilities, so deletion through it is refused (`confirmation_required`). Delete over stdio or in revision
|
|
2781
|
-
2026-07-28.
|
|
2782
|
-
- A confirmation in revision 2026-07-28 is signed with a process key: a retried call that lands in another process of
|
|
2783
|
-
the application (several instances behind a load balancer) is refused with `confirmation_required`.
|
|
2784
|
-
- `versions` returns the latest versions within `maxRows` and `maxBytes` (at most `maxRows` + 1 versions are read
|
|
2785
|
-
from the database) and `more`; earlier ones, through `diff` and `blame` with `from`, `to`, `at`. A mode where the
|
|
2786
|
-
agent writes code against the API (code mode) is not part of the package.
|
|
2787
|
-
|
|
2788
|
-
**Import from 0.21**
|
|
2789
|
-
|
|
2790
|
-
- 1.0 is a new major version, not a continuation of 0.21: `count()`, `paths()`, `create`, roles, multiple ends and
|
|
2791
|
-
`reclass` behave in a new way, and the data is moved by an import; see [MIGRATION.md](MIGRATION.md) (in Russian).
|
|
2792
|
-
- Import: the steps (classes, accounts, credentials, permissions, the data of each tenant) are separate transactions,
|
|
2793
|
-
and the source is read into memory entirely; a row owner that is not found among the accounts becomes the tenant;
|
|
2794
|
-
0.21 access rules move to System and apply in all tenants; the `onDelete` of ends is `cascade`, the way 0.21 deleted.
|
|
2795
|
-
|
|
2796
|
-
## 19. Tests
|
|
2797
|
-
|
|
2798
|
-
This section is for those who develop letopis itself. Run: `cd lib && npm test`. The harness starts the
|
|
2799
|
-
`letopis-pg18` container (`postgres:18`, port 15433) by itself if the database does not respond; each test file works in its own schema `v2.t_<name>`.
|
|
2800
|
-
|
|
2801
|
-
```
|
|
2802
|
-
# connect — connecting under a session, refusing roles that bypass RLS, the registry, the deferred signal, error translation (stage 2)
|
|
2803
|
-
# context — sessions, service sign-in by key, the RLS policies of stage 1 (stage 1)
|
|
2804
|
-
# access-rest — the remaining rows of stage 5: loading on behalf of a user, balance No. 20, 22, 29, upsert and { rev } under ACL, inlining of acl_ok (stage 5)
|
|
2805
|
-
# aggregate — aggregates in the database: 20 writers of one wallet, raw SQL, a cascade, reclass, restore, the loader, account deletion, restore and rekey of the target, a description change under writes and race No. 18, aggregate_field, repair and verify, a System row (plan 1.1, stage 4b)
|
|
2806
|
-
# acl — ACL permissions: the 0.21 suite ported, row conditions, weight and default deny, cascade, deny on a descendant, deep, an invisible target, raw SQL versus the library; a change of parent and alias of a class only with permissions on all its rows, an alias is not the name of another class, a name is not another class's alias (stage 5; security review No. 2)
|
|
2807
|
-
# acl-parity — 1000 random rule sets: the database permission plan and endpoint decisions versus decide from 0.21 (stage 5)
|
|
2808
|
-
# auth — credentials (password, key, key + secret, OTP, TOTP, identity), timing equalization, case, session revocation, recent sign-in (stage 5)
|
|
2809
|
-
# balance — top-up and debit: idempotency by opId, races, deadlock, connection drop (TCP proxy), adjustment under { rev } and forUpdate (stage 4)
|
|
2810
|
-
# chain-api — guards of the removed API, reserved names, build errors, the plan cache, onQuery (stage 3)
|
|
2811
|
-
# chain-read — read chains on the demo domain: hops, roles, multiple ends, 18 operators, keyset, aggregates, deep, asOf, withDeleted, pivot (stage 3)
|
|
2812
|
-
# chain-types — types of steps, rows and filters from models: connect({ models }) (stages 3, 4)
|
|
2813
|
-
# chain-write — writing with chains: create, update, upsert, delete with preview, anonymize, reclass, slots, inc, { rev }, fan-out, plan rollback (stage 4)
|
|
2814
|
-
# demo — the booking demo domain on Zod: seed from models, writing data, id by key, agreement between Zod and the database (stage 2)
|
|
2815
|
-
# docs — documentation examples are executed: the marked blocks of the README and the cheatsheet (Russian and English) run as scenarios against PostgreSQL 18 (stage 8)
|
|
2816
|
-
# engine-attacks — raw SQL attacks from the application role: the rows of §7.3.1 for stages 1 and 3 with specific outcomes; the signal key without permission, flooding with fake signals, Class included; writing to a derived class and a reference to it, a projector checkpoint through the credential functions; an agent through MCP without --write and without a human confirmation; a change of parent, alias and derivedBy of a class with the WRITE permission on Class alone, deleting and re-creating a class with a history of rows, a trial tightening as an oracle, reclass without DELETE and rewriting references deeper than the first level, a retention policy with the WRITE permission on Class alone; an agent through MCP grants a membership or a rule, a system administrator session; a tenant user creates a derived class on their own, `rekeyClass` of an ancestor of a derived class; moving a row in a way that would rewrite references of derived class rows; the application role holds advisory locks with the former keys of internal locks, while lock rows are out of its reach; a forged note of the family locks already taken (stages 1, 3; plan 1.1, stages 2, 4a, 5, checkpoint A, security review No. 2, checkpoint B and the work after it, security review No. 3)
|
|
2817
|
-
# engine-class — classes as entities: metaschema, inheritance, recomputing descendants, tightening, metadata; a class with a history of rows is deleted only after purge() (stage 1; the work after checkpoint B)
|
|
2818
|
-
# engine-write — the write pipeline in raw SQL: v5 ids, defaults, merge, inc, deletion, resurrection, journal, signals (stage 1)
|
|
2819
|
-
# engine-races — races on two connections in both orders, without timed pauses; tightening during continuous writes to the family does not starve; locking families whose roots differ only in case, and a planted row of kind LOCK (stage 1; security review No. 3)
|
|
2820
|
-
# explain — chain indexes from the application role: outside, the primary key and (class, tenant); inside find_*(), GIN; an asOf slice by class does not read another tenant's journal (stage 3; the work after checkpoint B)
|
|
2821
|
-
# find-ids-parity — 30 chains through find_*() and through a direct read under RLS give the same answers; own memberships in other tenants and the Class rows of System — now, asOf and withDeleted (stage 3; the work after checkpoint B)
|
|
2822
|
-
# hash — the version hash chain, journal tampering and gaps, verify() (stage 1)
|
|
2823
|
-
# harness — the test harness: starting the database, a schema per file, two connection strings (stage 0)
|
|
2824
|
-
# jsonschema — compiling JSON Schema to jsonpath and checking data in the database (stage 1)
|
|
2825
|
-
# impersonation — impersonation: letopis.account only for a service with the permission, target restrictions, author and agent in the journal, sessionFor; the db.as() registry is one per tenant, does not empty, is re-read on a signal of its tenant (stage 5; the work after checkpoint B); the facade registry is not shared with the root, an empty registry is not cached (security review No. 3 leftovers)
|
|
2826
|
-
# install — the installer and roles: a repeated installation, two installations in one database, three tables (stage 1)
|
|
2827
|
-
# member — memberships: working in a tenant by roles, auth.switch, leaving with ownership transfer, system rows, tenant roles without System permissions (stage 5)
|
|
2828
|
-
# model — Zod models, the JSON Schema normalizer, pattern translation, agreement with the database metaschema (stage 2)
|
|
2829
|
-
# load — the loader: several classes, cycles, $key and $ext, repeats, existing, portions, history mode, the query for targets outside the load, CLI (stage 4)
|
|
2830
|
-
# roles — roles and privileges: installation without a superuser, owner functions, triggers by catalog, the log without the changed column (stage 1; plan 1.1, checkpoint A)
|
|
2831
|
-
# write-races — write races: 20 concurrent edits of different fields without losses and strict rev, concurrent add/remove on a multiple end, the loader versus class tightening and versus a concurrent edit (existing: update) in both orders (stage 4)
|
|
2832
|
-
# validator-parity — the TypeScript validator versus the database check: the official suite, 10,000 random documents, defaults (stage 2)
|
|
2833
|
-
# scenario-barbershop — end-to-end barbershop scenario: 9 scenes from 0.21 (integration) on 1.0 chains — catalog, free slots in a batch, bookings under lock, multi-slot and multi-specialist, versions, cancellation and booking again on the same id, closing a month with restrict (stage 4)
|
|
2834
|
-
# scenario-real-life — the "real life" of a salon: 16 scenes from 0.21 (real-life) — a booking service on top of chains, races of double booking, rescheduling and cancellation, walk-in, refusals, reports and daily invariants (stage 4)
|
|
2835
|
-
# system-classes — system classes instead of the 0.21 facades: the System seed, accounts, credentials, resources and rules, transactions (stage 5)
|
|
2836
|
-
# sync — letopis sync (dry run, apply, tightening, lost refine) and letopis types (stage 2)
|
|
2837
|
-
# tx — explicit db.begin() transactions: a broken transaction, lock(), 40P01; forUpdate(); the batch and its merging, a batch queue per facade; db.as(…, { owner }) (stage 4)
|
|
2838
|
-
# v5 — v5 identifiers on the SQL side versus the vectors of an independent implementation (stage 1)
|
|
2839
|
-
# zod-output — a snapshot of the z.toJSONSchema() output by field kind of the demo domain versus a reference (stage 0)
|
|
2840
|
-
# time — history: versions({ follow }) across reclass and rekey, withDeleted, restore, purge, history trimming, account deletion (stage 6)
|
|
2841
|
-
# rekey — a new key value of an object and a new key of a class: new ids, moving references, recursion through links with the end in the key (stage 6)
|
|
2842
|
-
# reset — reset of sessions, tenant, all: the allowReset flag, the letopis.reset permission, confirmation by the schema name, the reset record, a new service key (stage 6)
|
|
2843
|
-
# verify — legitimate gaps and tampering, verify({ data: true }): data, ends, tenants, id by key; permissions and CLI (stage 6)
|
|
2844
|
-
# retention — retention policy: tiers, a finite term, special versions, deleted rows, asOf, cursor_expired, time zone, catch-up run, inheritance and default — versus a reference in TypeScript (stage 6)
|
|
2845
|
-
# watch — cursor subscription: commit order, a long transaction, resuming, listener drop, two tenants (stage 6)
|
|
2846
|
-
# cache — result cache: hits, own and others' writes, the key with the user, permissions, revoked sessions, listener drop, ttl, class marking, without LISTEN, a race, db.sql, the limit of signal counters (stage 6; plan 1.1, checkpoint A)
|
|
2847
|
-
# maintain — maintenance: expired sessions and codes, permissions, one process under a lock, schedule (stage 6)
|
|
2848
|
-
# up — the full installer: fresh installation, re-apply, custom seeds, automatic database creation, UTF8, version 17, upgrade with migrations, PGlite, the first tenant and user (up({ tenant }), createTenant) (stages 6, 8)
|
|
2849
|
-
# upgrade — schema upgrade from 1.1.0 (upgrade: true, migration 0007) and from 1.0.1 (upgrade: true, migrations 0006 and 0007) with the letopis-11 and letopis-1 packages: versions and hashes written by the old engine, verify(), the catalog against a clean installation, lock rows, secret rows without an account erased, the application, a subscription and a projector over old data, a repeated up() (work after 1.2.0, stage 1; revision 7)
|
|
2850
|
-
# scenario-salon — end-to-end salon scenario: 21 acts from 0.21 (salon) with a coverage matrix of the 1.0 API (stage 6)
|
|
2851
|
-
# telemetry — onQuery and slowMs, describe() and class metadata with inheritance, describeText (stage 6)
|
|
2852
|
-
# history — diff and blame over the log, the reason of a change for verbs, db.begin() and batches; a version hidden by an ACL condition, the changed column, a missing version, row_changes over 20 thousand targets (plan 1.1, stage 1 and checkpoint A)
|
|
2853
|
-
# signals — signal modes: the payload seen by a foreign role, cache precision per tenant, System, permission classes, a process without the key, the loader, subscriptions, the key, none, coalescing, the server version, a class change, a resend, the mode after up() and reset({ all }) (plan 1.1, stage 2)
|
|
2854
|
-
# initial-load — the initial load: equality with the regular loader, structure and trigger modes, system rows, a non-empty schema, an insert during the start, rollback, MULTI_INSERT in WAL, commitEvery, import --initial, aggregates (agg_fill) (plan 1.1, stages 3 and 4b)
|
|
2855
|
-
# projector — projectors from the journal: catching up, exactly once on a break, one writer, a long transaction, a transaction on the boundary of the rebuild snapshot, no writes to derived classes and no references to them, the bounds of out, atomic and batch rebuilds during source writes, purge of a source, lag and cursor_expired, the credential functions; the version, the name, an admin connection; a race of two processes; a data error of one tenant drops out only this tenant (failed, rebuild({ tenant }), the tenants option), the checkpoint is only a System row, a rebuild touches only the rows of participants and into families, before after rekey, the all tier and the journal of sources per tenant; folding of out.inc by key in a batch, batch rebuilds on the automatic paths of run() and a second process during them; rebuilding one tenant in batches while the tail goes on; onExpired in batches in another process, a data error without retries, the tier from the start of the rebuild; an empty last pass of a batch rebuild, a tenant that drops out in the middle of its rebuild keeps no partial rows, a class change that keeps the id out of the source family and into it, purge of an unread tombstone (plan 1.1, stage 4a; security review No. 2; checkpoint B and the work after it; security review No. 3)
|
|
2856
|
-
# mcp — MCP tools: the model and meta only in results, tenant rights, the JSON chain and hints of similar names, row and byte limits with a cursor, the untrusted-data wrapper, writing only with write, update only with rev, deletion with a human confirmation (elicitation in revisions 2025 and 2026-07-28), the reason in blame, credentials, tool schemas, letopis without the SDK; letopis mcp over stdio, the HTTP handler; the confirmation window without the alias and meta, countBy and versions limited in the query itself, limits of steps and $or conditions; the permission classes only with allowAclWrite, a session in the System tenant only with allowSystemAdmin; structuredContent with a note, a requestState signature of the server user with an expiry (plan 1.1, stage 5; security review No. 2; checkpoint B and the work after it)
|
|
2857
|
-
# import — import from 0.21 with a fixture of a real 0.21: key violation, --keyless, per-class counters, verify, self-reference, repeat, ids without a key, a dangling reference, $$strict and fields starting with $, update, credentials, CLI (stage 7)
|
|
2858
|
-
# security-final — the final security review (stage 8, §7.10 No. 8.4): a family lock by the tenant of the root, account_purge without permission and restore/purge of a foreign id give neither a leak nor a lock; the functions of stage 6, log_refs_at, journal-based find_* reads and watch_read stay within their own tenant; NOTIFY carries only names or mode tokens; the cache key distinguishes the token, the actor and the tenant (stage 8)
|
|
2859
|
-
```
|