letopis 0.20.3 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENT-CHEATSHEET.en.md +368 -0
- package/AGENT-CHEATSHEET.md +354 -0
- package/CHANGELOG.md +348 -0
- package/MIGRATION.md +190 -0
- package/README.en.md +1937 -0
- package/README.md +1493 -3466
- package/dist/acl.d.ts +26 -50
- package/dist/acl.js +22 -267
- package/dist/admin.d.ts +138 -0
- package/dist/admin.js +170 -0
- package/dist/auth.d.ts +120 -73
- package/dist/auth.js +121 -306
- package/dist/cache.d.ts +73 -0
- package/dist/cache.js +148 -0
- package/dist/chain.d.ts +124 -191
- package/dist/chain.js +369 -551
- package/dist/cli.d.ts +2 -0
- package/dist/cli.js +164 -0
- package/dist/demo/booking.d.ts +289 -0
- package/dist/demo/booking.js +159 -0
- package/dist/errors.d.ts +29 -0
- package/dist/errors.js +70 -0
- package/dist/import.d.ts +179 -0
- package/dist/import.js +792 -0
- package/dist/index.d.ts +172 -26
- package/dist/index.js +304 -178
- package/dist/jsonschema.d.ts +22 -0
- package/dist/jsonschema.js +167 -0
- package/dist/load.d.ts +76 -0
- package/dist/load.js +884 -0
- package/dist/model.d.ts +166 -0
- package/dist/model.js +224 -0
- package/dist/ops.d.ts +7 -6
- package/dist/ops.js +7 -51
- package/dist/pglite.d.ts +22 -0
- package/dist/pglite.js +45 -0
- package/dist/registry.d.ts +57 -0
- package/dist/registry.js +82 -0
- package/dist/sql.d.ts +59 -142
- package/dist/sql.js +568 -654
- package/dist/sync.d.ts +31 -0
- package/dist/sync.js +108 -0
- package/dist/tx.d.ts +129 -8
- package/dist/tx.js +300 -73
- package/dist/typed.d.ts +97 -0
- package/dist/typed.js +1 -0
- package/dist/types.d.ts +71 -250
- package/dist/types.js +27 -108
- package/dist/up.d.ts +140 -47
- package/dist/up.js +339 -267
- package/dist/uuid.d.ts +21 -6
- package/dist/uuid.js +48 -64
- package/dist/validate.d.ts +24 -0
- package/dist/validate.js +251 -0
- package/dist/watch.d.ts +62 -0
- package/dist/watch.js +168 -0
- package/dist/write.d.ts +117 -74
- package/dist/write.js +658 -720
- package/llms.txt +26 -0
- package/package.json +49 -19
- package/sql/10-core.sql +136 -0
- package/sql/15-errors.sql +60 -0
- package/sql/20-context.sql +153 -0
- package/sql/30-validate.sql +423 -0
- package/sql/40-class.sql +259 -0
- package/sql/50-acl.sql +539 -0
- package/sql/60-write.sql +1369 -0
- package/sql/70-read.sql +245 -0
- package/sql/80-auth.sql +827 -0
- package/sql/90-time.sql +957 -0
- package/sql/95-seed.system.sql +178 -0
- package/sql/99-revision.sql +3 -0
- package/sql/README.md +56 -0
- package/sql/seed.booking.sql +39 -112
- package/dist/schema.d.ts +0 -15
- package/dist/schema.js +0 -351
- package/dist/sessions.d.ts +0 -32
- package/dist/sessions.js +0 -114
- package/dist/tables.d.ts +0 -105
- package/dist/tables.js +0 -248
- package/docker/Dockerfile +0 -40
- package/docker/start.sh +0 -18
- package/scripts/check-docs.mjs +0 -375
- package/scripts/gen-api-contract.mjs +0 -226
- package/scripts/gen-types.mjs +0 -350
- package/scripts/release-notes.mjs +0 -76
- package/scripts/schema-sync.mjs +0 -185
- package/sql/ddl.sql +0 -600
- package/sql/seed.auth.sql +0 -73
package/README.en.md
ADDED
|
@@ -0,0 +1,1937 @@
|
|
|
1
|
+
# letopis 1.0
|
|
2
|
+
|
|
3
|
+
**letopis** is a Node.js library: a store of versioned entities on PostgreSQL 18. Every change is saved as a version
|
|
4
|
+
in a journal. So every object has a history, a state as of any date and a change subscription. The database itself
|
|
5
|
+
checks data integrity, tenant isolation and permissions, with triggers and RLS policies. The rules work the same for
|
|
6
|
+
the library and for raw SQL. Data is read and written with chains:
|
|
7
|
+
`db.Shop(shop).Product({ price: lt(1000) }).rows()`.
|
|
8
|
+
|
|
9
|
+
[Русская версия](README.md) · [Cheatsheet for AI agents](AGENT-CHEATSHEET.en.md) · [Migration from 0.21 (in Russian)](MIGRATION.md) · [Changelog (in Russian)](CHANGELOG.md) · [llms.txt](llms.txt)
|
|
10
|
+
|
|
11
|
+
What letopis can do:
|
|
12
|
+
|
|
13
|
+
- **Zod models** are stored in the database as classes: hubs (objects) and links (relations with their own data) — [section 4](#4-data-model).
|
|
14
|
+
- **Writing with chains**: `create`, `update` with merge, `upsert`, deletion with preview and cascade, the `inc` increment
|
|
15
|
+
for money and counters — [sections 5](#5-writing), [7](#7-deleting-and-restoring) and [8](#8-transactions-batches-and-money).
|
|
16
|
+
- **Reading with chains**: hops along links, filters, trees, aggregates, pagination — [section 6](#6-reading).
|
|
17
|
+
- **History**: versions, point-in-time reads, restoring deleted data, retention policy, hash chain verification —
|
|
18
|
+
[section 11](#11-history-and-point-in-time-reads).
|
|
19
|
+
- **Tenants, sign-in and permissions** in the database itself: passwords, API keys, one-time codes, TOTP, sessions,
|
|
20
|
+
impersonation, access rules — [section 10](#10-users-sign-in-and-permissions).
|
|
21
|
+
- **Change subscription** by cursor without losses, and a **result cache** — [section 12](#12-change-subscription-and-cache).
|
|
22
|
+
- **Bulk loading** of millions of rows and **import from 0.21** — [sections 9](#9-bulk-loading) and [14](#14-migrating-from-021).
|
|
23
|
+
|
|
24
|
+
Requirements:
|
|
25
|
+
|
|
26
|
+
- PostgreSQL 18 or newer with the pgcrypto extension (it is part of the standard distribution), and a database in UTF8
|
|
27
|
+
encoding.
|
|
28
|
+
- Node.js 22 or newer; the package is an ES module.
|
|
29
|
+
- One runtime dependency: `postgres`. Zod 4 is an optional peer dependency: only the `letopis/model` subpath needs it,
|
|
30
|
+
and `import { connect } from 'letopis'` works without it. For development without PostgreSQL there are the optional
|
|
31
|
+
`@electric-sql/pglite` and `@electric-sql/pglite-socket`.
|
|
32
|
+
|
|
33
|
+
## Contents
|
|
34
|
+
|
|
35
|
+
1. [Quick start](#1-quick-start)
|
|
36
|
+
2. [How letopis works](#2-how-letopis-works)
|
|
37
|
+
3. [Installation and connection](#3-installation-and-connection)
|
|
38
|
+
4. [Data model](#4-data-model)
|
|
39
|
+
5. [Writing](#5-writing)
|
|
40
|
+
6. [Reading](#6-reading)
|
|
41
|
+
7. [Deleting and restoring](#7-deleting-and-restoring)
|
|
42
|
+
8. [Transactions, batches and money](#8-transactions-batches-and-money)
|
|
43
|
+
9. [Bulk loading](#9-bulk-loading)
|
|
44
|
+
10. [Users, sign-in and permissions](#10-users-sign-in-and-permissions)
|
|
45
|
+
11. [History and point-in-time reads](#11-history-and-point-in-time-reads)
|
|
46
|
+
12. [Change subscription and cache](#12-change-subscription-and-cache)
|
|
47
|
+
13. [Operations](#13-operations)
|
|
48
|
+
14. [Migrating from 0.21](#14-migrating-from-021)
|
|
49
|
+
15. [API reference](#15-api-reference)
|
|
50
|
+
16. [Errors](#16-errors)
|
|
51
|
+
17. [Performance](#17-performance)
|
|
52
|
+
18. [Limitations](#18-limitations)
|
|
53
|
+
19. [Tests](#19-tests)
|
|
54
|
+
|
|
55
|
+
Sections 3–13 are built on one example, a shop: installation, model, writing, reading and so on. The code blocks of
|
|
56
|
+
these sections go in order and together form one program. The `docs` test runs this program, like the quick start
|
|
57
|
+
example, against a real database, so the examples do not drift away from the code.
|
|
58
|
+
|
|
59
|
+
## 1. Quick start
|
|
60
|
+
|
|
61
|
+
You need PostgreSQL 18 and an admin role: a superuser or a role with the `CREATEROLE` privilege. Install the package:
|
|
62
|
+
|
|
63
|
+
```sh
|
|
64
|
+
npm install letopis zod postgres # postgres: for createTenant and your own SQL queries
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
The application connects to the database with a separate role that is a member of `letopis_app`. The installation
|
|
68
|
+
creates `letopis_app`. So after the first installation, create the application role once:
|
|
69
|
+
|
|
70
|
+
```sql
|
|
71
|
+
create role shop_app login password 'app-secret';
|
|
72
|
+
grant letopis_app to shop_app;
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
The example installs the schema, creates a shop with its first user, describes two classes and works with them:
|
|
76
|
+
|
|
77
|
+
<!-- run: quickstart -->
|
|
78
|
+
```ts
|
|
79
|
+
import { up, connect, syncModels, inc, gte } from 'letopis'
|
|
80
|
+
import { hub } from 'letopis/model'
|
|
81
|
+
import { z } from 'zod'
|
|
82
|
+
|
|
83
|
+
const ADMIN = 'postgres://postgres:secret@localhost:5432/shop' // admin: installs the schema
|
|
84
|
+
const APP = 'postgres://shop_app:app-secret@localhost:5432/shop' // application: a member of letopis_app
|
|
85
|
+
|
|
86
|
+
// 1. Install the v2.shop schema. tenant is the first tenant (the shop) and its user Anna.
|
|
87
|
+
const r = await up({ dsn: ADMIN, schema: 'shop',
|
|
88
|
+
tenant: { name: 'Chamomile', user: { name: 'Anna', login: 'anna@shop.ru', password: 'Chamomile-2026!' } } })
|
|
89
|
+
// r.serviceKey is the application's service key: it is issued on the first installation, so save it
|
|
90
|
+
|
|
91
|
+
// 2. The application service signs in with the key, checks Anna's password and works on her behalf in the shop.
|
|
92
|
+
const svc = await connect({ dsn: APP, schema: r.schema, apiKey: r.serviceKey! })
|
|
93
|
+
const login = await svc.auth.verifyPassword({ identifier: 'anna@shop.ru', password: 'Chamomile-2026!' })
|
|
94
|
+
const db = await svc.as(login!.account, { tenant: r.tenant!.tenant })
|
|
95
|
+
|
|
96
|
+
// 3. Model: a shop and a product. syncModels writes the class descriptions to the database.
|
|
97
|
+
const Shop = hub('Shop', { name: z.string() }, { key: ['name'] })
|
|
98
|
+
const Product = hub('Product', { sku: z.string(), title: z.string(), price: z.number().int().min(0) },
|
|
99
|
+
{ ends: { Shop }, key: ['Shop', 'sku'] })
|
|
100
|
+
await syncModels(db, [Shop, Product], { apply: true })
|
|
101
|
+
|
|
102
|
+
// 4. Writing and reading with chains.
|
|
103
|
+
const [shop] = await db.Shop().create({ name: 'Chamomile' }).rows()
|
|
104
|
+
const [kettle] = await db.Shop(shop).Product().create({ sku: 'K-1', title: 'Tea kettle', price: 2500 }).rows()
|
|
105
|
+
await db.Product(kettle).update({ price: inc(-500) }).rows() // discount: the database subtracts inside the statement
|
|
106
|
+
const expensive = await db.Shop(shop).Product({ price: gte(1000) }).rows()
|
|
107
|
+
const history = await db.Product(kettle).versions() // two versions: creation and edit
|
|
108
|
+
|
|
109
|
+
await svc.close()
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
What happened here:
|
|
113
|
+
|
|
114
|
+
- `up()` installed the `v2.shop` schema in the database: three tables, triggers, RLS policies, functions and the
|
|
115
|
+
system seed. The `tenant` option created the tenant "Chamomile" with the rule "authenticated users may do
|
|
116
|
+
everything", and the user Anna with the `owner` role in it.
|
|
117
|
+
- `connect({ apiKey })` exchanged the service key for a session. `verifyPassword` returned `{ account, token }` (or
|
|
118
|
+
`null` if the password is wrong). `svc.as(account, { tenant })` is the same connection pool, but acting as Anna in
|
|
119
|
+
the shop; the database checks that the service has the right to do this.
|
|
120
|
+
- `hub('Product', fields, options)` described an object class. `ends: { Shop }` is a required reference to the shop.
|
|
121
|
+
`key: ['Shop', 'sku']` is the key: the product id is computed from the shop and the SKU, and the database does not
|
|
122
|
+
accept a second product with the same key.
|
|
123
|
+
- `db.Shop(shop).Product().create(…)` created a product and took the reference to the shop from the chain path.
|
|
124
|
+
`inc(-500)` lowered the price inside the `UPDATE` statement, without reading it first. `versions()` read the history
|
|
125
|
+
from the journal.
|
|
126
|
+
|
|
127
|
+
Calling `up()` again is safe: it re-applies the same revision and does not touch the data, and it does not issue the
|
|
128
|
+
service key again while the installation still has a valid API key. Next, section 2 explains the design, and sections
|
|
129
|
+
3–13 go through the tasks in order.
|
|
130
|
+
|
|
131
|
+
## 2. How letopis works
|
|
132
|
+
|
|
133
|
+
### Core concepts
|
|
134
|
+
|
|
135
|
+
| Concept | What it is |
|
|
136
|
+
|---|---|
|
|
137
|
+
| installation | a PostgreSQL schema `v2.<name>` with the letopis tables and functions; one database can hold several |
|
|
138
|
+
| class | a data type: name, fields (JSON Schema), ends and key. A class description is also a row in the database (class `Class`) |
|
|
139
|
+
| hub | an object class: a shop, a product, a customer |
|
|
140
|
+
| link | a relation class between objects, with its own data: an order line, a specialist's skill |
|
|
141
|
+
| end (role) | a reference from one row to another row. An end has a name, the role: `Shop`, `Parent`. Ends are in `row.links`, fields are in `row.data` |
|
|
142
|
+
| key | the fields and ends that identify an object within a class and a tenant. The id is computed from them, so uniqueness is the key |
|
|
143
|
+
| id | a version 5 UUID: a hash of the tenant, the class and the key values; for a class without a key, a hash of a random part |
|
|
144
|
+
| account | a user, a service or an organization: a row of class `Account` |
|
|
145
|
+
| tenant | the account whose data is being worked with. Data of different tenants is isolated |
|
|
146
|
+
| membership | a `member` link "user — tenant" with roles; it allows work in another tenant |
|
|
147
|
+
| session | a token by which the database identifies the account and the tenant. A service gets it with an API key, a user by signing in |
|
|
148
|
+
| version, journal | each change of a row is a version in the `log` journal, with author, time and hash |
|
|
149
|
+
| tombstone | a "deleted" version with the last snapshot of the object |
|
|
150
|
+
| chain | a query like `db.Shop(shop).Product().rows()`: steps, modifiers, write verbs and a terminal |
|
|
151
|
+
| terminal | a method that executes the chain: `rows`, `first`, `ids`, `count`, `paths`, aggregates |
|
|
152
|
+
|
|
153
|
+
### What is stored in the database
|
|
154
|
+
|
|
155
|
+
There are three tables. `entity` is the current state: one row per live entity, including the rows of system classes
|
|
156
|
+
(`Class`, `Account`, `Resource`, `member`, `rule`). `log` is the journal: a full snapshot of every version with author,
|
|
157
|
+
time, kind of operation and hash. `secret` holds credentials and sessions: the application sees none of its rows; only
|
|
158
|
+
the sign-in functions work with it.
|
|
159
|
+
|
|
160
|
+
Classes are entities too. A class description is stored in a `Class` row; the database also keeps the expanded
|
|
161
|
+
description (with ancestors applied) and the compiled data check there. The version hash is a generated column: sha256 of the
|
|
162
|
+
previous version's hash and the row snapshot. So the next version exposes a tampered version, and for the last version
|
|
163
|
+
an anchor outside the database does this ([section 11](#11-history-and-point-in-time-reads)).
|
|
164
|
+
|
|
165
|
+
### Who checks the data
|
|
166
|
+
|
|
167
|
+
There are two roles. `letopis_owner` owns the schema, tables and functions. `letopis_app` is the application role: the
|
|
168
|
+
application's LOGIN role is a member of this role only. Everything the application runs, both the library and raw SQL,
|
|
169
|
+
goes through triggers and RLS.
|
|
170
|
+
|
|
171
|
+
- **Triggers.** On insert, a trigger computes the id, fills in defaults and checks the data against the class JSON
|
|
172
|
+
Schema (the schema is compiled to jsonpath). It checks the ends and their targets, fills in the service columns and
|
|
173
|
+
writes a version to the journal. An update merges the patch inside the statement (`merge`, `inc`). After a delete,
|
|
174
|
+
the trigger applies the `restrict`, `cascade` and `unset` rules and writes tombstones.
|
|
175
|
+
- **RLS and permissions.** Policies isolate tenants and apply access rules. At the start of a query, the database
|
|
176
|
+
builds the actor's permission plan once (`acl_plan()`); each row then needs only the cheap `acl_ok` check.
|
|
177
|
+
- **Query context** is the transaction settings: `letopis.token` (the session), `letopis.tenant` and `letopis.account`
|
|
178
|
+
(the tenant and impersonation), `letopis.reason` (the reason for the change) and `letopis.acl` (the permission pass,
|
|
179
|
+
see [section 10](#10-users-sign-in-and-permissions)). The database finds the session by the sha256 of the token, so
|
|
180
|
+
without a real token the settings give nothing.
|
|
181
|
+
- **Bypassing the checks** is possible only for code that runs as the owner: owner functions (`security definer` with
|
|
182
|
+
a fixed `search_path`) and the loader on an admin connection. `connect()` refuses a role that bypasses RLS (a
|
|
183
|
+
superuser, `BYPASSRLS`, a member of `letopis_owner`) unless `allowBypassRls: true` is set.
|
|
184
|
+
|
|
185
|
+
### How reads work
|
|
186
|
+
|
|
187
|
+
Under RLS, the indexes on `data` and on back references are not directly available to the policy. So the ids for chain
|
|
188
|
+
steps are found by the owner functions `find_ids()`, `find_hop()` and `find_deep()`. They apply the same tenant and
|
|
189
|
+
permission conditions as the policy. The rows themselves are then read by primary key under RLS. A point-in-time read
|
|
190
|
+
is built from the journal.
|
|
191
|
+
|
|
192
|
+
### Signals
|
|
193
|
+
|
|
194
|
+
A write reports the changed classes through `NOTIFY`. Raw SQL sends the signal from a trigger, and PostgreSQL delivers
|
|
195
|
+
it on commit. Library transactions defer their signals: the process notifier sends them every 100 ms
|
|
196
|
+
(`notifyIntervalMs`). A signal carries only a class name or the start of the token hash of a revoked session: no ids,
|
|
197
|
+
no data. On signals, the class registry is re-read, the result cache is cleared and subscriptions wake up.
|
|
198
|
+
|
|
199
|
+
## 3. Installation and connection
|
|
200
|
+
|
|
201
|
+
### Installation: `up()`
|
|
202
|
+
|
|
203
|
+
`up()` waits for the database and creates it if it does not exist. It checks for PostgreSQL 18 and UTF8 encoding,
|
|
204
|
+
installs the roles `letopis_owner` and `letopis_app`, the pgcrypto extension and the `v2.<name>` schema with all
|
|
205
|
+
functions and the system seed. On the first installation, it issues a service key. Call it from an admin role.
|
|
206
|
+
|
|
207
|
+
| Option | What it does |
|
|
208
|
+
|---|---|
|
|
209
|
+
| `dsn` or `sql` | an admin connection string or a ready postgres.js pool |
|
|
210
|
+
| `schema` | the installation name: `'shop'` gives the schema `v2.shop`; you can also pass `'v2.shop'` directly |
|
|
211
|
+
| `tenant` | the first tenant and its user (see below) |
|
|
212
|
+
| `seeds` | seeds after the core: `'booking'` is the salon demo domain; or paths to your own `.sql` files |
|
|
213
|
+
| `history` | the history retention policy for classes without their own ([section 11](#11-history-and-point-in-time-reads)) |
|
|
214
|
+
| `serviceKey` | your own service key instead of a random one: `lts_` and 32–128 hexadecimal characters |
|
|
215
|
+
| `allowReset` | allow `db.reset()` ([section 13](#13-operations)); off by default |
|
|
216
|
+
| `fresh` | drop the schema and install it again — **the schema data is lost** |
|
|
217
|
+
| `upgrade` | upgrade a schema of an older revision: migrations and a re-apply |
|
|
218
|
+
| `pglite` | a database for development without PostgreSQL (see below) |
|
|
219
|
+
| `quiet`, `waitTimeoutMs` | no console messages; how long to wait for the database (60,000 ms by default) |
|
|
220
|
+
| `owner`, `app` | the names of the owner and application roles (`letopis_owner`, `letopis_app` by default) |
|
|
221
|
+
|
|
222
|
+
Result: `schema` (the full name), `serviceKey` (on the first installation, and on a re-run when the installation has no
|
|
223
|
+
valid API key left; otherwise `null`), `created`, `upgraded`, `revision`, `migrations`, `analyzed`, `owner`, `app`,
|
|
224
|
+
and also `tenant` and `pglite` if they were requested.
|
|
225
|
+
|
|
226
|
+
<!-- run: guide -->
|
|
227
|
+
```ts
|
|
228
|
+
import { up, connect } from 'letopis'
|
|
229
|
+
|
|
230
|
+
const ADMIN = 'postgres://postgres:secret@localhost:5432/shop'
|
|
231
|
+
const APP = 'postgres://shop_app:app-secret@localhost:5432/shop'
|
|
232
|
+
|
|
233
|
+
const r = await up({ dsn: ADMIN, schema: 'shop',
|
|
234
|
+
tenant: { name: 'Chamomile', user: { name: 'Anna', login: 'anna@shop.ru', password: 'Chamomile-2026!' } } })
|
|
235
|
+
const SHOP = r.tenant!.tenant // the tenant: the Chamomile shop
|
|
236
|
+
const ANNA = r.tenant!.user! // its first user
|
|
237
|
+
```
|
|
238
|
+
|
|
239
|
+
### Application role
|
|
240
|
+
|
|
241
|
+
The application works as a LOGIN role that is a member of `letopis_app` only:
|
|
242
|
+
`create role shop_app login password '…'; grant letopis_app to shop_app;`. Then RLS and triggers apply to it, as to
|
|
243
|
+
any raw SQL from this role. `connect()` rejects a role that bypasses RLS with the error `bypass_rls`. The admin
|
|
244
|
+
connection (`allowBypassRls: true`) is needed only for the loader, verification and maintenance; keep it in a separate
|
|
245
|
+
process, not in the web application.
|
|
246
|
+
|
|
247
|
+
### First tenant and user
|
|
248
|
+
|
|
249
|
+
Permissions in letopis are always on and deny everything by default. So a new tenant needs rules. The
|
|
250
|
+
`up({ tenant })` option and the `createTenant(sql, schema, opts)` function do this in one transaction:
|
|
251
|
+
|
|
252
|
+
- the tenant: an account in System with an id from the name (`tenant:<name>`);
|
|
253
|
+
- the rule "authenticated users may do everything" in this tenant (`grantAll`, on by default, weight 10);
|
|
254
|
+
- a user (id from the login) with a membership in the tenant with the roles `user.roles` (`['owner']` by default), a
|
|
255
|
+
password and a session in the tenant.
|
|
256
|
+
|
|
257
|
+
A repeat with the same name and login duplicates nothing and does not change the password; it only issues a new
|
|
258
|
+
session. The login goes into the id as is: `Anna@shop.ru` and `anna@shop.ru` would give two accounts, and the second one
|
|
259
|
+
would have no password, so write logins in one case. The `created` field in the result tells whether this call created
|
|
260
|
+
the tenant: for an existing tenant with a new user it is `false`. The roles are set by `user.roles`, a non-empty list
|
|
261
|
+
of names (not masks), otherwise `invalid_data`. A repeat does not change the roles of an existing membership: they are
|
|
262
|
+
changed by editing the membership ([Granting a permission](#granting-a-permission)). A repeat does not bring back
|
|
263
|
+
what was deleted: if the tenant, the user account or the membership is deleted (the row is gone but its history is in
|
|
264
|
+
the journal), `createTenant` answers `invalid_data` and writes nothing. The error text names what was deleted and
|
|
265
|
+
advises bringing it back with `restore()`: an account with `db.Account(id).restore()` on behalf of the system
|
|
266
|
+
administrator, a membership with `db.member(id).restore()` in that tenant; the other way out is a different name or
|
|
267
|
+
login. The session from `createTenant` has no sign-in method, so the user cannot change their own password with it:
|
|
268
|
+
for that the user signs in with the password. `createTenant` needs an admin postgres.js pool:
|
|
269
|
+
|
|
270
|
+
<!-- run: guide -->
|
|
271
|
+
```ts
|
|
272
|
+
import postgres from 'postgres'
|
|
273
|
+
import { createTenant } from 'letopis'
|
|
274
|
+
|
|
275
|
+
const pg = postgres(ADMIN)
|
|
276
|
+
const lutik = await createTenant(pg, r.schema, {
|
|
277
|
+
name: 'Buttercup', user: { name: 'Boris', login: 'boris@lutik.ru', password: 'Buttercup-2026!' } })
|
|
278
|
+
await pg.end()
|
|
279
|
+
const BORIS = lutik.user! // lutik: { tenant, user, token, created: true }
|
|
280
|
+
```
|
|
281
|
+
|
|
282
|
+
### Connecting: `connect()`
|
|
283
|
+
|
|
284
|
+
A connection gets a session in one of two ways: with an API key (`apiKey`) or with a token (`token`). An application
|
|
285
|
+
service usually keeps one connection by key and acts on behalf of users through `as()`:
|
|
286
|
+
|
|
287
|
+
<!-- run: guide -->
|
|
288
|
+
```ts
|
|
289
|
+
// service: the API key is exchanged for a service session
|
|
290
|
+
const svc = await connect({ dsn: APP, schema: r.schema, apiKey: r.serviceKey! })
|
|
291
|
+
// sign-in: the service checks the password and acts on behalf of the user in the tenant
|
|
292
|
+
const login = await svc.auth.verifyPassword({ identifier: 'anna@shop.ru', password: 'Chamomile-2026!' })
|
|
293
|
+
const db = await svc.as(login!.account, { tenant: SHOP })
|
|
294
|
+
// a separate connection by session token, for example in another process
|
|
295
|
+
const anna = await connect({ dsn: APP, schema: r.schema, token: r.tenant!.token! })
|
|
296
|
+
```
|
|
297
|
+
|
|
298
|
+
| `connect()` option | What it does |
|
|
299
|
+
|---|---|
|
|
300
|
+
| `dsn` or `sql` | a connection string or a ready postgres.js pool |
|
|
301
|
+
| `schema` | the full installation name: `'v2.shop'` (it is handy to take `r.schema`) |
|
|
302
|
+
| `token`, `apiKey` | a session token or an API key (`service` is a synonym of `apiKey`) |
|
|
303
|
+
| `models` | `letopis/model` models, only for chain types ([section 4](#4-data-model)) |
|
|
304
|
+
| `max` | pool size (10 by default) |
|
|
305
|
+
| `listen` | listen to database signals (yes by default; `false` for PgBouncer in transaction mode) |
|
|
306
|
+
| `cache` | result cache: `{ ttl, max, ttlOnly }` or `false` ([section 12](#12-change-subscription-and-cache)) |
|
|
307
|
+
| `maintain` | run `maintain()` on a schedule: `'1h'`, `'15m'` or milliseconds |
|
|
308
|
+
| `onQuery`, `slowMs` | a hook on every chain query and the slow query threshold |
|
|
309
|
+
| `refreshSession` | extend the session once per minute of activity (yes by default) |
|
|
310
|
+
| `notifyIntervalMs` | the interval for sending deferred signals (100 ms) |
|
|
311
|
+
| `allowBypassRls`, `owner` | allow a role that bypasses RLS (admin connection); the name of the owner role |
|
|
312
|
+
|
|
313
|
+
A connection has `db.schema`, `db.token`, `db.tenant` (the session tenant) and `db.close()`. Connections obtained
|
|
314
|
+
through `as()` share the pool with the original one: close the original one.
|
|
315
|
+
|
|
316
|
+
### Admin connection
|
|
317
|
+
|
|
318
|
+
The loader needs a connection of a role that can become the owner, with `allowBypassRls: true`. `verify()` and
|
|
319
|
+
`maintain()` work on it too, and also for a system administrator on a regular connection. The admin connection prints
|
|
320
|
+
the warning "RLS не действует" ("RLS does not apply"); this is expected. The session sets the tenant it works with. A
|
|
321
|
+
separate loading process is better off with a service key:
|
|
322
|
+
`connect({ dsn: ADMIN, schema, apiKey, allowBypassRls: true })`, and the tenant of the rows is set by the
|
|
323
|
+
`load(…, { tenant })` option. The example uses the session from `up()`:
|
|
324
|
+
|
|
325
|
+
<!-- run: guide -->
|
|
326
|
+
```ts
|
|
327
|
+
const adm = await connect({ dsn: ADMIN, schema: r.schema, token: r.tenant!.token!, allowBypassRls: true })
|
|
328
|
+
```
|
|
329
|
+
|
|
330
|
+
### Upgrading the package and schema
|
|
331
|
+
|
|
332
|
+
The schema has an engine revision (currently 5). If a new package version brings a new revision, `connect()` only warns
|
|
333
|
+
and names the command. It is `up()` that refuses: a lagging schema without `upgrade: true`, and a schema newer than the
|
|
334
|
+
package. So a startup script that calls `up()` on every start will stop after a package update until it gets
|
|
335
|
+
`upgrade: true`. `up({ dsn, schema, upgrade: true })` applies the migrations from `sql/migrate/` and re-applies the
|
|
336
|
+
functions; the data is kept.
|
|
337
|
+
|
|
338
|
+
### Development without PostgreSQL: PGlite
|
|
339
|
+
|
|
340
|
+
`up({ pglite: true })` starts PGlite (PostgreSQL 18 on WebAssembly) with pgcrypto and a socket on a free port. It needs
|
|
341
|
+
the `@electric-sql/pglite` and `@electric-sql/pglite-socket` packages. There is only one connection, so:
|
|
342
|
+
|
|
343
|
+
```ts
|
|
344
|
+
const dev = await up({ schema: 'dev', pglite: true })
|
|
345
|
+
const local = await connect({ dsn: dev.pglite!.dsn, schema: dev.schema, apiKey: dev.serviceKey!,
|
|
346
|
+
listen: false, max: 1, allowBypassRls: true })
|
|
347
|
+
// … work …
|
|
348
|
+
await local.close()
|
|
349
|
+
await dev.pglite!.close()
|
|
350
|
+
```
|
|
351
|
+
|
|
352
|
+
The connection to PGlite is a superuser, so it needs `allowBypassRls: true`, and RLS does not apply on it: tenant
|
|
353
|
+
isolation and permissions cannot be checked on PGlite. It is not fit for race tests and CI either: there is one
|
|
354
|
+
connection and no `LISTEN`.
|
|
355
|
+
|
|
356
|
+
## 4. Data model
|
|
357
|
+
|
|
358
|
+
Classes are described with Zod models from the `letopis/model` subpath and written to the database. The database
|
|
359
|
+
stores a normalized JSON Schema and checks the data itself on every write.
|
|
360
|
+
|
|
361
|
+
### Hubs and links
|
|
362
|
+
|
|
363
|
+
`hub()` describes an object, `link()` describes a link with its own data. Both have two forms:
|
|
364
|
+
|
|
365
|
+
- short: `hub(name, fields, options)`: fields as an object of Zod types, options as the third argument;
|
|
366
|
+
- full: `hub(name, { fields, ends, key, extends, … })`: it is needed, for example, for inheritance (`extends`), so
|
|
367
|
+
that TypeScript infers the data types of the descendant.
|
|
368
|
+
|
|
369
|
+
The shop model that the following examples are built on:
|
|
370
|
+
|
|
371
|
+
<!-- run: guide -->
|
|
372
|
+
```ts
|
|
373
|
+
import { z } from 'zod'
|
|
374
|
+
import { hub, link, one, many } from 'letopis/model'
|
|
375
|
+
import { syncModels } from 'letopis'
|
|
376
|
+
|
|
377
|
+
const Shop = hub('Shop', { name: z.string(), city: z.string().optional() }, { key: ['name'] })
|
|
378
|
+
const Category = hub('Category', { name: z.string() }, {
|
|
379
|
+
key: ['name'],
|
|
380
|
+
ends: { Parent: one('Category').optional().onDelete('unset') }, // tree: a reference to the parent
|
|
381
|
+
})
|
|
382
|
+
const Product = hub('Product', {
|
|
383
|
+
sku: z.string(), title: z.string(), price: z.number().int().min(0), views: z.number().int().optional(),
|
|
384
|
+
}, {
|
|
385
|
+
ends: { Shop, Category: many(Category, { max: 5, onDelete: 'unset' }) },
|
|
386
|
+
key: ['Shop', 'sku'],
|
|
387
|
+
meta: { title: 'Product', icon: 'mdi:package-variant' },
|
|
388
|
+
})
|
|
389
|
+
const Customer = hub('Customer', { name: z.string(), email: z.string() }, {}) // no key: the database assigns the id
|
|
390
|
+
const Vip = hub('Vip', { extends: Customer, fields: { discount: z.number().int().min(0).max(50) } })
|
|
391
|
+
const Order = hub('Order', {
|
|
392
|
+
number: z.string(), status: z.enum(['new', 'paid', 'cancelled']).default('new'),
|
|
393
|
+
}, { ends: { Customer, Shop }, key: ['Shop', 'number'] })
|
|
394
|
+
const line = link('line', { qty: z.number().int().min(1), price: z.number().int().min(0) }, {
|
|
395
|
+
ends: { Order: one(Order).onDelete('cascade'), Product },
|
|
396
|
+
key: ['Order', 'Product'],
|
|
397
|
+
})
|
|
398
|
+
const Wallet = hub('Wallet', { number: z.string(), balance: z.number().int().min(0).default(0) },
|
|
399
|
+
{ ends: { Customer }, key: ['number'] })
|
|
400
|
+
const Movement = link('Movement', {
|
|
401
|
+
opId: z.string(), kind: z.enum(['topup', 'debit', 'adjust']), amount: z.number().int(), target: z.number().int().optional(),
|
|
402
|
+
}, { ends: { Wallet }, key: ['Wallet', 'opId'] })
|
|
403
|
+
|
|
404
|
+
const models = [Shop, Category, Product, Customer, Vip, Order, line, Wallet, Movement]
|
|
405
|
+
const plan = await syncModels(db, models) // dry run: what will be created and changed
|
|
406
|
+
const applied = await syncModels(db, models, { apply: true }) // { applied: true, created: ['Shop', …], … }
|
|
407
|
+
```
|
|
408
|
+
|
|
409
|
+
### Fields
|
|
410
|
+
|
|
411
|
+
Fields are Zod types: strings, numbers (`int`, `min`, `max`), booleans, enums (`z.enum`), dates and times
|
|
412
|
+
(`z.iso.date()`, `z.iso.datetime()`), nested objects of any depth, arrays and `z.record`. `.optional()` makes a field
|
|
413
|
+
optional. `.default(…)` sets a default that the database fills in on insert. The database stores the `.meta()`
|
|
414
|
+
annotations of a field (`title`, `description`, your own keys), but they do not affect data checks.
|
|
415
|
+
|
|
416
|
+
The `refine`, `superRefine` and `transform` rules cannot be expressed in JSON Schema, and the database does not run
|
|
417
|
+
them. `syncModels` returns them in `lost`: check such rules in the application. A description with a JSON Schema
|
|
418
|
+
keyword that the database does not know is rejected, so no rule is lost silently. `pattern` regexes are a limited
|
|
419
|
+
subset of regular expressions shared by Zod, the validator and the database.
|
|
420
|
+
|
|
421
|
+
### Ends
|
|
422
|
+
|
|
423
|
+
- `ends: { Shop }` is the same as `one(Shop)`: a single required end with the role `Shop`.
|
|
424
|
+
- `one(A, B)` is a union: the end accepts rows of several classes. `one('Category')` is a target by name: this is how
|
|
425
|
+
a class refers to itself.
|
|
426
|
+
- `.optional()` makes the end optional.
|
|
427
|
+
- `many(X, { min, max, onDelete })` is a multiple end: a list of targets in one end.
|
|
428
|
+
- `.onDelete(…)` sets what happens when a target is deleted: `restrict` (the default: deleting the target is
|
|
429
|
+
forbidden), `cascade` (delete this row too), `unset` (remove the reference; only for an optional single end and for a
|
|
430
|
+
multiple end without `min`).
|
|
431
|
+
|
|
432
|
+
The database checks the targets: the target exists, belongs to the same tenant or to System, and is of the declared
|
|
433
|
+
class or its descendant. A role name points to one class across the whole schema. So the roles `User` and `Tenant`
|
|
434
|
+
(pointing to `Account`), `Group` and `Object` (pointing to `Resource`) are taken by system classes: your own end with
|
|
435
|
+
such a name is possible only with the same target. In a result row, a single end is the target id (`row.links.Shop`),
|
|
436
|
+
and a multiple end is a list of ids.
|
|
437
|
+
|
|
438
|
+
### Key and id
|
|
439
|
+
|
|
440
|
+
A key is made of required string fields and required single ends. Numbers, booleans, objects, optional fields and
|
|
441
|
+
multiple ends cannot be part of a key. The id of a class with a key is a version 5 UUID of the tenant, the class and
|
|
442
|
+
the key values. The values are taken as they are, without normalization: an email in a different case gives a
|
|
443
|
+
different id. Set the required format of a key field in the model, and the database will check it.
|
|
444
|
+
|
|
445
|
+
- You do not pass the id: the database computes it. A different id gives `id_mismatch`; an id for a class without a
|
|
446
|
+
key gives `id_from_db`.
|
|
447
|
+
- `db.idOf(class, key)` computes the id in advance, for example `db.idOf('Shop', { name: 'Chamomile' })`.
|
|
448
|
+
- A class without a key (`Customer` above) gets an id from a random part; you can learn it from the write result.
|
|
449
|
+
- A key field cannot be changed by a regular update (`immutable_key`): a new key means a new id, and that is `rekey`
|
|
450
|
+
([section 5](#5-writing)).
|
|
451
|
+
|
|
452
|
+
If something other than the key must be unique, make it a separate class. For example, the email is the key of an
|
|
453
|
+
`Email` hub, and the link from a person to it has the key `['Email']`. It is better not to make personal data the key
|
|
454
|
+
of the person itself: an id from an email can be found by brute force, and it stays in references and in the journal.
|
|
455
|
+
|
|
456
|
+
### Inheritance and aliases
|
|
457
|
+
|
|
458
|
+
`extends`: the descendant gets the fields, ends and key of the ancestor. Its own field with the same name replaces the
|
|
459
|
+
ancestor's field entirely. `abstract: true`: the class never has its own rows. A chain step by class also returns the
|
|
460
|
+
rows of descendants: `db.Customer()` also finds `Vip`, and `.exact()` keeps only the class itself. `alias` is a second
|
|
461
|
+
name of the class for a chain step, for example a name in another language.
|
|
462
|
+
|
|
463
|
+
### Class settings
|
|
464
|
+
|
|
465
|
+
- `history: false`: no journal is kept for the class: no versions, no point-in-time reads, no restore and no
|
|
466
|
+
subscription events. A retention policy is `history: { all: '1 day', daily: '3 months', … }`
|
|
467
|
+
([section 11](#11-history-and-point-in-time-reads)).
|
|
468
|
+
- `ownerDefault: 'actor'`: the owner of new rows is the user, not the tenant (the default is `'tenant'`). This is
|
|
469
|
+
useful for the rule "everyone sees their own records".
|
|
470
|
+
- `cache: { ttl }`: reads of this class alone are cached without `.cache()` ([section 12](#12-change-subscription-and-cache)).
|
|
471
|
+
- `meta`: free-form metadata for the application: a title, an icon, a color, an order in a menu. Up to 64 KB,
|
|
472
|
+
inherited key by key (`null` removes an inherited value), visible to anyone who has a session. For an end, use
|
|
473
|
+
`.meta()`.
|
|
474
|
+
|
|
475
|
+
### Writing the model to the database
|
|
476
|
+
|
|
477
|
+
`syncModels(db, models)` without `apply` is a dry run; with `{ apply: true }` it applies the changes. The result is
|
|
478
|
+
`{ applied, created, updated, unchanged, skipped, lost }`; `skipped` lists System classes, which only the installation
|
|
479
|
+
changes. `syncModels` touches only the classes passed to it and deletes nothing. Safe changes (a new optional field or
|
|
480
|
+
end, metadata, an alias) are applied at once. A tightening checks all rows of the family; if some rows conflict with it, you get
|
|
481
|
+
`tightening_conflict` with a list. Classes are written to the session tenant, and other tenants do not see them.
|
|
482
|
+
Classes shared by everyone are in System: the installation seed or the import puts them there.
|
|
483
|
+
|
|
484
|
+
The same from the command line:
|
|
485
|
+
|
|
486
|
+
```sh
|
|
487
|
+
npx letopis sync ./models.js --dsn "$APP" --schema v2.shop --token "$TOKEN" # dry run
|
|
488
|
+
npx letopis sync ./models.js --dsn "$APP" --schema v2.shop --token "$TOKEN" --apply # apply
|
|
489
|
+
npx letopis types --dsn "$APP" --schema v2.shop --token "$TOKEN" --out types.ts # class types from the database
|
|
490
|
+
```
|
|
491
|
+
|
|
492
|
+
### Chain types
|
|
493
|
+
|
|
494
|
+
`connect({ …, models })` gives types of steps, rows and filters from the models. The library still takes the class
|
|
495
|
+
descriptions from the database; only TypeScript needs the models. The row data type is `Infer<typeof Product>`, the
|
|
496
|
+
ends type is `InferLinks<typeof Product>`. Your own `meta` shape is typed by extending the `ClassMeta` interface.
|
|
497
|
+
|
|
498
|
+
### Class names
|
|
499
|
+
|
|
500
|
+
A chain step is named after a class, its alias or a role name. The Proxy parses the names of chain and root methods
|
|
501
|
+
(`count`, `sort`, `set`, `sql`, `auth` and others) before classes. So a class with such a name cannot be reached as a
|
|
502
|
+
step, and the database does not accept a link with such a name. The full list is in the
|
|
503
|
+
["Reserved names"](#reserved-names) section.
|
|
504
|
+
|
|
505
|
+
## 5. Writing
|
|
506
|
+
|
|
507
|
+
The write verbs — `create`, `update`, `upsert`, `delete`, `anonymize`, `reclass`, `rekey`, `restore`, `purge` — are
|
|
508
|
+
parts of a plan. A verb does nothing by itself: a terminal (`rows`, `first`, `ids`, `count`) executes the plan, and the
|
|
509
|
+
whole plan is one transaction.
|
|
510
|
+
|
|
511
|
+
### Create: `create`
|
|
512
|
+
|
|
513
|
+
<!-- run: guide -->
|
|
514
|
+
```ts
|
|
515
|
+
import { LetopisError } from 'letopis'
|
|
516
|
+
|
|
517
|
+
const [shop] = await db.Shop().create({ name: 'Chamomile', city: 'Kazan' }).rows()
|
|
518
|
+
const [tea] = await db.Category().create({ name: 'Tea' }).rows()
|
|
519
|
+
const [green] = await db.Category().create({ name: 'Green tea' }).Parent.set(tea).rows()
|
|
520
|
+
const [kettle] = await db.Shop(shop).Product().create({ sku: 'K-1', title: 'Tea kettle', price: 2500 }).rows()
|
|
521
|
+
const [sencha] = await db.Shop(shop).Product()
|
|
522
|
+
.create({ sku: 'T-7', title: 'Sencha', price: 900 }).Category.add(green).rows()
|
|
523
|
+
const [olga] = await db.Customer().create({ name: 'Olga', email: 'olga@mail.ru' }).rows()
|
|
524
|
+
|
|
525
|
+
try {
|
|
526
|
+
await db.Shop().create({ name: 'Chamomile' }).rows()
|
|
527
|
+
} catch (e) {
|
|
528
|
+
if (!(e instanceof LetopisError) || e.code !== 'exists') throw e // e.detail.id === shop.id: this shop already exists
|
|
529
|
+
}
|
|
530
|
+
```
|
|
531
|
+
|
|
532
|
+
- `create` only creates. A taken key gives the error `exists` with `detail.id`: the library computes the id itself. So
|
|
533
|
+
for a class with a key, a retry after a network failure shows at once that the first attempt went through. A class
|
|
534
|
+
without a key has no such protection: a retry creates a second object.
|
|
535
|
+
- The tags of a new row are set by `.tags([…])` before the verb, the owner by `.owner(id)`; an explicit owner must be
|
|
536
|
+
allowed by the permissions.
|
|
537
|
+
- A deleted object can be created again with the same key: its history continues.
|
|
538
|
+
- The result is the rows of the created objects with all columns ([section 15](#15-api-reference), "Result row").
|
|
539
|
+
|
|
540
|
+
### Update: `update`
|
|
541
|
+
|
|
542
|
+
<!-- run: guide -->
|
|
543
|
+
```ts
|
|
544
|
+
import { gte } from 'letopis'
|
|
545
|
+
|
|
546
|
+
await db.Product(kettle).update({ title: 'Glass tea kettle' }).rows()
|
|
547
|
+
await db.Shop(shop).Product({ price: gte(2000) }).update({ title: 'Glass tea kettle, 1 L' }).rows()
|
|
548
|
+
```
|
|
549
|
+
|
|
550
|
+
`update` edits every target of the step and never creates. The patch is merged with the data inside the statement:
|
|
551
|
+
nested objects are merged recursively, while arrays, scalars and `null` are replaced. So concurrent edits of different
|
|
552
|
+
fields are not lost. If the step has no filter of its own, the target is taken from `data.id` of the patch, and without
|
|
553
|
+
`data.id` all entities of the step are edited: `db.Product().update(…)` edits all products of the tenant. If the step
|
|
554
|
+
found nothing, the result is empty and there is no error, so check the length of the result. If `update` affected fewer
|
|
555
|
+
rows than the step found, the whole plan is rolled back: a row deleted in the meantime gives `target_not_found`, and a
|
|
556
|
+
row that is visible but may not be edited gives `acl_denied`. The `.tags(…)` modifier before `update` both selects the
|
|
557
|
+
rows and replaces their tags entirely.
|
|
558
|
+
|
|
559
|
+
### Create or replace: `upsert`
|
|
560
|
+
|
|
561
|
+
<!-- run: guide -->
|
|
562
|
+
```ts
|
|
563
|
+
const [kept] = await db.Shop().upsert({ name: 'Chamomile', city: 'Kazan' }).rows() // kept.$upsert: 'unchanged'
|
|
564
|
+
const [moved] = await db.Shop().upsert({ name: 'Chamomile', city: 'Ufa' }).rows() // moved.$upsert: 'updated'
|
|
565
|
+
```
|
|
566
|
+
|
|
567
|
+
`upsert` creates an object, and if an object with this key already exists, it replaces its data, ends and tags
|
|
568
|
+
entirely. This is one `INSERT … ON CONFLICT` statement, so the race "checked that it does not exist, then created it"
|
|
569
|
+
cannot happen. Rules:
|
|
570
|
+
|
|
571
|
+
- only for a class with a key, otherwise `no_key` without a call to the database;
|
|
572
|
+
- the object is passed whole: fields that are not in the passed object disappear;
|
|
573
|
+
- the result carries `$upsert`: `created`, `updated` or `unchanged`;
|
|
574
|
+
- `upsert` cannot increment and overwrites concurrent increments, so it is not suitable for money;
|
|
575
|
+
- to sync large reference tables, the loader with `existing: 'update'` is cheaper ([section 9](#9-bulk-loading)).
|
|
576
|
+
|
|
577
|
+
### Ends: from the path and slots
|
|
578
|
+
|
|
579
|
+
The ends of a created row are taken from the chain path: `db.Shop(shop).Product().create(…)` references `shop`. If
|
|
580
|
+
several roles fit, the role whose slot is free is chosen, then the role named after the step class; otherwise you get
|
|
581
|
+
`invalid_query`. A slot after the verb sets an end explicitly: `.RoleName.set(value)` and `.RoleName.unset()`, and for
|
|
582
|
+
a multiple end also `.add()` and `.remove()`. The value is an id, a list of ids, a result row or a nested chain. Slots
|
|
583
|
+
change `links` atomically, without reading the list:
|
|
584
|
+
|
|
585
|
+
<!-- run: guide -->
|
|
586
|
+
```ts
|
|
587
|
+
await db.Product(sencha).update({}).Category.add(tea).rows() // one more category
|
|
588
|
+
await db.Product(sencha).update({}).Category.remove(tea).rows()
|
|
589
|
+
await db.Category(green).update({}).Parent.unset().rows() // the category is now a root
|
|
590
|
+
await db.Category(green).update({}).Parent.set(tea).rows() // and inside "Tea" again
|
|
591
|
+
```
|
|
592
|
+
|
|
593
|
+
### Numbers and counters: `inc`
|
|
594
|
+
|
|
595
|
+
<!-- run: guide -->
|
|
596
|
+
```ts
|
|
597
|
+
import { inc } from 'letopis'
|
|
598
|
+
|
|
599
|
+
await db.Product(kettle).update({ price: inc(-500) }).rows() // the price went down by 500
|
|
600
|
+
await db.Product(kettle).update({ views: inc(1, { start: 0 }) }).rows() // the field does not exist yet: start from 0
|
|
601
|
+
```
|
|
602
|
+
|
|
603
|
+
`inc(n)` adds the amount inside the `UPDATE` statement, over the fresh locked row. So concurrent increments are not
|
|
604
|
+
lost, and a class rule like `min(0)` is checked against the result: going below zero is rejected with `invalid_data`,
|
|
605
|
+
without reads or locks in the application. Rules:
|
|
606
|
+
|
|
607
|
+
- only in `update`; a marker in `create`, `upsert`, the loader or inside an array gives `invalid_data` before sending;
|
|
608
|
+
- the field must already be a number; if it is missing, you get `invalid_data`, or `start` is used. `null` and a
|
|
609
|
+
non-number are an error even with `start`;
|
|
610
|
+
- the path goes only through existing objects;
|
|
611
|
+
- an `inc` marker cannot come from JSON: `{ "$inc": 5 }` that came from the network stays data;
|
|
612
|
+
- store money as an integer number of minor units (cents), no greater than 2^53 − 1.
|
|
613
|
+
|
|
614
|
+
### Protection against concurrent edits: `{ rev }`
|
|
615
|
+
|
|
616
|
+
<!-- run: guide -->
|
|
617
|
+
```ts
|
|
618
|
+
const cur = await db.Product(kettle).first()
|
|
619
|
+
await db.Product(kettle).update({ price: 1900 }, { rev: cur!.rev }).rows() // the version matched: written
|
|
620
|
+
try {
|
|
621
|
+
await db.Product(kettle).update({ price: 1800 }, { rev: cur!.rev }).rows() // the version is already different
|
|
622
|
+
} catch (e) {
|
|
623
|
+
if (!(e instanceof LetopisError) || e.code !== 'conflict') throw e // re-read and decide again
|
|
624
|
+
}
|
|
625
|
+
```
|
|
626
|
+
|
|
627
|
+
The strict mode is needed when the new value is computed from what was read: if the row was changed in the meantime,
|
|
628
|
+
you get `conflict`, and nothing is written. The mode works when the step has one target (otherwise `rev_ambiguous`).
|
|
629
|
+
The library does not retry `conflict` by itself: the application decides what to do with the changed data. Another way
|
|
630
|
+
is a read with a `forUpdate()` lock ([section 8](#8-transactions-batches-and-money)).
|
|
631
|
+
|
|
632
|
+
### Changing the key or class
|
|
633
|
+
|
|
634
|
+
<!-- run: guide -->
|
|
635
|
+
```ts
|
|
636
|
+
const [mug] = await db.Shop(shop).Product().create({ sku: 'M-1', title: 'Mug', price: 400 }).rows()
|
|
637
|
+
const [mug2] = await db.Product(mug).rekey({ sku: 'M-100' }).rows() // new key, new id; references moved
|
|
638
|
+
const [vip] = await db.Customer(olga).reclass('Vip', { name: 'Olga', email: 'olga@mail.ru', discount: 10 }).rows()
|
|
639
|
+
```
|
|
640
|
+
|
|
641
|
+
- `rekey(patch)` sets a new key value: a row with a new id, all references moved, the old row deleted. If a referencing
|
|
642
|
+
row includes this end in its own key, its id changes too, and the move goes on. The journal links the old and the new
|
|
643
|
+
id (`moved`).
|
|
644
|
+
- `db.rekeyClass(class, key)` sets a new key for all rows of the family, in one transaction.
|
|
645
|
+
- `reclass(class, data)` changes the class; the data is optional, and if given, it is given whole. If the new class has
|
|
646
|
+
no key, the id is kept (as for `vip` above); if it has one, the id is computed anew and the references are moved.
|
|
647
|
+
The roles of referencing rows must accept the new class, otherwise `reclass_denied`. A raw `UPDATE … SET class`
|
|
648
|
+
passes only if neither the old nor the new class has a key (otherwise `use_reclass`).
|
|
649
|
+
- Deleting, changing the class and changing the key work only in `read committed`: library transactions always open at
|
|
650
|
+
this level.
|
|
651
|
+
|
|
652
|
+
### Anonymize: `anonymize`
|
|
653
|
+
|
|
654
|
+
<!-- run: guide -->
|
|
655
|
+
```ts
|
|
656
|
+
await db.Customer(olga).anonymize(['email']).rows() // email → '[erased]', tag anonymized
|
|
657
|
+
```
|
|
658
|
+
|
|
659
|
+
`anonymize(fields)` overwrites string fields with a new version and marks the row with the `anonymized` tag. The
|
|
660
|
+
earlier versions stay in the journal. For complete erasure, the next steps are history trimming
|
|
661
|
+
([section 11](#11-history-and-point-in-time-reads)), then deletion and `purge` ([section 7](#7-deleting-and-restoring)).
|
|
662
|
+
|
|
663
|
+
### How a plan runs
|
|
664
|
+
|
|
665
|
+
One plan can create several related objects. The chain after a verb continues from the verb's result: after `create`
|
|
666
|
+
and `upsert`, from the created row; after other verbs, from all rows.
|
|
667
|
+
|
|
668
|
+
<!-- run: guide -->
|
|
669
|
+
```ts
|
|
670
|
+
const [l1] = await db.Customer(olga).Order()
|
|
671
|
+
.create({ number: 'A-1' }).Shop.set(shop) // order: Customer from the path, Shop via a slot
|
|
672
|
+
.line().create({ qty: 2, price: 2000 }).Product.set(kettle) // order line: Order from the path
|
|
673
|
+
.rows() // rows of the last step: the order line
|
|
674
|
+
const order = l1.links.Order as string
|
|
675
|
+
await db.Order(order).line().create({ qty: 1, price: 900 }).Product.set(sencha).rows()
|
|
676
|
+
```
|
|
677
|
+
|
|
678
|
+
- The whole plan is one `read committed` transaction. On a deadlock (`40P01`), a serialization failure (`40001`) and a
|
|
679
|
+
journal conflict, the library retries the whole plan, up to three times.
|
|
680
|
+
- An error in any part rolls back the whole plan. `exists` and `conflict` are not retried: they are answers for the
|
|
681
|
+
application.
|
|
682
|
+
- Modifiers before a verb (`limit`, `sort`) narrow its targets.
|
|
683
|
+
- A verb right after a verb writes to the same entities.
|
|
684
|
+
|
|
685
|
+
## 6. Reading
|
|
686
|
+
|
|
687
|
+
### Steps and hops
|
|
688
|
+
|
|
689
|
+
A chain starts with a class: `db.Shop()` is all shops, `db.Shop(shop)` is one shop. The next step is a class, an alias
|
|
690
|
+
or an end role. A hop goes along ends: a role step goes along its own end; other steps go along the ends of all classes
|
|
691
|
+
of both families. From an object to the rows that reference it, the hop goes backward; to the target of a reference,
|
|
692
|
+
forward; to the same class, to the children.
|
|
693
|
+
|
|
694
|
+
<!-- run: guide -->
|
|
695
|
+
```ts
|
|
696
|
+
const products = await db.Shop(shop).Product().rows() // products of the shop: backward along the Shop end
|
|
697
|
+
const shopOf = await db.Product(kettle).Shop().first() // the shop of the product: forward
|
|
698
|
+
const inOrder = await db.Order(order).line().Product().rows() // products of the order: hub → link → hub
|
|
699
|
+
const parent = await db.Category(green).Parent().first() // role step: the parent category
|
|
700
|
+
const allCustomers = await db.Customer().rows() // together with descendants (Vip)
|
|
701
|
+
const onlyCustomer = await db.Customer().exact().rows() // only the class itself
|
|
702
|
+
```
|
|
703
|
+
|
|
704
|
+
### Filters
|
|
705
|
+
|
|
706
|
+
A step filter is an id, a list of ids, a result row, an object of fields or `or(…)`. An object of fields can have any
|
|
707
|
+
depth. A plain value means equality through containment in `data`, without type casting: `{ price: 900 }` will not find
|
|
708
|
+
a record with the price `"900"`. A list as a field value means "the array contains all the elements". Operators cast
|
|
709
|
+
their value to the field type from the JSON Schema. An end role in a filter takes a target id or `null`.
|
|
710
|
+
|
|
711
|
+
<!-- run: guide -->
|
|
712
|
+
```ts
|
|
713
|
+
import { lt, ilike, between, or, not } from 'letopis'
|
|
714
|
+
|
|
715
|
+
const cheap = await db.Product({ price: lt(1000) }).rows()
|
|
716
|
+
const teaCount = await db.Product({ title: ilike('%tea%'), price: between(500, 3000) }).count()
|
|
717
|
+
const twoIds = await db.Product(or({ sku: 'K-1' }, { sku: 'T-7' })).ids()
|
|
718
|
+
const active = await db.Order({ status: not('cancelled') }).rows()
|
|
719
|
+
const ofShop = await db.Product({ Shop: shop.id }).rows()
|
|
720
|
+
const roots = await db.Category({ Parent: null }).rows() // root categories
|
|
721
|
+
```
|
|
722
|
+
|
|
723
|
+
| Operator | Condition |
|
|
724
|
+
|---|---|
|
|
725
|
+
| `ne(v)`, `gt(v)`, `gte(v)`, `lt(v)`, `lte(v)` | ≠ (taking `null` into account), >, ≥, <, ≤ |
|
|
726
|
+
| `between(a, b)`, `inList([…])` | a ≤ x ≤ b; a value from the list |
|
|
727
|
+
| `like(s)`, `ilike(s)`, `starts(s)`, `ends(s)` | a pattern with `%` and `_`; case-insensitive; starts with; ends with |
|
|
728
|
+
| `has(v)`, `hasAny([…])`, `hasAll([…])` | an array field contains the value; at least one; all |
|
|
729
|
+
| `exists(true \| false)`, `isNull()` | the field is present or not; the field is `null` or missing |
|
|
730
|
+
| `not(v)` | negation of an operator; for a scalar, "not equal" |
|
|
731
|
+
| `or(f1, f2, …)` | any of the step filters |
|
|
732
|
+
|
|
733
|
+
Tags and the row owner are filtered with the `.tags(…)` and `.owner(…)` modifiers.
|
|
734
|
+
|
|
735
|
+
### Terminals and the result row
|
|
736
|
+
|
|
737
|
+
| Terminal | What it returns |
|
|
738
|
+
|---|---|
|
|
739
|
+
| `rows()` | the rows of the entities of the last step, without duplicates |
|
|
740
|
+
| `first()` | the first row or `null` |
|
|
741
|
+
| `ids()` | the ids of the entities of the last step |
|
|
742
|
+
| `count()` | the number of entities of the last step; `count({ paths: true })` is the number of paths |
|
|
743
|
+
| `paths()` | paths: `[{ Step: Row, … }]`, a node for each step |
|
|
744
|
+
| `versions()` | all versions of the entities of the last step ([section 11](#11-history-and-point-in-time-reads)) |
|
|
745
|
+
| `sum`, `avg`, `min`, `max`, `countBy` | aggregates over the field `'data.<path>'`; on an empty set, `null`, and `{}` for `countBy` |
|
|
746
|
+
|
|
747
|
+
Result row: `id`, `class`, `rev` (version number), `tenant`, `owner`, `links` (ends), `data` (fields), `tags`, `at`
|
|
748
|
+
(version time), `author`, `op` (kind of operation) and, if present, `agent`, `reason`, `moved`. Service fields start
|
|
749
|
+
with `$`: `$deleted`, `$depth`, `$upsert`, `$action`, `$versions`, `$purged`.
|
|
750
|
+
|
|
751
|
+
### Sorting and pagination
|
|
752
|
+
|
|
753
|
+
<!-- run: guide -->
|
|
754
|
+
```ts
|
|
755
|
+
import { cursorOf } from 'letopis'
|
|
756
|
+
|
|
757
|
+
const page1 = await db.Product().sort('data.price', 'desc').limit(2).rows()
|
|
758
|
+
const page2 = await db.Product().sort('data.price', 'desc')
|
|
759
|
+
.after(cursorOf(page1.at(-1)!, 'data.price')).limit(2).rows() // strictly after the last row of the page
|
|
760
|
+
```
|
|
761
|
+
|
|
762
|
+
`sort(field, 'asc' | 'desc')` sorts by `'at'`, `'rev'`, `'id'` or `'data.<path>'`. `after(cursorOf(row, field))` is
|
|
763
|
+
pagination "after this row"; it requires `sort()`. Such pagination withstands inserts and deletes between pages, but a
|
|
764
|
+
row whose sort field has changed may drop out or repeat. A cursor is an object `{ v, id }` (the field value and the row
|
|
765
|
+
id): it survives JSON, so you can hand it to a client. The field in `cursorOf` must match the `sort` field. The cursor
|
|
766
|
+
value is checked against the type of the `sort` field before the query, and a wrong one is `invalid_query`: a numeric
|
|
767
|
+
field needs a finite number, `rev` an integer, a boolean field `true` or `false`, a text field a string, uuid fields
|
|
768
|
+
and `id` a uuid, time fields and `at` an ISO string with a time zone, a date field `YYYY-MM-DD`, a field without a
|
|
769
|
+
description a string, a number or `true`/`false`. `null` fits only data fields, and the cursor's `id` is always a uuid.
|
|
770
|
+
The check cannot tell apart a cursor of the same type taken from another field. `limit` and `offset` are available
|
|
771
|
+
too.
|
|
772
|
+
|
|
773
|
+
A data field may be `null` or missing. `sort` places such rows the way `order by` in PostgreSQL does: last when
|
|
774
|
+
ascending, first when descending, and `after()` treats `null` as a value after all others (ascending) or before them
|
|
775
|
+
(descending). So pages go through such rows without gaps or repeats, and a cursor with the value `null` gives the next
|
|
776
|
+
page.
|
|
777
|
+
|
|
778
|
+
### Trees: `deep`
|
|
779
|
+
|
|
780
|
+
<!-- run: guide -->
|
|
781
|
+
```ts
|
|
782
|
+
const subtree = await db.Category(tea).Category().deep().rows() // subcategories of any depth; $depth is the level
|
|
783
|
+
```
|
|
784
|
+
|
|
785
|
+
`deep(max)` repeats the hop to the same class up to the depth `max` (32 levels by default). The filter, tags and owner
|
|
786
|
+
of the step select nodes at any level but do not cut the traversal. Only a node invisible by permissions cuts it: such a
|
|
787
|
+
node also hides its subtree.
|
|
788
|
+
|
|
789
|
+
### Aggregates
|
|
790
|
+
|
|
791
|
+
<!-- run: guide -->
|
|
792
|
+
```ts
|
|
793
|
+
const total = await db.Order(order).line().sum('data.price') // sum of the order lines
|
|
794
|
+
const avgPrice = await db.Shop(shop).Product().avg('data.price')
|
|
795
|
+
const byStatus = await db.Order().countBy('data.status') // { new: 1 }
|
|
796
|
+
```
|
|
797
|
+
|
|
798
|
+
The database computes aggregates over the entities of the last step.
|
|
799
|
+
|
|
800
|
+
### Paths
|
|
801
|
+
|
|
802
|
+
<!-- run: guide -->
|
|
803
|
+
```ts
|
|
804
|
+
const orderPaths = await db.Order(order).line().Product().paths() // [{ Order: Row, line: Row, Product: Row }, …]
|
|
805
|
+
```
|
|
806
|
+
|
|
807
|
+
`paths()` returns the nodes of every variant of the path; the key of a node is the step name or the name from
|
|
808
|
+
`.alias(name)`. Repeating a link in the chain returns to its node: this way you can go from one link in several
|
|
809
|
+
directions. `db.entity(row | chain)` inserts a ready node into the path.
|
|
810
|
+
|
|
811
|
+
## 7. Deleting and restoring
|
|
812
|
+
|
|
813
|
+
### Preview and deletion
|
|
814
|
+
|
|
815
|
+
<!-- run: guide -->
|
|
816
|
+
```ts
|
|
817
|
+
const preview = await db.Order(order).delete().rows() // deletes nothing: the order and its lines, $action of each
|
|
818
|
+
await db.Order(order).delete({ confirm: true }).rows() // deleted: tombstones with $deleted: true
|
|
819
|
+
|
|
820
|
+
const blockers = (await db.Shop(shop).delete().rows()).filter((x) => x.$action === 'restrict')
|
|
821
|
+
try {
|
|
822
|
+
await db.Shop(shop).delete({ confirm: true }).rows() // products block the shop: the Shop end is restrict
|
|
823
|
+
} catch (e) {
|
|
824
|
+
if (!(e instanceof LetopisError) || e.code !== 'delete_restricted') throw e
|
|
825
|
+
}
|
|
826
|
+
```
|
|
827
|
+
|
|
828
|
+
`delete()` without `{ confirm: true }` is only a preview: the whole cascade closure. Every preview row has `$action`
|
|
829
|
+
(`delete`, `unset` or `restrict`) and `$depth`, the cascade level (0 is the target itself; `unset` and `restrict` rows
|
|
830
|
+
do not have it). With the confirmation, all targets of the step are deleted in one statement, and the cascade applies
|
|
831
|
+
the rules of the ends. The preview does not take permissions into account: a row with `$action: 'delete'` may fail to
|
|
832
|
+
be deleted for a user without the permission (`acl_denied`), so a user interface should ask
|
|
833
|
+
`acl.checkData(class, 'DELETE')`. The rules of the ends:
|
|
834
|
+
|
|
835
|
+
- `restrict`: deletion is forbidden while the target is referenced (`delete_restricted` with a list of the visible rows
|
|
836
|
+
and the number of the others);
|
|
837
|
+
- `cascade`: the referencing rows are deleted together with the target (the order lines above);
|
|
838
|
+
- `unset`: the reference is removed.
|
|
839
|
+
|
|
840
|
+
The cascade requires delete permission on every deleted row, and removing a reference requires write permission. A raw
|
|
841
|
+
`DELETE … WHERE id IN (…)` of any set of related rows works the same way.
|
|
842
|
+
|
|
843
|
+
### Restore: `restore`
|
|
844
|
+
|
|
845
|
+
<!-- run: guide -->
|
|
846
|
+
```ts
|
|
847
|
+
await db.Order(order).restore().rows() // the order, from its last snapshot, with the same id
|
|
848
|
+
await db.Order(order).line().withDeleted().restore().rows() // and its lines deleted by the cascade
|
|
849
|
+
```
|
|
850
|
+
|
|
851
|
+
`restore()` brings back the last snapshot of a deleted object as a new version and checks it again against the current
|
|
852
|
+
class description. Only the targets of the step are restored, not the whole cascade. A live object gives `not_deleted`.
|
|
853
|
+
|
|
854
|
+
### Erase history: `purge`
|
|
855
|
+
|
|
856
|
+
<!-- run: guide -->
|
|
857
|
+
```ts
|
|
858
|
+
const [temp] = await db.Customer().create({ name: 'Test', email: 'test@example.com' }).rows()
|
|
859
|
+
await db.Customer(temp).delete({ confirm: true }).rows()
|
|
860
|
+
const before = await db.Customer(temp).purge().rows() // preview: $versions is how many versions the journal has
|
|
861
|
+
await db.Customer(temp).purge({ confirm: true }).rows() // history erased; the journal has a purge record
|
|
862
|
+
```
|
|
863
|
+
|
|
864
|
+
`purge` works only for deleted objects (a live one gives `not_deleted`) and erases their whole history; in place of
|
|
865
|
+
version 1, a `purge` record stays with the number of erased versions. The history of system classes is not erased. To
|
|
866
|
+
delete an account, use `db.auth.purgeAccount` ([section 10](#10-users-sign-in-and-permissions)).
|
|
867
|
+
|
|
868
|
+
## 8. Transactions, batches and money
|
|
869
|
+
|
|
870
|
+
### Every operation is a transaction
|
|
871
|
+
|
|
872
|
+
Every chain terminal is its own `read committed` transaction, whatever the default level in the database is. At this
|
|
873
|
+
level, increments and the cascade see fresh rows. The library retries deadlocks, serialization failures and journal
|
|
874
|
+
conflicts by itself. If the connection drops during `COMMIT`, it does not retry and throws `commit_unknown`: the
|
|
875
|
+
transaction may have committed. A repeat of the same operation with the same key finds out the outcome: `exists` means
|
|
876
|
+
"already written".
|
|
877
|
+
|
|
878
|
+
### Explicit transaction: `db.begin()`
|
|
879
|
+
|
|
880
|
+
<!-- run: guide -->
|
|
881
|
+
```ts
|
|
882
|
+
const tr = await db.begin()
|
|
883
|
+
try {
|
|
884
|
+
const [o2] = await tr.Customer(olga).Order().create({ number: 'A-2' }).Shop.set(shop).rows()
|
|
885
|
+
await tr.Order(o2).line().create({ qty: 1, price: 2000 }).Product.set(kettle).rows()
|
|
886
|
+
await tr.commit()
|
|
887
|
+
} catch (e) {
|
|
888
|
+
await tr.rollback()
|
|
889
|
+
throw e
|
|
890
|
+
}
|
|
891
|
+
```
|
|
892
|
+
|
|
893
|
+
- Inside, you have the same chains, `db.sql`, `forUpdate()`, `lock()`, `commit()` and `rollback()`.
|
|
894
|
+
- An error in any query breaks the transaction: every next call and `commit()` throw the original error, and `commit()`
|
|
895
|
+
rolls back. `rollback()` goes through. If the error was caught inside `tr.transaction(fn)` and the library does not
|
|
896
|
+
know about it, PostgreSQL silently answers `ROLLBACK` to `COMMIT`, and the library turns this into the
|
|
897
|
+
`tx_rolled_back` error.
|
|
898
|
+
- There are no retries inside `db.begin()`: on `40P01`, repeat the whole block.
|
|
899
|
+
- `lock(...keys)` is an advisory lock until the end of the transaction. It protects only if everyone who edits this data
|
|
900
|
+
takes it.
|
|
901
|
+
|
|
902
|
+
### Batch
|
|
903
|
+
|
|
904
|
+
<!-- run: guide -->
|
|
905
|
+
```ts
|
|
906
|
+
const b = db.batch('storefront')
|
|
907
|
+
b.Shop(shop).Product().create({ sku: 'C-1', title: 'Cup', price: 300 })
|
|
908
|
+
b.Shop(shop).Product().create({ sku: 'C-2', title: 'Saucer', price: 200 })
|
|
909
|
+
const results = await b.run() // Row[][]: an array of rows for each plan, in one transaction
|
|
910
|
+
```
|
|
911
|
+
|
|
912
|
+
A batch collects plans, and `run()` executes them in one transaction. Consecutive simple `create` and `upsert` calls of
|
|
913
|
+
a class with a key are merged into one multi-row statement. `b.size()` is the number of queued plans, `b.discard()`
|
|
914
|
+
clears the queue. A terminal on a queued plan gives `invalid_query`: only `run()` executes. The name is the key of a
|
|
915
|
+
queue of the facade: the root connection, each `db.as()` and each `db.begin()` have their own queues. A plan runs on
|
|
916
|
+
behalf of the facade that queued it (actor, tenant, owner, transaction), and `run()` never executes plans of other
|
|
917
|
+
facades. Collect plans through one and the same facade object.
|
|
918
|
+
|
|
919
|
+
### Recipe: wallet
|
|
920
|
+
|
|
921
|
+
The balance is the `balance` field of the `Wallet` hub (integer cents, `min(0)`); operations are the `Movement` link
|
|
922
|
+
with the key `[Wallet, opId]` ([section 4](#4-data-model)). An operation is never edited or deleted; it is cancelled by
|
|
923
|
+
a reverse operation with a new `opId`. The key makes an operation idempotent: a repeat after a lost response gets
|
|
924
|
+
`exists`, and the recipe checks whether it is the same request.
|
|
925
|
+
|
|
926
|
+
<!-- run: guide -->
|
|
927
|
+
```ts
|
|
928
|
+
import { inc, type Db } from 'letopis'
|
|
929
|
+
|
|
930
|
+
type Kind = 'topup' | 'debit' | 'adjust'
|
|
931
|
+
|
|
932
|
+
/** Operation: write a Movement and increment the balance — one plan, one transaction. */
|
|
933
|
+
async function operate(db: Db, wallet: string, opId: string, kind: 'topup' | 'debit', amount: number) {
|
|
934
|
+
try {
|
|
935
|
+
await db.Wallet(wallet).Movement().create({ opId, kind, amount })
|
|
936
|
+
.Wallet().update({ balance: inc(kind === 'topup' ? amount : -amount) }).rows()
|
|
937
|
+
return 'done'
|
|
938
|
+
} catch (e) {
|
|
939
|
+
if (e instanceof LetopisError && e.code === 'exists') return same(db, wallet, opId, kind, amount) // repeat
|
|
940
|
+
throw e // invalid_data: not enough funds (min(0))
|
|
941
|
+
}
|
|
942
|
+
}
|
|
943
|
+
|
|
944
|
+
/** Repeat fingerprint: is the request already written under this opId the same request? */
|
|
945
|
+
async function same(db: Db, wallet: string, opId: string, kind: Kind, value: number) {
|
|
946
|
+
const m = await db.Movement(db.idOf('Movement', { Wallet: wallet, opId })).first()
|
|
947
|
+
if (!m || m.data.kind !== kind || (kind === 'adjust' ? m.data.target : m.data.amount) !== value)
|
|
948
|
+
throw new Error(`operation key ${opId} is already used by another request`)
|
|
949
|
+
return 'repeat'
|
|
950
|
+
}
|
|
951
|
+
|
|
952
|
+
/** Adjustment "balance = target": an adjust operation for the difference, the total under { rev }. */
|
|
953
|
+
async function adjust(db: Db, wallet: string, opId: string, target: number) {
|
|
954
|
+
for (;;) {
|
|
955
|
+
if (await db.Movement(db.idOf('Movement', { Wallet: wallet, opId })).first()) return same(db, wallet, opId, 'adjust', target)
|
|
956
|
+
const cur = (await db.Wallet(wallet).first())!
|
|
957
|
+
try {
|
|
958
|
+
await db.Wallet(wallet).Movement().create({ opId, kind: 'adjust', amount: target - (cur.data.balance as number), target })
|
|
959
|
+
.Wallet().update({ balance: target }, { rev: cur.rev }).rows()
|
|
960
|
+
return 'done'
|
|
961
|
+
} catch (e) {
|
|
962
|
+
if (e instanceof LetopisError && e.code === 'conflict') continue // the wallet changed: re-read
|
|
963
|
+
if (e instanceof LetopisError && e.code === 'exists') return same(db, wallet, opId, 'adjust', target)
|
|
964
|
+
throw e
|
|
965
|
+
}
|
|
966
|
+
}
|
|
967
|
+
}
|
|
968
|
+
|
|
969
|
+
const [wallet] = await db.Customer(olga).Wallet().create({ number: 'W-1' }).rows()
|
|
970
|
+
await operate(db, wallet.id, 'op-1', 'topup', 10_000) // 'done': balance 10,000
|
|
971
|
+
await operate(db, wallet.id, 'op-1', 'topup', 10_000) // 'repeat': a repeat after a lost response, the balance is the same
|
|
972
|
+
await operate(db, wallet.id, 'op-2', 'debit', 3_000) // balance 7,000
|
|
973
|
+
await adjust(db, wallet.id, 'op-3', 5_000) // adjustment: adjust by −2,000, balance 5,000
|
|
974
|
+
```
|
|
975
|
+
|
|
976
|
+
The same result can be written under a locking read, without retries on `conflict`:
|
|
977
|
+
|
|
978
|
+
<!-- run: guide -->
|
|
979
|
+
```ts
|
|
980
|
+
const [wallet2] = await db.Customer(olga).Wallet().create({ number: 'W-2' }).rows()
|
|
981
|
+
const tx = await db.begin()
|
|
982
|
+
try {
|
|
983
|
+
const w = (await tx.Wallet(wallet2).forUpdate().first())! // the wallet row is locked until commit
|
|
984
|
+
await tx.Wallet(wallet2).Movement()
|
|
985
|
+
.create({ opId: 'op-1', kind: 'adjust', amount: 6_000 - (w.data.balance as number), target: 6_000 })
|
|
986
|
+
.Wallet().update({ balance: 6_000 }).rows()
|
|
987
|
+
await tx.commit()
|
|
988
|
+
} catch (e) {
|
|
989
|
+
await tx.rollback()
|
|
990
|
+
throw e
|
|
991
|
+
}
|
|
992
|
+
```
|
|
993
|
+
|
|
994
|
+
- `inc` adds inside the statement, over the locked row: concurrent debits are not lost, and `min(0)` rejects a debit
|
|
995
|
+
beyond the balance.
|
|
996
|
+
- A plain `update({ balance: 700 })` writes the value; it does not add. A total computed from what was read, without
|
|
997
|
+
`{ rev }` or `forUpdate()`, loses concurrent debits. An advisory `lock()` does not help if not all writers take it.
|
|
998
|
+
`upsert` overwrites increments.
|
|
999
|
+
- `forUpdate()` works only inside `db.begin()` (otherwise `tx_required`) and only with `rows`, `first`, `ids`
|
|
1000
|
+
(otherwise `lock_unsupported`). Rows are locked in id order; a row that no longer matches the conditions of the step
|
|
1001
|
+
is not included in the result.
|
|
1002
|
+
- Choose one path for one wallet: under `{ rev }` and under `forUpdate()`, the wallet and the operation key are locked
|
|
1003
|
+
in a different order, and mixing the two in concurrent requests gives `40P01`. Inside a block with `forUpdate()`, do
|
|
1004
|
+
not wait for external calls: all writers wait for the wallet (`lock_timeout` and
|
|
1005
|
+
`idle_in_transaction_session_timeout` help here).
|
|
1006
|
+
- A transfer is two operations in one `db.begin()`, with the wallets edited in id order: then there are no deadlocks.
|
|
1007
|
+
- Insufficient funds are told apart from other data errors by `e.issues`: the field `balance`, the rule `minimum`.
|
|
1008
|
+
|
|
1009
|
+
## 9. Bulk loading
|
|
1010
|
+
|
|
1011
|
+
### When you need the loader
|
|
1012
|
+
|
|
1013
|
+
The loader is the path for import, seeding and syncing large reference tables. It checks rows in the library code
|
|
1014
|
+
with the same validator as the database, and writes them with the `COPY` command as the owner role, without triggers
|
|
1015
|
+
and RLS. In the benchmark, this is about 6,400 rows per second against 130 for row-by-row `create`
|
|
1016
|
+
([section 17](#17-performance)). It works only on an admin connection (`allowBypassRls` and the right to become the
|
|
1017
|
+
owner), otherwise you get `admin_required`.
|
|
1018
|
+
|
|
1019
|
+
### `db.load()`
|
|
1020
|
+
|
|
1021
|
+
<!-- run: guide -->
|
|
1022
|
+
```ts
|
|
1023
|
+
const rep = await adm.load({
|
|
1024
|
+
Shop: [{ name: 'Cornflower', city: 'Perm' }],
|
|
1025
|
+
Product: [
|
|
1026
|
+
{ sku: 'V-1', title: 'Mug', price: 450, Shop: { $key: ['Cornflower'] } },
|
|
1027
|
+
{ sku: 'V-2', title: 'Spoon', price: 150, Shop: { $key: ['Cornflower'] }, $tags: ['new'] },
|
|
1028
|
+
],
|
|
1029
|
+
Customer: [{ name: 'Petr', email: 'petr@mail.ru', $ext: 'crm:1001' }],
|
|
1030
|
+
})
|
|
1031
|
+
// rep: { rows: 4, loaded: 4, updated: 0, skipped: 0, duplicates: 0, byClass: { Shop: 1, Product: 2, Customer: 1 }, … }
|
|
1032
|
+
```
|
|
1033
|
+
|
|
1034
|
+
- The input is an object "class → rows", an array, or a stream of rows with a `$class` field.
|
|
1035
|
+
- A row has its fields and end roles at the top level. Service keys: `$class` is the class of the row, `$ext` is an
|
|
1036
|
+
external key of a class without a key (loading the same file again gives the same ids), `$tags` are the tags,
|
|
1037
|
+
`$owner` is the owner.
|
|
1038
|
+
- A reference is a target id, a key `{ $key: [...] }` or an external key `{ $ext: '…' }`; for a polymorphic end, add
|
|
1039
|
+
the `$class` of the target. The order of rows and classes does not matter, and circular references are allowed.
|
|
1040
|
+
- Keys that start with `$` inside the data are ordinary data.
|
|
1041
|
+
- Rows go to the session tenant (or to `tenant`). A load is one transaction: an error in any row cancels everything.
|
|
1042
|
+
Subscriptions see the rows after the commit.
|
|
1043
|
+
|
|
1044
|
+
### Repeats and existing objects
|
|
1045
|
+
|
|
1046
|
+
<!-- run: guide -->
|
|
1047
|
+
```ts
|
|
1048
|
+
const again = await adm.load({ Shop: [{ name: 'Cornflower', city: 'Perm' }] }, { existing: 'skip' }) // skipped: 1
|
|
1049
|
+
const changed = await adm.load({ Shop: [{ name: 'Cornflower', city: 'Samara' }] }, { existing: 'update' }) // updated: 1
|
|
1050
|
+
```
|
|
1051
|
+
|
|
1052
|
+
| Option | Values |
|
|
1053
|
+
|---|---|
|
|
1054
|
+
| `duplicates`: a repeated id within the load | `'error'` (default): cancel; `'first'` or `'last'`: keep the first or the last row |
|
|
1055
|
+
| `existing`: the object already exists in the database | `'error'` (default): cancel; `'skip'`: skip, do not resurrect deleted objects; `'update'`: a new version if the content differs (unchanged objects are counted in `skipped`), deleted objects are created again |
|
|
1056
|
+
|
|
1057
|
+
### Portions, history and verification
|
|
1058
|
+
|
|
1059
|
+
- `commitEvery: N` commits in portions of about N rows. Portions go by class in dependency order; the classes of one
|
|
1060
|
+
cycle go in one portion. An error cancels only the current portion, and a repeat with `existing: 'skip'` writes the
|
|
1061
|
+
rest.
|
|
1062
|
+
- `withHistory: true`: the rows are versions with `$rev`, `$at` and `$op`; this is how the import carries history over.
|
|
1063
|
+
- `verify: true`: at the end, check the loaded rows again with the database check.
|
|
1064
|
+
- `batch` (rows per `COPY` batch, 10,000), `class` (the class of rows without `$class`), `tenant`, `agent` (the agent of
|
|
1065
|
+
the versions).
|
|
1066
|
+
|
|
1067
|
+
### Loading on behalf of a user
|
|
1068
|
+
|
|
1069
|
+
<!-- run: guide -->
|
|
1070
|
+
```ts
|
|
1071
|
+
await adm.load({ Category: [{ name: 'Tableware' }] }, { as: ANNA, tenant: SHOP })
|
|
1072
|
+
```
|
|
1073
|
+
|
|
1074
|
+
With `as`, the user becomes the author of the versions, and the loader checks the same things as the database does on a
|
|
1075
|
+
regular write: the tenant is available to the user, write permission on each class, the permission conditions on rows,
|
|
1076
|
+
the visibility of reference targets, the owner per `ownerDefault`. Without `tenant`, the rows go to the user's own
|
|
1077
|
+
tenant. This is how a loading process can accept files sent by users.
|
|
1078
|
+
|
|
1079
|
+
### Command line
|
|
1080
|
+
|
|
1081
|
+
```sh
|
|
1082
|
+
npx letopis load shops.ndjson products.ndjson --dsn "$ADMIN" --schema v2.shop --token "$TOKEN" \
|
|
1083
|
+
[--class Product] [--duplicates first] [--existing skip] [--commit-every 100000] [--verify]
|
|
1084
|
+
# shops.ndjson: {"$class":"Shop","name":"South"}
|
|
1085
|
+
# products.ndjson: {"$class":"Product","sku":"Y-1","title":"Vase","price":800,"Shop":{"$key":["South"]}}
|
|
1086
|
+
```
|
|
1087
|
+
|
|
1088
|
+
NDJSON files have one object per line; all files form one load.
|
|
1089
|
+
|
|
1090
|
+
### Report and errors
|
|
1091
|
+
|
|
1092
|
+
Report: `rows`, `loaded`, `updated`, `skipped`, `duplicates`, `byClass`, `external` (how many targets outside the load
|
|
1093
|
+
were checked by a query at the end), `portions`, `ms`. An error is a `LetopisError` with the code of the first
|
|
1094
|
+
problem. Its `detail`: `rows` is the first 100 violations with the row number, code and details (`line`, `code`,
|
|
1095
|
+
`message`, `id`, `class`, `issues`), `total` is how many violations were found, `truncated` means the list is cut,
|
|
1096
|
+
`limit` is the list limit (100), `complete` means the checks that found violations went through all rows. Row codes:
|
|
1097
|
+
the codes of the database checks, and also `duplicate` (a repeated id), `exists` (the object already exists), `deleted`
|
|
1098
|
+
(an object with this id existed before) and `class_not_loadable` (the loader does not accept `Class` rows: classes are
|
|
1099
|
+
set by `syncModels`).
|
|
1100
|
+
|
|
1101
|
+
Violations are searched for in stages: parsing the rows (classes, ids, repeats), permissions on classes
|
|
1102
|
+
(`load({ as })`), data and ends in batches, and at the end of a portion the targets outside the load, owners and the
|
|
1103
|
+
tenant. The load is cancelled after the first stage that found violations, and `total` counts the violations of that
|
|
1104
|
+
stage. A target outside the load is one violation per target (the row is the first one that references it).
|
|
1105
|
+
"(и ещё N)" ("and N more") in the error text is the exact number of the remaining violations. If not all rows were
|
|
1106
|
+
checked, `complete` is `false`, and the text says "(и ещё не меньше N: причины)" ("at least N more: reasons") or, if
|
|
1107
|
+
one violation was found, "(возможно, не единственная: причины)" ("possibly not the only one: reasons"). The reasons:
|
|
1108
|
+
the following `commitEvery` portions were not checked; the ACL conditions of the following batches were not checked
|
|
1109
|
+
(`load({ as })`); the database check (`verify: true`) names at most 100 rows.
|
|
1110
|
+
|
|
1111
|
+
## 10. Users, sign-in and permissions
|
|
1112
|
+
|
|
1113
|
+
### Accounts, tenants and memberships
|
|
1114
|
+
|
|
1115
|
+
An account is a row of class `Account` in System: a name, categories (`User`, `Service` and your own) and an `enabled`
|
|
1116
|
+
flag. Every account is its own tenant. A `member` membership allows work in another tenant: it is a link from a user
|
|
1117
|
+
(`User`) to a tenant (`Tenant`) with roles in its data. Accounts and their categories are edited by the system
|
|
1118
|
+
administrator, a member of System with the `admin` role. There is no separate method to appoint one: a membership in
|
|
1119
|
+
System with the `admin` role is put in place by your own installation seed or by the loader on an admin connection (a
|
|
1120
|
+
`member` row with `tenant: <System id>`). New users of a tenant are easy to create with `createTenant`
|
|
1121
|
+
([section 3](#3-installation-and-connection)).
|
|
1122
|
+
|
|
1123
|
+
### Service and API keys
|
|
1124
|
+
|
|
1125
|
+
A service is an account of category `Service` with an API key. `up()` issues the installation key. The service checks
|
|
1126
|
+
passwords and codes, issues sessions and acts on behalf of users. It issues and revokes the keys of regular users
|
|
1127
|
+
itself (the `auth.credential` permission), as for Boris below. The service also issues and revokes its own API keys
|
|
1128
|
+
and "key + secret" pairs itself, with a session opened by its own key (`connect({ apiKey })`, `verifyApiKey`,
|
|
1129
|
+
`verifyKeySecret`). Its other credentials and the keys of other services, of System and of system administrators are
|
|
1130
|
+
issued and revoked only by a system administrator; a new service key is also registered by `up({ serviceKey })`.
|
|
1131
|
+
|
|
1132
|
+
There can be several keys. A session opened by a key remembers it, and revoking a key revokes only its sessions:
|
|
1133
|
+
connections by other keys keep working. So rotation goes without downtime: the service issues itself a new key
|
|
1134
|
+
(`issueApiKey({ account: <service id> })`), moves its processes to it and revokes the old one (`revokeCredential(id)`).
|
|
1135
|
+
The revocation keeps the current session; `keepCurrent: false` revokes it too. The last active key of a service
|
|
1136
|
+
cannot be revoked: `acl_denied` "последний действующий ключ сервиса не отзывается: сначала выпустите новый" ("the
|
|
1137
|
+
last active key of a service is not revoked: issue a new one first"). To disable a service, set `enabled: false` on
|
|
1138
|
+
its account.
|
|
1139
|
+
|
|
1140
|
+
<!-- run: guide -->
|
|
1141
|
+
```ts
|
|
1142
|
+
const { key, id: keyId } = await svc.auth.issueApiKey({ account: BORIS, name: 'integration' }) // the key is returned only once
|
|
1143
|
+
const viaKey = await svc.auth.verifyApiKey(key) // { account, token } | null
|
|
1144
|
+
await svc.auth.revokeCredential(keyId)
|
|
1145
|
+
```
|
|
1146
|
+
|
|
1147
|
+
A "key + secret" pair: `issueKeySecret({ account, name })` and `verifyKeySecret(key, secret)`.
|
|
1148
|
+
|
|
1149
|
+
### Passwords
|
|
1150
|
+
|
|
1151
|
+
<!-- run: guide -->
|
|
1152
|
+
```ts
|
|
1153
|
+
await svc.auth.setPassword({ account: BORIS, identifier: 'boris@lutik.ru', password: 'New-password-2026', confirmed: true })
|
|
1154
|
+
const failed = await svc.auth.verifyPassword({ identifier: 'boris@lutik.ru', password: 'Buttercup-2026!' }) // null: the password was changed
|
|
1155
|
+
```
|
|
1156
|
+
|
|
1157
|
+
- A password is stored as a scrypt hash; the service checks it, and the response time for a nonexistent login is
|
|
1158
|
+
equalized.
|
|
1159
|
+
- The identifier is compared case-insensitively: `Anna@Shop.ru` and `anna@shop.ru` are one login.
|
|
1160
|
+
- A password change revokes the other sessions of the account (`keepCurrent` keeps the current one).
|
|
1161
|
+
- By default, sign-in works only by a confirmed credential, otherwise `null`; `requireConfirmed: false` also lets in by
|
|
1162
|
+
an unconfirmed one. The service confirms a credential: `confirmed: true` in `setPassword` or `setFlags`.
|
|
1163
|
+
- The second factor is the `totp` option: `verifyPassword({ identifier, password, totp })` checks the account's TOTP
|
|
1164
|
+
code after a correct password and issues a session with the methods `[kind, 'TOTP']`, `['PASSWORD', 'TOTP']` by
|
|
1165
|
+
default (see below). Without `totp`, sign-in does not check the code: the service decides whether to require the
|
|
1166
|
+
second factor.
|
|
1167
|
+
|
|
1168
|
+
### One-time codes and TOTP
|
|
1169
|
+
|
|
1170
|
+
<!-- run: guide -->
|
|
1171
|
+
```ts
|
|
1172
|
+
import { totpCode } from 'letopis'
|
|
1173
|
+
|
|
1174
|
+
const { code } = await svc.auth.issueOtp({ account: BORIS, identifier: 'boris@lutik.ru', ttl: 300 }) // the service sends the code by email
|
|
1175
|
+
const byCode = await svc.auth.verifyOtp({ identifier: 'boris@lutik.ru', code }) // { account, token } | null
|
|
1176
|
+
const { secret, uri } = await svc.auth.enrollTotp({ account: BORIS, issuer: 'Buttercup' }) // uri is for a QR code
|
|
1177
|
+
const totpOk = await svc.auth.verifyTotp({ account: BORIS, code: totpCode(secret) }) // the first check enables the factor
|
|
1178
|
+
```
|
|
1179
|
+
|
|
1180
|
+
A code is single-use: a successful check and brute force (5 attempts by default) burn it. The attempt counters are
|
|
1181
|
+
rolled back together with the caller's transaction, so the service limits the rate of attempts. A repeat of the same
|
|
1182
|
+
TOTP step is rejected. `totpCode(secret)` is a helper for tests and services.
|
|
1183
|
+
|
|
1184
|
+
Sign-in with the second factor is `verifyPassword({ …, totp })` and `lookup({ …, totp })`. The code is checked only
|
|
1185
|
+
after a correct password or a found identity, by the same check as `verifyTotp`: the `auth.totp` permission is needed,
|
|
1186
|
+
the window is one step each way, and the failure counter is shared (five in a row close the check for 5 minutes). A
|
|
1187
|
+
correct code gives a session with the methods `[kind, 'TOTP']` and enables a not yet confirmed factor; a wrong or
|
|
1188
|
+
repeated code, like a code for an account without TOTP, gives `null`. With such a session, a user with TOTP changes
|
|
1189
|
+
their own credentials for 10 minutes.
|
|
1190
|
+
|
|
1191
|
+
### External identities
|
|
1192
|
+
|
|
1193
|
+
`link({ account, kind, identifier, confirmed })` links an external identity (Telegram, Google, SSO): the service checks
|
|
1194
|
+
its token, and the database stores the binding. `lookup({ kind, identifier })` signs in with it (with the `totp`
|
|
1195
|
+
option, also by a TOTP code, like `verifyPassword`):
|
|
1196
|
+
|
|
1197
|
+
<!-- run: guide -->
|
|
1198
|
+
```ts
|
|
1199
|
+
await svc.auth.link({ account: BORIS, kind: 'TELEGRAM', identifier: '100500', confirmed: true })
|
|
1200
|
+
const viaTelegram = await svc.auth.lookup({ kind: 'TELEGRAM', identifier: '100500' }) // { account, token } | null
|
|
1201
|
+
```
|
|
1202
|
+
|
|
1203
|
+
A user can link their own credentials only after a recent sign-in (10 minutes; an API key, a "key + secret" pair and a
|
|
1204
|
+
session from a service do not count), and cannot confirm them. With TOTP enabled, the database requires a session that
|
|
1205
|
+
has passed TOTP: `verifyPassword` and `lookup` with the `totp` option issue one. If the identifier is taken, linking
|
|
1206
|
+
answers with the same `acl_denied` as without a recent sign-in. But a user with a recent sign-in can tell from such a
|
|
1207
|
+
denial that the identifier is taken ([section 18](#18-limitations)).
|
|
1208
|
+
|
|
1209
|
+
### Sessions
|
|
1210
|
+
|
|
1211
|
+
<!-- run: guide -->
|
|
1212
|
+
```ts
|
|
1213
|
+
const shortToken = await svc.auth.sessionFor(BORIS, { ttl: 600 }) // a session for 10 minutes of idle time for another process
|
|
1214
|
+
const boris = await connect({ dsn: APP, schema: r.schema, token: shortToken })
|
|
1215
|
+
await boris.auth.revoke() // sign-out: the session is revoked
|
|
1216
|
+
await boris.close()
|
|
1217
|
+
```
|
|
1218
|
+
|
|
1219
|
+
A token is 256 random bits; the database stores only its sha256. The default lifetime is 7 days. `refresh()` extends
|
|
1220
|
+
the current session: the new lifetime is counted from the current moment, but it is not longer than the lifetime issued
|
|
1221
|
+
at sign-in; it returns the new expiration time. `connect()` extends the session itself once per minute of work
|
|
1222
|
+
(`refreshSession`), so a session in use does not expire: the lifetime is the idle time. `revoke()` signs out,
|
|
1223
|
+
`revokeAll()` signs out everywhere. Sessions are revoked when the account is disabled, the password is changed, a credential is deleted or the
|
|
1224
|
+
account is deleted. Deleting a credential revokes the other sessions of the account, while revoking an API key or a
|
|
1225
|
+
"key + secret" pair revokes only the sessions opened by that key (and old sessions of the same sign-in method without
|
|
1226
|
+
a key mark); the current session stays unless you pass `keepCurrent: false`.
|
|
1227
|
+
|
|
1228
|
+
### Working in another tenant
|
|
1229
|
+
|
|
1230
|
+
A service acts on behalf of a user through `svc.as(account, { tenant, owner })`. The database checks the service's
|
|
1231
|
+
`auth.impersonate` permission (by System rules), that the user exists and is enabled, and that the tenant is available
|
|
1232
|
+
to the user, either their own or through membership. You cannot act on behalf of System, services, system
|
|
1233
|
+
administrators or disabled accounts. The user's visibility under the access rules is not checked. In the journal,
|
|
1234
|
+
`author` is the user and `agent` is the service. A user's connection by their own token switches between tenants like
|
|
1235
|
+
this:
|
|
1236
|
+
|
|
1237
|
+
<!-- run: guide -->
|
|
1238
|
+
```ts
|
|
1239
|
+
const mine = await anna.auth.tenants() // [{ tenant, roles }]: her own tenant and the shop via membership
|
|
1240
|
+
await anna.auth.switch(SHOP) // the session tenant is now the shop
|
|
1241
|
+
```
|
|
1242
|
+
|
|
1243
|
+
### Permissions
|
|
1244
|
+
|
|
1245
|
+
Permissions are always on and deny by default. A rule connects a group with an object:
|
|
1246
|
+
|
|
1247
|
+
- **group**: a `Resource` row of category `ACCOUNT`: accounts by category (`{ categories: '{User}' }`) or members by
|
|
1248
|
+
membership role (`{ roles: '{staff}' }`);
|
|
1249
|
+
- **object**: a `Resource` row of category `API` (an endpoint by mask, `{ endpoint: 'shop.reports.*' }`), or of
|
|
1250
|
+
category `READ`, `WRITE`, `DELETE` (the rows of a class by mask, with descendants, `{ class: 'Product' }`, and a
|
|
1251
|
+
condition on the columns `id`, `owner`, `tenant`, `tags`, `data`, `links` with the values `$actor` and `$tenant`);
|
|
1252
|
+
- **rule**: a `rule` link group → object: `permission` (`allow` or `deny`) and `weight`.
|
|
1253
|
+
|
|
1254
|
+
Exactly one rule decides: the highest weight; at equal weight, deny beats allow. A tenant rule weight above 999 is
|
|
1255
|
+
cut down to 999, so System rules with a weight of 1000 or more are an administrative layer that a tenant cannot
|
|
1256
|
+
outweigh. Tenant rules apply to the tenant's rows, System rules apply everywhere. Permissions apply to raw SQL too: rows
|
|
1257
|
+
without permission are invisible, and an edit changes nothing or fails with `acl_denied`.
|
|
1258
|
+
|
|
1259
|
+
### Granting a permission
|
|
1260
|
+
|
|
1261
|
+
Example: Oleg is a shop employee with the `staff` role, and employees may not delete products.
|
|
1262
|
+
|
|
1263
|
+
<!-- run: guide -->
|
|
1264
|
+
```ts
|
|
1265
|
+
const pg2 = postgres(ADMIN)
|
|
1266
|
+
const OLEG = (await createTenant(pg2, r.schema, { // the same shop: only the user is created
|
|
1267
|
+
name: 'Chamomile', user: { name: 'Oleg', login: 'oleg@shop.ru', password: 'Oleg-2026!' } })).user!
|
|
1268
|
+
await pg2.end()
|
|
1269
|
+
await db.member(db.idOf('member', { User: OLEG, Tenant: SHOP })).update({ roles: ['staff'] }).rows()
|
|
1270
|
+
|
|
1271
|
+
const [staff] = await db.Resource().create({ alias: 'staff:ACCOUNT', category: 'ACCOUNT', pattern: { roles: '{staff}' } }).rows()
|
|
1272
|
+
const [delProduct] = await db.Resource().create({ alias: 'product:DELETE', category: 'DELETE', pattern: { class: 'Product' } }).rows()
|
|
1273
|
+
await db.rule().create({ permission: 'deny', weight: 50 }).Group.set(staff).Object.set(delProduct).rows() // stronger than "may do everything" (10)
|
|
1274
|
+
const [reports] = await db.Resource().create({ alias: 'reports:API', category: 'API', pattern: { endpoint: 'shop.reports.*' } }).rows()
|
|
1275
|
+
await db.rule().create({ permission: 'allow', weight: 50 }).Group.set(staff).Object.set(reports).rows()
|
|
1276
|
+
|
|
1277
|
+
const asOleg = await svc.as(OLEG, { tenant: SHOP })
|
|
1278
|
+
const canDelete = await asOleg.acl.checkData('Product', 'DELETE') // { allow: false }
|
|
1279
|
+
const canRead = await asOleg.acl.checkData('Product', 'READ') // { allow: true }
|
|
1280
|
+
const canReport = await asOleg.acl.check('shop.reports.daily') // { allow: true, rule: { weight: 50, … } }
|
|
1281
|
+
```
|
|
1282
|
+
|
|
1283
|
+
- `acl.check(endpoint)` → `{ allow, rule, code, message }`: the `code` and `message` fields of the denying rule set the
|
|
1284
|
+
endpoint's own denial.
|
|
1285
|
+
- `acl.checkData(class, 'READ' | 'WRITE' | 'DELETE')` → `{ allow, filter }`: `filter` is the condition on rows.
|
|
1286
|
+
- The database recomputes permissions by itself. To remove a permission, delete the rule:
|
|
1287
|
+
`db.rule(id).delete({ confirm: true })`.
|
|
1288
|
+
- The library takes the actor's signed permission plan — the permission pass — from the database once and passes it
|
|
1289
|
+
in every transaction, so a permission check costs tens of microseconds. Raw SQL without the pass pays 0.3–1.3 ms per
|
|
1290
|
+
query.
|
|
1291
|
+
- The question "who had access at a given date" is answered by an `asOf` read over the classes `rule`, `member` and
|
|
1292
|
+
`Resource`.
|
|
1293
|
+
|
|
1294
|
+
Fields of the system permission classes:
|
|
1295
|
+
|
|
1296
|
+
- `member`: the ends `User` and `Tenant`, the field `roles` (a list of roles);
|
|
1297
|
+
- `Resource`: `alias` (the unique name of the resource and its key; this is not a class alias), `category` (`ACCOUNT`,
|
|
1298
|
+
`API`, `READ`, `WRITE`, `DELETE`) and `pattern`, the mask. A group's mask is `categories` or `roles`, and only a
|
|
1299
|
+
string: `'{staff}'`, `'{a,b}'`, negation `'!{a}'`. An object's mask is `endpoint` or `class` (also strings only) plus
|
|
1300
|
+
conditions on the columns. **The database rejects a non-string mask (for example, the array `['staff']`, a number or
|
|
1301
|
+
`null`)** with `invalid_data`, whether written by the library, raw SQL or the loader. Old rows with such a mask
|
|
1302
|
+
(written before this check or around it) grant no permissions, while a denial by them applies to everyone: by a
|
|
1303
|
+
group, to all accounts, by a `class` mask, to all classes (a non-string `endpoint` mask, as before, matches no
|
|
1304
|
+
endpoint). The write check comes with the description of the `Resource` class: it arrives with a fresh installation
|
|
1305
|
+
and after `reset({ all })`; in an existing schema, the corrected matching works after `up()` is run again;
|
|
1306
|
+
- `rule`: the ends `Group` and `Object` (`Resource` rows), the fields `permission` (`allow` or `deny`), `weight`, and,
|
|
1307
|
+
for the endpoint's own denial, `code` and `message`.
|
|
1308
|
+
|
|
1309
|
+
### Deleting an account
|
|
1310
|
+
|
|
1311
|
+
`db.auth.purgeAccount(account)` deletes an account and the data of its tenant. The system administrator or the owner of
|
|
1312
|
+
its tenant can call it; you cannot delete yourself or the last owner of a tenant. Ownership of rows in other tenants
|
|
1313
|
+
passes to those tenants, sessions are revoked, credentials are deleted.
|
|
1314
|
+
|
|
1315
|
+
## 11. History and point-in-time reads
|
|
1316
|
+
|
|
1317
|
+
### Versions
|
|
1318
|
+
|
|
1319
|
+
<!-- run: guide -->
|
|
1320
|
+
```ts
|
|
1321
|
+
const versions = await db.Product(kettle).versions() // all versions: op, rev, author, at, data
|
|
1322
|
+
const followed = await db.Product(mug2).versions({ follow: true }) // also across id changes (rekey, reclass)
|
|
1323
|
+
```
|
|
1324
|
+
|
|
1325
|
+
Every version has an author, an agent, a time, a number, a kind of operation (`create`, `update`, `delete`, `reclass`,
|
|
1326
|
+
`rekey`, `restore`, `purge`, `trim`, `reset`) and a reason (`reason`), if the `letopis.reason` setting of the
|
|
1327
|
+
transaction set one; there is no separate API for the reason. A deletion version has `$deleted`.
|
|
1328
|
+
|
|
1329
|
+
### Point-in-time read
|
|
1330
|
+
|
|
1331
|
+
<!-- run: guide -->
|
|
1332
|
+
```ts
|
|
1333
|
+
const createdAt = versions[0].at
|
|
1334
|
+
const atCreation = await db.Product(kettle).asOf(createdAt).first() // the price at creation time: 2500
|
|
1335
|
+
const assortment = await db.Shop(shop).Product().asOf(createdAt).rows()
|
|
1336
|
+
```
|
|
1337
|
+
|
|
1338
|
+
`asOf(moment)` is the state "as it was", from the journal; the moment is an ISO string or a `Date`. A point-in-time
|
|
1339
|
+
read works at any step of a chain, including with `deep()`. Only the current state is cheap: a point-in-time read is
|
|
1340
|
+
rebuilt from the journal. The version time `at` stores microseconds, while a `Date` holds only milliseconds: pass the
|
|
1341
|
+
`at` string to `asOf` as is, otherwise a read at the moment of a version may miss it.
|
|
1342
|
+
|
|
1343
|
+
### Deleted rows
|
|
1344
|
+
|
|
1345
|
+
<!-- run: guide -->
|
|
1346
|
+
```ts
|
|
1347
|
+
const withGone = await db.Customer().withDeleted().rows() // live and deleted; deleted rows have $deleted: true
|
|
1348
|
+
```
|
|
1349
|
+
|
|
1350
|
+
### Retention policy
|
|
1351
|
+
|
|
1352
|
+
Without a policy, the journal keeps every version. A policy sets how long to keep old versions and with what
|
|
1353
|
+
granularity:
|
|
1354
|
+
|
|
1355
|
+
```ts
|
|
1356
|
+
history: { all: '1 day', daily: '1 week', weekly: '1 month', monthly: '1 year', yearly: 'forever', tz: 'Europe/Moscow' }
|
|
1357
|
+
```
|
|
1358
|
+
|
|
1359
|
+
This means: all versions for the last day; for the last week, one per day; for the last month, one per week; for the
|
|
1360
|
+
last year, one per month; older, one per year, forever. Each period keeps its last version. A week starts on Monday;
|
|
1361
|
+
period boundaries are in the `tz` time zone (UTC by default).
|
|
1362
|
+
|
|
1363
|
+
- A policy is set on a class (descendants inherit it) or for all classes without their own policy: `up({ history })`.
|
|
1364
|
+
- If the last tier has a term instead of `'forever'`, the history older than the term is deleted: `{ all: '30 days' }`.
|
|
1365
|
+
- While the object's history is alive, the current version, version 1 and versions of special kinds (`create`,
|
|
1366
|
+
`delete`, `reclass`, `rekey`, `restore`, `purge`, `reset`, `trim`) are not deleted. If the tombstone of a deleted
|
|
1367
|
+
object is older than the last tier with a term, its history is erased entirely. The history of system classes is not
|
|
1368
|
+
thinned.
|
|
1369
|
+
- `maintain()` does the thinning in batches of 10,000 versions ([section 13](#13-operations)).
|
|
1370
|
+
- In the thinned part, a point-in-time read shows the last kept version before the moment, so its precision equals the
|
|
1371
|
+
period of the tier. A subscriber that lags behind by more than the `all` tier gets `cursor_expired`.
|
|
1372
|
+
|
|
1373
|
+
### Trimming history
|
|
1374
|
+
|
|
1375
|
+
<!-- run: guide -->
|
|
1376
|
+
```ts
|
|
1377
|
+
const trimmed = await db.trimHistory(kettle, new Date()) // erase the versions of the kettle before this moment
|
|
1378
|
+
```
|
|
1379
|
+
|
|
1380
|
+
`db.trimHistory(target, moment)` erases the history before the moment right away. The target is a class with its
|
|
1381
|
+
descendants (by name), an id, a list of ids or rows. This is the middle step of the "right to be forgotten":
|
|
1382
|
+
`anonymize` → history trimming → deletion and `purge`. In place of version 1, a `trim` record stays with its time, so a
|
|
1383
|
+
point-in-time read knows when the object appeared.
|
|
1384
|
+
|
|
1385
|
+
### Integrity check
|
|
1386
|
+
|
|
1387
|
+
<!-- run: guide -->
|
|
1388
|
+
```ts
|
|
1389
|
+
const audit = await adm.verify({ data: true }) // { ok, issues, anchor }
|
|
1390
|
+
```
|
|
1391
|
+
|
|
1392
|
+
`verify()` recomputes the hash chains of all versions and returns an anchor: a hash of all chain heads. Keep the anchor
|
|
1393
|
+
outside the database: a mismatch with it exposes tampering with the latest versions. `{ data: true }` also checks again
|
|
1394
|
+
the data, ends, tenants, owners and ids of all rows. The system administrator or an admin connection can call it; from
|
|
1395
|
+
the command line, use `letopis verify [--data]` (exit code 1 if something is found). Legitimate gaps (`trim` and
|
|
1396
|
+
`purge` records, thinning and `reset`) are not counted as errors.
|
|
1397
|
+
|
|
1398
|
+
## 12. Change subscription and cache
|
|
1399
|
+
|
|
1400
|
+
### Subscription
|
|
1401
|
+
|
|
1402
|
+
<!-- run: guide -->
|
|
1403
|
+
```ts
|
|
1404
|
+
const sub = db.watch({ from: 'start', classes: ['Product'] }) // the whole journal of the class, then new events
|
|
1405
|
+
await db.Product(kettle).update({ price: 2100 }).rows()
|
|
1406
|
+
let cursor = ''
|
|
1407
|
+
for await (const ev of sub) { // ev: { cursor, row, id, class, op, rev, at }
|
|
1408
|
+
cursor = ev.cursor // save the cursor: watch({ from: cursor }) continues after it
|
|
1409
|
+
if (ev.id === kettle.id && ev.row.data.price === 2100) break // break (or sub.close()) stops the subscription
|
|
1410
|
+
}
|
|
1411
|
+
```
|
|
1412
|
+
|
|
1413
|
+
- `from` is `'now'` (the default: only changes after subscribing), `'start'` (the whole journal) or a saved cursor.
|
|
1414
|
+
`db.watch()` returns at once, and the subscription takes the "now" moment with its first query to the database: a
|
|
1415
|
+
write made right after `db.watch()` may commit earlier and miss the stream. That is why the example above reads from
|
|
1416
|
+
`'start'`.
|
|
1417
|
+
- `classes`: only these classes with their descendants; `pollMs`: fallback polling (1000 ms).
|
|
1418
|
+
- Events come without losses or repeats, under the reader's permissions, in the order of transaction numbers, and
|
|
1419
|
+
within a transaction in the order of writing. This is not the commit order: compare versions of one object by `rev`.
|
|
1420
|
+
- A long write transaction in any database of the cluster delays the stream: `await sub.lag()` shows the lag in ms.
|
|
1421
|
+
- A cursor older than the `all` tier of the retention policy gives `cursor_expired`: read the current state again and
|
|
1422
|
+
subscribe with `'now'`.
|
|
1423
|
+
- Without `LISTEN` (`connect({ listen: false })`), the subscription polls the journal.
|
|
1424
|
+
- A subscription without losses is a saved cursor: store `ev.cursor` of the last processed event and resume from it
|
|
1425
|
+
after a restart.
|
|
1426
|
+
|
|
1427
|
+
### Result cache
|
|
1428
|
+
|
|
1429
|
+
<!-- run: guide -->
|
|
1430
|
+
```ts
|
|
1431
|
+
const cached = await db.Product().cache({ ttl: 30 }).rows() // the result is kept in process memory for 30 s
|
|
1432
|
+
const stats = db.cache.stats() // { size, hits, misses, stores, raced, … }
|
|
1433
|
+
```
|
|
1434
|
+
|
|
1435
|
+
The cache is enabled only explicitly: `.cache()` in a chain or `cache: { ttl }` on a class in the model. This is how it
|
|
1436
|
+
stays correct:
|
|
1437
|
+
|
|
1438
|
+
- the result key includes the query, the parameters, the session, the actor and the tenant, so one user never gets the
|
|
1439
|
+
result of another user;
|
|
1440
|
+
- your own writes clear the results of the changed classes right after commit (including the cascade classes); other
|
|
1441
|
+
writers' changes clear them on a database signal;
|
|
1442
|
+
- changes of permissions and classes, writes through `db.sql`, a revoked session and a listener reconnect clear the
|
|
1443
|
+
whole cache or the results of the session;
|
|
1444
|
+
- the default term is 60 seconds; the cache does not work inside `db.begin()`;
|
|
1445
|
+
- without `LISTEN` the cache is off, and `connect({ cache: { ttlOnly: true } })` enables it by term only.
|
|
1446
|
+
|
|
1447
|
+
Between processes, the cache goes stale for the signal sending interval (`notifyIntervalMs`, 100 ms) plus delivery;
|
|
1448
|
+
`db.flushSignals()` sends the accumulated signals at once. `db.cache.clear()` clears the cache.
|
|
1449
|
+
|
|
1450
|
+
## 13. Operations
|
|
1451
|
+
|
|
1452
|
+
### Maintenance: `maintain()`
|
|
1453
|
+
|
|
1454
|
+
<!-- run: guide -->
|
|
1455
|
+
```ts
|
|
1456
|
+
const report = await adm.maintain() // { sessions, codes, thinned, batches, more, queue, skipped }
|
|
1457
|
+
```
|
|
1458
|
+
|
|
1459
|
+
`maintain()` deletes expired sessions and one-time codes, thins the journal by the retention policy and checks the
|
|
1460
|
+
PostgreSQL signal queue: if it is more than half full, you get a warning. The system administrator, a role with the
|
|
1461
|
+
`letopis.maintain` permission or an admin connection can call it. An external scheduler runs it (`letopis maintain`),
|
|
1462
|
+
or the library itself does: `connect({ maintain: '1h' })`. Of several processes, only one works: each batch is its own
|
|
1463
|
+
transaction, and the first thing it does is take a transaction-level advisory lock (`pg_try_advisory_xact_lock`),
|
|
1464
|
+
which is released when the batch commits. If another process holds the lock, the call answers `skipped: true` before
|
|
1465
|
+
the first batch, and between batches it stops (that process continues) and reports what it has done with
|
|
1466
|
+
`more: true`. There is no session-level lock, so `maintain()` works behind any pool, including PgBouncer in
|
|
1467
|
+
transaction mode.
|
|
1468
|
+
|
|
1469
|
+
### Reset: `reset()`
|
|
1470
|
+
|
|
1471
|
+
```ts
|
|
1472
|
+
await adm.reset({ level: 'tenant', confirm: 'v2.shop', tenant: SHOP }) // sessions | tenant | all
|
|
1473
|
+
```
|
|
1474
|
+
|
|
1475
|
+
Reset is meant for tests and test environments. It works only if the installation was set up with
|
|
1476
|
+
`up({ allowReset: true })` (otherwise `reset_disabled`, even for System), the caller has the `letopis.reset`
|
|
1477
|
+
permission, and `confirm` matches the schema name (`confirm_mismatch`). Levels: `sessions` resets all sessions
|
|
1478
|
+
except the current one; `tenant` the tenant's data; `all` everything, with the system seed laid down again and a new
|
|
1479
|
+
service key. The first journal record after a reset is `reset`.
|
|
1480
|
+
|
|
1481
|
+
### Model description for people and AI
|
|
1482
|
+
|
|
1483
|
+
<!-- run: guide -->
|
|
1484
|
+
```ts
|
|
1485
|
+
import { describeText } from 'letopis'
|
|
1486
|
+
|
|
1487
|
+
const model = db.describe() // { schema, tenant, classes: [...], methods }
|
|
1488
|
+
const modelText = describeText(model) // the same as text: for the application's llms.txt or a prompt for an AI agent
|
|
1489
|
+
```
|
|
1490
|
+
|
|
1491
|
+
`db.describe()` returns a description of the classes available to the session: fields, ends, keys, policies and
|
|
1492
|
+
metadata. From the command line: `letopis describe --format text --out llms.txt`.
|
|
1493
|
+
|
|
1494
|
+
### Raw SQL
|
|
1495
|
+
|
|
1496
|
+
<!-- run: guide -->
|
|
1497
|
+
```ts
|
|
1498
|
+
const [{ n }] = await db.sql`select count(*)::int as n from "v2.shop".entity where class = 'Product'`
|
|
1499
|
+
```
|
|
1500
|
+
|
|
1501
|
+
`` db.sql`…` `` runs a query in one transaction under the session of the connection and under RLS: it sees the same
|
|
1502
|
+
rows as chains do. `db.transaction(fn)` gives a postgres.js connection for the whole transaction. On a deadlock or a
|
|
1503
|
+
serialization failure, the query and `fn` are retried up to three times, so `fn` must withstand a retry. You can also
|
|
1504
|
+
write with raw SQL: the same triggers check the data, and the database functions `merge` and `inc` are available to the
|
|
1505
|
+
application.
|
|
1506
|
+
|
|
1507
|
+
### Command line
|
|
1508
|
+
|
|
1509
|
+
```sh
|
|
1510
|
+
letopis sync <module> --dsn <dsn> --schema <v2.name> (--token <token> | --api-key <key>) [--apply]
|
|
1511
|
+
letopis types --dsn <dsn> --schema <v2.name> (--token <token> | --api-key <key>) [--out <file>]
|
|
1512
|
+
letopis load <files…> --dsn <dsn> --schema <v2.name> (--token <token> | --api-key <key>) [--class <Class>]
|
|
1513
|
+
[--duplicates error|first|last] [--existing error|skip|update] [--commit-every <N>] [--verify]
|
|
1514
|
+
letopis verify --dsn <dsn> --schema <v2.name> [--token <token> | --api-key <key>] [--data]
|
|
1515
|
+
letopis maintain --dsn <dsn> --schema <v2.name> [--token <token> | --api-key <key>] [--batches <N>]
|
|
1516
|
+
letopis describe --dsn <dsn> --schema <v2.name> (--token <token> | --api-key <key>) [--format json|text] [--out <file>]
|
|
1517
|
+
letopis import --from <0.21 dsn> --from-schema <v1.name> --dsn <dsn> --schema <v2.name> [--tenant <name>] [--keyless <Class>]…
|
|
1518
|
+
[--rename <Old>=<New>]… [--update] [--commit-every <N>] [--partition <name>]
|
|
1519
|
+
```
|
|
1520
|
+
|
|
1521
|
+
### Monitoring
|
|
1522
|
+
|
|
1523
|
+
- `connect({ onQuery, slowMs })`: for every chain query, the hook gets the mode, the classes, the duration, the row
|
|
1524
|
+
count and a slow query flag; without a hook, a slow query prints a warning.
|
|
1525
|
+
- `sub.lag()` is the lag of a subscription; the `maintain()` report shows how full the signal queue is.
|
|
1526
|
+
- Hot reference targets (an organization that almost everything references) accumulate `for key share` locks and
|
|
1527
|
+
MultiXact, as with regular foreign keys. This is worth monitoring.
|
|
1528
|
+
|
|
1529
|
+
### Connection pools and PgBouncer
|
|
1530
|
+
|
|
1531
|
+
With PgBouncer in transaction mode, `LISTEN` is not available: use `connect({ listen: false })`. Then the registry is
|
|
1532
|
+
re-read with `db.refresh()`, the subscription polls the journal, and the result cache is off or works by term only.
|
|
1533
|
+
Prepared statements work with PgBouncer 1.21 and newer; with older versions, pass your own postgres.js pool with
|
|
1534
|
+
`prepare: false` in `connect({ sql })`. `maintain()` takes a transaction-level lock in each batch and works behind
|
|
1535
|
+
such a pool.
|
|
1536
|
+
|
|
1537
|
+
<!-- run: guide -->
|
|
1538
|
+
```ts
|
|
1539
|
+
await adm.close()
|
|
1540
|
+
await anna.close()
|
|
1541
|
+
await svc.close() // and the connections obtained through svc.as()
|
|
1542
|
+
```
|
|
1543
|
+
|
|
1544
|
+
## 14. Migrating from 0.21
|
|
1545
|
+
|
|
1546
|
+
1.0 is a new implementation, not a continuation of 0.21: a 0.21 schema is not upgraded in place. 1.0 is installed into
|
|
1547
|
+
a new schema, the `letopis import` command moves the data, and the code is ported with the replacement table in
|
|
1548
|
+
[MIGRATION.md](MIGRATION.md) (in Russian). The main differences in behavior: `create` does not overwrite (a taken key
|
|
1549
|
+
gives `exists`), `count()` counts entities, paths are `paths()`, permissions and tenant isolation are always on.
|
|
1550
|
+
|
|
1551
|
+
```sh
|
|
1552
|
+
npx letopis import --from postgres://…/old --from-schema v1.salon --dsn postgres://…/new --schema v2.salon \
|
|
1553
|
+
--tenant salon [--keyless booking] [--rename link=link_] [--update] [--commit-every 100000]
|
|
1554
|
+
```
|
|
1555
|
+
|
|
1556
|
+
- The 1.0 schema is installed in advance (`up()`); both connections are admin connections. The source is the 0.21
|
|
1557
|
+
tables (`Schema`, `Entity`, `Account`, `Credential`, `Resource`, `Rule`) in any database: PostgreSQL 17 with
|
|
1558
|
+
TimescaleDB or a dump of plain tables. The loader (history mode, `existing: 'skip'`, agent `letopis import`) and the
|
|
1559
|
+
owner functions do the writing.
|
|
1560
|
+
- Classes become `Class` rows in System. fastest-validator rules are translated to JSON Schema (`optional` lets `null`
|
|
1561
|
+
through, `$$strict` → `additionalProperties: false`, `record` → `propertyNames`). Ends become roles named after the
|
|
1562
|
+
target class; a union of all descendants of a class becomes a polymorphic end. The `Entity` and `link` roots are
|
|
1563
|
+
dropped; `description`, `appearance` and `order` move to `meta`. For names taken by 1.0 methods, use `--rename` (by
|
|
1564
|
+
default, the name with `_`).
|
|
1565
|
+
- Ids: for a class with a key (the v5 rule from 0.21), v5 of the key; for a class without a key, v5 of the old id.
|
|
1566
|
+
System data moves to the `--tenant` tenant, accounts to `Account` rows, credentials to `secret` as they are (scrypt,
|
|
1567
|
+
keys); one-time codes are not carried over. Resources and rules go to System.
|
|
1568
|
+
- The import stops before the first write if two 0.21 objects give one 1.0 id (the data violates the key; use
|
|
1569
|
+
`--keyless <Class>`), if a key changed between versions, or if it references a missing object. A dangling reference
|
|
1570
|
+
in a live row gives `target_not_found` with the 0.21 object.
|
|
1571
|
+
- If the loader rejects the data, the error starts with `import: арендатор …`, and its `detail` is the loader's
|
|
1572
|
+
([Report and errors](#report-and-errors)), except that rows are named by 0.21 objects: `rows` are the violations
|
|
1573
|
+
with the `class`, `id` and `rev` of the 0.21 object, `code` and `message`; `total`, `truncated`, `limit` and
|
|
1574
|
+
`complete` are as in the loader's report; `notes` are the untranslated rules.
|
|
1575
|
+
- A repeat adds what is missing without duplicates; `--update` also adds new versions of changed objects. The report is
|
|
1576
|
+
JSON at the end of the output: objects and versions by class in 0.21 and in 1.0, `verify()`, untranslated rules,
|
|
1577
|
+
classes without a key, and uuids in `data` that match old ids (the import does not rewrite them). The exit code is 1
|
|
1578
|
+
if `verify()` found discrepancies.
|
|
1579
|
+
|
|
1580
|
+
## 15. API reference
|
|
1581
|
+
|
|
1582
|
+
The full machine-readable list of exports, methods and error texts is `docs/api-contract.json` in the repository.
|
|
1583
|
+
|
|
1584
|
+
### Package exports
|
|
1585
|
+
|
|
1586
|
+
| Entry point | What it exports |
|
|
1587
|
+
|---|---|
|
|
1588
|
+
| `letopis` | `connect`, `up`, `createTenant`; operators `ne`, `gt`, `gte`, `lt`, `lte`, `between`, `inList`, `like`, `ilike`, `starts`, `ends`, `has`, `hasAny`, `hasAll`, `exists`, `isNull`, `not`, `or`, `isOp`; `inc`, `isInc`, `cursorOf`; `idOf`, `uuidv5`, `v5Name`, `v5Id`; `LetopisError`, `ValidationError`, `fromDbError`; `syncModels`, `generateTypes`, `schemaToTs`; `compileValidator`, `applyDefaults`, `Registry`; `totpCode`, `hashPassword`, `verifyHash`; `describeText`; constants `DDL_REVISION`, `RESERVED_CLASS_NAMES`, `CHAIN_METHODS`, `DB_METHODS`; types `Db`, `DbTx`, `TypedDb`, `Chain`, `Row`, `Filter`, `Cursor`, `ConnectOptions`, `UpOptions`, `LoadOptions`, `LoadReport`, `WatchOptions`, `WatchEvent`, `AuthApi`, `AclApi` and others |
|
|
1589
|
+
| `letopis/model` | `hub`, `link`, `one`, `many`, `End`, types `Infer`, `InferLinks`, `Model`, `ClassDef`, `ClassMeta`, `HistoryPolicy`; `normalizeSchema`, `translatePattern`, type `JsonSchema` |
|
|
1590
|
+
| CLI `letopis` | `sync`, `types`, `load`, `verify`, `maintain`, `describe`, `import` ([section 13](#13-operations)) |
|
|
1591
|
+
|
|
1592
|
+
Removed 0.21 names (`uuidv7`, `LETOPIS_NS` and the methods listed below) throw `removed` with the replacement.
|
|
1593
|
+
|
|
1594
|
+
### The `db` root
|
|
1595
|
+
|
|
1596
|
+
| Member | What it does |
|
|
1597
|
+
|---|---|
|
|
1598
|
+
| `db.ClassName(filter)` | the start of a chain |
|
|
1599
|
+
| `schema`, `token`, `tenant`, `registry` | the installation name, the token, the session tenant, the class registry (`resolve`, `find`, `has`, `all`, `family`) |
|
|
1600
|
+
| `auth`, `acl` | sign-in, credentials and sessions; permission decisions ([section 10](#10-users-sign-in-and-permissions)) |
|
|
1601
|
+
| `as(account, { tenant, owner })` | the same pool on behalf of an account (async) |
|
|
1602
|
+
| `begin()`, `commit(tr)`, `rollback(tr)`, `lock(…keys)` | an explicit transaction ([section 8](#8-transactions-batches-and-money)) |
|
|
1603
|
+
| `batch(name)` | a queue of plans: `run()`, `discard()`, `size()` |
|
|
1604
|
+
| `load(input, options)` | the loader ([section 9](#9-bulk-loading)) |
|
|
1605
|
+
| `watch(options)`, `cache` | subscription; result cache (`clear()`, `stats()`) |
|
|
1606
|
+
| `` sql`…` ``, `transaction(fn)` | raw SQL under the session and RLS |
|
|
1607
|
+
| `idOf(class, key)` | the id of an object of a class with a key, in the session tenant |
|
|
1608
|
+
| `entity(row \| chain)` | start a path from a ready node |
|
|
1609
|
+
| `describe()` | a machine-readable description of the model |
|
|
1610
|
+
| `verify(options)`, `maintain(options)`, `reset(options)` | verification, maintenance, reset ([sections 11](#11-history-and-point-in-time-reads) and [13](#13-operations)) |
|
|
1611
|
+
| `trimHistory(target, moment)`, `rekeyClass(class, key)` | history trimming; a new class key |
|
|
1612
|
+
| `refresh()`, `flushSignals()`, `close()` | re-read the classes; send the signals now; close |
|
|
1613
|
+
|
|
1614
|
+
### Chain
|
|
1615
|
+
|
|
1616
|
+
| Group | Methods |
|
|
1617
|
+
|---|---|
|
|
1618
|
+
| steps | `.ClassName(filter)`, `.AliasName(filter)`, `.RoleName(filter)` |
|
|
1619
|
+
| modifiers | `limit(n)`, `offset(n)`, `sort(field, 'asc' \| 'desc')`, `after(cursor)`, `asOf(moment)`, `withDeleted()`, `deep(max)`, `exact()`, `alias(name)`, `tags(…)`, `owner(…)`, `entity(…)`, `forUpdate()`, `cache({ ttl })` |
|
|
1620
|
+
| verbs | `create(data)`, `update(patch, { rev })`, `upsert(data)`, `delete({ confirm })`, `anonymize(fields)`, `reclass(class, data)`, `rekey(patch)`, `restore()`, `purge({ confirm })` |
|
|
1621
|
+
| slots | after a verb: `.RoleName.set(value)`, `.RoleName.unset()`, `.RoleName.add(value)`, `.RoleName.remove(value)` |
|
|
1622
|
+
| terminals | `rows()`, `first()`, `ids()`, `count({ paths })`, `paths()`, `versions({ follow })`, `sum(field)`, `avg(field)`, `min(field)`, `max(field)`, `countBy(field)` |
|
|
1623
|
+
|
|
1624
|
+
### `db.auth`
|
|
1625
|
+
|
|
1626
|
+
| Method | Parameters → result |
|
|
1627
|
+
|---|---|
|
|
1628
|
+
| `setPassword` | `{ account, identifier, password, kind?, confirmed?, keepCurrent? }` → credential id |
|
|
1629
|
+
| `verifyPassword` | `{ identifier, password, kind?, requireConfirmed?, ttl?, totp? }` → `{ account, token }` or `null` |
|
|
1630
|
+
| `issueApiKey`, `verifyApiKey` | `{ account, name?, ttl? }` → `{ key, id }`; `(key)` → sign-in or `null` |
|
|
1631
|
+
| `issueKeySecret`, `verifyKeySecret` | `{ account, name? }` → `{ key, secret, id }`; `(key, secret)` → sign-in or `null` |
|
|
1632
|
+
| `enrollTotp`, `verifyTotp`, `totpEnabled` | `{ account, issuer?, label? }` → `{ secret, uri }`; `{ account, code, window? }` → `boolean`; `(account)` → `boolean` |
|
|
1633
|
+
| `issueOtp`, `verifyOtp` | `{ account, identifier, kind?, ttl?, digits?, maxAttempts? }` → `{ code }`; `{ identifier, code, kind? }` → sign-in or `null` |
|
|
1634
|
+
| `link`, `lookup` | `{ account, kind, identifier, meta?, confirmed? }` → id; `{ kind, identifier, requireConfirmed?, ttl?, totp? }` → sign-in or `null` |
|
|
1635
|
+
| `credentials`, `revokeCredential`, `setFlags` | `(account)` → credentials without secrets; `(id, { keepCurrent? })`; `(id, { confirmed?, enabled? })` |
|
|
1636
|
+
| `sessionFor`, `refresh`, `revoke`, `revokeAll` | `(account, { ttl? })` → token; `({ ttl? })` → the new expiration time (the token stays the same); `()`; `(account?)` → number of sessions |
|
|
1637
|
+
| `tenants`, `switch` | `()` → `[{ tenant, roles }]`; `(tenant)` |
|
|
1638
|
+
| `purgeAccount` | `(account)` → `{ account, rows, versions, memberships, owned }` |
|
|
1639
|
+
|
|
1640
|
+
Password, code and TOTP checks, session issuing and credential confirmation are called by the service (`auth.*`
|
|
1641
|
+
permissions); a regular user gets `acl_denied`. The credentials of System, services and system administrators can be
|
|
1642
|
+
changed only by the account itself after a recent interactive sign-in, or by the system administrator. The exception is
|
|
1643
|
+
the service's keys: the service issues and revokes its own API keys and "key + secret" pairs itself with a session of
|
|
1644
|
+
its own key, but not its other credentials; nobody can revoke the last active key of a service (`acl_denied`). With
|
|
1645
|
+
`totp`, `verifyPassword` and `lookup` also check the TOTP code (the `auth.totp` permission) and issue a session with the
|
|
1646
|
+
`TOTP` method. The credential kind `kind` is `^[A-Z][A-Z0-9_]{1,31}$` (`PASSWORD`, `EMAIL`,
|
|
1647
|
+
`TELEGRAM` …); `SESSION`, `APIKEY`, `KEYSECRET`, `OTP`, `TOTP` and `KEY` are taken.
|
|
1648
|
+
|
|
1649
|
+
### Result row
|
|
1650
|
+
|
|
1651
|
+
| Field | Meaning |
|
|
1652
|
+
|---|---|
|
|
1653
|
+
| `id`, `class`, `rev` | id, class, version number (for `update(patch, { rev })`) |
|
|
1654
|
+
| `tenant`, `owner` | the tenant and the owner of the row |
|
|
1655
|
+
| `links`, `data`, `tags` | ends, fields, tags |
|
|
1656
|
+
| `at`, `author`, `agent`, `op`, `reason` | version time (ISO, UTC, microseconds), author, agent (the service under impersonation), kind of operation, reason |
|
|
1657
|
+
| `moved` | the reference to the old or the new id after `reclass` and `rekey` |
|
|
1658
|
+
| `$deleted` | tombstone: a deleted entity or a deletion version |
|
|
1659
|
+
| `$depth` | the depth of a node in `deep()`; the cascade level in a deletion preview |
|
|
1660
|
+
| `$upsert`, `$action`, `$versions`, `$purged` | what `upsert` did; what the deletion will do; the number of versions in `purge`; the history is erased |
|
|
1661
|
+
|
|
1662
|
+
### Reserved names
|
|
1663
|
+
|
|
1664
|
+
The Proxy parses the names below before classes, so a class with such a name cannot be reached as a step, and the
|
|
1665
|
+
database does not accept a link with such a name (`reserved_names()`; the installer fills in the list from
|
|
1666
|
+
`RESERVED_CLASS_NAMES`).
|
|
1667
|
+
|
|
1668
|
+
- chain methods: `then`, `entity`, `paths`, `rows`, `first`, `ids`, `count`, `limit`, `offset`, `sort`, `asOf`,
|
|
1669
|
+
`withDeleted`, `deep`, `exact`, `sum`, `avg`, `min`, `max`, `countBy`, `after`, `versions`, `create`, `update`,
|
|
1670
|
+
`upsert`, `set`, `unset`, `add`, `remove`, `delete`, `anonymize`, `reclass`, `restore`, `purge`, `alias`, `tags`,
|
|
1671
|
+
`owner`, `forUpdate`, `cache`, `rekey`; guards of the removed API: `run`, `execute`, `account`;
|
|
1672
|
+
- the `db` root: `as`, `begin`, `commit`, `rollback`, `lock`, `batch`, `watch`, `close`, `registry`, `sql`, `load`,
|
|
1673
|
+
`verify`, `maintain`, `reset`, `describe`, `auth`, `acl`, `idOf`, `refresh`, `transaction`, `flushSignals`,
|
|
1674
|
+
`schema`, `token`, `tenant`, `trimHistory`, `rekeyClass`, `cache`; guards: `reloadSchema`, `accounts`,
|
|
1675
|
+
`credentials`, `resources`, `rules`, `compact`;
|
|
1676
|
+
- batch: `discard`, `size`.
|
|
1677
|
+
|
|
1678
|
+
## 16. Errors
|
|
1679
|
+
|
|
1680
|
+
The text of an error is `letopis: <code>: <message>`. The code is in `err.code`, the details are in `err.detail`.
|
|
1681
|
+
Errors are `LetopisError` objects; data errors are `ValidationError` with `err.issues: [{ path, keyword, message }]`.
|
|
1682
|
+
|
|
1683
|
+
Database codes. Raw SQL gets them too: SQLSTATE class `LT`, the details are JSON in `DETAIL`.
|
|
1684
|
+
|
|
1685
|
+
| Code | SQLSTATE | When |
|
|
1686
|
+
|---|---|---|
|
|
1687
|
+
| `no_session` | LT001 | there is no valid session |
|
|
1688
|
+
| `target_not_found` | LT002 | the target is not found; the same for a nonexistent, a foreign and an invisible target |
|
|
1689
|
+
| `invalid_data` | LT003 | the data does not match the class description; `err.issues` gives the path, keyword and message |
|
|
1690
|
+
| `undeclared_end` | LT004 | the end is not declared in the class |
|
|
1691
|
+
| `tenant_denied` | LT005 | a foreign tenant or no membership |
|
|
1692
|
+
| `acl_denied` | LT006 | denied by the access rules |
|
|
1693
|
+
| `id_mismatch`, `id_from_db` | LT007, LT008 | the id is not v5 of the key; the id of a class without a key is assigned by the database |
|
|
1694
|
+
| `immutable_key` | LT009 | a key field, the id, the tenant and the owner do not change (a new key is `rekey`) |
|
|
1695
|
+
| `invalid_class` | LT010 | the class description is rejected, the class is not found or is abstract |
|
|
1696
|
+
| `use_reclass`, `reclass_denied` | LT011, LT012 | changing the class with `update` when the old or the new class has a key: the id changes, so use `reclass()`; the roles of referencing rows do not accept the new class |
|
|
1697
|
+
| `tightening_conflict` | LT013 | rows prevent tightening the description |
|
|
1698
|
+
| `isolation_level` | LT014 | deleting, changing the class and changing the key work only in `read committed` |
|
|
1699
|
+
| `recreated_in_statement` | LT015 | an id was deleted and created again in one statement |
|
|
1700
|
+
| `cursor_expired` | LT016 | the subscription cursor is older than the `all` tier of the retention policy |
|
|
1701
|
+
| `delete_restricted` | LT017 | referencing rows prevent the deletion (`onDelete: 'restrict'`) |
|
|
1702
|
+
| `not_deleted` | LT018 | `restore` and `purge` work only for deleted objects |
|
|
1703
|
+
| `reset_disabled` | LT019 | reset is disabled in the installation (`allowReset`) |
|
|
1704
|
+
| `confirm_mismatch` | LT020 | the reset confirmation does not match the schema name |
|
|
1705
|
+
|
|
1706
|
+
Library codes:
|
|
1707
|
+
|
|
1708
|
+
| Code | When |
|
|
1709
|
+
|---|---|
|
|
1710
|
+
| `exists` | the object already exists: `create` (`detail.id` is its id), `rekey` (the new id is taken) or the loader |
|
|
1711
|
+
| `duplicate`, `deleted` | the loader: a repeated id in the load; an object with this id existed before. The loader throws the code of the first problem row, the first 100 are in `detail.rows`, the total is `detail.total` |
|
|
1712
|
+
| `conflict` | `update(patch, { rev })`: the row was changed in the meantime (`detail`: `id`, `rev`, `expected`) |
|
|
1713
|
+
| `rev_ambiguous` | `{ rev }` on a step that does not have exactly one target |
|
|
1714
|
+
| `tx_required` | `forUpdate()`, `lock()`, `commit()` or `rollback()` outside an active `db.begin()` |
|
|
1715
|
+
| `lock_unsupported` | `forUpdate()` with an aggregate, `asOf`, `versions`, `withDeleted`, `deep` or a verb |
|
|
1716
|
+
| `commit_unknown` | the connection dropped during `COMMIT`: the outcome is unknown |
|
|
1717
|
+
| `tx_rolled_back` | PostgreSQL answered `ROLLBACK` to `COMMIT`: the error was caught inside `tr.transaction(fn)` |
|
|
1718
|
+
| `no_key` | `upsert`, `rekey` or `idOf` on a class without a key |
|
|
1719
|
+
| `no_path` | there is no hop between the classes of neighboring steps |
|
|
1720
|
+
| `invalid_query` | the chain or the API is used incorrectly (a step modifier `deep`, `exact`, `alias`, `tags` or `owner` after a verb, a slot not after a verb, an `after()` cursor with a value of a different type than the `sort` field, without `v` or with an `id` that is not a uuid, and so on) |
|
|
1721
|
+
| `admin_required` | the loader needs an admin connection |
|
|
1722
|
+
| `class_not_loadable` | the loader does not accept `Class` rows |
|
|
1723
|
+
| `bypass_rls` | `connect()` with a role that bypasses RLS, without `allowBypassRls` |
|
|
1724
|
+
| `removed` | a removed 0.21 API; the text gives the replacement |
|
|
1725
|
+
|
|
1726
|
+
A PostgreSQL permission denial (`42501`) comes, for example, on an `upsert` over someone else's row that you may not
|
|
1727
|
+
edit.
|
|
1728
|
+
|
|
1729
|
+
## 17. Performance
|
|
1730
|
+
|
|
1731
|
+
Benchmark of 1.0 against 0.21 on the same datasets (`cd lib && npx tsx bench/compare.bench.mjs`, 2026-10-09,
|
|
1732
|
+
PostgreSQL 18, and 0.21 on PostgreSQL 17 with TimescaleDB, one machine): salon has ~980 thousand rows (in 0.21 they
|
|
1733
|
+
belong to System without isolation, in 1.0 they are in a tenant under RLS and permissions); tenants has 500 thousand
|
|
1734
|
+
customers with 2 versions each, across ten tenants. The numbers depend on the machine; what matters is the ratio.
|
|
1735
|
+
|
|
1736
|
+
| Measurement | 1.0 | 0.21 | Unit | Note |
|
|
1737
|
+
|---|---|---|---|---|
|
|
1738
|
+
| loading the salon dataset | 6359 | 3609 | rows/s | 1.0: the loader in history mode with checks; 0.21: raw INSERTs into Entity |
|
|
1739
|
+
| loading the tenants dataset | 8566 | 35186 | rows/s | 0.21: INSERT … SELECT with triggers disabled |
|
|
1740
|
+
| row-by-row create | 128 | 156 | rows/s | |
|
|
1741
|
+
| read by key | 2.2 | 3.81 | ms (p50) | |
|
|
1742
|
+
| three-step chain (count) | 222.79 | 1233.83 | ms (p50) | organization → specialists → bookings |
|
|
1743
|
+
| point-in-time query (asOf) | 202.77 | 327.98 | ms (p50) | a specialist's bookings as of mid-year |
|
|
1744
|
+
| 20 concurrent writers | 982 | 522 | rows/s | |
|
|
1745
|
+
| class change (reclass) | 9.5 | — | ms (p50) | 0.21 has no such verb |
|
|
1746
|
+
| count() of bookings: a tenant under RLS and permissions | 272.25 | 890.73 | ms (p50) | 0.21: without isolation and permissions (enforceAccount: false); 1.0 without RLS (admin): 18.59 ms |
|
|
1747
|
+
| count() of customers of tenant 1 (50.0%) | 114.09 | 667.19 | ms (p50) | |
|
|
1748
|
+
| count() of customers of tenant 3 (10.0%) | 23.8 | 229.94 | ms (p50) | |
|
|
1749
|
+
| count() of customers of tenant 8 (1.0%) | 5.3 | 88.02 | ms (p50) | |
|
|
1750
|
+
| count() of customers of tenant 10 (0.3%) | 3.54 | 26.08 | ms (p50) | |
|
|
1751
|
+
|
|
1752
|
+
A full import of the salon dataset from 0.21 (`npm run test:import-salon`): 539,580 objects and 980,180 versions in
|
|
1753
|
+
193 s (5,084 rows/s); the per-class counters matched, and `verify()` is clean. Checkpoint B tuning (revision 5): the
|
|
1754
|
+
journal index on reference targets, `log_ends`, made an `asOf` read across a hop 16 times faster (3.37 s → 0.20 s) and
|
|
1755
|
+
costs the loader 15–17% of its speed. Reading `letopis.changed` and `COMMIT` in one pipeline removed a round trip to the
|
|
1756
|
+
database from every write (row-by-row writes: 119 → 128–147 rows/s in different runs). Row-by-row `create` is still
|
|
1757
|
+
12–18% slower than 0.21 (the measurements vary): most of the time goes to the check triggers in the database and to the
|
|
1758
|
+
commit with a disk write.
|
|
1759
|
+
|
|
1760
|
+
## 18. Limitations
|
|
1761
|
+
|
|
1762
|
+
This section lists what letopis 1.0 deliberately does not do or does with caveats, and where you need to watch the
|
|
1763
|
+
behavior.
|
|
1764
|
+
|
|
1765
|
+
**Platform**
|
|
1766
|
+
|
|
1767
|
+
- PostgreSQL 18 or newer is required; there are no workarounds for 15–17. AWS RDS, Google Cloud SQL, Azure Flexible
|
|
1768
|
+
Server and Neon support 18, while Supabase, as of October 2026, offers only 15 and 17, so 1.0 will not work there.
|
|
1769
|
+
- The database must be in UTF8 encoding (`casefold()` and the `pg_unicode_fast` collation). If a new major PostgreSQL
|
|
1770
|
+
version updates the Unicode tables, the unique index of credential identifiers is rebuilt with the `reindex` command.
|
|
1771
|
+
- PGlite has one superuser connection: no concurrent writers and no `LISTEN`, and RLS does not apply
|
|
1772
|
+
(`allowBypassRls: true`); it does not replace `postgres:18` for permissions, race tests and CI.
|
|
1773
|
+
- A pool in transaction mode (PgBouncer): `LISTEN` is not available, so use `connect({ listen: false })`; the
|
|
1774
|
+
subscription polls the journal, and the result cache is off. Prepared statements work with PgBouncer 1.21; with older
|
|
1775
|
+
versions, pass a postgres.js pool with `prepare: false` in `connect({ sql })`.
|
|
1776
|
+
- The NUL character (`\u0000`) is impossible in data strings: PostgreSQL does not store it in `jsonb`; such data is
|
|
1777
|
+
rejected already when the JSON is parsed (in the official JSON-Schema-Test-Suite, the "nul characters" groups are
|
|
1778
|
+
skipped).
|
|
1779
|
+
|
|
1780
|
+
**Writing and integrity**
|
|
1781
|
+
|
|
1782
|
+
- Row-by-row writing costs more than a plain insert: triggers check every row (data, ends, id, journal, hash, RLS).
|
|
1783
|
+
Row-by-row `create` is 12–18% slower than 0.21 ([section 17](#17-performance)); for bulk writing there is the loader.
|
|
1784
|
+
- Deleting, changing the class and changing the key work only in `read committed` (`isolation_level`); library
|
|
1785
|
+
transactions always open at this level.
|
|
1786
|
+
- PostgreSQL does not run `COPY` into tables under RLS; bulk loading works only through the loader on an admin
|
|
1787
|
+
connection.
|
|
1788
|
+
- The database trusts the loader: the data, ends and ids of loaded rows are checked by the library code (the TypeScript
|
|
1789
|
+
validator), not by the triggers; the database checks only the targets outside the load, the tenants and the owners,
|
|
1790
|
+
with one query at the end. The safeguards: the loader is available only to an admin, differential tests of the
|
|
1791
|
+
validator against the database, `load({ verify: true })` and `db.verify({ data: true })`. Keep the admin connection
|
|
1792
|
+
in a separate loading process, not in the web application: an SQL injection there would get the owner's rights.
|
|
1793
|
+
- The schema owner and a superuser can disable the triggers and change anything. letopis does not prevent this; it
|
|
1794
|
+
detects it: `verify()` recomputes the hash chains, `verify({ data: true })` checks the rows again, and the anchor (a
|
|
1795
|
+
hash of all chain heads) should be kept outside the database.
|
|
1796
|
+
- The database checks a subset of JSON Schema: Zod `refine`, `superRefine` and `transform` work only in the library,
|
|
1797
|
+
and the database does not accept a description with an unknown keyword; `pattern` is a limited subset of regular
|
|
1798
|
+
expressions shared by Zod, the validator and the database. The output of `z.toJSONSchema()` changes between Zod
|
|
1799
|
+
versions: the database stores an already normalized schema, and a `syncModels` dry run shows the difference before it
|
|
1800
|
+
is applied.
|
|
1801
|
+
- The key is the only way to get uniqueness: the id is computed from the key values without normalization, so the same
|
|
1802
|
+
value written in a different way (an email in a different case, a time with and without milliseconds) gives different
|
|
1803
|
+
ids. The model sets the format of a key field, and the database checks it.
|
|
1804
|
+
- Frequent changes of a key value (email, phone) require `rekey`. Values that change often and personal values are
|
|
1805
|
+
better moved to a separate hub with a link: an id from an email can be found by brute force and stays in references and in the journal
|
|
1806
|
+
even after `anonymize`, while a separate hub is deleted together with the link through `purge`.
|
|
1807
|
+
- `create` with a taken key answers `exists` with the id, and `upsert` without permission on the existing row answers
|
|
1808
|
+
with a `42501` denial: this reveals that the key is taken, but only within your own tenant, because in other tenants
|
|
1809
|
+
the same keys give other ids.
|
|
1810
|
+
- Changing the class and the key rewrites all incoming references (and, recursively, the links whose key includes this
|
|
1811
|
+
end), so its cost is proportional to the number of references. Tightening a class description and `rekeyClass` pause
|
|
1812
|
+
writes to the family while they check or move rows.
|
|
1813
|
+
- `rekeyClass` moves rows in one transaction under an exclusive lock of the family, not in batches: between batches,
|
|
1814
|
+
rows with old ids would break the rule "the id is v5 of the key".
|
|
1815
|
+
- v5 ids land in the primary key index in scattered places; on very large tables with heavy inserts this is more
|
|
1816
|
+
noticeable than with sequential ids.
|
|
1817
|
+
- Hot reference targets (an organization that almost everything references) accumulate `for key share` locks and
|
|
1818
|
+
MultiXact, as with regular foreign keys; this needs monitoring. Operations on one wallet go one after another under a
|
|
1819
|
+
lock of its row ([section 8](#8-transactions-batches-and-money), "Recipe: wallet").
|
|
1820
|
+
- A link cannot have a reserved name, and a hub with such a name cannot be reached as a step
|
|
1821
|
+
(["Reserved names"](#reserved-names)).
|
|
1822
|
+
|
|
1823
|
+
**Access**
|
|
1824
|
+
|
|
1825
|
+
- The service key is the root of trust: its compromise allows acting on behalf of any enabled user of the
|
|
1826
|
+
installation except System, services and system administrators; the access rules do not narrow this scope. The key
|
|
1827
|
+
is reissued (`db.auth.issueApiKey`) and revoked (`db.auth.revokeCredential`) by the service itself with a session of
|
|
1828
|
+
its own key, or by the system administrator; with its key the service also issues itself new keys, so after a leak
|
|
1829
|
+
revoke all its keys except the new one (`credentials` lists them). Nobody can revoke the last active key of a
|
|
1830
|
+
service: to cut a service off, set `enabled: false` on its account. A database password alone, without a token,
|
|
1831
|
+
gives nothing.
|
|
1832
|
+
- The permission pass reveals its own plan to the client (the classes and rule conditions that apply to it).
|
|
1833
|
+
- Raw SQL without the permission pass pays for the permission plan on every query: 0.3–1.3 ms depending on the number
|
|
1834
|
+
of rules.
|
|
1835
|
+
- The end names `User`, `Tenant` (membership), `Group` and `Object` (rule) are taken by system classes across the
|
|
1836
|
+
whole schema.
|
|
1837
|
+
- OTP and TOTP attempt counters are rolled back together with the caller's transaction: the service limits the rate of
|
|
1838
|
+
attempts.
|
|
1839
|
+
- When users link a taken identifier themselves, the answer is `acl_denied`, so a user with a recent sign-in learns
|
|
1840
|
+
this way that the identifier is taken. Permission changes in all tenants are serialized through one counter row.
|
|
1841
|
+
- The number of invisible rows that block a deletion, and the existence of an account by id (as a reference target),
|
|
1842
|
+
are not hidden.
|
|
1843
|
+
- Class metadata (`meta`) is visible to anyone who has a session: anyone can read `Class` rows. Secrets do not belong
|
|
1844
|
+
there.
|
|
1845
|
+
- An application role can slow the installation down for everyone: `LOCK TABLE entity`, advisory locks with
|
|
1846
|
+
predictable keys, fake `NOTIFY` signals (they only clear the cache), frequent permission changes (one counter row per
|
|
1847
|
+
installation). The global journal number reveals the overall write activity.
|
|
1848
|
+
|
|
1849
|
+
**History, signals, cache**
|
|
1850
|
+
|
|
1851
|
+
- An `asOf` read is rebuilt from the journal; only the current state is cheap. In the thinned part of the journal, a
|
|
1852
|
+
point-in-time read shows the last kept version before the moment: its precision there equals the period of the tier,
|
|
1853
|
+
a day, a week, a month or a year. A history import into a class with a policy is thinned right away, and a subscriber
|
|
1854
|
+
may not see the old versions.
|
|
1855
|
+
- The thinned part of the journal proves neither that the history is complete nor that old versions are authentic: for
|
|
1856
|
+
a class with a policy, `verify()` cannot tell the tampering of a version older than the `all` tier from thinning.
|
|
1857
|
+
- The journal has no partitions: thinning deletes versions in batches, and the table file does not shrink by itself
|
|
1858
|
+
(new versions take the space after autovacuum); to return the space to the system, use `VACUUM FULL` or `pg_repack`
|
|
1859
|
+
in a maintenance window.
|
|
1860
|
+
- Signals reveal activity: PostgreSQL lets any role listen to a channel, and the class names in signals show when and
|
|
1861
|
+
to which classes writes happen (a signal has no ids, no tenant and no data). A fake signal only clears the cache one
|
|
1862
|
+
extra time.
|
|
1863
|
+
- The `NOTIFY` queue (8 GB) is shared by the whole cluster: a stuck listener fills it up, and then transactions with
|
|
1864
|
+
`NOTIFY` fail on commit. The library reads signals right away, and `maintain()` warns when the queue is more than
|
|
1865
|
+
half full.
|
|
1866
|
+
- The subscription horizon is shared by the whole cluster: a long write transaction delays the events of all
|
|
1867
|
+
subscribers (`sub.lag()`); long operations are better done in batches.
|
|
1868
|
+
- Between processes, the result cache goes stale for the signal sending interval (100 ms) plus delivery, and for the
|
|
1869
|
+
time of a listener reconnect; the `ttl` term is a safeguard.
|
|
1870
|
+
- A "from now" subscription cursor carries a snapshot of the subscription moment until it passes that boundary: such a
|
|
1871
|
+
cursor is longer than a regular one.
|
|
1872
|
+
|
|
1873
|
+
**Import from 0.21**
|
|
1874
|
+
|
|
1875
|
+
- 1.0 is a new major version, not a continuation of 0.21: `count()`, `paths()`, `create`, roles, multiple ends and
|
|
1876
|
+
`reclass` behave in a new way, and the data is moved by an import; see [MIGRATION.md](MIGRATION.md) (in Russian).
|
|
1877
|
+
- Import: the steps (classes, accounts, credentials, permissions, the data of each tenant) are separate transactions,
|
|
1878
|
+
and the source is read into memory entirely; a row owner that is not found among the accounts becomes the tenant;
|
|
1879
|
+
0.21 access rules move to System and apply in all tenants; the `onDelete` of ends is `cascade`, the way 0.21 deleted.
|
|
1880
|
+
|
|
1881
|
+
## 19. Tests
|
|
1882
|
+
|
|
1883
|
+
This section is for those who develop letopis itself. Run: `cd lib && npm test`. The harness starts the
|
|
1884
|
+
`letopis-pg18` container (`postgres:18`, port 15433) by itself if the database does not respond; each test file works in its own schema `v2.t_<name>`.
|
|
1885
|
+
|
|
1886
|
+
```
|
|
1887
|
+
# connect — connecting under a session, refusing roles that bypass RLS, the registry, the deferred signal, error translation (stage 2)
|
|
1888
|
+
# context — sessions, service sign-in by key, the RLS policies of stage 1 (stage 1)
|
|
1889
|
+
# access-rest — the remaining rows of stage 5: loading on behalf of a user, balance No. 20, 22, 29, upsert and { rev } under ACL, inlining of acl_ok (stage 5)
|
|
1890
|
+
# acl — ACL permissions: the 0.21 suite ported, row conditions, weight and default deny, cascade, deny on a descendant, deep, an invisible target, raw SQL versus the library (stage 5)
|
|
1891
|
+
# acl-parity — 1000 random rule sets: the database permission plan and endpoint decisions versus decide from 0.21 (stage 5)
|
|
1892
|
+
# auth — credentials (password, key, key + secret, OTP, TOTP, identity), timing equalization, case, session revocation, recent sign-in (stage 5)
|
|
1893
|
+
# balance — top-up and debit: idempotency by opId, races, deadlock, connection drop (TCP proxy), adjustment under { rev } and forUpdate (stage 4)
|
|
1894
|
+
# chain-api — guards of the removed API, reserved names, build errors, the plan cache, onQuery (stage 3)
|
|
1895
|
+
# chain-read — read chains on the demo domain: hops, roles, multiple ends, 18 operators, keyset, aggregates, deep, asOf, withDeleted, pivot (stage 3)
|
|
1896
|
+
# chain-types — types of steps, rows and filters from models: connect({ models }) (stages 3, 4)
|
|
1897
|
+
# chain-write — writing with chains: create, update, upsert, delete with preview, anonymize, reclass, slots, inc, { rev }, fan-out, plan rollback (stage 4)
|
|
1898
|
+
# demo — the booking demo domain on Zod: seed from models, writing data, id by key, agreement between Zod and the database (stage 2)
|
|
1899
|
+
# docs — documentation examples are executed: the marked blocks of the README and the cheatsheet (Russian and English) run as scenarios against PostgreSQL 18 (stage 8)
|
|
1900
|
+
# engine-attacks — raw SQL attacks from the application role: the rows of §7.3.1 for stages 1 and 3 with specific outcomes (stages 1, 3)
|
|
1901
|
+
# engine-class — classes as entities: metaschema, inheritance, recomputing descendants, tightening, metadata (stage 1)
|
|
1902
|
+
# engine-write — the write pipeline in raw SQL: v5 ids, defaults, merge, inc, deletion, resurrection, journal, signals (stage 1)
|
|
1903
|
+
# engine-races — races on two connections in both orders, without timed pauses (stage 1)
|
|
1904
|
+
# explain — chain indexes from the application role: outside, the primary key and (class, tenant); inside find_*(), GIN (stage 3)
|
|
1905
|
+
# find-ids-parity — 30 chains through find_*() and through a direct read under RLS give the same answers (stage 3)
|
|
1906
|
+
# hash — the version hash chain, journal tampering and gaps, verify() (stage 1)
|
|
1907
|
+
# harness — the test harness: starting the database, a schema per file, two connection strings (stage 0)
|
|
1908
|
+
# jsonschema — compiling JSON Schema to jsonpath and checking data in the database (stage 1)
|
|
1909
|
+
# impersonation — impersonation: letopis.account only for a service with the permission, target restrictions, author and agent in the journal, sessionFor (stage 5)
|
|
1910
|
+
# install — the installer and roles: a repeated installation, two installations in one database, three tables (stage 1)
|
|
1911
|
+
# member — memberships: working in a tenant by roles, auth.switch, leaving with ownership transfer, system rows, tenant roles without System permissions (stage 5)
|
|
1912
|
+
# model — Zod models, the JSON Schema normalizer, pattern translation, agreement with the database metaschema (stage 2)
|
|
1913
|
+
# load — the loader: several classes, cycles, $key and $ext, repeats, existing, portions, history mode, the query for targets outside the load, CLI (stage 4)
|
|
1914
|
+
# roles — roles and privileges: installation without a superuser, owner functions, triggers by catalog (stage 1)
|
|
1915
|
+
# write-races — write races: 20 concurrent edits of different fields without losses and strict rev, concurrent add/remove on a multiple end, the loader versus class tightening and versus a concurrent edit (existing: update) in both orders (stage 4)
|
|
1916
|
+
# validator-parity — the TypeScript validator versus the database check: the official suite, 10,000 random documents, defaults (stage 2)
|
|
1917
|
+
# scenario-barbershop — end-to-end barbershop scenario: 9 scenes from 0.21 (integration) on 1.0 chains — catalog, free slots in a batch, bookings under lock, multi-slot and multi-specialist, versions, cancellation and booking again on the same id, closing a month with restrict (stage 4)
|
|
1918
|
+
# scenario-real-life — the "real life" of a salon: 16 scenes from 0.21 (real-life) — a booking service on top of chains, races of double booking, rescheduling and cancellation, walk-in, refusals, reports and daily invariants (stage 4)
|
|
1919
|
+
# system-classes — system classes instead of the 0.21 facades: the System seed, accounts, credentials, resources and rules, transactions (stage 5)
|
|
1920
|
+
# sync — letopis sync (dry run, apply, tightening, lost refine) and letopis types (stage 2)
|
|
1921
|
+
# tx — explicit db.begin() transactions: a broken transaction, lock(), 40P01; forUpdate(); the batch and its merging, a batch queue per facade; db.as(…, { owner }) (stage 4)
|
|
1922
|
+
# v5 — v5 identifiers on the SQL side versus the vectors of an independent implementation (stage 1)
|
|
1923
|
+
# zod-output — a snapshot of the z.toJSONSchema() output by field kind of the demo domain versus a reference (stage 0)
|
|
1924
|
+
# time — history: versions({ follow }) across reclass and rekey, withDeleted, restore, purge, history trimming, account deletion (stage 6)
|
|
1925
|
+
# rekey — a new key value of an object and a new key of a class: new ids, moving references, recursion through links with the end in the key (stage 6)
|
|
1926
|
+
# reset — reset of sessions, tenant, all: the allowReset flag, the letopis.reset permission, confirmation by the schema name, the reset record, a new service key (stage 6)
|
|
1927
|
+
# verify — legitimate gaps and tampering, verify({ data: true }): data, ends, tenants, id by key; permissions and CLI (stage 6)
|
|
1928
|
+
# retention — retention policy: tiers, a finite term, special versions, deleted rows, asOf, cursor_expired, time zone, catch-up run, inheritance and default — versus a reference in TypeScript (stage 6)
|
|
1929
|
+
# watch — cursor subscription: commit order, a long transaction, resuming, listener drop, two tenants (stage 6)
|
|
1930
|
+
# cache — result cache: hits, own and others' writes, the key with the user, permissions, revoked sessions, listener drop, ttl, class marking, without LISTEN, a race, db.sql (stage 6)
|
|
1931
|
+
# maintain — maintenance: expired sessions and codes, permissions, one process under a lock, schedule (stage 6)
|
|
1932
|
+
# up — the full installer: fresh installation, re-apply, custom seeds, automatic database creation, UTF8, version 17, upgrade with migrations, PGlite, the first tenant and user (up({ tenant }), createTenant) (stages 6, 8)
|
|
1933
|
+
# scenario-salon — end-to-end salon scenario: 21 acts from 0.21 (salon) with a coverage matrix of the 1.0 API (stage 6)
|
|
1934
|
+
# telemetry — onQuery and slowMs, describe() and class metadata with inheritance, describeText (stage 6)
|
|
1935
|
+
# import — import from 0.21 with a fixture of a real 0.21: key violation, --keyless, per-class counters, verify, self-reference, repeat, ids without a key, a dangling reference, $$strict and fields starting with $, update, credentials, CLI (stage 7)
|
|
1936
|
+
# security-final — the final security review (stage 8, §7.10 No. 8.4): a family lock by the tenant of the root, account_purge without permission and restore/purge of a foreign id give neither a leak nor a lock; the functions of stage 6, log_refs_at, journal-based find_* reads and watch_read stay within their own tenant; NOTIFY carries only names; the cache key distinguishes the token, the actor and the tenant (stage 8)
|
|
1937
|
+
```
|