letopis 1.0.1 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENT-CHEATSHEET.en.md +117 -10
- package/AGENT-CHEATSHEET.md +110 -8
- package/CHANGELOG.md +739 -0
- package/README.en.md +873 -68
- package/README.md +830 -66
- package/dist/admin.d.ts +24 -4
- package/dist/admin.js +39 -9
- package/dist/cache.d.ts +33 -9
- package/dist/cache.js +81 -32
- package/dist/chain.d.ts +29 -7
- package/dist/chain.js +61 -12
- package/dist/cli.js +59 -7
- package/dist/errors.js +4 -2
- package/dist/import.d.ts +6 -0
- package/dist/import.js +196 -175
- package/dist/index.d.ts +27 -5
- package/dist/index.js +191 -29
- package/dist/internal.d.ts +16 -0
- package/dist/internal.js +14 -0
- package/dist/load.d.ts +63 -0
- package/dist/load.js +331 -44
- package/dist/mcp.d.ts +28 -0
- package/dist/mcp.js +810 -0
- package/dist/model.d.ts +34 -0
- package/dist/model.js +17 -0
- package/dist/project.d.ts +309 -0
- package/dist/project.js +1097 -0
- package/dist/registry.d.ts +17 -0
- package/dist/registry.js +13 -0
- package/dist/signals.d.ts +73 -0
- package/dist/signals.js +169 -0
- package/dist/sql.d.ts +1 -1
- package/dist/sql.js +26 -7
- package/dist/tx.d.ts +9 -7
- package/dist/tx.js +15 -5
- package/dist/typed.d.ts +21 -6
- package/dist/types.d.ts +77 -4
- package/dist/types.js +8 -3
- package/dist/up.d.ts +16 -2
- package/dist/up.js +34 -4
- package/dist/uuid.js +19 -8
- package/dist/watch.d.ts +23 -2
- package/dist/watch.js +44 -7
- package/dist/write.d.ts +11 -2
- package/dist/write.js +57 -10
- package/llms.txt +9 -2
- package/package.json +13 -2
- package/sql/10-core.sql +13 -2
- package/sql/15-errors.sql +4 -1
- package/sql/40-class.sql +57 -2
- package/sql/60-write.sql +611 -82
- package/sql/70-read.sql +33 -8
- package/sql/75-history.sql +358 -0
- package/sql/80-auth.sql +7 -7
- package/sql/88-derived.sql +1154 -0
- package/sql/90-time.sql +134 -21
- package/sql/95-seed.system.sql +24 -6
- package/sql/99-revision.sql +1 -1
- package/sql/README.md +5 -1
- package/sql/migrate/0006-changed-paths.sql +3 -0
package/README.en.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
# letopis 1.
|
|
1
|
+
# letopis 1.2
|
|
2
2
|
|
|
3
3
|
**letopis** is a Node.js library: a store of versioned entities on PostgreSQL 18. Every change is saved as a version
|
|
4
4
|
in a journal. So every object has a history, a state as of any date and a change subscription. The database itself
|
|
@@ -14,11 +14,15 @@ What letopis can do:
|
|
|
14
14
|
- **Writing with chains**: `create`, `update` with merge, `upsert`, deletion with preview and cascade, the `inc` increment
|
|
15
15
|
for money and counters — [sections 5](#5-writing), [7](#7-deleting-and-restoring) and [8](#8-transactions-batches-and-money).
|
|
16
16
|
- **Reading with chains**: hops along links, filters, trees, aggregates, pagination — [section 6](#6-reading).
|
|
17
|
-
- **History**: versions, point-in-time reads,
|
|
18
|
-
[section 11](#11-history-and-point-in-time-reads).
|
|
17
|
+
- **History**: versions, point-in-time reads, what changed and who changed it (`diff`, `blame`), restoring deleted
|
|
18
|
+
data, retention policy, hash chain verification — [section 11](#11-history-and-point-in-time-reads).
|
|
19
19
|
- **Tenants, sign-in and permissions** in the database itself: passwords, API keys, one-time codes, TOTP, sessions,
|
|
20
20
|
impersonation, access rules — [section 10](#10-users-sign-in-and-permissions).
|
|
21
21
|
- **Change subscription** by cursor without losses, and a **result cache** — [section 12](#12-change-subscription-and-cache).
|
|
22
|
+
- **Derived data**: aggregates in the database, sums and counts that the database itself maintains, and projectors
|
|
23
|
+
that build summaries from the journal — [sections 4](#aggregates-in-the-database) and [12](#projectors).
|
|
24
|
+
- **Agents and MCP**: an MCP server gives an AI agent the model, reads, history and, if allowed, writes, with the
|
|
25
|
+
rights of one user — [section 13](#agents-and-mcp).
|
|
22
26
|
- **Bulk loading** of millions of rows and **import from 0.21** — [sections 9](#9-bulk-loading) and [14](#14-migrating-from-021).
|
|
23
27
|
|
|
24
28
|
Requirements:
|
|
@@ -28,7 +32,8 @@ Requirements:
|
|
|
28
32
|
- Node.js 22 or newer; the package is an ES module.
|
|
29
33
|
- One runtime dependency: `postgres`. Zod 4 is an optional peer dependency: only the `letopis/model` subpath needs it,
|
|
30
34
|
and `import { connect } from 'letopis'` works without it. For development without PostgreSQL there are the optional
|
|
31
|
-
`@electric-sql/pglite` and `@electric-sql/pglite-socket`.
|
|
35
|
+
`@electric-sql/pglite` and `@electric-sql/pglite-socket`. The MCP server (`letopis/mcp` and `letopis mcp`) needs
|
|
36
|
+
the optional `@modelcontextprotocol/server`, version 2.4 or newer in the 2.x line.
|
|
32
37
|
|
|
33
38
|
## Contents
|
|
34
39
|
|
|
@@ -154,8 +159,8 @@ service key again while the installation still has a valid API key. Next, sectio
|
|
|
154
159
|
|
|
155
160
|
There are three tables. `entity` is the current state: one row per live entity, including the rows of system classes
|
|
156
161
|
(`Class`, `Account`, `Resource`, `member`, `rule`). `log` is the journal: a full snapshot of every version with author,
|
|
157
|
-
time, kind of operation and hash. `secret` holds credentials and
|
|
158
|
-
the sign-in functions work with it.
|
|
162
|
+
time, kind of operation and hash. `secret` holds credentials, sessions and projector checkpoints: the application sees
|
|
163
|
+
none of its rows; only the sign-in and projector functions work with it.
|
|
159
164
|
|
|
160
165
|
Classes are entities too. A class description is stored in a `Class` row; the database also keeps the expanded
|
|
161
166
|
description (with ancestors applied) and the compiled data check there. The version hash is a generated column: sha256 of the
|
|
@@ -191,10 +196,27 @@ is built from the journal.
|
|
|
191
196
|
|
|
192
197
|
### Signals
|
|
193
198
|
|
|
194
|
-
A write reports
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
199
|
+
A write reports changes through `NOTIFY`. The channel is the full name of the installation schema, version included,
|
|
200
|
+
for example `v2.shop`; because of the dot, SQL takes the name in double quotes: `LISTEN "v2.shop"`. A raw SQL write on
|
|
201
|
+
its own connection sends the signal from a trigger, and PostgreSQL delivers it on commit. Library transactions,
|
|
202
|
+
`db.sql` and `db.transaction()` included, defer their signals: the process notifier sends them every 100 ms
|
|
203
|
+
(`notifyIntervalMs`) and retries the ones that failed to send. What a write signal carries (its payload) is set by the
|
|
204
|
+
installation mode `up({ signals })`:
|
|
205
|
+
|
|
206
|
+
| Mode | Write signal payload | What any role that can connect to the database sees |
|
|
207
|
+
|---|---|---|
|
|
208
|
+
| `class` (default) | the class name: `Booking` | class names and write times |
|
|
209
|
+
| `tenant` | `Class@tenant token`: `Booking@3f9c…` | also tenant pseudonyms |
|
|
210
|
+
| `opaque` | `class token@tenant token`: `9a1e…@3f9c…` | only write times and frequency per pseudonym |
|
|
211
|
+
| `none` | no signals | nothing |
|
|
212
|
+
|
|
213
|
+
A token is 24 hexadecimal characters of a keyed HMAC: only a process with the signal key can match a pseudonym to a
|
|
214
|
+
name, and the key is given to services (the `letopis.signals` permission). Besides write signals, there are service
|
|
215
|
+
ones: a revoked session sends `session:<16 hex>` (the first 16 hexadecimal characters of the sha256 of its token), a
|
|
216
|
+
session reset sends `session:*`, and a reset of a tenant or of the whole installation sends `*` (a reset of the whole
|
|
217
|
+
installation also sends `session:*`). In the `none` mode there are none of them either. Signals carry no ids and no
|
|
218
|
+
data. On signals, the class registry is re-read, the result cache is cleared and subscriptions wake up; what the mode
|
|
219
|
+
changes for the cache and subscriptions is in [section 12](#12-change-subscription-and-cache).
|
|
198
220
|
|
|
199
221
|
## 3. Installation and connection
|
|
200
222
|
|
|
@@ -213,6 +235,7 @@ functions and the system seed. On the first installation, it issues a service ke
|
|
|
213
235
|
| `history` | the history retention policy for classes without their own ([section 11](#11-history-and-point-in-time-reads)) |
|
|
214
236
|
| `serviceKey` | your own service key instead of a random one: `lts_` and 32–128 hexadecimal characters |
|
|
215
237
|
| `allowReset` | allow `db.reset()` ([section 13](#13-operations)); off by default |
|
|
238
|
+
| `signals` | the signal mode: `'class'` (default), `'tenant'`, `'opaque'` or `'none'` (["Signals"](#signals)); the mode is stored in the schema — `up()` without the option keeps it |
|
|
216
239
|
| `fresh` | drop the schema and install it again — **the schema data is lost** |
|
|
217
240
|
| `upgrade` | upgrade a schema of an older revision: migrations and a re-apply |
|
|
218
241
|
| `pglite` | a database for development without PostgreSQL (see below) |
|
|
@@ -221,7 +244,17 @@ functions and the system seed. On the first installation, it issues a service ke
|
|
|
221
244
|
|
|
222
245
|
Result: `schema` (the full name), `serviceKey` (on the first installation, and on a re-run when the installation has no
|
|
223
246
|
valid API key left; otherwise `null`), `created`, `upgraded`, `revision`, `migrations`, `analyzed`, `owner`, `app`,
|
|
224
|
-
|
|
247
|
+
`signals` (the signal mode of the installation), `warnings` (without `quiet` they are printed), and also `tenant` and
|
|
248
|
+
`pglite` if they were requested. On PostgreSQL below 18.2, `up()` warns: such a server could truncate CLOG that unread
|
|
249
|
+
signals still need — upgrade it.
|
|
250
|
+
|
|
251
|
+
To hide class names from signals, use the `opaque` mode (or `tenant` if the names are not secret but you want the
|
|
252
|
+
cache to be precise per tenant); after changing the mode, restart the application processes — they read the mode on
|
|
253
|
+
connect:
|
|
254
|
+
|
|
255
|
+
```ts
|
|
256
|
+
await up({ dsn: ADMIN, schema: 'shop', signals: 'opaque' }) // later up({ dsn, schema }) keeps the mode
|
|
257
|
+
```
|
|
225
258
|
|
|
226
259
|
<!-- run: guide -->
|
|
227
260
|
```ts
|
|
@@ -241,8 +274,8 @@ const ANNA = r.tenant!.user! // its first user
|
|
|
241
274
|
The application works as a LOGIN role that is a member of `letopis_app` only:
|
|
242
275
|
`create role shop_app login password '…'; grant letopis_app to shop_app;`. Then RLS and triggers apply to it, as to
|
|
243
276
|
any raw SQL from this role. `connect()` rejects a role that bypasses RLS with the error `bypass_rls`. The admin
|
|
244
|
-
connection (`allowBypassRls: true`) is needed only for the loader,
|
|
245
|
-
process, not in the web application.
|
|
277
|
+
connection (`allowBypassRls: true`) is needed only for the loader, projectors, aggregate repair (`rebuildAggregates`),
|
|
278
|
+
verification and maintenance; keep it in a separate process, not in the web application.
|
|
246
279
|
|
|
247
280
|
### First tenant and user
|
|
248
281
|
|
|
@@ -315,7 +348,8 @@ through `as()` share the pool with the original one: close the original one.
|
|
|
315
348
|
|
|
316
349
|
### Admin connection
|
|
317
350
|
|
|
318
|
-
The loader
|
|
351
|
+
The loader, projectors and `rebuildAggregates` need a connection of a role that can become the owner, with
|
|
352
|
+
`allowBypassRls: true`. `verify()` and
|
|
319
353
|
`maintain()` work on it too, and also for a system administrator on a regular connection. The admin connection prints
|
|
320
354
|
the warning "RLS не действует" ("RLS does not apply"); this is expected. The session sets the tenant it works with. A
|
|
321
355
|
separate loading process is better off with a service key:
|
|
@@ -329,12 +363,18 @@ const adm = await connect({ dsn: ADMIN, schema: r.schema, token: r.tenant!.token
|
|
|
329
363
|
|
|
330
364
|
### Upgrading the package and schema
|
|
331
365
|
|
|
332
|
-
The schema has an engine revision (currently
|
|
366
|
+
The schema has an engine revision (currently 6). If a new package version brings a new revision, `connect()` only warns
|
|
333
367
|
and names the command. It is `up()` that refuses: a lagging schema without `upgrade: true`, and a schema newer than the
|
|
334
368
|
package. So a startup script that calls `up()` on every start will stop after a package update until it gets
|
|
335
369
|
`upgrade: true`. `up({ dsn, schema, upgrade: true })` applies the migrations from `sql/migrate/` and re-applies the
|
|
336
370
|
functions; the data is kept.
|
|
337
371
|
|
|
372
|
+
The first `up()` after moving to 1.2 builds the journal index `log_class_tenant_at (class, tenant, at)` instead of
|
|
373
|
+
`log_class_at`: an `asOf` and `withDeleted` slice by class no longer reads the journal of other tenants with the same
|
|
374
|
+
class name. While the index is being built, writes to the journal wait. On a large installation, build it in advance
|
|
375
|
+
without stopping writes: `create index concurrently log_class_tenant_at on "v2.<schema>".log (class, tenant, at)` as
|
|
376
|
+
the owner role — then `up()` only drops the old one.
|
|
377
|
+
|
|
338
378
|
### Development without PostgreSQL: PGlite
|
|
339
379
|
|
|
340
380
|
`up({ pglite: true })` starts PGlite (PostgreSQL 18 on WebAssembly) with pgcrypto and a socket on a free port. It needs
|
|
@@ -464,13 +504,76 @@ name of the class for a chain step, for example a name in another language.
|
|
|
464
504
|
|
|
465
505
|
- `history: false`: no journal is kept for the class: no versions, no point-in-time reads, no restore and no
|
|
466
506
|
subscription events. A retention policy is `history: { all: '1 day', daily: '3 months', … }`
|
|
467
|
-
([section 11](#11-history-and-point-in-time-reads)).
|
|
507
|
+
([section 11](#11-history-and-point-in-time-reads)). Only someone with the `READ` and `DELETE` permissions without
|
|
508
|
+
conditions on all rows of the class and its descendants may change `history` of an existing class: `maintain()` thins
|
|
509
|
+
the journal by the policy, and `false` turns it off.
|
|
468
510
|
- `ownerDefault: 'actor'`: the owner of new rows is the user, not the tenant (the default is `'tenant'`). This is
|
|
469
511
|
useful for the rule "everyone sees their own records".
|
|
470
512
|
- `cache: { ttl }`: reads of this class alone are cached without `.cache()` ([section 12](#12-change-subscription-and-cache)).
|
|
471
513
|
- `meta`: free-form metadata for the application: a title, an icon, a color, an order in a menu. Up to 64 KB,
|
|
472
514
|
inherited key by key (`null` removes an inherited value), visible to anyone who has a session. For an end, use
|
|
473
515
|
`.meta()`.
|
|
516
|
+
- `derivedBy: 'projector-name'`: a derived class: only this projector writes its rows ([Projectors](#projectors)).
|
|
517
|
+
The name is lowercase Latin letters, digits and `_.:-` (the first character is a letter or a digit), up to 64
|
|
518
|
+
characters. The option is inherited, and a descendant does not change it. The ends of a derived class are only
|
|
519
|
+
`onDelete: 'cascade'` or `'unset'`: a derived row does not prevent deleting its target. Nothing refers to the rows
|
|
520
|
+
of a derived class. The application, raw SQL and the loader do not write to such a class: writing, editing and
|
|
521
|
+
deleting its rows gives `derived_class`. The derived class itself (own or inherited) is created, edited and deleted
|
|
522
|
+
only by an administrator — an admin connection or an account with the System permission `letopis.projector`; for an
|
|
523
|
+
application session it is `acl_denied` ([Projectors](#projectors)).
|
|
524
|
+
- `aggregate: { field: { sum | count, of, via, where } }`: an aggregate in the database: the database itself keeps
|
|
525
|
+
the sum or count of source rows ([Aggregates in the database](#aggregates-in-the-database)).
|
|
526
|
+
|
|
527
|
+
### Aggregates in the database
|
|
528
|
+
|
|
529
|
+
An aggregate is a field that the database itself maintains: the sum of a field or the number of live rows of a
|
|
530
|
+
source class whose end leads to the row. It is needed where the total must match the source even for raw SQL: a
|
|
531
|
+
wallet balance, the number of order lines.
|
|
532
|
+
|
|
533
|
+
```ts
|
|
534
|
+
const Wallet = hub('Wallet', { number: z.string(), balance: z.number().int().min(0), deposits: z.number().int().optional() }, {
|
|
535
|
+
key: ['number'],
|
|
536
|
+
aggregate: {
|
|
537
|
+
balance: { sum: 'amount', of: 'Movement', via: 'Wallet' }, // sum of movements
|
|
538
|
+
deposits: { count: true, of: 'Movement', via: 'Wallet', where: { properties: { amount: { minimum: 1 } } } }, // number of top-ups
|
|
539
|
+
},
|
|
540
|
+
})
|
|
541
|
+
const Movement = link('Movement', { amount: z.number().int() }, { ends: { Wallet } })
|
|
542
|
+
```
|
|
543
|
+
|
|
544
|
+
- `sum: 'field'` is the sum of a numeric field of the source, `count: true` is the number of rows. `of` is the source
|
|
545
|
+
class (a name or a model) together with its descendants, `via` is its single end that leads to the target row.
|
|
546
|
+
`where` is a JSON Schema over the source data: rows that do not pass it are not counted.
|
|
547
|
+
- Only live rows of the same tenant as the target are counted. A reference to a System row is not counted: a shared
|
|
548
|
+
directory does not count the data of all tenants. An aggregate of a tenant class, including one inherited from a
|
|
549
|
+
System class, counts only classes of its own tenant: a System class named `of` is not counted, while the tenant's
|
|
550
|
+
own descendant of a System class is. An aggregate of a System class counts System classes and the classes of the
|
|
551
|
+
target row's tenant.
|
|
552
|
+
- A trigger recalculates the field after every statement: writes by the library, raw SQL, a delete cascade,
|
|
553
|
+
`restore`, `reclass`, `rekey`, the loader, account deletion. The operations of one statement add up to one increment
|
|
554
|
+
of the target, and the class rule (`min(0)`) is checked on the total: an operation that would take the balance below
|
|
555
|
+
zero is rejected with `invalid_data`. Every change of the target is its new version with the author and reason of
|
|
556
|
+
the operation.
|
|
557
|
+
- Only the database writes the field. A value in `create`, `update`, `upsert`, `rekey`, `reclass` or raw SQL is an
|
|
558
|
+
`aggregate_field` error. A new row gets the total of already existing sources, usually 0. Replacing the data as a
|
|
559
|
+
whole without the field keeps the previous value. The loader does not write a passed value. No permission to write
|
|
560
|
+
the target is needed: the permission to write the source is enough.
|
|
561
|
+
- The database checks the description (`invalid_class`): the field is in the schema, it is numeric and not part of the
|
|
562
|
+
key; the `via` end is single and accepts the target class; the `sum` field is a number; aggregates do not form a
|
|
563
|
+
cycle; the target and the source are not system or derived classes. A source described after the target is checked
|
|
564
|
+
when it appears.
|
|
565
|
+
- Rows get into an aggregate only with the knowledge of someone who reads them all: the field also counts rows that
|
|
566
|
+
the author cannot see. Only someone who reads all rows of the source and its descendants without conditions can
|
|
567
|
+
describe or change an aggregate. Only someone who reads all rows of a class and its descendants without conditions,
|
|
568
|
+
both before the change and after it, can make the class a source (a new class, a `parent` change, a source described
|
|
569
|
+
after the target). Otherwise `acl_denied`.
|
|
570
|
+
- The option is inherited, and a descendant replaces it as a whole. Adding or changing an aggregate is a tightening:
|
|
571
|
+
writes to the target and to the sources wait while the field is recalculated for all rows.
|
|
572
|
+
- `db.rebuildAggregates(class)` recalculates the field from the current rows, for repair. A version is written only
|
|
573
|
+
where the value has changed. `db.verify({ data: true })` shows discrepancies. It needs an admin connection.
|
|
574
|
+
- A raw `INSERT … ON CONFLICT DO UPDATE SET data = excluded.data` for a target is rejected: the insert trigger has
|
|
575
|
+
already set the field in `excluded.data`. Write `excluded.data - array['balance']`, and the field stays as it was;
|
|
576
|
+
the library's `upsert` does this itself.
|
|
474
577
|
|
|
475
578
|
### Writing the model to the database
|
|
476
579
|
|
|
@@ -612,6 +715,13 @@ without reads or locks in the application. Rules:
|
|
|
612
715
|
- an `inc` marker cannot come from JSON: `{ "$inc": 5 }` that came from the network stays data;
|
|
613
716
|
- store money as an integer number of minor units (cents), no greater than 2^53 − 1.
|
|
614
717
|
|
|
718
|
+
A counter that must match the source even for raw SQL is easier to keep as an aggregate
|
|
719
|
+
([Aggregates in the database](#aggregates-in-the-database)): the database recalculates it itself, and you cannot forget
|
|
720
|
+
an `inc`. An aggregate and `inc` cost the same: operations on one object go one after another under the lock of its
|
|
721
|
+
row, and each writes its version. On the benchmark machine, one object took 150–190 operations per second this way, no
|
|
722
|
+
matter how many writers were waiting for it ([section 17](#17-performance)). For hotter counters, use a projector that
|
|
723
|
+
folds a batch ([Projectors](#projectors)): the total lags behind the source, but writers do not wait for each other.
|
|
724
|
+
|
|
615
725
|
### Protection against concurrent edits: `{ rev }`
|
|
616
726
|
|
|
617
727
|
<!-- run: guide -->
|
|
@@ -642,9 +752,12 @@ const [vip] = await db.Customer(olga).reclass('Vip', { name: 'Olga', email: 'olg
|
|
|
642
752
|
- `rekey(patch)` sets a new key value: a row with a new id, all references moved, the old row deleted. If a referencing
|
|
643
753
|
row includes this end in its own key, its id changes too, and the move goes on. The journal links the old and the new
|
|
644
754
|
id (`moved`).
|
|
645
|
-
- `db.rekeyClass(class, key)` sets a new key for all rows of the family, in one transaction.
|
|
646
|
-
|
|
647
|
-
|
|
755
|
+
- `db.rekeyClass(class, key)` sets a new key for all rows of the family, in one transaction. For a derived class and
|
|
756
|
+
for a class with a derived descendant — `derived_class`: only its projector writes the rows of a derived class.
|
|
757
|
+
- `reclass(class, data)` changes the class; the data is optional, and if given, it is given whole. The row leaves its
|
|
758
|
+
previous class, so it needs the `DELETE` permission on it, and `WRITE` on the new class and on every row whose
|
|
759
|
+
references the move rewrites, at any depth. If the new class has no key, the id is kept (as for `vip` above); if
|
|
760
|
+
it has one, the id is computed anew and the references are moved.
|
|
648
761
|
The roles of referencing rows must accept the new class, otherwise `reclass_denied`. A raw `UPDATE … SET class`
|
|
649
762
|
passes only if neither the old nor the new class has a key (otherwise `use_reclass`).
|
|
650
763
|
- Deleting, changing the class and changing the key work only in `read committed`: library transactions always open at
|
|
@@ -796,7 +909,8 @@ const avgPrice = await db.Shop(shop).Product().avg('data.price')
|
|
|
796
909
|
const byStatus = await db.Order().countBy('data.status') // { new: 1 }
|
|
797
910
|
```
|
|
798
911
|
|
|
799
|
-
The database computes aggregates over the entities of the last step.
|
|
912
|
+
The database computes aggregates over the entities of the last step. `limit(n)` and `offset(n)` before `countBy()`
|
|
913
|
+
apply to the groups: `db.Order().limit(3).countBy('data.status')` gives the three most frequent values.
|
|
800
914
|
|
|
801
915
|
### Paths
|
|
802
916
|
|
|
@@ -863,8 +977,11 @@ await db.Customer(temp).purge({ confirm: true }).rows() // history erased
|
|
|
863
977
|
```
|
|
864
978
|
|
|
865
979
|
`purge` works only for deleted objects (a live one gives `not_deleted`) and erases their whole history; in place of
|
|
866
|
-
version 1, a `purge` record stays with the number of erased versions. The history of system classes is not erased.
|
|
867
|
-
|
|
980
|
+
version 1, a `purge` record stays with the number of erased versions. The history of system classes is not erased.
|
|
981
|
+
While a projector whose `from` sources include the class of the object ([“Projectors”](#projectors)) has not read
|
|
982
|
+
the deletion, `purge` refuses (`projector_pending`): the projector would never see an erased tombstone, and the
|
|
983
|
+
object's contribution would stay in the totals. So a deletion and a `purge` in one transaction do not go through in
|
|
984
|
+
such a class. To delete an account, use `db.auth.purgeAccount` ([section 10](#10-users-sign-in-and-permissions)).
|
|
868
985
|
|
|
869
986
|
## 8. Transactions, batches and money
|
|
870
987
|
|
|
@@ -1007,6 +1124,67 @@ try {
|
|
|
1007
1124
|
- A transfer is two operations in one `db.begin()`, with the wallets edited in id order: then there are no deadlocks.
|
|
1008
1125
|
- Insufficient funds are told apart from other data errors by `e.issues`: the field `balance`, the rule `minimum`.
|
|
1009
1126
|
|
|
1127
|
+
### Recipe: balance as an aggregate
|
|
1128
|
+
|
|
1129
|
+
The same wallet, but the database itself keeps the balance: `balance` is an aggregate, the sum of `amount` over the
|
|
1130
|
+
live operations ([Aggregates in the database](#aggregates-in-the-database)). An operation is one `Movement` record;
|
|
1131
|
+
there is no `inc` increment in the code, so it cannot be forgotten. The balance stays correct even when raw SQL or the
|
|
1132
|
+
loader writes the operations and a cascade deletes them.
|
|
1133
|
+
|
|
1134
|
+
```ts
|
|
1135
|
+
const Wallet = hub('Wallet', { number: z.string(), balance: z.number().int().min(0) }, {
|
|
1136
|
+
ends: { Customer }, key: ['number'],
|
|
1137
|
+
aggregate: { balance: { sum: 'amount', of: 'Movement', via: 'Wallet' } }, // the balance is the sum of live operations
|
|
1138
|
+
})
|
|
1139
|
+
const Movement = link('Movement', { opId: z.string(), amount: z.number().int() },
|
|
1140
|
+
{ ends: { Wallet }, key: ['Wallet', 'opId'] }) // the operation key gives idempotency
|
|
1141
|
+
|
|
1142
|
+
/** Operation: an amount above zero tops up, below zero debits; the database recalculates the balance in the same transaction. */
|
|
1143
|
+
async function operate(db: Db, wallet: string, opId: string, amount: number) {
|
|
1144
|
+
try {
|
|
1145
|
+
await db.Wallet(wallet).Movement().create({ opId, amount }).rows()
|
|
1146
|
+
return 'done'
|
|
1147
|
+
} catch (e) {
|
|
1148
|
+
if (e instanceof LetopisError && e.code === 'exists') return 'repeat' // a repeat after a lost response
|
|
1149
|
+
throw e // invalid_data: the balance would go below zero (min(0))
|
|
1150
|
+
}
|
|
1151
|
+
}
|
|
1152
|
+
|
|
1153
|
+
/** Adjustment "balance = target": an operation for the difference, with the wallet locked until commit. */
|
|
1154
|
+
async function adjust(db: Db, wallet: string, opId: string, target: number) {
|
|
1155
|
+
const tx = await db.begin()
|
|
1156
|
+
try {
|
|
1157
|
+
const w = (await tx.Wallet(wallet).forUpdate().first())!
|
|
1158
|
+
await tx.Wallet(wallet).Movement().create({ opId, amount: target - (w.data.balance as number) }).rows()
|
|
1159
|
+
await tx.commit()
|
|
1160
|
+
return 'done'
|
|
1161
|
+
} catch (e) {
|
|
1162
|
+
await tx.rollback()
|
|
1163
|
+
if (e instanceof LetopisError && e.code === 'exists') return 'repeat' // an adjustment with this opId is already written
|
|
1164
|
+
throw e
|
|
1165
|
+
}
|
|
1166
|
+
}
|
|
1167
|
+
|
|
1168
|
+
const [wallet] = await db.Customer(olga).Wallet().create({ number: 'W-1' }).rows() // balance is 0: the database sets it
|
|
1169
|
+
await operate(db, wallet.id, 'op-1', 10_000) // balance 10,000
|
|
1170
|
+
await operate(db, wallet.id, 'op-2', -3_000) // balance 7,000
|
|
1171
|
+
await adjust(db, wallet.id, 'op-3', 5_000) // an operation of −2,000, balance 5,000
|
|
1172
|
+
```
|
|
1173
|
+
|
|
1174
|
+
- The model here is its own, and the block is not part of the example program: there the balance grows through `inc`,
|
|
1175
|
+
and an aggregate field does not change that way.
|
|
1176
|
+
- `balance` is passed neither to `create` nor to `update` (`aggregate_field`), so an adjustment is an operation too,
|
|
1177
|
+
not an `update` under `{ rev }`. Writing an operation needs no permission to write the wallet: the permission to
|
|
1178
|
+
write `Movement` is enough.
|
|
1179
|
+
- Insufficient funds are `invalid_data`, as with `inc`: `e.issues` holds the field `balance` and the rule `minimum`.
|
|
1180
|
+
Check a repeat as `same()` in the recipe above does: `exists` only says that an operation with this `opId` already
|
|
1181
|
+
exists.
|
|
1182
|
+
- Editing and deleting an operation change the balance too: the database subtracts the previous value. It is still
|
|
1183
|
+
better to cancel an operation with a reverse one, so that the history of the wallet stays complete.
|
|
1184
|
+
- Every operation locks the wallet row and writes its version, so the operations of one wallet go one after another,
|
|
1185
|
+
as with `inc`: about 150 per second on the benchmark machine ([section 17](#17-performance)). If one object gets
|
|
1186
|
+
more, compute the total with a projector instead ([Recipe: daily revenue](#recipe-daily-revenue)).
|
|
1187
|
+
|
|
1010
1188
|
## 9. Bulk loading
|
|
1011
1189
|
|
|
1012
1190
|
### When you need the loader
|
|
@@ -1077,6 +1255,23 @@ regular write: the tenant is available to the user, write permission on each cla
|
|
|
1077
1255
|
the visibility of reference targets, the owner per `ownerDefault`. Without `tenant`, the rows go to the user's own
|
|
1078
1256
|
tenant. This is how a loading process can accept files sent by users.
|
|
1079
1257
|
|
|
1258
|
+
### Initial load
|
|
1259
|
+
|
|
1260
|
+
```ts
|
|
1261
|
+
const r = await adm.load(rows, { initial: true }) // empty schema: one transaction, indexes and triggers at the end
|
|
1262
|
+
```
|
|
1263
|
+
|
|
1264
|
+
`db.load(rows, { initial: true })` (`letopis load --initial`) loads data into an empty schema faster than the regular
|
|
1265
|
+
loader — about 2.2 times on a million objects: everything goes in one transaction, the tables are closed for reading and
|
|
1266
|
+
writing during the load, indexes, constraints and triggers are dropped and rebuilt at the end, and rows are written with
|
|
1267
|
+
`COPY … FREEZE`. Empty means that the schema has nothing besides the rows of system classes (class descriptions,
|
|
1268
|
+
accounts, memberships, resources, rules) and `reset` records; otherwise you get `not_empty` before any change. The mode
|
|
1269
|
+
is incompatible with `commitEvery`, `existing` (other than `'error'`) and `{ as }`; an error in any row rolls
|
|
1270
|
+
everything back, indexes and triggers included. `letopis import --initial` runs the whole import this way: in one
|
|
1271
|
+
transaction, without `--update` and `--commit-every`. WAL is written here too (it is skipped only with
|
|
1272
|
+
`wal_level = minimal`), but the indexes are built once, so there is almost three times less of it; after `FREEZE`, the
|
|
1273
|
+
first read and the first vacuum do not rewrite the table.
|
|
1274
|
+
|
|
1080
1275
|
### Command line
|
|
1081
1276
|
|
|
1082
1277
|
```sh
|
|
@@ -1097,7 +1292,8 @@ problem. Its `detail`: `rows` is the first 100 violations with the row number, c
|
|
|
1097
1292
|
`limit` is the list limit (100), `complete` means the checks that found violations went through all rows. Row codes:
|
|
1098
1293
|
the codes of the database checks, and also `duplicate` (a repeated id), `exists` (the object already exists), `deleted`
|
|
1099
1294
|
(an object with this id existed before) and `class_not_loadable` (the loader does not accept `Class` rows: classes are
|
|
1100
|
-
set by `syncModels`).
|
|
1295
|
+
set by `syncModels`). The loader rejects rows of derived classes and references to them with the code
|
|
1296
|
+
`derived_class`: only a projector writes such rows.
|
|
1101
1297
|
|
|
1102
1298
|
Violations are searched for in stages: parsing the rows (classes, ids, repeats), permissions on classes
|
|
1103
1299
|
(`load({ as })`), data and ends in batches, and at the end of a portion the targets outside the load, owners and the
|
|
@@ -1257,6 +1453,26 @@ cut down to 999, so System rules with a weight of 1000 or more are an administra
|
|
|
1257
1453
|
outweigh. Tenant rules apply to the tenant's rows, System rules apply everywhere. Permissions apply to raw SQL too: rows
|
|
1258
1454
|
without permission are invisible, and an edit changes nothing or fails with `acl_denied`.
|
|
1259
1455
|
|
|
1456
|
+
A class mask in a `READ`, `WRITE` or `DELETE` object matches the descendants of the class too, by the names of the
|
|
1457
|
+
ancestors and their aliases. So an edit of a class that changes `parent`, `alias` or `derivedBy` needs the `READ`,
|
|
1458
|
+
`WRITE` and `DELETE` permissions without conditions on all rows of the class and of each of its descendants,
|
|
1459
|
+
otherwise `acl_denied`: the `WRITE` permission on `Class` alone is not enough to put the rows of a class under other
|
|
1460
|
+
rules. An alias does not match the name of another class (including `Class` and the system ones) or another class's
|
|
1461
|
+
alias in the tenant and System, and the name of a new class does not match another class's alias: otherwise a rule
|
|
1462
|
+
written for one class would act on the rows of another too. A violation gives `invalid_class`.
|
|
1463
|
+
|
|
1464
|
+
By the same names, the rules also find the old versions of rows in the journal. So a class is deleted only when it has
|
|
1465
|
+
no rows, no descendants and no history of rows: `purge()` erases the history of deleted rows first. A class deleted
|
|
1466
|
+
with history before is created again only with the same `parent`, `alias` and `derivedBy` — otherwise `invalid_class`:
|
|
1467
|
+
the old versions of its rows would fall under other rules. If the journal holds rows of a class with this name but not
|
|
1468
|
+
its tombstone (for example, rows of a System class deleted in 1.1), a class with this name is not created until
|
|
1469
|
+
`purge()` erases the history.
|
|
1470
|
+
|
|
1471
|
+
A derived class (`derivedBy`) is created, edited and deleted only by an administrator: an admin connection or an
|
|
1472
|
+
account with the System permission `letopis.projector` (the system administrator has it through the `letopis.*`
|
|
1473
|
+
mask). The projector writes totals over all source rows of the tenant into such a class, without read permissions,
|
|
1474
|
+
so a user who created it on their own would read totals over rows they may not read.
|
|
1475
|
+
|
|
1260
1476
|
### Granting a permission
|
|
1261
1477
|
|
|
1262
1478
|
Example: Oleg is a shop employee with the `staff` role, and employees may not delete products.
|
|
@@ -1324,8 +1540,14 @@ const followed = await db.Product(mug2).versions({ follow: true }) // also acros
|
|
|
1324
1540
|
```
|
|
1325
1541
|
|
|
1326
1542
|
Every version has an author, an agent, a time, a number, a kind of operation (`create`, `update`, `delete`, `reclass`,
|
|
1327
|
-
`rekey`, `restore`, `purge`, `trim`, `reset`) and a reason (`reason`)
|
|
1328
|
-
|
|
1543
|
+
`rekey`, `restore`, `purge`, `trim`, `reset`) and a reason (`reason`). The reason comes from the `reason` option of a verb
|
|
1544
|
+
(`update(patch, { reason })`, `create(data, { reason })`, `delete({ confirm, reason })` and others); for every
|
|
1545
|
+
operation of a block, from `db.begin({ reason })` and `db.batch(name, { reason })`; in raw SQL, from the
|
|
1546
|
+
`letopis.reason` setting of the transaction. A deletion version has `$deleted`.
|
|
1547
|
+
|
|
1548
|
+
Without `sort()`, versions go by id and version number (with `follow`, by time). `sort()` sets the order of versions
|
|
1549
|
+
too, and `limit()` limits them in the query itself: `db.Product(kettle).sort('at', 'desc').limit(5).versions()` gives
|
|
1550
|
+
the five latest versions, newest first.
|
|
1329
1551
|
|
|
1330
1552
|
### Point-in-time read
|
|
1331
1553
|
|
|
@@ -1371,6 +1593,75 @@ period boundaries are in the `tz` time zone (UTC by default).
|
|
|
1371
1593
|
- In the thinned part, a point-in-time read shows the last kept version before the moment, so its precision equals the
|
|
1372
1594
|
period of the tier. A subscriber that lags behind by more than the `all` tier gets `cursor_expired`.
|
|
1373
1595
|
|
|
1596
|
+
### What changed and who: `diff` and `blame`
|
|
1597
|
+
|
|
1598
|
+
```ts
|
|
1599
|
+
const [d] = await db.Order(id).diff() // previous version → latest
|
|
1600
|
+
await db.Order(id).diff({ from: 3, to: 7 }) // by version numbers
|
|
1601
|
+
await db.Order(id).diff({ from: '2026-10-01' }) // state at a moment → latest
|
|
1602
|
+
await db.Order(id).diff({ follow: true }) // across id changes (reclass, rekey)
|
|
1603
|
+
const [b] = await db.Order(id).blame() // every field of the latest version
|
|
1604
|
+
await db.Order(id).blame({ at: '2026-10-01', limit: 200 }) // the version at a moment; scan 200 versions
|
|
1605
|
+
```
|
|
1606
|
+
|
|
1607
|
+
Both terminals return an array, one item per object of the step. This is what the answers look like for a kettle
|
|
1608
|
+
whose price was lowered with a reason:
|
|
1609
|
+
|
|
1610
|
+
```ts
|
|
1611
|
+
import type { BlameResult, DiffResult } from 'letopis'
|
|
1612
|
+
|
|
1613
|
+
// db.Product(kettle).blame(): version 2 changed the price, the other leaves date from the creation (version 1)
|
|
1614
|
+
const blamed: BlameResult = {
|
|
1615
|
+
id: '6f1c…', class: 'Product', rev: 2, complete: true,
|
|
1616
|
+
fields: [ // in path order
|
|
1617
|
+
{ path: ['class'], rev: 1, at: '2026-10-27T09:15:02.118204+00:00', author: '<Anna id>', agent: '<service id>',
|
|
1618
|
+
op: 'create', exact: true },
|
|
1619
|
+
{ path: ['data', 'price'], rev: 2, at: '2026-10-27T09:20:41.502617+00:00', author: '<Anna id>', agent: '<service id>',
|
|
1620
|
+
op: 'update', reason: 'promotion discount', exact: true },
|
|
1621
|
+
// ['data', 'sku'], ['data', 'title'], ['links', 'Shop'], ['owner'] — also version 1, like ['class']
|
|
1622
|
+
],
|
|
1623
|
+
}
|
|
1624
|
+
// db.Product(kettle).diff(): previous version → latest
|
|
1625
|
+
const diffed: DiffResult = {
|
|
1626
|
+
id: '6f1c…', class: 'Product',
|
|
1627
|
+
from: { rev: 1, at: '2026-10-27T09:15:02.118204+00:00' },
|
|
1628
|
+
to: { rev: 2, at: '2026-10-27T09:20:41.502617+00:00' },
|
|
1629
|
+
changes: [{ path: ['data', 'price'], kind: 'changed', before: 2500, after: 2000 }],
|
|
1630
|
+
}
|
|
1631
|
+
```
|
|
1632
|
+
|
|
1633
|
+
`agent` here is the service that writes on behalf of Anna (`db = await svc.as(…)`); a write without impersonation has
|
|
1634
|
+
none.
|
|
1635
|
+
|
|
1636
|
+
- `diff()` returns an array of `DiffResult`: `{ id, class, from, to, changes }`. `from` and `to` are the compared
|
|
1637
|
+
versions `{ rev, at }`. They have `id` if the version belongs to another id (`follow: true` across `reclass` or
|
|
1638
|
+
`rekey`), and `to` has `deleted: true` if the object was deleted at that moment; then `changes` is empty. `from` is
|
|
1639
|
+
`null` if the object did not exist yet: every path is `added`. If there is no version at all, `class` and `to` are
|
|
1640
|
+
`null` too.
|
|
1641
|
+
- A change, `DiffChange`, is `{ path, kind, before, after }`. `path` is an array of segments, not a dotted string;
|
|
1642
|
+
the first segment is the section of the row: `data`, `links`, `tags`, `owner`, `class`. `kind` is `added` (there
|
|
1643
|
+
was no value), `removed` (the value is gone) or `changed`. A missing value is omitted: `added` has no `before`,
|
|
1644
|
+
`removed` has no `after`. Data objects are compared by keys, arrays as a whole. A multi-valued end and tags are
|
|
1645
|
+
sets: one change per id or tag, the path is `['links', role]` or `['tags']`, and the id or tag itself is in
|
|
1646
|
+
`before` or `after`.
|
|
1647
|
+
- `blame()` returns an array of `BlameResult`: `{ id, class, rev, complete, fields }`, plus `deleted: true` if that
|
|
1648
|
+
version is a tombstone. An object with no versions at that moment gives no item. `fields` is an array of
|
|
1649
|
+
`BlameField`, one item per leaf of the version, ordered by path: `{ path, rev, at, author, op, exact }` and, when
|
|
1650
|
+
present, `agent`, `reason`, `id`. A leaf is a scalar or an array in the data, an empty object, the role of a
|
|
1651
|
+
single-valued end, every id of a multi-valued end (`['links', role, id]`), every tag (`['tags', tag]`), `owner` and
|
|
1652
|
+
`class`. `rev`, `at`, `author`, `agent`, `op` and `reason` come from the version that changed the leaf last. Empty
|
|
1653
|
+
`agent` and `reason` are omitted; `id` is present only if it is a version of the former id (the history went
|
|
1654
|
+
through `reclass` or `rekey`).
|
|
1655
|
+
- `exact: false` means there are gaps between the current version and the found one (the history is thinned,
|
|
1656
|
+
trimmed or partly invisible): a missing version may have changed the leaf too. `complete: true` means the whole
|
|
1657
|
+
history was scanned without gaps; by default at most the 1000 latest versions are scanned.
|
|
1658
|
+
- The database computes both with the caller's rights: raw SQL gets the same, `select "v2.shop".blame('<id>')`,
|
|
1659
|
+
`select "v2.shop".diff_versions('<id>')`.
|
|
1660
|
+
- The journal column `changed` (the paths changed by a version, which make `blame` fast) is not granted to the
|
|
1661
|
+
application: the paths describe the difference from the previous version, and the reader may not see that one.
|
|
1662
|
+
So `select * from log` does not work for the application — list the columns; `blame` gets the paths through a
|
|
1663
|
+
database function only when both versions are visible.
|
|
1664
|
+
|
|
1374
1665
|
### Trimming history
|
|
1375
1666
|
|
|
1376
1667
|
<!-- run: guide -->
|
|
@@ -1413,8 +1704,10 @@ for await (const ev of sub) { // ev: { cursor, row, id, class, op, r
|
|
|
1413
1704
|
|
|
1414
1705
|
- `from` is `'now'` (the default: only changes after subscribing), `'start'` (the whole journal) or a saved cursor.
|
|
1415
1706
|
`db.watch()` returns at once, and the subscription takes the "now" moment with its first query to the database: a
|
|
1416
|
-
write made right after `db.watch()` may commit earlier and miss the stream.
|
|
1417
|
-
`'start'`.
|
|
1707
|
+
write made right after `db.watch()` may commit earlier and miss the stream. `await sub.ready()` waits for that
|
|
1708
|
+
moment: a write after it comes to the stream. The example above reads from `'start'`.
|
|
1709
|
+
- Calls of `sub.next()` go in turn: parallel calls get different events. A call abandoned by a timeout
|
|
1710
|
+
(`Promise.race`) still gets its event — keep its promise and wait for it again.
|
|
1418
1711
|
- `classes`: only these classes with their descendants; `pollMs`: fallback polling (1000 ms).
|
|
1419
1712
|
- Events come without losses or repeats, under the reader's permissions, in the order of transaction numbers, and
|
|
1420
1713
|
within a transaction in the order of writing. This is not the commit order: compare versions of one object by `rev`.
|
|
@@ -1425,6 +1718,210 @@ for await (const ev of sub) { // ev: { cursor, row, id, class, op, r
|
|
|
1425
1718
|
- A subscription without losses is a saved cursor: store `ev.cursor` of the last processed event and resume from it
|
|
1426
1719
|
after a restart.
|
|
1427
1720
|
|
|
1721
|
+
### Projectors
|
|
1722
|
+
|
|
1723
|
+
A projector builds rows of derived classes from the journal: summaries, denormalized cards and anything else that may
|
|
1724
|
+
be computed with a lag behind the writes. The application writes the source classes (`from`), and only the projector
|
|
1725
|
+
writes the rows of the `into` classes. Each `into` class is marked with the option `derivedBy` holding the projector
|
|
1726
|
+
name ([Class settings](#class-settings)). Only an administrator creates derived classes: sync them with an admin
|
|
1727
|
+
connection holding a tenant session — `adm` from section 3 (`connect({ dsn: ADMIN, schema, token, allowBypassRls:
|
|
1728
|
+
true })`) or `as()` of a service on such a connection; in the CLI, `letopis sync` with the `--dsn` of the admin role.
|
|
1729
|
+
For an application session it is `acl_denied`. In the example below, the projector counts product sales from the
|
|
1730
|
+
order lines `line`.
|
|
1731
|
+
|
|
1732
|
+
```ts
|
|
1733
|
+
const ProductSales = hub('ProductSales', { qty: z.number().int(), revenue: z.number().int() }, {
|
|
1734
|
+
ends: { Product: one(Product).onDelete('cascade') }, key: ['Product'], derivedBy: 'product-sales', history: false,
|
|
1735
|
+
}) // the derived class is in the model; syncModels — on an admin connection
|
|
1736
|
+
|
|
1737
|
+
const sales = adm.projector('product-sales', { // adm is an admin connection
|
|
1738
|
+
from: ['line'], // source classes with descendants; they must keep a journal
|
|
1739
|
+
into: ['ProductSales'], // the derived classes of this projector
|
|
1740
|
+
version: 1, // the handle logic changed: a new version and a rebuild
|
|
1741
|
+
withBefore: true, // c.before is the previous version of the object or null
|
|
1742
|
+
handle(changes, out) {
|
|
1743
|
+
for (const c of changes) {
|
|
1744
|
+
if (c.op === 'purge' || c.op === 'trim' || c.op === 'reset') continue // service changes, not versions
|
|
1745
|
+
const was = c.before // the previous version of the order line: subtract it
|
|
1746
|
+
const now = c.row.$deleted ? null : c.row // the new version: add it; a deletion has none
|
|
1747
|
+
if (was) out.inc('ProductSales', { Product: was.links.Product },
|
|
1748
|
+
{ qty: -Number(was.data.qty), revenue: -Number(was.data.qty) * Number(was.data.price) }, { tenant: c.tenant })
|
|
1749
|
+
if (now) out.inc('ProductSales', { Product: now.links.Product },
|
|
1750
|
+
{ qty: Number(now.data.qty), revenue: Number(now.data.qty) * Number(now.data.price) }, { tenant: c.tenant })
|
|
1751
|
+
}
|
|
1752
|
+
},
|
|
1753
|
+
})
|
|
1754
|
+
const report = await sales.run({ once: true }) // catch up with the journal: { batches, changes, created, updated, … }
|
|
1755
|
+
const state = await sales.status() // { name, state, version, cursor, at, lag, failed, rebuildingTenants }
|
|
1756
|
+
await sales.rebuild() // rebuild from the current state; { chunked: true } works in batches
|
|
1757
|
+
await sales.rebuild({ tenant: tenantId }) // rebuild one tenant, for example a dropped one (failed)
|
|
1758
|
+
await sales.rebuild({ tenant: tenantId, chunked: true }) // the same in batches: meanwhile the journal tail goes on for all
|
|
1759
|
+
sales.run().catch((err) => console.error('the projector stopped', err)) // follow the journal until stop()
|
|
1760
|
+
sales.close() // stop() after the current batch and unsubscribe from signals
|
|
1761
|
+
```
|
|
1762
|
+
|
|
1763
|
+
- A projector needs an admin connection: `connect({ allowBypassRls: true })` with a role that may become the owner of
|
|
1764
|
+
the schema. Otherwise you get `admin_required`, and inside `db.begin()` you get `invalid_query`. The projector reads
|
|
1765
|
+
the journal of all tenants.
|
|
1766
|
+
- The first `run()` rebuilds the projection from the current state: the journal is thinned by the retention policy
|
|
1767
|
+
and cut by `purge`, so the projection cannot be assembled from the start of the journal.
|
|
1768
|
+
- Each batch is one transaction: the journal changes, `handle`, the written results and the cursor move are committed
|
|
1769
|
+
together. A break in the middle of a batch rolls the whole batch back, and the retry starts from the same cursor, so
|
|
1770
|
+
every change is applied exactly once. A projector has one writer: a second process of the same projector skips the
|
|
1771
|
+
batch (the `skipped` counter of the report), and `rebuild()` started while another process writes a batch gives
|
|
1772
|
+
`projector_busy`.
|
|
1773
|
+
- A batch is split by tenant: `handle` gets the changes of one tenant, and its results are written under a savepoint
|
|
1774
|
+
of its own. If the database rejects writing the results of a tenant (the data do not pass the description of its
|
|
1775
|
+
derived class, the target of an end is not found and so on) or `out` rejects a row or a delta built from its data
|
|
1776
|
+
(a delta that is not a number), only its changes are rolled back: the tenant drops out, and the others are written
|
|
1777
|
+
in the same batch. A dropped tenant gets into `status().failed` — `{ [tenant id]: { cursor, code, message } }`, and
|
|
1778
|
+
`message` names the tenant and the row id — and its changes are skipped until it is rebuilt. The derived rows of a
|
|
1779
|
+
dropped tenant no longer change: a tenant dropped in `run()` keeps them as they were before the failed batch, and a
|
|
1780
|
+
tenant dropped in the middle of a rebuild is left without them — the partly built rows are deleted. An exception
|
|
1781
|
+
from `handle`, a breach of the `out` limits and a failure of the environment (a deadlock, a cancelled query, a
|
|
1782
|
+
broken connection) roll the whole batch back, and `run()` is rejected. So catch `run()` without `once`: after an
|
|
1783
|
+
error it stops.
|
|
1784
|
+
- A change (`ProjectorChange`) is `cursor`, `id`, `class`, `tenant`, `op`, `rev`, `at`, `author`, `agent`, `reason`,
|
|
1785
|
+
`moved` and `row`, the version from the journal (for a deletion, a tombstone with `$deleted`). During a rebuild, the
|
|
1786
|
+
changes are rows of the current state: `rebuild: true`, `op` is `create`, `cursor` is empty.
|
|
1787
|
+
- A class change that keeps the id (classes without a key: a raw `UPDATE … SET class` and `reclass()`) is written to
|
|
1788
|
+
the journal as a version of the new class. If it takes the row out of the `from` family, the projector gets a
|
|
1789
|
+
deletion: `op` is `delete`, `row` is the last version of the row in the family with `$deleted`, and `rev`, `at` and
|
|
1790
|
+
the author are those of the class change. A row that enters the family this way comes as a `reclass` change with
|
|
1791
|
+
`before` = `null`.
|
|
1792
|
+
- With `withBefore`, a change has `before`, the previous version of the same id. `before` is `null` if there is no
|
|
1793
|
+
previous version (a creation and the first version of a new id after `reclass` and `rekey`), if the previous
|
|
1794
|
+
version is a deletion (this happens after `restore`) or lies outside the `from` family, and for every change of a
|
|
1795
|
+
rebuild. With `reclass` and `rekey`, the former data are subtracted through the tombstone of the former id: it comes
|
|
1796
|
+
as a change with `$deleted`, its `before` is the last version of the former id, and `moved` of both changes points
|
|
1797
|
+
to the other id. So the fold of the example gives the same sum after `rekey`. The service changes `purge`, `trim`
|
|
1798
|
+
and `reset` have no `before` field at all. So subtract `before` only after checking it for `null`, as in the example
|
|
1799
|
+
above.
|
|
1800
|
+
- The service journal records `purge`, `trim` and `reset` also come as changes, and their `rev` is 1. So check `op`
|
|
1801
|
+
first, and only then compare `rev`.
|
|
1802
|
+
- `out.upsert(class, row, { tenant })` creates a row or replaces it entirely by the class key.
|
|
1803
|
+
`out.inc(class, row, deltas, { tenant })` creates a row or completes it and adds the numbers without reading the row
|
|
1804
|
+
(missing fields count from 0). `out.delete(class, key or id, { tenant })` deletes a row and does nothing if there is
|
|
1805
|
+
none. `await out.get(class, key or id, { tenant })` reads the current row together with the writes of the batch.
|
|
1806
|
+
`out` writes only to the `into` classes and only to the tenant of the changes passed to `handle`; otherwise you get
|
|
1807
|
+
`derived_class` or `tenant_denied`, and the batch is rolled back.
|
|
1808
|
+
- `out` adds up the increments of a row given by key fields only (or by `$id` alone) before sending them to the
|
|
1809
|
+
database, as long as there is no `upsert`, `delete` or `get` between them: one edit of a row per batch, and the class
|
|
1810
|
+
rule is checked on the total of its increments. `inc` does not fold a row with other fields (data, `$tags`,
|
|
1811
|
+
`$owner`): its fields edit the row, and the order of such edits is kept.
|
|
1812
|
+
- A row for `out` is in the loader format: fields and end roles at the top level, the service keys `$tags`, `$owner`
|
|
1813
|
+
and `$id` (`$id` is needed only by a class without a key). The value of a role is the id of the target. An edit that
|
|
1814
|
+
changes nothing gives no version. Versions of derived rows are written by System with the agent
|
|
1815
|
+
`letopis projector <name>`; the journal is kept according to the `history` of the class, and the change signal is
|
|
1816
|
+
always sent.
|
|
1817
|
+
- `rebuild()` rebuilds the projection atomically, in one transaction: until it commits, readers see the previous rows,
|
|
1818
|
+
but the subscription horizon stands still. `rebuild({ chunked: true })` works in batches: the derived rows are
|
|
1819
|
+
deleted at once, the data stays incomplete until the rebuild ends, and `status().state` is `'rebuilding'`. The next
|
|
1820
|
+
start continues an interrupted batch rebuild. `run()` itself, on the first start of a new projection and with
|
|
1821
|
+
`onExpired: 'rebuild'`, rebuilds in batches: the subscription horizon of the cluster stands still no longer than a
|
|
1822
|
+
batch. Meanwhile a second process of the same projector waits or continues the same rebuild instead of starting it
|
|
1823
|
+
anew. A rebuild deletes the rows of the participants only, and only of the classes of their `into` families, and
|
|
1824
|
+
starts all participants anew: the marks of dropped tenants are removed, and a tenant whose data fail again drops
|
|
1825
|
+
out again.
|
|
1826
|
+
- `rebuild({ tenant: id })` rebuilds one tenant in one transaction: it deletes its derived rows, builds them from its
|
|
1827
|
+
current source rows and removes its mark in `status().failed`; the projector does not apply again the changes the
|
|
1828
|
+
rebuild has already counted. Fix the data or the class description of the tenant, then rebuild it. If the database
|
|
1829
|
+
rejects the write again, the rebuild is rolled back with this error, and the tenant stays dropped. No projection yet
|
|
1830
|
+
or a batch rebuild in progress gives `rebuild_required`; a tenant that is not a participant gives `tenant_denied`.
|
|
1831
|
+
With `chunked: true` — in batches, each in a transaction of its own with its own snapshot, while the journal tail goes
|
|
1832
|
+
on for all: the tail skips the tenant's changes above the last batch (a batch will read them) and, in a passed batch,
|
|
1833
|
+
those whose transaction is visible in its snapshot. The tenant's derived rows are deleted at once and are incomplete
|
|
1834
|
+
until the end of the rebuild (`status().rebuildingTenants`); the rebuild waits out a lock held by `run()` of another
|
|
1835
|
+
process.
|
|
1836
|
+
- `status().state` describes the projection in the database, not a process: `'new'` means there is no projection yet,
|
|
1837
|
+
and the first `run()` or `rebuild()` builds it; `'running'` means the projection is built and kept up with the
|
|
1838
|
+
journal (it does not mean that some process is executing `run()` right now); `'rebuilding'` means a batch rebuild
|
|
1839
|
+
is in progress or was interrupted.
|
|
1840
|
+
- `version` is the version of the `handle` logic. If the projection was built by another version, `run()` throws
|
|
1841
|
+
`rebuild_required`, and a rebuild with `rebuild()` is needed.
|
|
1842
|
+
- If the cursor is older than the `all` tier of the retention policy of the sources and there are changes after it,
|
|
1843
|
+
some changes may have been thinned out. Each tenant has a tier of its own: it is the description of its classes. If
|
|
1844
|
+
the cursor is older than the tier of every participant, `run()` throws `cursor_expired`, and with
|
|
1845
|
+
`onExpired: 'rebuild'` the projector rebuilds entirely, in batches. If it is older than the tier of only some of
|
|
1846
|
+
them, they drop out (`status().failed`, code `cursor_expired`) and the others go on, and with
|
|
1847
|
+
`onExpired: 'rebuild'` the dropped tenants rebuild themselves, in batches: the rebuild state lies in the checkpoint,
|
|
1848
|
+
and `run()` of any process continues it; a lock held by another process is only a retry on the next pass, and a data
|
|
1849
|
+
error drops the tenant with its code, without retries. The tier of a rebuilt tenant counts from the start of its
|
|
1850
|
+
rebuild until the cursor catches up with it: a lagging projector does not drop it again. A projector without new
|
|
1851
|
+
changes does not expire. `status().lag` is the age of the oldest unprocessed change in ms.
|
|
1852
|
+
- A projector works with the tenants in which every `into` class is marked with its `derivedBy`; it skips the changes
|
|
1853
|
+
of other tenants, so a class with the same name and no mark in another tenant does not get in its way. No such tenant
|
|
1854
|
+
at all — `derived_class`. Only an administrator may mark the classes of a tenant, so it is the administrator who
|
|
1855
|
+
chooses the participants: a tenant user cannot join a projector on their own. The option `tenants: [id, …]`
|
|
1856
|
+
is an explicit list of participants: the projector works only with these tenants (their `into` classes must be
|
|
1857
|
+
marked too) and ignores the marks of the others. The sources are the `from` families of these tenants; a class
|
|
1858
|
+
cannot be a source and a result at the same time. A projector does not see the changes of a class with
|
|
1859
|
+
`history: false`: it has no journal. So a tenant with such a class among the sources drops out with the code
|
|
1860
|
+
`invalid_class`, and the projector does not build its derived rows. Turn the journal of the class on and rebuild the
|
|
1861
|
+
tenant with `rebuild({ tenant })`.
|
|
1862
|
+
- `batch` is how many changes go into a batch (500), `pollMs` is fallback polling (1000 ms): source signals wake the
|
|
1863
|
+
projector earlier.
|
|
1864
|
+
- Every edit that `out` sends to the database is a separate edit of a row, and the speed of a projector is set by the
|
|
1865
|
+
number of edits, not by the number of changes. The folding of `inc` brings the edits of a row down to one per batch,
|
|
1866
|
+
so the handle from the example above (`out.inc` for every change) caught up with 6 thousand changes per second with
|
|
1867
|
+
`batch` 500 and 33 thousand with `batch` 2000 on the benchmark machine ([section 17](#17-performance)): the larger
|
|
1868
|
+
the batch, the more increments of one row fold into one edit.
|
|
1869
|
+
|
|
1870
|
+
### Recipe: daily revenue
|
|
1871
|
+
|
|
1872
|
+
An aggregate in the database and a projector solve similar tasks in different ways:
|
|
1873
|
+
|
|
1874
|
+
| | Aggregate in the database | Projector |
|
|
1875
|
+
|---|---|---|
|
|
1876
|
+
| when it is computed | in the same transaction as the write of the source | after the commit, with a lag (`status().lag`) |
|
|
1877
|
+
| what it computes | the sum of a field or the number of source rows for one target row | anything: summaries by day and by group, denormalized cards |
|
|
1878
|
+
| matches the source | always, raw SQL included | when the projector has caught up with the journal |
|
|
1879
|
+
| cost of a write | every operation locks the target row and writes its version, so the writers of one target queue up | writers do not wait for each other: the projector applies changes in batches after the commit |
|
|
1880
|
+
| rule over the total | `min(0)` rejects an operation that would take the total beyond the rule | the total does not block writes: a data error drops the tenant out of the projection |
|
|
1881
|
+
| what to run | nothing: the database keeps the field | a process on an admin connection |
|
|
1882
|
+
|
|
1883
|
+
A wallet balance that must not go below zero is an aggregate
|
|
1884
|
+
([Recipe: balance as an aggregate](#recipe-balance-as-an-aggregate)). The revenue of a shop by day is a total over many
|
|
1885
|
+
sales that may lag a little, which is a job for a projector:
|
|
1886
|
+
|
|
1887
|
+
```ts
|
|
1888
|
+
const Sale = hub('Sale', { day: z.iso.date(), amount: z.number().int().min(0) }, { ends: { Shop } }) // a sale: day and amount
|
|
1889
|
+
const DayRevenue = hub('DayRevenue', { day: z.iso.date(), revenue: z.number().int(), sales: z.number().int() }, {
|
|
1890
|
+
ends: { Shop: one(Shop).onDelete('cascade') }, key: ['Shop', 'day'], derivedBy: 'day-revenue', history: false,
|
|
1891
|
+
})
|
|
1892
|
+
await syncModels(db, [Sale], { apply: true })
|
|
1893
|
+
await syncModels(adm, [DayRevenue], { apply: true }) // the derived class is created by an administrator: adm holds a tenant session
|
|
1894
|
+
|
|
1895
|
+
const revenue = adm.projector('day-revenue', {
|
|
1896
|
+
from: ['Sale'], into: ['DayRevenue'], version: 1, withBefore: true,
|
|
1897
|
+
handle(changes, out) {
|
|
1898
|
+
for (const c of changes) {
|
|
1899
|
+
if (c.op === 'purge' || c.op === 'trim' || c.op === 'reset') continue // service changes
|
|
1900
|
+
const was = c.before // the previous version of the sale: subtract it from its day
|
|
1901
|
+
const now = c.row.$deleted ? null : c.row // the new one: add it to its day; a deletion has none
|
|
1902
|
+
if (was) out.inc('DayRevenue', { Shop: was.links.Shop, day: was.data.day },
|
|
1903
|
+
{ revenue: -Number(was.data.amount), sales: -1 }, { tenant: c.tenant })
|
|
1904
|
+
if (now) out.inc('DayRevenue', { Shop: now.links.Shop, day: now.data.day },
|
|
1905
|
+
{ revenue: Number(now.data.amount), sales: 1 }, { tenant: c.tenant })
|
|
1906
|
+
}
|
|
1907
|
+
},
|
|
1908
|
+
})
|
|
1909
|
+
revenue.run().catch((err) => console.error('the revenue projector stopped', err)) // follows the journal until stop()
|
|
1910
|
+
const week = await db.Shop(shop).DayRevenue().sort('data.day', 'desc').limit(7).rows() // the last seven days
|
|
1911
|
+
```
|
|
1912
|
+
|
|
1913
|
+
- The day is a field of the sale, not the version time `at`: with the version time, an edit, `restore` or `rekey`
|
|
1914
|
+
would move the amount of the sale to its own day.
|
|
1915
|
+
- `out.inc` adds without reading the row, so the total does not depend on the order of changes: the journal gives
|
|
1916
|
+
transactions in the order of their first write, not of their commit. Editing a sale subtracts the previous amount
|
|
1917
|
+
from the previous day and adds the new one to the new day; a deletion only subtracts. A day with no sales left stays
|
|
1918
|
+
as a row of zeros. `out` folds the increments of one day in a batch into one edit.
|
|
1919
|
+
- `DayRevenue` has `history: false`: the derived rows do not double the journal, and a rebuild assembles them without
|
|
1920
|
+
history anyway. `Sale` needs a journal: a projector does not see the changes of a class with `history: false`.
|
|
1921
|
+
- The application reads `DayRevenue` rows like any class, and only the projector writes them: `create` or `update` of
|
|
1922
|
+
such a row gives `derived_class`. The total lags behind the sales until the next batch: write signals wake the
|
|
1923
|
+
projector, and without them it polls every `pollMs`.
|
|
1924
|
+
|
|
1428
1925
|
### Result cache
|
|
1429
1926
|
|
|
1430
1927
|
<!-- run: guide -->
|
|
@@ -1439,11 +1936,21 @@ stays correct:
|
|
|
1439
1936
|
- the result key includes the query, the parameters, the session, the actor and the tenant, so one user never gets the
|
|
1440
1937
|
result of another user;
|
|
1441
1938
|
- your own writes clear the results of the changed classes right after commit (including the cascade classes); other
|
|
1442
|
-
writers' changes clear them on a database signal
|
|
1939
|
+
writers' changes clear them on a database signal. In the `tenant` and `opaque` modes, a process with the signal key
|
|
1940
|
+
clears a class only in the tenant of the write (System rows — in all tenants); a process without the key clears it
|
|
1941
|
+
in all tenants, and in the `opaque` mode any signal clears its whole cache;
|
|
1443
1942
|
- changes of permissions and classes, writes through `db.sql`, a revoked session and a listener reconnect clear the
|
|
1444
1943
|
whole cache or the results of the session;
|
|
1445
1944
|
- the default term is 60 seconds; the cache does not work inside `db.begin()`;
|
|
1446
|
-
- without `LISTEN` the cache is off, and `connect({ cache: { ttlOnly: true } })` enables it by
|
|
1945
|
+
- without `LISTEN` and in the `none` mode the cache is off, and `connect({ cache: { ttlOnly: true } })` enables it by
|
|
1946
|
+
term only.
|
|
1947
|
+
|
|
1948
|
+
A process re-reads the class registry right away on a class change signal (one re-read at a time), and the registry
|
|
1949
|
+
and the signal key on `*`, on an unknown token and on a payload of another mode (and without the key in the `opaque`
|
|
1950
|
+
mode — on any signal) not more often than once in 5 seconds: a stream of signals, fake ones included, does not turn
|
|
1951
|
+
into a stream of queries. In the `none` mode, the registry checks the class
|
|
1952
|
+
fingerprint every 5 seconds, and subscriptions poll the log. A subscription wakes up only for its own classes and its
|
|
1953
|
+
own tenant, and without the key — for any signal; it always reads the events from the log, so nothing is lost.
|
|
1447
1954
|
|
|
1448
1955
|
Between processes, the cache goes stale for the signal sending interval (`notifyIntervalMs`, 100 ms) plus delivery;
|
|
1449
1956
|
`db.flushSignals()` sends the accumulated signals at once. `db.cache.clear()` clears the cache.
|
|
@@ -1461,11 +1968,11 @@ const report = await adm.maintain() // { sessions, codes, thinned, batches,
|
|
|
1461
1968
|
PostgreSQL signal queue: if it is more than half full, you get a warning. The system administrator, a role with the
|
|
1462
1969
|
`letopis.maintain` permission or an admin connection can call it. An external scheduler runs it (`letopis maintain`),
|
|
1463
1970
|
or the library itself does: `connect({ maintain: '1h' })`. Of several processes, only one works: each batch is its own
|
|
1464
|
-
transaction, and the first thing it does is take a
|
|
1465
|
-
|
|
1466
|
-
the first batch, and between batches it stops (that
|
|
1467
|
-
`more: true`. There is no session-level lock, so `maintain()`
|
|
1468
|
-
transaction mode.
|
|
1971
|
+
transaction, and the first thing it does is take the maintenance lock — a row of the `secret` table that neither the
|
|
1972
|
+
application role nor any other database role can take; the lock is released when the batch commits. If another
|
|
1973
|
+
process holds the lock, the call answers `skipped: true` before the first batch, and between batches it stops (that
|
|
1974
|
+
process continues) and reports what it has done with `more: true`. There is no session-level lock, so `maintain()`
|
|
1975
|
+
works behind any pool, including PgBouncer in transaction mode.
|
|
1469
1976
|
|
|
1470
1977
|
### Reset: `reset()`
|
|
1471
1978
|
|
|
@@ -1490,7 +1997,157 @@ const modelText = describeText(model) // the same as text: for the application's
|
|
|
1490
1997
|
```
|
|
1491
1998
|
|
|
1492
1999
|
`db.describe()` returns a description of the classes available to the session: fields, ends, keys, policies and
|
|
1493
|
-
metadata
|
|
2000
|
+
metadata, plus the multi-field conditions `anyOf` and `oneOf` with inheritance applied, the derived class mark
|
|
2001
|
+
`derivedBy` and the aggregates `aggregate`, when a class has them. From the command line:
|
|
2002
|
+
`letopis describe --format text --out llms.txt`.
|
|
2003
|
+
|
|
2004
|
+
### Agents and MCP
|
|
2005
|
+
|
|
2006
|
+
An agent — Claude, Cursor or your own — works with the data through MCP tools: it learns the model, reads with
|
|
2007
|
+
chains, looks at the history and, if allowed, writes. The MCP server acts on behalf of one user and only with that
|
|
2008
|
+
user's rights: every call is a chain under the user's session, and the database checks rights and tenants. The agent
|
|
2009
|
+
has no raw SQL.
|
|
2010
|
+
|
|
2011
|
+
| Tool | What it does |
|
|
2012
|
+
|---|---|
|
|
2013
|
+
| `letopis_model` | the model: classes, fields (JSON Schema), the `anyOf` and `oneOf` conditions, ends, keys, aliases, metadata; with `class`, one class in detail |
|
|
2014
|
+
| `letopis_query` | reading with a JSON chain: the start is `class` and `filter`, `id` or `ids`; `steps` are hops by roles and classes; `asOf`, `withDeleted`, `sort`, `limit`, `cursor`; the terminal is `rows`, `first`, `ids`, `count`, `sum`, `avg`, `min`, `max` or `countBy` |
|
|
2015
|
+
| `letopis_history` | `versions`, `diff` or `blame` of one object |
|
|
2016
|
+
| `letopis_write` | `create`, `update` (only with `rev`), `upsert`, `delete`, only with `write: true` (`letopis mcp --write`) |
|
|
2017
|
+
|
|
2018
|
+
A `letopis_query` request is the same chain as `db.Product({ price: lt(1000) }).Shop().sort('data.name').rows()`, written as JSON:
|
|
2019
|
+
|
|
2020
|
+
```json
|
|
2021
|
+
{ "class": "Product", "filter": { "price": { "$lt": 1000 } }, "steps": [{ "step": "Shop" }], "sort": { "field": "data.name" } }
|
|
2022
|
+
```
|
|
2023
|
+
|
|
2024
|
+
A filter value means equality, a nested object addresses the fields of nested objects, `{ "Role": id }` is an end.
|
|
2025
|
+
Operators: `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$between`, `$in`, `$like`, `$ilike`, `$starts`, `$ends`, `$has`,
|
|
2026
|
+
`$hasAny`, `$hasAll`, `$exists`, `$isNull`, `$not`; OR is `{ "$or": [filter, …] }`. An unknown class, role, parameter
|
|
2027
|
+
or operator gives an `isError` answer with a hint of similar names. There are at most 16 `steps` and at most 64
|
|
2028
|
+
conditions in the `$or` of one filter; otherwise `invalid_query`. These limits do not bound the time of a query: set
|
|
2029
|
+
`statement_timeout` for the database role the MCP server connects as (`alter role … set statement_timeout = '5s'`).
|
|
2030
|
+
|
|
2031
|
+
The answer is `structuredContent` and its text copy. `structuredContent` is `{ notice, data }`: `notice` is a constant
|
|
2032
|
+
text of the package saying that `data` holds database data, not instructions, and `data` is the answer itself. The
|
|
2033
|
+
shape of `data` of `letopis_query` depends on the terminal:
|
|
2034
|
+
|
|
2035
|
+
| Terminal | `structuredContent.data` |
|
|
2036
|
+
|---|---|
|
|
2037
|
+
| `rows` (default) | `{ rows, count, more, cursor? }` |
|
|
2038
|
+
| `ids` | `{ ids, count, more, cursor? }` |
|
|
2039
|
+
| `first` | `{ row }`, a row or `null` |
|
|
2040
|
+
| `count` | `{ count }` |
|
|
2041
|
+
| `countBy` | `{ counts, truncated? }` |
|
|
2042
|
+
| `sum`, `avg`, `min`, `max` | `{ value }` |
|
|
2043
|
+
|
|
2044
|
+
For `rows` and `ids`, `count` is the number of rows on this page, not the total. A page holds at most `limit`,
|
|
2045
|
+
`maxRows` rows (100) and `maxBytes` bytes of row JSON (50,000); a row that is itself larger than `maxBytes` comes
|
|
2046
|
+
alone on its page. `more: true` means there are more rows, and then the same request with `cursor` returns the next
|
|
2047
|
+
page. `counts` is `{ value: number }`, the most frequent values; if the groups did not fit into the limits,
|
|
2048
|
+
`truncated: true` comes. `countBy` reads at most `maxRows` + 1 groups from the database. The
|
|
2049
|
+
field of an aggregate and of `countBy` is `field: 'data.<path>'`. Sorting is `sort: { field, dir }`: `field` is `at`,
|
|
2050
|
+
`rev`, `id` or `data.<path>`, `dir` is `'asc'` (default) or `'desc'`; without `sort`, rows go by `id`. `sort`, `limit`
|
|
2051
|
+
and `cursor` work only with `rows`, `first` and `ids`.
|
|
2052
|
+
|
|
2053
|
+
`letopis_model` without `class` answers (in `data`) `{ schema, tenant, classes }`, the same classes as in `db.describe()`. If such
|
|
2054
|
+
a model is larger than `maxBytes`, a brief one comes: `brief: true`, and the classes have only the names of their
|
|
2055
|
+
fields and ends; `letopis_model` with `class` returns each class in detail. `letopis_history` takes `class`, `id` and
|
|
2056
|
+
`kind`. `versions` answers `{ versions, more }`, the latest versions within the answer limits, from the oldest to the
|
|
2057
|
+
newest; `diff` with `from` and `to` (a version number or a moment) answers `{ diff }`; `blame` with `at` and `limit`
|
|
2058
|
+
answers `{ blame }`. If the changes or the leaves did not fit, the answer is cut, and it has `truncated: true` and
|
|
2059
|
+
`total`. `follow` works with `versions` and `diff`.
|
|
2060
|
+
|
|
2061
|
+
A tool refusal is `isError: true` with a `structuredContent` of the form `{ notice, error }`, where `error` is `{ code, message }`, plus `similar` (similar names),
|
|
2062
|
+
`detail` (details) and `sqlstate` (the PostgreSQL code of an error not from letopis) when present. A parameter that
|
|
2063
|
+
does not match the tool input schema (a wrong type, a value outside the list, a missing required one) is rejected by
|
|
2064
|
+
the SDK before letopis: then it is `isError: true` with text only, without `structuredContent`.
|
|
2065
|
+
|
|
2066
|
+
In the text of the answer, everything that came from the database (rows, names and descriptions of classes, error
|
|
2067
|
+
texts, hints) sits inside `<untrusted-data-…>` with a random suffix and a note that this is data, not instructions.
|
|
2068
|
+
Tool descriptions and the server's `instructions` are fixed text of the package: tenants write the `meta` of classes,
|
|
2069
|
+
so it only appears in results.
|
|
2070
|
+
|
|
2071
|
+
Writing. Without `write`, the `letopis_write` tool is not in the tool list, and a call to it is an SDK protocol error
|
|
2072
|
+
(`Tool letopis_write not found`), not a tool answer with `isError`. The ends of a write are set by
|
|
2073
|
+
`links: { Role: id | [id] | null }`, and `create`, `update` and `upsert` answer `{ rows }`. `update` takes only the
|
|
2074
|
+
`rev` of the version that was read: someone else's version gives `conflict`, and no update is lost. `delete` without
|
|
2075
|
+
`confirm` returns a preview of the cascade and the operation code: `{ preview, operation }`. With `confirm: true`, a
|
|
2076
|
+
human confirms the deletion in the client through elicitation (in protocol revision 2025 as a request to the client,
|
|
2077
|
+
in 2026-07-28 as an `input_required` answer), the confirmation is bound to the hash of the operation, and the answer
|
|
2078
|
+
is `{ deleted, operation }`. In revision 2026-07-28, the `requestState` signature with which the client repeats the
|
|
2079
|
+
call is also bound to the user of the server and is valid for 10 minutes: a signature of another user's server and an
|
|
2080
|
+
expired one give the refusal `confirmation_required`. The confirmation window shows the canonical class name and the ids of the targets (the
|
|
2081
|
+
first 10, then «и ещё K», that is "and K more"); a tenant writes the alias and the `meta` of a class, so they do not
|
|
2082
|
+
get into the window. An operation code sent by the agent is not a confirmation; a client without elicitation gets the
|
|
2083
|
+
refusal `confirmation_required`. Every write gets the reason `mcp: letopis_write`, and `blame()` shows it. By default,
|
|
2084
|
+
`letopis_write` does not create, edit or delete the permission classes, `rule`, `Resource`, `member`, `Account`,
|
|
2085
|
+
`Class` and their descendants (rules, memberships, class descriptions): the refusal is `acl_denied`. An agent under an
|
|
2086
|
+
injection in the data would grant a stranger a membership or a rule without a human confirmation. Only the option
|
|
2087
|
+
`allowAclWrite: true` allows it.
|
|
2088
|
+
|
|
2089
|
+
**stdio.** `letopis mcp` is a stdio server for MCP clients. It needs the SDK package: `npm i @modelcontextprotocol/server`
|
|
2090
|
+
(an optional peer dependency). The client passes the connection string, the schema and the token as flags or as the
|
|
2091
|
+
variables `LETOPIS_DSN`, `LETOPIS_SCHEMA`, `LETOPIS_TOKEN`; `--write` turns writing on. Only a user session token is
|
|
2092
|
+
accepted, and the application issues it: `svc.auth.sessionFor(user, { ttl })`, while `auth.switch(tenant)` sets the
|
|
2093
|
+
working tenant of the session. The command rejects the service key (`--api-key`): it grants all `auth.*` rights and
|
|
2094
|
+
would lie on the agent's machine. Without a token and with `--api-key`, the command exits with code 2 and writes the
|
|
2095
|
+
reason to stderr. The command also rejects the token of a session in the System tenant (this is how a system
|
|
2096
|
+
administrator works): it has no `allowSystemAdmin` and `allowAclWrite` options. Client configuration (Claude Desktop,
|
|
2097
|
+
Claude Code, Cursor):
|
|
2098
|
+
|
|
2099
|
+
```json
|
|
2100
|
+
{
|
|
2101
|
+
"mcpServers": {
|
|
2102
|
+
"letopis": {
|
|
2103
|
+
"command": "npx",
|
|
2104
|
+
"args": ["letopis", "mcp"],
|
|
2105
|
+
"env": {
|
|
2106
|
+
"LETOPIS_DSN": "postgres://shop_app:app-secret@localhost:5432/shop",
|
|
2107
|
+
"LETOPIS_SCHEMA": "v2.shop",
|
|
2108
|
+
"LETOPIS_TOKEN": "<user session token>"
|
|
2109
|
+
}
|
|
2110
|
+
}
|
|
2111
|
+
}
|
|
2112
|
+
}
|
|
2113
|
+
```
|
|
2114
|
+
|
|
2115
|
+
**HTTP.** There is no built-in HTTP server. `createMcpServer(db, { write, maxRows, maxBytes })` from `letopis/mcp`
|
|
2116
|
+
returns an MCP server, and the application plugs it into its own `createMcpHandler` handler from the SDK with its own
|
|
2117
|
+
OAuth 2.1 authorization: the application checks the request token and finds the letopis user by it. The connection
|
|
2118
|
+
is a user session (`connect({ token })`) or the facade `svc.as(user)`: the service key stays in the application
|
|
2119
|
+
process. A connection that bypasses RLS is refused with `bypass_rls`; the service root without `as()` and a connection
|
|
2120
|
+
without a session, with `user_session_required`; the facade of an explicit transaction `db.begin()`, with
|
|
2121
|
+
`invalid_query`. The server also refuses a session in the System tenant with `user_session_required` unless the option
|
|
2122
|
+
`allowSystemAdmin: true` is passed: this is how a system administrator works, and their `admin` role grants rights to
|
|
2123
|
+
the accounts, memberships and rules of the installation.
|
|
2124
|
+
|
|
2125
|
+
<!-- run: mcp -->
|
|
2126
|
+
```ts
|
|
2127
|
+
import { up, connect } from 'letopis'
|
|
2128
|
+
import { createMcpServer } from 'letopis/mcp'
|
|
2129
|
+
import { createMcpHandler } from '@modelcontextprotocol/server'
|
|
2130
|
+
|
|
2131
|
+
const ADMIN = 'postgres://postgres:secret@localhost:5432/shop'
|
|
2132
|
+
const APP = 'postgres://shop_app:app-secret@localhost:5432/shop'
|
|
2133
|
+
const r = await up({ dsn: ADMIN, schema: 'shop', quiet: true,
|
|
2134
|
+
tenant: { name: 'Chamomile', user: { name: 'Anna', login: 'anna@shop.ru', password: 'Chamomile-2026!' } } })
|
|
2135
|
+
const svc = await connect({ dsn: APP, schema: r.schema, apiKey: r.serviceKey! })
|
|
2136
|
+
|
|
2137
|
+
// authInfo is the result of the application's check of the request token; it puts the letopis account id into extra
|
|
2138
|
+
const mcp = createMcpHandler(async ({ authInfo }) =>
|
|
2139
|
+
createMcpServer(await svc.as(String(authInfo?.extra?.account), { tenant: r.tenant!.tenant })))
|
|
2140
|
+
// in the application's HTTP handler, after checking the token: return mcp.fetch(request, { authInfo })
|
|
2141
|
+
```
|
|
2142
|
+
|
|
2143
|
+
The handler calls the factory on every HTTP request, so `createMcpServer` checks the connection every time (one query
|
|
2144
|
+
to the database).
|
|
2145
|
+
|
|
2146
|
+
The handler can be checked without a network: the SDK client from the `@modelcontextprotocol/client` package reaches
|
|
2147
|
+
it through the application's `fetch`: `new StreamableHTTPClientTransport(new URL('http://localhost/mcp'), { fetch:
|
|
2148
|
+
(url, init) => mcp.fetch(new Request(url, init), { authInfo }) })`. This is how the `docs` test checks the example
|
|
2149
|
+
above (the `mcp` scenario in `lib/test/docs.test.ts` of the repository): without `write` the client sees three tools,
|
|
2150
|
+
and `letopis_model` answers on behalf of Anna in the shop.
|
|
1494
2151
|
|
|
1495
2152
|
### Raw SQL
|
|
1496
2153
|
|
|
@@ -1511,12 +2168,14 @@ application.
|
|
|
1511
2168
|
letopis sync <module> --dsn <dsn> --schema <v2.name> (--token <token> | --api-key <key>) [--apply]
|
|
1512
2169
|
letopis types --dsn <dsn> --schema <v2.name> (--token <token> | --api-key <key>) [--out <file>]
|
|
1513
2170
|
letopis load <files…> --dsn <dsn> --schema <v2.name> (--token <token> | --api-key <key>) [--class <Class>]
|
|
1514
|
-
[--duplicates error|first|last] [--existing error|skip|update] [--commit-every <N>] [--verify]
|
|
2171
|
+
[--duplicates error|first|last] [--existing error|skip|update] [--commit-every <N>] [--verify] [--initial]
|
|
1515
2172
|
letopis verify --dsn <dsn> --schema <v2.name> [--token <token> | --api-key <key>] [--data]
|
|
1516
2173
|
letopis maintain --dsn <dsn> --schema <v2.name> [--token <token> | --api-key <key>] [--batches <N>]
|
|
1517
2174
|
letopis describe --dsn <dsn> --schema <v2.name> (--token <token> | --api-key <key>) [--format json|text] [--out <file>]
|
|
1518
2175
|
letopis import --from <0.21 dsn> --from-schema <v1.name> --dsn <dsn> --schema <v2.name> [--tenant <name>] [--keyless <Class>]…
|
|
1519
|
-
[--rename <Old>=<New>]… [--update] [--commit-every <N>] [--partition <name>]
|
|
2176
|
+
[--rename <Old>=<New>]… [--update] [--commit-every <N>] [--partition <name>] [--initial]
|
|
2177
|
+
letopis mcp --dsn <dsn> --schema <v2.name> --token <user token> [--write]
|
|
2178
|
+
(or LETOPIS_DSN, LETOPIS_SCHEMA, LETOPIS_TOKEN; the service key --api-key is not accepted)
|
|
1520
2179
|
```
|
|
1521
2180
|
|
|
1522
2181
|
### Monitoring
|
|
@@ -1586,9 +2245,10 @@ The full machine-readable list of exports, methods and error texts is `docs/api-
|
|
|
1586
2245
|
|
|
1587
2246
|
| Entry point | What it exports |
|
|
1588
2247
|
|---|---|
|
|
1589
|
-
| `letopis` | `connect`, `up`, `createTenant`; operators `ne`, `gt`, `gte`, `lt`, `lte`, `between`, `inList`, `like`, `ilike`, `starts`, `ends`, `has`, `hasAny`, `hasAll`, `exists`, `isNull`, `not`, `or`, `isOp`; `inc`, `isInc`, `cursorOf`; `idOf`, `uuidv5`, `v5Name`, `v5Id`; `LetopisError`, `ValidationError`, `fromDbError`; `syncModels`, `generateTypes`, `schemaToTs`; `compileValidator`, `applyDefaults`, `Registry`; `totpCode`, `hashPassword`, `verifyHash`; `describeText`; constants `DDL_REVISION`, `RESERVED_CLASS_NAMES`, `CHAIN_METHODS`, `DB_METHODS`; types `Db`, `DbTx`, `TypedDb`, `Chain`, `Row`, `Filter`, `Cursor`, `ConnectOptions`, `UpOptions`, `LoadOptions`, `LoadReport`, `WatchOptions`, `WatchEvent`, `AuthApi`, `AclApi` and others |
|
|
1590
|
-
| `letopis/model` | `hub`, `link`, `one`, `many`, `End`, types `Infer`, `InferLinks`, `Model`, `ClassDef`, `ClassMeta`, `HistoryPolicy`; `normalizeSchema`, `translatePattern`, type `JsonSchema` |
|
|
1591
|
-
|
|
|
2248
|
+
| `letopis` | `connect`, `up`, `createTenant`; operators `ne`, `gt`, `gte`, `lt`, `lte`, `between`, `inList`, `like`, `ilike`, `starts`, `ends`, `has`, `hasAny`, `hasAll`, `exists`, `isNull`, `not`, `or`, `isOp`; `inc`, `isInc`, `cursorOf`; `idOf`, `uuidv5`, `v5Name`, `v5Id`; `LetopisError`, `ValidationError`, `fromDbError`; `syncModels`, `generateTypes`, `schemaToTs`; `compileValidator`, `applyDefaults`, `Registry`; `totpCode`, `hashPassword`, `verifyHash`; `describeText`; constants `DDL_REVISION`, `RESERVED_CLASS_NAMES`, `CHAIN_METHODS`, `DB_METHODS`; types `Db`, `DbTx`, `TypedDb`, `Chain`, `Row`, `Filter`, `Cursor`, `ConnectOptions`, `UpOptions`, `LoadOptions`, `LoadReport`, `WatchOptions`, `WatchEvent`, `AuthApi`, `AclApi`, `DiffResult`, `DiffChange`, `BlameResult`, `BlameField`, `Projector`, `ProjectorOptions`, `ProjectorChange`, `ProjectorOut`, `ProjectorRow`, `ProjectorReport`, `ProjectorStatus`, `ProjectorFailure` ([Projector](#projector)), `AggregateInfo` and others |
|
|
2249
|
+
| `letopis/model` | `hub`, `link`, `one`, `many`, `End`, types `Infer`, `InferLinks`, `Model`, `ClassDef`, `ClassMeta`, `HistoryPolicy`, `AggregateDef`; `normalizeSchema`, `translatePattern`, type `JsonSchema` |
|
|
2250
|
+
| `letopis/mcp` | `createMcpServer(db, options)`: an MCP SDK server with the letopis tools; the type `McpOptions`: `write` adds the `letopis_write` tool (absent by default), `maxRows` is the rows in an answer (100, from 1 to 10,000), `maxBytes` is the bytes in an answer (50,000, from 1,000 to 50,000,000); `allowAclWrite` lets `letopis_write` write the permission classes too (by default `acl_denied`); `allowSystemAdmin` accepts a session in the System tenant (by default `user_session_required`); a value out of range gives `invalid_data`. Needs the `@modelcontextprotocol/server` package ([section 13](#agents-and-mcp)) |
|
|
2251
|
+
| CLI `letopis` | `sync`, `types`, `load`, `verify`, `maintain`, `describe`, `import`, `mcp` ([section 13](#13-operations)); `mcp` takes `--dsn`, `--schema`, `--token` (or `LETOPIS_DSN`, `LETOPIS_SCHEMA`, `LETOPIS_TOKEN`) and `--write` |
|
|
1592
2252
|
|
|
1593
2253
|
Removed 0.21 names (`uuidv7`, `LETOPIS_NS` and the methods listed below) throw `removed` with the replacement.
|
|
1594
2254
|
|
|
@@ -1603,6 +2263,8 @@ Removed 0.21 names (`uuidv7`, `LETOPIS_NS` and the methods listed below) throw `
|
|
|
1603
2263
|
| `begin()`, `commit(tr)`, `rollback(tr)`, `lock(…keys)` | an explicit transaction ([section 8](#8-transactions-batches-and-money)) |
|
|
1604
2264
|
| `batch(name)` | a queue of plans: `run()`, `discard()`, `size()` |
|
|
1605
2265
|
| `load(input, options)` | the loader ([section 9](#9-bulk-loading)) |
|
|
2266
|
+
| `projector(name, options)` | a projector from the journal: `run()`, `rebuild({ chunked, tenant })`, `status()`, `stop()`, `close()` ([Projectors](#projectors)) |
|
|
2267
|
+
| `rebuildAggregates(class)` | repair of the aggregates of a class and its descendants in all tenants: the field anew from live sources, a version only for corrected rows; → their number ([Aggregates in the database](#aggregates-in-the-database)) |
|
|
1606
2268
|
| `watch(options)`, `cache` | subscription; result cache (`clear()`, `stats()`) |
|
|
1607
2269
|
| `` sql`…` ``, `transaction(fn)` | raw SQL under the session and RLS |
|
|
1608
2270
|
| `idOf(class, key)` | the id of an object of a class with a key, in the session tenant |
|
|
@@ -1620,7 +2282,7 @@ Removed 0.21 names (`uuidv7`, `LETOPIS_NS` and the methods listed below) throw `
|
|
|
1620
2282
|
| modifiers | `limit(n)`, `offset(n)`, `sort(field, 'asc' \| 'desc')`, `after(cursor)`, `asOf(moment)`, `withDeleted()`, `deep(max)`, `exact()`, `alias(name)`, `tags(…)`, `owner(…)`, `entity(…)`, `forUpdate()`, `cache({ ttl })` |
|
|
1621
2283
|
| verbs | `create(data)`, `update(patch, { rev })`, `upsert(data)`, `delete({ confirm })`, `anonymize(fields)`, `reclass(class, data)`, `rekey(patch)`, `restore()`, `purge({ confirm })` |
|
|
1622
2284
|
| slots | after a verb: `.RoleName.set(value)`, `.RoleName.unset()`, `.RoleName.add(value)`, `.RoleName.remove(value)` |
|
|
1623
|
-
| terminals | `rows()`, `first()`, `ids()`, `count({ paths })`, `paths()`, `versions({ follow })`, `sum(field)`, `avg(field)`, `min(field)`, `max(field)`, `countBy(field)` |
|
|
2285
|
+
| terminals | `rows()`, `first()`, `ids()`, `count({ paths })`, `paths()`, `versions({ follow })`, `diff({ from, to, follow })`, `blame({ at, limit })`, `sum(field)`, `avg(field)`, `min(field)`, `max(field)`, `countBy(field)` |
|
|
1624
2286
|
|
|
1625
2287
|
### `db.auth`
|
|
1626
2288
|
|
|
@@ -1647,6 +2309,23 @@ its own key, but not its other credentials; nobody can revoke the last active ke
|
|
|
1647
2309
|
`TOTP` method. The credential kind `kind` is `^[A-Z][A-Z0-9_]{1,31}$` (`PASSWORD`, `EMAIL`,
|
|
1648
2310
|
`TELEGRAM` …); `SESSION`, `APIKEY`, `KEYSECRET`, `OTP`, `TOTP` and `KEY` are taken.
|
|
1649
2311
|
|
|
2312
|
+
### Projector
|
|
2313
|
+
|
|
2314
|
+
`db.projector(name, options)` works only on an admin connection ([Projectors](#projectors)). The name is lowercase
|
|
2315
|
+
Latin letters, digits and `_.:-`, up to 64 characters; it is also the `derivedBy` of the derived classes. Wrong options
|
|
2316
|
+
give `invalid_data`.
|
|
2317
|
+
|
|
2318
|
+
| Member | What it does |
|
|
2319
|
+
|---|---|
|
|
2320
|
+
| options `ProjectorOptions` | `from`, `into`: classes with descendants; `version`: an integer from 1; `handle(changes, out)`; `withBefore`; `onExpired: 'rebuild' \| 'error'` (`'rebuild'` rebuilds in batches); `batch`: changes in a batch (500, from 1 to 10,000); `pollMs`: fallback polling (1000 ms); `tenants`: ids of the participant tenants |
|
|
2321
|
+
| `run({ once })` | catch up with the journal (`once`) or follow it until `stop()` → `ProjectorReport` |
|
|
2322
|
+
| `rebuild({ chunked, tenant })` | rebuild from the current state: atomically, in batches or for one tenant → `ProjectorReport` |
|
|
2323
|
+
| `status()` | → `ProjectorStatus`: `{ name, state, version, cursor, at, lag, failed, rebuildingTenants }`; `failed` is `{ [tenant id]: ProjectorFailure }`, where `ProjectorFailure` is `{ cursor, code, message }` |
|
|
2324
|
+
| `stop()`, `close()` | stop `run()` after the current batch; the same plus unsubscribing from signals |
|
|
2325
|
+
| `ProjectorReport` | `{ batches, changes, created, updated, deleted, unchanged, skipped, rebuilds }` |
|
|
2326
|
+
| `ProjectorChange` | `{ cursor, id, class, tenant, op, rev, at, author, row }` and, when present, `agent`, `reason`, `moved`, `before`, `rebuild` |
|
|
2327
|
+
| `ProjectorOut` | `upsert(class, row, { tenant })`, `inc(class, row, deltas, { tenant })`, `delete(class, key or id, { tenant })`, `get(class, key or id, { tenant })` → `Row \| null`; a row is a `ProjectorRow` in the loader format |
|
|
2328
|
+
|
|
1650
2329
|
### Result row
|
|
1651
2330
|
|
|
1652
2331
|
| Field | Meaning |
|
|
@@ -1669,11 +2348,11 @@ database does not accept a link with such a name (`reserved_names()`; the instal
|
|
|
1669
2348
|
- chain methods: `then`, `entity`, `paths`, `rows`, `first`, `ids`, `count`, `limit`, `offset`, `sort`, `asOf`,
|
|
1670
2349
|
`withDeleted`, `deep`, `exact`, `sum`, `avg`, `min`, `max`, `countBy`, `after`, `versions`, `create`, `update`,
|
|
1671
2350
|
`upsert`, `set`, `unset`, `add`, `remove`, `delete`, `anonymize`, `reclass`, `restore`, `purge`, `alias`, `tags`,
|
|
1672
|
-
`owner`, `forUpdate`, `cache`, `rekey`; guards of the removed API: `run`, `execute`, `account`;
|
|
2351
|
+
`owner`, `forUpdate`, `cache`, `rekey`, `diff`, `blame`; guards of the removed API: `run`, `execute`, `account`;
|
|
1673
2352
|
- the `db` root: `as`, `begin`, `commit`, `rollback`, `lock`, `batch`, `watch`, `close`, `registry`, `sql`, `load`,
|
|
1674
2353
|
`verify`, `maintain`, `reset`, `describe`, `auth`, `acl`, `idOf`, `refresh`, `transaction`, `flushSignals`,
|
|
1675
|
-
`schema`, `token`, `tenant`, `trimHistory`, `rekeyClass`, `cache`; guards: `reloadSchema`,
|
|
1676
|
-
`credentials`, `resources`, `rules`, `compact`;
|
|
2354
|
+
`schema`, `token`, `tenant`, `trimHistory`, `rekeyClass`, `cache`, `projector`, `rebuildAggregates`; guards: `reloadSchema`,
|
|
2355
|
+
`accounts`, `credentials`, `resources`, `rules`, `compact`;
|
|
1677
2356
|
- batch: `discard`, `size`.
|
|
1678
2357
|
|
|
1679
2358
|
## 16. Errors
|
|
@@ -1693,16 +2372,19 @@ Database codes. Raw SQL gets them too: SQLSTATE class `LT`, the details are JSON
|
|
|
1693
2372
|
| `acl_denied` | LT006 | denied by the access rules |
|
|
1694
2373
|
| `id_mismatch`, `id_from_db` | LT007, LT008 | the id is not v5 of the key; the id of a class without a key is assigned by the database |
|
|
1695
2374
|
| `immutable_key` | LT009 | a key field, the id, the tenant and the owner do not change (a new key is `rekey`) |
|
|
1696
|
-
| `invalid_class` | LT010 | the class description is rejected, the class is not found or is abstract |
|
|
2375
|
+
| `invalid_class` | LT010 | the class description is rejected, the class is not found or is abstract; a class with rows, descendants or a history of rows is not deleted |
|
|
1697
2376
|
| `use_reclass`, `reclass_denied` | LT011, LT012 | changing the class with `update` when the old or the new class has a key: the id changes, so use `reclass()`; the roles of referencing rows do not accept the new class |
|
|
1698
2377
|
| `tightening_conflict` | LT013 | rows prevent tightening the description |
|
|
1699
2378
|
| `isolation_level` | LT014 | deleting, changing the class and changing the key work only in `read committed` |
|
|
1700
2379
|
| `recreated_in_statement` | LT015 | an id was deleted and created again in one statement |
|
|
1701
|
-
| `cursor_expired` | LT016 | the subscription
|
|
2380
|
+
| `cursor_expired` | LT016 | the cursor of a subscription or a projector is older than the `all` tier of the retention policy |
|
|
1702
2381
|
| `delete_restricted` | LT017 | referencing rows prevent the deletion (`onDelete: 'restrict'`) |
|
|
1703
2382
|
| `not_deleted` | LT018 | `restore` and `purge` work only for deleted objects |
|
|
1704
2383
|
| `reset_disabled` | LT019 | reset is disabled in the installation (`allowReset`) |
|
|
1705
2384
|
| `confirm_mismatch` | LT020 | the reset confirmation does not match the schema name |
|
|
2385
|
+
| `derived_class` | LT021 | a row of a derived class (`derivedBy`) is written, edited or deleted by someone other than its projector; an end leads to such a row or to a derived class; a projector writes to a class outside its `into` or has found no tenant in which every `into` class is marked with its `derivedBy` |
|
|
2386
|
+
| `aggregate_field` | LT022 | the application passed or changed an aggregate field (`aggregate`): the database calculates it |
|
|
2387
|
+
| `projector_pending` | LT023 | `purge` of a deleted object whose deletion a projector of its class has not read yet: run the projector first |
|
|
1706
2388
|
|
|
1707
2389
|
Library codes:
|
|
1708
2390
|
|
|
@@ -1719,9 +2401,14 @@ Library codes:
|
|
|
1719
2401
|
| `no_key` | `upsert`, `rekey` or `idOf` on a class without a key |
|
|
1720
2402
|
| `no_path` | there is no hop between the classes of neighboring steps |
|
|
1721
2403
|
| `invalid_query` | the chain or the API is used incorrectly (a step modifier `deep`, `exact`, `alias`, `tags` or `owner` after a verb, a slot not after a verb, an `after()` cursor with a value of a different type than the `sort` field, without `v` or with an `id` that is not a uuid, and so on) |
|
|
1722
|
-
| `admin_required` | the loader
|
|
2404
|
+
| `admin_required` | the loader, a projector and `rebuildAggregates` need an admin connection |
|
|
2405
|
+
| `rebuild_required` | the projection was built by another version of the projector (`version`): a rebuild with `rebuild()` is needed; `rebuild({ tenant })` while there is no projection or a batch rebuild is in progress |
|
|
2406
|
+
| `projector_busy` | a rebuild cannot start: the projector is running in another process right now |
|
|
2407
|
+
| `not_empty` | the initial load (`initial`) goes only into an empty schema |
|
|
1723
2408
|
| `class_not_loadable` | the loader does not accept `Class` rows |
|
|
1724
|
-
| `bypass_rls` | `connect()` with a role that bypasses RLS, without `allowBypassRls` |
|
|
2409
|
+
| `bypass_rls` | `connect()` with a role that bypasses RLS, without `allowBypassRls`; `createMcpServer` with a connection that bypasses RLS |
|
|
2410
|
+
| `user_session_required` | `createMcpServer` and `letopis mcp`: a user session is needed — the connection has no session, it is the service root without `as()`, a session in the System tenant without `allowSystemAdmin`, or the command line got the service key or no token |
|
|
2411
|
+
| `confirmation_required` | `letopis_write` (MCP): a deletion without a human confirmation — the client has no elicitation, the human declined, the answer belongs to another operation, or the data changed after the preview |
|
|
1725
2412
|
| `removed` | a removed 0.21 API; the text gives the replacement |
|
|
1726
2413
|
|
|
1727
2414
|
A PostgreSQL permission denial (`42501`) comes, for example, on an `upsert` over someone else's row that you may not
|
|
@@ -1758,9 +2445,32 @@ database from every write (row-by-row writes: 119 → 128–147 rows/s in differ
|
|
|
1758
2445
|
12–18% slower than 0.21 (the measurements vary): most of the time goes to the check triggers in the database and to the
|
|
1759
2446
|
commit with a disk write.
|
|
1760
2447
|
|
|
2448
|
+
Initial load (`cd lib && npx tsx bench/initial-load.mjs`, 2026-10-11, median of three rounds): the regular loader
|
|
2449
|
+
writes 1.07 million objects of the salon dataset without history in 169 s (6,351 rows/s, 4.0 GB of WAL), the initial
|
|
2450
|
+
load in 77 s (13,902 rows/s, 1.4 GB of WAL). Most of the difference comes from the GIN indexes: maintaining them during
|
|
2451
|
+
the load costs 36 s, building them at the end about 5 s. Preparing rows in the loader itself (checks, ends, ids) runs
|
|
2452
|
+
alongside the `COPY` of the previous batch and now sets the pace: about 60 s of the 77.
|
|
2453
|
+
|
|
2454
|
+
Derived data in 1.2 (`cd lib && npx tsx bench/derived.bench.mjs`, 2026-10-27, PostgreSQL 18.6, one machine; for
|
|
2455
|
+
aggregates, the median of three rounds; the projector rows are a repeat of series 2 and 3 after the folding of `inc` in
|
|
2456
|
+
`out`, `BENCH_SERIES=2,3`):
|
|
2457
|
+
|
|
2458
|
+
| Measurement | 1.2 | For comparison |
|
|
2459
|
+
|---|---|---|
|
|
2460
|
+
| an operation with an aggregate, one writer | 94 operations/s, p50 10.7 ms | without an aggregate 137 and 6.9 ms; the `inc` recipe 69 |
|
|
2461
|
+
| operations on one wallet with an aggregate, 20 writers | 152 operations/s, p95 0.47 s | without an aggregate 1,987; the `inc` recipe 190 |
|
|
2462
|
+
| operations on 20 wallets with an aggregate, 20 writers | 1,379 operations/s | without an aggregate 2,023 |
|
|
2463
|
+
| projector catch-up, `out.inc` for every change | 6,076 and 33,395 changes/s (`batch` 500 and 2000) | before the folding of `inc` in `out`: 992 and 1,203 |
|
|
2464
|
+
| rebuild of 75 thousand rows | atomic 13.3–13.5 s, in batches 13.0–13.2 s | before the folding of `inc` in `out`: 101 s and 58–63 s |
|
|
2465
|
+
| delay of a subscription event during a rebuild | atomic up to 13.4 s, in batches up to 0.21 s | without a rebuild up to 0.12 s |
|
|
2466
|
+
|
|
2467
|
+
The aggregate triggers fire on every write statement, but for classes without aggregates the measurement does not see
|
|
2468
|
+
their cost: the p50 difference is from −0.28 to +0.21 ms. The benchmark against 1.0.1 (`bench/vs-101.bench.mjs`, four
|
|
2469
|
+
runs) showed no write or read regressions against 1.1: two outliers did not repeat in separate checks.
|
|
2470
|
+
|
|
1761
2471
|
## 18. Limitations
|
|
1762
2472
|
|
|
1763
|
-
This section lists what letopis 1.
|
|
2473
|
+
This section lists what letopis 1.2 deliberately does not do or does with caveats, and where you need to watch the
|
|
1764
2474
|
behavior.
|
|
1765
2475
|
|
|
1766
2476
|
**Platform**
|
|
@@ -1777,6 +2487,9 @@ behavior.
|
|
|
1777
2487
|
- The NUL character (`\u0000`) is impossible in data strings: PostgreSQL does not store it in `jsonb`; such data is
|
|
1778
2488
|
rejected already when the JSON is parsed (in the official JSON-Schema-Test-Suite, the "nul characters" groups are
|
|
1779
2489
|
skipped).
|
|
2490
|
+
- On PostgreSQL before 18.6, a parallel GIN index build could write a wrong row count into the table statistics
|
|
2491
|
+
(`pg_class.reltuples`), after which autovacuum stopped processing the table. That is why the initial load always ends
|
|
2492
|
+
with `ANALYZE`, but it is better to upgrade the server to 18.6 or newer.
|
|
1780
2493
|
|
|
1781
2494
|
**Writing and integrity**
|
|
1782
2495
|
|
|
@@ -1786,6 +2499,10 @@ behavior.
|
|
|
1786
2499
|
transactions always open at this level.
|
|
1787
2500
|
- PostgreSQL does not run `COPY` into tables under RLS; bulk loading works only through the loader on an admin
|
|
1788
2501
|
connection.
|
|
2502
|
+
- The initial load closes the schema's tables for the whole load, for writing and for reading: it is meant for an
|
|
2503
|
+
empty schema (seeding, migration, import), not for a running system. Its gain is in the database (indexes and
|
|
2504
|
+
constraints are built once); it does not speed up the row preparation in the loader, only runs it alongside `COPY`,
|
|
2505
|
+
so on a million objects it is about 2.2 times faster than the regular load.
|
|
1789
2506
|
- The database trusts the loader: the data, ends and ids of loaded rows are checked by the library code (the TypeScript
|
|
1790
2507
|
validator), not by the triggers; the database checks only the targets outside the load, the tenants and the owners,
|
|
1791
2508
|
with one query at the end. The safeguards: the loader is available only to an admin, differential tests of the
|
|
@@ -1839,13 +2556,20 @@ behavior.
|
|
|
1839
2556
|
attempts.
|
|
1840
2557
|
- When users link a taken identifier themselves, the answer is `acl_denied`, so a user with a recent sign-in learns
|
|
1841
2558
|
this way that the identifier is taken. Permission changes in all tenants are serialized through one counter row.
|
|
1842
|
-
- The number of invisible rows that block a deletion, and the existence of an
|
|
1843
|
-
are not hidden.
|
|
2559
|
+
- The number of invisible rows that block a deletion or a class change (`reclass_denied`), and the existence of an
|
|
2560
|
+
account by id (as a reference target), are not hidden.
|
|
2561
|
+
- Tightening a class description and `rekeyClass` check all rows of the family, so they need the `READ` permission
|
|
2562
|
+
without conditions on all rows of the class and its descendants: otherwise a trial tightening (`maximum`, `enum`,
|
|
2563
|
+
`pattern`) would answer with the number of invisible rows that do not match it. A system administrator who tightens
|
|
2564
|
+
a System class gets the number of violating rows of other tenants (`others`): this is the installation level.
|
|
1844
2565
|
- Class metadata (`meta`) is visible to anyone who has a session: anyone can read `Class` rows. Secrets do not belong
|
|
1845
2566
|
there.
|
|
1846
|
-
- An application role can slow the installation down for everyone: `LOCK TABLE entity`, advisory locks with
|
|
1847
|
-
|
|
1848
|
-
installation). The global journal number reveals the overall write activity.
|
|
2567
|
+
- An application role can slow the installation down for everyone: `LOCK TABLE entity`, advisory locks with the keys
|
|
2568
|
+
of the application's `lock()`, fake `NOTIFY` signals (they only clear the cache), frequent permission changes (one
|
|
2569
|
+
counter row per installation). The global journal number reveals the overall write activity. Internal locks — class
|
|
2570
|
+
families, projectors and maintenance — are rows of the `secret` table, not advisory locks: neither the application
|
|
2571
|
+
role nor any other database role can take them. A family writer passes the tightening gate in a subtransaction — one
|
|
2572
|
+
per transaction for the first write to the family.
|
|
1849
2573
|
|
|
1850
2574
|
**History, signals, cache**
|
|
1851
2575
|
|
|
@@ -1858,19 +2582,94 @@ behavior.
|
|
|
1858
2582
|
- The journal has no partitions: thinning deletes versions in batches, and the table file does not shrink by itself
|
|
1859
2583
|
(new versions take the space after autovacuum); to return the space to the system, use `VACUUM FULL` or `pg_repack`
|
|
1860
2584
|
in a maintenance window.
|
|
1861
|
-
- Signals reveal activity: PostgreSQL lets any role listen to a channel
|
|
1862
|
-
|
|
1863
|
-
|
|
2585
|
+
- Signals reveal activity: PostgreSQL lets any role listen to a channel. In the `class` mode one sees when and to which
|
|
2586
|
+
classes writes happen; `tenant` adds tenant pseudonyms; `opaque` hides what was written but not when: the write
|
|
2587
|
+
time and frequency per pseudonym stay visible, and a deterministic token is the basis of frequency attacks. Write
|
|
2588
|
+
times are visible without signals too: `pg_stat_all_tables` and `pg_locks` are open to PUBLIC. The `none` mode
|
|
2589
|
+
removes signals, but then a revoked session leaves the cache of other processes only by term, and a new class
|
|
2590
|
+
appears in their registry within 5 seconds. Real protection from foreign roles is a separate database. A fake
|
|
2591
|
+
signal only clears the cache one extra time: on `*` and unknown tokens the registry is re-read not more often than
|
|
2592
|
+
once in 5 seconds, and a fake class change signal re-reads it like a real one, but with at most one re-read at a
|
|
2593
|
+
time.
|
|
2594
|
+
- The signal mode is changed by `up({ signals })`, and the processes have to be restarted after that: a process reads
|
|
2595
|
+
the mode and the signal key on connect (it treats a payload of another mode as foreign and clears the whole cache).
|
|
2596
|
+
- PostgreSQL below 18.2 could truncate CLOG that unread signals still need: `up()` warns about it.
|
|
1864
2597
|
- The `NOTIFY` queue (8 GB) is shared by the whole cluster: a stuck listener fills it up, and then transactions with
|
|
1865
2598
|
`NOTIFY` fail on commit. The library reads signals right away, and `maintain()` warns when the queue is more than
|
|
1866
2599
|
half full.
|
|
2600
|
+
- PostgreSQL commits transactions with `NOTIFY` one at a time: a commit holds the cluster-wide lock of the signal queue
|
|
2601
|
+
together with the disk write. A raw SQL write on its own connection sends the signal from a trigger in the same
|
|
2602
|
+
transaction, so concurrent raw SQL writers commit one after another: on the benchmark machine, 20 such writers did
|
|
2603
|
+
about 520 inserts per second. Library transactions, `db.sql` and `db.transaction()` included, commit the data
|
|
2604
|
+
without `NOTIFY` and send the signals later, separately, so their writers do not wait in this queue: 20 library
|
|
2605
|
+
writers did about 2,000 operations per second.
|
|
1867
2606
|
- The subscription horizon is shared by the whole cluster: a long write transaction delays the events of all
|
|
1868
2607
|
subscribers (`sub.lag()`); long operations are better done in batches.
|
|
1869
2608
|
- Between processes, the result cache goes stale for the signal sending interval (100 ms) plus delivery, and for the
|
|
1870
|
-
time of a listener reconnect
|
|
2609
|
+
time of a listener reconnect: postgres.js does not report a broken listener connection, so the cache works until the
|
|
2610
|
+
reconnect and is cleared completely after it. The `ttl` term is a safeguard.
|
|
1871
2611
|
- A "from now" subscription cursor carries a snapshot of the subscription moment until it passes that boundary: such a
|
|
1872
2612
|
cursor is longer than a regular one.
|
|
1873
2613
|
|
|
2614
|
+
**Projectors**
|
|
2615
|
+
|
|
2616
|
+
- A projector is an admin process: it works only on an admin connection, reads the journal of all tenants bypassing
|
|
2617
|
+
RLS and writes its results as the owner role. Keep it in a separate process, like the loader. A participant of a
|
|
2618
|
+
projector is a tenant in which all `into` classes are marked with its `derivedBy`; only an administrator may mark a
|
|
2619
|
+
class (the `letopis.projector` permission).
|
|
2620
|
+
- The totals of a derived class are seen by everyone whom the tenant rules give `READ` on it — even by those who do not
|
|
2621
|
+
read the source rows: decide who reads it with rules on the derived class itself. With `history: true`, its journal
|
|
2622
|
+
shows the change of every batch, that is, the contribution of single source rows too.
|
|
2623
|
+
- Only their projector writes the rows of derived classes: the application, raw SQL and the loader get
|
|
2624
|
+
`derived_class`, and ends may not refer to such rows.
|
|
2625
|
+
- An atomic rebuild is one long write transaction: while it runs, the subscription horizon stands still, and events
|
|
2626
|
+
are delayed for the subscribers and projectors of all processes. On the benchmark machine, a rebuild of 75 thousand
|
|
2627
|
+
rows took 13 s, and the subscriber got no event until it ended. That is why `run()` itself rebuilds in batches, and
|
|
2628
|
+
only an explicit `rebuild()` runs an atomic rebuild.
|
|
2629
|
+
- A batch rebuild holds the horizon only for the time of a batch (in the same measurements, events were delayed by up
|
|
2630
|
+
to 0.2 s), but the derived data is incomplete until it ends: the rows are deleted at once and reappear as the
|
|
2631
|
+
rebuild goes on.
|
|
2632
|
+
- A projector sees only the changes from the journal, so a class with `history: false` cannot be a source: a tenant
|
|
2633
|
+
with such a source drops out of the projector (`invalid_class`) until the journal of the class is turned on and the
|
|
2634
|
+
tenant is rebuilt.
|
|
2635
|
+
- Thinning the journal older than the `all` tier leads to a rebuild: a projector that lags behind by more than that
|
|
2636
|
+
tier gets `cursor_expired` or, with `onExpired: 'rebuild'`, rebuilds itself. The tier is per tenant: if it has
|
|
2637
|
+
expired only for some participants, only they drop out and rebuild.
|
|
2638
|
+
- A projector works only on a schema with the engine revision the library expects; on a schema of another revision it
|
|
2639
|
+
answers `invalid_data`.
|
|
2640
|
+
- `purge()` waits for projectors: while a projector whose sources include the class of the object, and which has a
|
|
2641
|
+
derived class in the object's tenant, has not read the deletion, `purge` answers `projector_pending`. A projector
|
|
2642
|
+
that is no longer run holds `purge` of its sources while the tenant has its derived classes: run it or delete them.
|
|
2643
|
+
|
|
2644
|
+
**Aggregates in the database**
|
|
2645
|
+
|
|
2646
|
+
- An aggregate queues the writers of one object: every operation locks the target row and writes its version. On the
|
|
2647
|
+
benchmark machine, 20 writers of one wallet with an aggregate did 152 operations per second (p95 is 0.47 s), without
|
|
2648
|
+
an aggregate 1,987. For hot counters, use a projector that folds a batch.
|
|
2649
|
+
- `reclass` and `rekey` accumulate contributions until the end of the move, while a delete cascade and the loader apply
|
|
2650
|
+
them per statement: the class rule is checked on the total of each cascade statement and of each loader batch. So a
|
|
2651
|
+
load in which debits come in a batch before top-ups gets `invalid_data`, although the total of the whole load is
|
|
2652
|
+
valid. Every batch that touches a target writes its version.
|
|
2653
|
+
- A tenant aggregate does not count rows of a System class, and the field of a System row counts only System sources.
|
|
2654
|
+
|
|
2655
|
+
**Agents and MCP**
|
|
2656
|
+
|
|
2657
|
+
- The `<untrusted-data-…>` wrapper is a mitigation, not a protection: the model may follow the text inside it. The
|
|
2658
|
+
protection is the rights in the database: the agent reads and writes only what its user is allowed to. In
|
|
2659
|
+
`structuredContent`, the database data lies in `data` and `error` next to the `notice`, without marks: a client that
|
|
2660
|
+
gives the model that instead of the text passes the data with the note but without a random mark.
|
|
2661
|
+
- By default the server only reads; writing is `write: true` (`--write`); deletion is only with a human confirmation.
|
|
2662
|
+
The server writes the permission classes only with `allowAclWrite` and accepts a session in the System tenant only
|
|
2663
|
+
with `allowSystemAdmin`; `create`, `update` and `upsert` of the other classes need no human confirmation.
|
|
2664
|
+
- The HTTP handler serves revision 2025 statelessly: the per-request instance does not know the client's
|
|
2665
|
+
capabilities, so deletion through it is refused (`confirmation_required`). Delete over stdio or in revision
|
|
2666
|
+
2026-07-28.
|
|
2667
|
+
- A confirmation in revision 2026-07-28 is signed with a process key: a retried call that lands in another process of
|
|
2668
|
+
the application (several instances behind a load balancer) is refused with `confirmation_required`.
|
|
2669
|
+
- `versions` returns the latest versions within `maxRows` and `maxBytes` (at most `maxRows` + 1 versions are read
|
|
2670
|
+
from the database) and `more`; earlier ones, through `diff` and `blame` with `from`, `to`, `at`. A mode where the
|
|
2671
|
+
agent writes code against the API (code mode) is not part of the package.
|
|
2672
|
+
|
|
1874
2673
|
**Import from 0.21**
|
|
1875
2674
|
|
|
1876
2675
|
- 1.0 is a new major version, not a continuation of 0.21: `count()`, `paths()`, `create`, roles, multiple ends and
|
|
@@ -1888,7 +2687,8 @@ This section is for those who develop letopis itself. Run: `cd lib && npm test`.
|
|
|
1888
2687
|
# connect — connecting under a session, refusing roles that bypass RLS, the registry, the deferred signal, error translation (stage 2)
|
|
1889
2688
|
# context — sessions, service sign-in by key, the RLS policies of stage 1 (stage 1)
|
|
1890
2689
|
# access-rest — the remaining rows of stage 5: loading on behalf of a user, balance No. 20, 22, 29, upsert and { rev } under ACL, inlining of acl_ok (stage 5)
|
|
1891
|
-
#
|
|
2690
|
+
# aggregate — aggregates in the database: 20 writers of one wallet, raw SQL, a cascade, reclass, restore, the loader, account deletion, restore and rekey of the target, a description change under writes and race No. 18, aggregate_field, repair and verify, a System row (plan 1.1, stage 4b)
|
|
2691
|
+
# acl — ACL permissions: the 0.21 suite ported, row conditions, weight and default deny, cascade, deny on a descendant, deep, an invisible target, raw SQL versus the library; a change of parent and alias of a class only with permissions on all its rows, an alias is not the name of another class, a name is not another class's alias (stage 5; security review No. 2)
|
|
1892
2692
|
# acl-parity — 1000 random rule sets: the database permission plan and endpoint decisions versus decide from 0.21 (stage 5)
|
|
1893
2693
|
# auth — credentials (password, key, key + secret, OTP, TOTP, identity), timing equalization, case, session revocation, recent sign-in (stage 5)
|
|
1894
2694
|
# balance — top-up and debit: idempotency by opId, races, deadlock, connection drop (TCP proxy), adjustment under { rev } and forUpdate (stage 4)
|
|
@@ -1898,21 +2698,21 @@ This section is for those who develop letopis itself. Run: `cd lib && npm test`.
|
|
|
1898
2698
|
# chain-write — writing with chains: create, update, upsert, delete with preview, anonymize, reclass, slots, inc, { rev }, fan-out, plan rollback (stage 4)
|
|
1899
2699
|
# demo — the booking demo domain on Zod: seed from models, writing data, id by key, agreement between Zod and the database (stage 2)
|
|
1900
2700
|
# docs — documentation examples are executed: the marked blocks of the README and the cheatsheet (Russian and English) run as scenarios against PostgreSQL 18 (stage 8)
|
|
1901
|
-
# engine-attacks — raw SQL attacks from the application role: the rows of §7.3.1 for stages 1 and 3 with specific outcomes (stages 1, 3)
|
|
1902
|
-
# engine-class — classes as entities: metaschema, inheritance, recomputing descendants, tightening, metadata (stage 1)
|
|
2701
|
+
# engine-attacks — raw SQL attacks from the application role: the rows of §7.3.1 for stages 1 and 3 with specific outcomes; the signal key without permission, flooding with fake signals, Class included; writing to a derived class and a reference to it, a projector checkpoint through the credential functions; an agent through MCP without --write and without a human confirmation; a change of parent, alias and derivedBy of a class with the WRITE permission on Class alone, deleting and re-creating a class with a history of rows, a trial tightening as an oracle, reclass without DELETE and rewriting references deeper than the first level, a retention policy with the WRITE permission on Class alone; an agent through MCP grants a membership or a rule, a system administrator session; a tenant user creates a derived class on their own, `rekeyClass` of an ancestor of a derived class; moving a row in a way that would rewrite references of derived class rows; the application role holds advisory locks with the former keys of internal locks, while lock rows are out of its reach; a forged note of the family locks already taken (stages 1, 3; plan 1.1, stages 2, 4a, 5, checkpoint A, security review No. 2, checkpoint B and the work after it, security review No. 3)
|
|
2702
|
+
# engine-class — classes as entities: metaschema, inheritance, recomputing descendants, tightening, metadata; a class with a history of rows is deleted only after purge() (stage 1; the work after checkpoint B)
|
|
1903
2703
|
# engine-write — the write pipeline in raw SQL: v5 ids, defaults, merge, inc, deletion, resurrection, journal, signals (stage 1)
|
|
1904
|
-
# engine-races — races on two connections in both orders, without timed pauses (stage 1)
|
|
1905
|
-
# explain — chain indexes from the application role: outside, the primary key and (class, tenant); inside find_*(), GIN (stage 3)
|
|
1906
|
-
# find-ids-parity — 30 chains through find_*() and through a direct read under RLS give the same answers (stage 3)
|
|
2704
|
+
# engine-races — races on two connections in both orders, without timed pauses; tightening during continuous writes to the family does not starve; locking families whose roots differ only in case, and a planted row of kind LOCK (stage 1; security review No. 3)
|
|
2705
|
+
# explain — chain indexes from the application role: outside, the primary key and (class, tenant); inside find_*(), GIN; an asOf slice by class does not read another tenant's journal (stage 3; the work after checkpoint B)
|
|
2706
|
+
# find-ids-parity — 30 chains through find_*() and through a direct read under RLS give the same answers; own memberships in other tenants and the Class rows of System — now, asOf and withDeleted (stage 3; the work after checkpoint B)
|
|
1907
2707
|
# hash — the version hash chain, journal tampering and gaps, verify() (stage 1)
|
|
1908
2708
|
# harness — the test harness: starting the database, a schema per file, two connection strings (stage 0)
|
|
1909
2709
|
# jsonschema — compiling JSON Schema to jsonpath and checking data in the database (stage 1)
|
|
1910
|
-
# impersonation — impersonation: letopis.account only for a service with the permission, target restrictions, author and agent in the journal, sessionFor (stage 5)
|
|
2710
|
+
# impersonation — impersonation: letopis.account only for a service with the permission, target restrictions, author and agent in the journal, sessionFor; the db.as() registry is one per tenant, does not empty, is re-read on a signal of its tenant (stage 5; the work after checkpoint B)
|
|
1911
2711
|
# install — the installer and roles: a repeated installation, two installations in one database, three tables (stage 1)
|
|
1912
2712
|
# member — memberships: working in a tenant by roles, auth.switch, leaving with ownership transfer, system rows, tenant roles without System permissions (stage 5)
|
|
1913
2713
|
# model — Zod models, the JSON Schema normalizer, pattern translation, agreement with the database metaschema (stage 2)
|
|
1914
2714
|
# load — the loader: several classes, cycles, $key and $ext, repeats, existing, portions, history mode, the query for targets outside the load, CLI (stage 4)
|
|
1915
|
-
# roles — roles and privileges: installation without a superuser, owner functions, triggers by catalog (stage 1)
|
|
2715
|
+
# roles — roles and privileges: installation without a superuser, owner functions, triggers by catalog, the log without the changed column (stage 1; plan 1.1, checkpoint A)
|
|
1916
2716
|
# write-races — write races: 20 concurrent edits of different fields without losses and strict rev, concurrent add/remove on a multiple end, the loader versus class tightening and versus a concurrent edit (existing: update) in both orders (stage 4)
|
|
1917
2717
|
# validator-parity — the TypeScript validator versus the database check: the official suite, 10,000 random documents, defaults (stage 2)
|
|
1918
2718
|
# scenario-barbershop — end-to-end barbershop scenario: 9 scenes from 0.21 (integration) on 1.0 chains — catalog, free slots in a batch, bookings under lock, multi-slot and multi-specialist, versions, cancellation and booking again on the same id, closing a month with restrict (stage 4)
|
|
@@ -1928,11 +2728,16 @@ This section is for those who develop letopis itself. Run: `cd lib && npm test`.
|
|
|
1928
2728
|
# verify — legitimate gaps and tampering, verify({ data: true }): data, ends, tenants, id by key; permissions and CLI (stage 6)
|
|
1929
2729
|
# retention — retention policy: tiers, a finite term, special versions, deleted rows, asOf, cursor_expired, time zone, catch-up run, inheritance and default — versus a reference in TypeScript (stage 6)
|
|
1930
2730
|
# watch — cursor subscription: commit order, a long transaction, resuming, listener drop, two tenants (stage 6)
|
|
1931
|
-
# cache — result cache: hits, own and others' writes, the key with the user, permissions, revoked sessions, listener drop, ttl, class marking, without LISTEN, a race, db.sql (stage 6)
|
|
2731
|
+
# cache — result cache: hits, own and others' writes, the key with the user, permissions, revoked sessions, listener drop, ttl, class marking, without LISTEN, a race, db.sql, the limit of signal counters (stage 6; plan 1.1, checkpoint A)
|
|
1932
2732
|
# maintain — maintenance: expired sessions and codes, permissions, one process under a lock, schedule (stage 6)
|
|
1933
2733
|
# up — the full installer: fresh installation, re-apply, custom seeds, automatic database creation, UTF8, version 17, upgrade with migrations, PGlite, the first tenant and user (up({ tenant }), createTenant) (stages 6, 8)
|
|
1934
2734
|
# scenario-salon — end-to-end salon scenario: 21 acts from 0.21 (salon) with a coverage matrix of the 1.0 API (stage 6)
|
|
1935
2735
|
# telemetry — onQuery and slowMs, describe() and class metadata with inheritance, describeText (stage 6)
|
|
2736
|
+
# history — diff and blame over the log, the reason of a change for verbs, db.begin() and batches; a version hidden by an ACL condition, the changed column, a missing version, row_changes over 20 thousand targets (plan 1.1, stage 1 and checkpoint A)
|
|
2737
|
+
# signals — signal modes: the payload seen by a foreign role, cache precision per tenant, System, permission classes, a process without the key, the loader, subscriptions, the key, none, coalescing, the server version, a class change, a resend, the mode after up() and reset({ all }) (plan 1.1, stage 2)
|
|
2738
|
+
# initial-load — the initial load: equality with the regular loader, structure and trigger modes, system rows, a non-empty schema, an insert during the start, rollback, MULTI_INSERT in WAL, commitEvery, import --initial, aggregates (agg_fill) (plan 1.1, stages 3 and 4b)
|
|
2739
|
+
# projector — projectors from the journal: catching up, exactly once on a break, one writer, a long transaction, a transaction on the boundary of the rebuild snapshot, no writes to derived classes and no references to them, the bounds of out, atomic and batch rebuilds during source writes, purge of a source, lag and cursor_expired, the credential functions; the version, the name, an admin connection; a race of two processes; a data error of one tenant drops out only this tenant (failed, rebuild({ tenant }), the tenants option), the checkpoint is only a System row, a rebuild touches only the rows of participants and into families, before after rekey, the all tier and the journal of sources per tenant; folding of out.inc by key in a batch, batch rebuilds on the automatic paths of run() and a second process during them; rebuilding one tenant in batches while the tail goes on; onExpired in batches in another process, a data error without retries, the tier from the start of the rebuild; an empty last pass of a batch rebuild, a tenant that drops out in the middle of its rebuild keeps no partial rows, a class change that keeps the id out of the source family and into it, purge of an unread tombstone (plan 1.1, stage 4a; security review No. 2; checkpoint B and the work after it; security review No. 3)
|
|
2740
|
+
# mcp — MCP tools: the model and meta only in results, tenant rights, the JSON chain and hints of similar names, row and byte limits with a cursor, the untrusted-data wrapper, writing only with write, update only with rev, deletion with a human confirmation (elicitation in revisions 2025 and 2026-07-28), the reason in blame, credentials, tool schemas, letopis without the SDK; letopis mcp over stdio, the HTTP handler; the confirmation window without the alias and meta, countBy and versions limited in the query itself, limits of steps and $or conditions; the permission classes only with allowAclWrite, a session in the System tenant only with allowSystemAdmin; structuredContent with a note, a requestState signature of the server user with an expiry (plan 1.1, stage 5; security review No. 2; checkpoint B and the work after it)
|
|
1936
2741
|
# import — import from 0.21 with a fixture of a real 0.21: key violation, --keyless, per-class counters, verify, self-reference, repeat, ids without a key, a dangling reference, $$strict and fields starting with $, update, credentials, CLI (stage 7)
|
|
1937
|
-
# security-final — the final security review (stage 8, §7.10 No. 8.4): a family lock by the tenant of the root, account_purge without permission and restore/purge of a foreign id give neither a leak nor a lock; the functions of stage 6, log_refs_at, journal-based find_* reads and watch_read stay within their own tenant; NOTIFY carries only names; the cache key distinguishes the token, the actor and the tenant (stage 8)
|
|
2742
|
+
# security-final — the final security review (stage 8, §7.10 No. 8.4): a family lock by the tenant of the root, account_purge without permission and restore/purge of a foreign id give neither a leak nor a lock; the functions of stage 6, log_refs_at, journal-based find_* reads and watch_read stay within their own tenant; NOTIFY carries only names or mode tokens; the cache key distinguishes the token, the actor and the tenant (stage 8)
|
|
1938
2743
|
```
|