letopis 1.1.0 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENT-CHEATSHEET.en.md +113 -16
- package/AGENT-CHEATSHEET.md +106 -14
- package/CHANGELOG.md +702 -0
- package/README.en.md +798 -74
- package/README.md +756 -68
- package/dist/admin.d.ts +20 -4
- package/dist/admin.js +33 -9
- package/dist/cli.js +53 -4
- package/dist/errors.js +4 -2
- package/dist/index.d.ts +16 -2
- package/dist/index.js +94 -23
- package/dist/internal.d.ts +16 -0
- package/dist/internal.js +14 -0
- package/dist/load.js +42 -11
- package/dist/mcp.d.ts +28 -0
- package/dist/mcp.js +810 -0
- package/dist/model.d.ts +34 -0
- package/dist/model.js +17 -0
- package/dist/project.d.ts +318 -0
- package/dist/project.js +1142 -0
- package/dist/registry.d.ts +17 -0
- package/dist/registry.js +13 -0
- package/dist/sql.js +9 -5
- package/dist/sync.js +11 -2
- package/dist/types.d.ts +1 -1
- package/dist/types.js +1 -1
- package/dist/uuid.js +19 -8
- package/dist/watch.d.ts +12 -0
- package/dist/watch.js +48 -4
- package/dist/write.d.ts +1 -1
- package/dist/write.js +25 -5
- package/llms.txt +6 -2
- package/package.json +13 -2
- package/sql/10-core.sql +8 -2
- package/sql/15-errors.sql +4 -1
- package/sql/40-class.sql +57 -2
- package/sql/60-write.sql +629 -56
- package/sql/70-read.sql +33 -8
- package/sql/80-auth.sql +4 -4
- package/sql/88-derived.sql +1154 -0
- package/sql/90-time.sql +154 -13
- package/sql/95-seed.system.sql +12 -2
- package/sql/README.md +4 -1
package/README.en.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
# letopis 1.
|
|
1
|
+
# letopis 1.3
|
|
2
2
|
|
|
3
3
|
**letopis** is a Node.js library: a store of versioned entities on PostgreSQL 18. Every change is saved as a version
|
|
4
4
|
in a journal. So every object has a history, a state as of any date and a change subscription. The database itself
|
|
@@ -14,11 +14,15 @@ What letopis can do:
|
|
|
14
14
|
- **Writing with chains**: `create`, `update` with merge, `upsert`, deletion with preview and cascade, the `inc` increment
|
|
15
15
|
for money and counters — [sections 5](#5-writing), [7](#7-deleting-and-restoring) and [8](#8-transactions-batches-and-money).
|
|
16
16
|
- **Reading with chains**: hops along links, filters, trees, aggregates, pagination — [section 6](#6-reading).
|
|
17
|
-
- **History**: versions, point-in-time reads,
|
|
18
|
-
[section 11](#11-history-and-point-in-time-reads).
|
|
17
|
+
- **History**: versions, point-in-time reads, what changed and who changed it (`diff`, `blame`), restoring deleted
|
|
18
|
+
data, retention policy, hash chain verification — [section 11](#11-history-and-point-in-time-reads).
|
|
19
19
|
- **Tenants, sign-in and permissions** in the database itself: passwords, API keys, one-time codes, TOTP, sessions,
|
|
20
20
|
impersonation, access rules — [section 10](#10-users-sign-in-and-permissions).
|
|
21
21
|
- **Change subscription** by cursor without losses, and a **result cache** — [section 12](#12-change-subscription-and-cache).
|
|
22
|
+
- **Derived data**: aggregates in the database, sums and counts that the database itself maintains, and projectors
|
|
23
|
+
that build summaries from the journal — [sections 4](#aggregates-in-the-database) and [12](#projectors).
|
|
24
|
+
- **Agents and MCP**: an MCP server gives an AI agent the model, reads, history and, if allowed, writes, with the
|
|
25
|
+
rights of one user — [section 13](#agents-and-mcp).
|
|
22
26
|
- **Bulk loading** of millions of rows and **import from 0.21** — [sections 9](#9-bulk-loading) and [14](#14-migrating-from-021).
|
|
23
27
|
|
|
24
28
|
Requirements:
|
|
@@ -28,7 +32,8 @@ Requirements:
|
|
|
28
32
|
- Node.js 22 or newer; the package is an ES module.
|
|
29
33
|
- One runtime dependency: `postgres`. Zod 4 is an optional peer dependency: only the `letopis/model` subpath needs it,
|
|
30
34
|
and `import { connect } from 'letopis'` works without it. For development without PostgreSQL there are the optional
|
|
31
|
-
`@electric-sql/pglite` and `@electric-sql/pglite-socket`.
|
|
35
|
+
`@electric-sql/pglite` and `@electric-sql/pglite-socket`. The MCP server (`letopis/mcp` and `letopis mcp`) needs
|
|
36
|
+
the optional `@modelcontextprotocol/server`, version 2.4 or newer in the 2.x line.
|
|
32
37
|
|
|
33
38
|
## Contents
|
|
34
39
|
|
|
@@ -154,8 +159,8 @@ service key again while the installation still has a valid API key. Next, sectio
|
|
|
154
159
|
|
|
155
160
|
There are three tables. `entity` is the current state: one row per live entity, including the rows of system classes
|
|
156
161
|
(`Class`, `Account`, `Resource`, `member`, `rule`). `log` is the journal: a full snapshot of every version with author,
|
|
157
|
-
time, kind of operation and hash. `secret` holds credentials and
|
|
158
|
-
the sign-in functions work with it.
|
|
162
|
+
time, kind of operation and hash. `secret` holds credentials, sessions and projector checkpoints: the application sees
|
|
163
|
+
none of its rows; only the sign-in and projector functions work with it.
|
|
159
164
|
|
|
160
165
|
Classes are entities too. A class description is stored in a `Class` row; the database also keeps the expanded
|
|
161
166
|
description (with ancestors applied) and the compiled data check there. The version hash is a generated column: sha256 of the
|
|
@@ -191,23 +196,27 @@ is built from the journal.
|
|
|
191
196
|
|
|
192
197
|
### Signals
|
|
193
198
|
|
|
194
|
-
A write reports changes through `NOTIFY
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
199
|
+
A write reports changes through `NOTIFY`. The channel is the full name of the installation schema, version included,
|
|
200
|
+
for example `v2.shop`; because of the dot, SQL takes the name in double quotes: `LISTEN "v2.shop"`. A raw SQL write on
|
|
201
|
+
its own connection sends the signal from a trigger, and PostgreSQL delivers it on commit. Library transactions,
|
|
202
|
+
`db.sql` and `db.transaction()` included, defer their signals: the process notifier sends them every 100 ms
|
|
203
|
+
(`notifyIntervalMs`) and retries the ones that failed to send. What a write signal carries (its payload) is set by the
|
|
204
|
+
installation mode `up({ signals })`:
|
|
198
205
|
|
|
199
|
-
| Mode | Write signal | What any role that can connect to the database sees |
|
|
206
|
+
| Mode | Write signal payload | What any role that can connect to the database sees |
|
|
200
207
|
|---|---|---|
|
|
201
208
|
| `class` (default) | the class name: `Booking` | class names and write times |
|
|
202
|
-
| `tenant` |
|
|
203
|
-
| `opaque` |
|
|
209
|
+
| `tenant` | `Class@tenant token`: `Booking@3f9c…` | also tenant pseudonyms |
|
|
210
|
+
| `opaque` | `class token@tenant token`: `9a1e…@3f9c…` | only write times and frequency per pseudonym |
|
|
204
211
|
| `none` | no signals | nothing |
|
|
205
212
|
|
|
206
|
-
A
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
and
|
|
210
|
-
|
|
213
|
+
A token is 24 hexadecimal characters of a keyed HMAC: only a process with the signal key can match a pseudonym to a
|
|
214
|
+
name, and the key is given to services (the `letopis.signals` permission). Besides write signals, there are service
|
|
215
|
+
ones: a revoked session sends `session:<16 hex>` (the first 16 hexadecimal characters of the sha256 of its token), a
|
|
216
|
+
session reset sends `session:*`, and a reset of a tenant or of the whole installation sends `*` (a reset of the whole
|
|
217
|
+
installation also sends `session:*`). In the `none` mode there are none of them either. Signals carry no ids and no
|
|
218
|
+
data. On signals, the class registry is re-read, the result cache is cleared and subscriptions wake up; what the mode
|
|
219
|
+
changes for the cache and subscriptions is in [section 12](#12-change-subscription-and-cache).
|
|
211
220
|
|
|
212
221
|
## 3. Installation and connection
|
|
213
222
|
|
|
@@ -265,8 +274,8 @@ const ANNA = r.tenant!.user! // its first user
|
|
|
265
274
|
The application works as a LOGIN role that is a member of `letopis_app` only:
|
|
266
275
|
`create role shop_app login password '…'; grant letopis_app to shop_app;`. Then RLS and triggers apply to it, as to
|
|
267
276
|
any raw SQL from this role. `connect()` rejects a role that bypasses RLS with the error `bypass_rls`. The admin
|
|
268
|
-
connection (`allowBypassRls: true`) is needed only for the loader,
|
|
269
|
-
process, not in the web application.
|
|
277
|
+
connection (`allowBypassRls: true`) is needed only for the loader, projectors, aggregate repair (`rebuildAggregates`),
|
|
278
|
+
verification and maintenance; keep it in a separate process, not in the web application.
|
|
270
279
|
|
|
271
280
|
### First tenant and user
|
|
272
281
|
|
|
@@ -339,7 +348,8 @@ through `as()` share the pool with the original one: close the original one.
|
|
|
339
348
|
|
|
340
349
|
### Admin connection
|
|
341
350
|
|
|
342
|
-
The loader
|
|
351
|
+
The loader, projectors and `rebuildAggregates` need a connection of a role that can become the owner, with
|
|
352
|
+
`allowBypassRls: true`. `verify()` and
|
|
343
353
|
`maintain()` work on it too, and also for a system administrator on a regular connection. The admin connection prints
|
|
344
354
|
the warning "RLS не действует" ("RLS does not apply"); this is expected. The session sets the tenant it works with. A
|
|
345
355
|
separate loading process is better off with a service key:
|
|
@@ -353,12 +363,23 @@ const adm = await connect({ dsn: ADMIN, schema: r.schema, token: r.tenant!.token
|
|
|
353
363
|
|
|
354
364
|
### Upgrading the package and schema
|
|
355
365
|
|
|
356
|
-
The schema has an engine revision (currently
|
|
366
|
+
The schema has an engine revision (currently 6). If a new package version brings a new revision, `connect()` only warns
|
|
357
367
|
and names the command. It is `up()` that refuses: a lagging schema without `upgrade: true`, and a schema newer than the
|
|
358
368
|
package. So a startup script that calls `up()` on every start will stop after a package update until it gets
|
|
359
369
|
`upgrade: true`. `up({ dsn, schema, upgrade: true })` applies the migrations from `sql/migrate/` and re-applies the
|
|
360
370
|
functions; the data is kept.
|
|
361
371
|
|
|
372
|
+
The first `up()` after moving to 1.2 builds the journal index `log_class_tenant_at (class, tenant, at)` instead of
|
|
373
|
+
`log_class_at`: an `asOf` and `withDeleted` slice by class no longer reads the journal of other tenants with the same
|
|
374
|
+
class name. While the index is being built, writes to the journal wait. On a large installation, build it in advance
|
|
375
|
+
without stopping writes: `create index concurrently log_class_tenant_at on "v2.<schema>".log (class, tenant, at)` as
|
|
376
|
+
the owner role — then `up()` only drops the old one.
|
|
377
|
+
|
|
378
|
+
The `upgrade` test checks upgrading 1.1.0 and 1.0.1 schemas. The old package installs the schema and the old engine
|
|
379
|
+
writes the data, then the current version's `up()` runs. After it, the versions and hashes written by the old engine
|
|
380
|
+
are in place, `verify({ data: true })` is clean, and the schema catalog matches a clean installation. A 1.1.0 schema is
|
|
381
|
+
upgraded with a plain `up()`, since its revision is the same. A 1.0.1 schema is upgraded with `up({ upgrade: true })`.
|
|
382
|
+
|
|
362
383
|
### Development without PostgreSQL: PGlite
|
|
363
384
|
|
|
364
385
|
`up({ pglite: true })` starts PGlite (PostgreSQL 18 on WebAssembly) with pgcrypto and a socket on a free port. It needs
|
|
@@ -488,13 +509,78 @@ name of the class for a chain step, for example a name in another language.
|
|
|
488
509
|
|
|
489
510
|
- `history: false`: no journal is kept for the class: no versions, no point-in-time reads, no restore and no
|
|
490
511
|
subscription events. A retention policy is `history: { all: '1 day', daily: '3 months', … }`
|
|
491
|
-
([section 11](#11-history-and-point-in-time-reads)).
|
|
512
|
+
([section 11](#11-history-and-point-in-time-reads)). Only someone with the `READ` and `DELETE` permissions without
|
|
513
|
+
conditions on all rows of the class and its descendants may change `history` of an existing class: `maintain()` thins
|
|
514
|
+
the journal by the policy, and `false` turns it off.
|
|
492
515
|
- `ownerDefault: 'actor'`: the owner of new rows is the user, not the tenant (the default is `'tenant'`). This is
|
|
493
516
|
useful for the rule "everyone sees their own records".
|
|
494
517
|
- `cache: { ttl }`: reads of this class alone are cached without `.cache()` ([section 12](#12-change-subscription-and-cache)).
|
|
495
518
|
- `meta`: free-form metadata for the application: a title, an icon, a color, an order in a menu. Up to 64 KB,
|
|
496
519
|
inherited key by key (`null` removes an inherited value), visible to anyone who has a session. For an end, use
|
|
497
520
|
`.meta()`.
|
|
521
|
+
- `derivedBy: 'projector-name'`: a derived class: only this projector writes its rows ([Projectors](#projectors)).
|
|
522
|
+
The name is lowercase Latin letters, digits and `_.:-` (the first character is a letter or a digit), up to 64
|
|
523
|
+
characters. The option is inherited, and a descendant does not change it. The ends of a derived class are only
|
|
524
|
+
`onDelete: 'cascade'` or `'unset'`: a derived row does not prevent deleting its target. Nothing refers to the rows
|
|
525
|
+
of a derived class. The application, raw SQL and the loader do not write to such a class: writing, editing and
|
|
526
|
+
deleting its rows gives `derived_class`. The derived class itself (own or inherited) is created, edited and deleted
|
|
527
|
+
only by an administrator — an admin connection or an account with the System permission `letopis.projector`; for an
|
|
528
|
+
application session it is `acl_denied` ([Projectors](#projectors)).
|
|
529
|
+
- `aggregate: { field: { sum | count, of, via, where } }`: an aggregate in the database: the database itself keeps
|
|
530
|
+
the sum or count of source rows ([Aggregates in the database](#aggregates-in-the-database)).
|
|
531
|
+
|
|
532
|
+
### Aggregates in the database
|
|
533
|
+
|
|
534
|
+
An aggregate is a field that the database itself maintains: the sum of a field or the number of live rows of a
|
|
535
|
+
source class whose end leads to the row. It is needed where the total must match the source even for raw SQL: a
|
|
536
|
+
wallet balance, the number of order lines.
|
|
537
|
+
|
|
538
|
+
```ts
|
|
539
|
+
const Wallet = hub('Wallet', { number: z.string(), balance: z.number().int().min(0), deposits: z.number().int().optional() }, {
|
|
540
|
+
key: ['number'],
|
|
541
|
+
aggregate: {
|
|
542
|
+
balance: { sum: 'amount', of: 'Movement', via: 'Wallet' }, // sum of movements
|
|
543
|
+
deposits: { count: true, of: 'Movement', via: 'Wallet', where: { properties: { amount: { minimum: 1 } } } }, // number of top-ups
|
|
544
|
+
},
|
|
545
|
+
})
|
|
546
|
+
const Movement = link('Movement', { amount: z.number().int() }, { ends: { Wallet } })
|
|
547
|
+
```
|
|
548
|
+
|
|
549
|
+
- `sum: 'field'` is the sum of a numeric field of the source, `count: true` is the number of rows. `of` is the source
|
|
550
|
+
class (a name or a model) together with its descendants, `via` is its single end that leads to the target row.
|
|
551
|
+
`where` is a JSON Schema over the source data: rows that do not pass it are not counted.
|
|
552
|
+
- Only live rows of the same tenant as the target are counted. A reference to a System row is not counted: a shared
|
|
553
|
+
directory does not count the data of all tenants. An aggregate of a tenant class, including one inherited from a
|
|
554
|
+
System class, counts only classes of its own tenant: a System class named `of` is not counted, while the tenant's
|
|
555
|
+
own descendant of a System class is. An aggregate of a System class counts System classes and the classes of the
|
|
556
|
+
target row's tenant.
|
|
557
|
+
- A trigger recalculates the field after every statement: writes by the library, raw SQL, a delete cascade,
|
|
558
|
+
`restore`, `reclass`, `rekey`, the loader, account deletion. The operations of one statement add up to one increment
|
|
559
|
+
of the target, and the class rule (`min(0)`) is checked on the total: an operation that would take the balance below
|
|
560
|
+
zero is rejected with `invalid_data`. Every change of the target is its new version with the author and reason of
|
|
561
|
+
the operation.
|
|
562
|
+
- Only the database writes the field. A value in `create`, `update`, `upsert`, `rekey`, `reclass` or raw SQL is an
|
|
563
|
+
`aggregate_field` error. A new row gets the total of already existing sources, usually 0. Replacing the data as a
|
|
564
|
+
whole without the field keeps the previous value. The loader does not write a passed value. No permission to write
|
|
565
|
+
the target is needed: the permission to write the source is enough.
|
|
566
|
+
- The database checks the description (`invalid_class`): the field is in the schema, it is numeric and not part of the
|
|
567
|
+
key; the `via` end is single and accepts the target class; the `sum` field is a number; aggregates do not form a
|
|
568
|
+
cycle; the target and the source are not system or derived classes. A source class is described before the
|
|
569
|
+
aggregate: an aggregate over a class that does not exist yet is refused. For a tenant class, the source is looked up
|
|
570
|
+
among the tenant's classes. `syncModels` describes sources before targets itself. The `of` class is not deleted while
|
|
571
|
+
an aggregate of a class of the same tenant names it: first the aggregate is removed from the target's description.
|
|
572
|
+
- Rows get into an aggregate only with the knowledge of someone who reads them all: the field also counts rows that
|
|
573
|
+
the author cannot see. Only someone who reads all rows of the source and its descendants without conditions can
|
|
574
|
+
describe or change an aggregate. Only someone who reads all rows of a class and its descendants without conditions,
|
|
575
|
+
both before the change and after it, can make the class a source (a new descendant of a source, a `parent` change).
|
|
576
|
+
Otherwise `acl_denied`.
|
|
577
|
+
- The option is inherited, and a descendant replaces it as a whole. Adding or changing an aggregate is a tightening:
|
|
578
|
+
writes to the target and to the sources wait while the field is recalculated for all rows.
|
|
579
|
+
- `db.rebuildAggregates(class)` recalculates the field from the current rows, for repair. A version is written only
|
|
580
|
+
where the value has changed. `db.verify({ data: true })` shows discrepancies. It needs an admin connection.
|
|
581
|
+
- A raw `INSERT … ON CONFLICT DO UPDATE SET data = excluded.data` for a target is rejected: the insert trigger has
|
|
582
|
+
already set the field in `excluded.data`. Write `excluded.data - array['balance']`, and the field stays as it was;
|
|
583
|
+
the library's `upsert` does this itself.
|
|
498
584
|
|
|
499
585
|
### Writing the model to the database
|
|
500
586
|
|
|
@@ -636,6 +722,13 @@ without reads or locks in the application. Rules:
|
|
|
636
722
|
- an `inc` marker cannot come from JSON: `{ "$inc": 5 }` that came from the network stays data;
|
|
637
723
|
- store money as an integer number of minor units (cents), no greater than 2^53 − 1.
|
|
638
724
|
|
|
725
|
+
A counter that must match the source even for raw SQL is easier to keep as an aggregate
|
|
726
|
+
([Aggregates in the database](#aggregates-in-the-database)): the database recalculates it itself, and you cannot forget
|
|
727
|
+
an `inc`. An aggregate and `inc` cost the same: operations on one object go one after another under the lock of its
|
|
728
|
+
row, and each writes its version. On the benchmark machine, one object took 150–190 operations per second this way, no
|
|
729
|
+
matter how many writers were waiting for it ([section 17](#17-performance)). For hotter counters, use a projector that
|
|
730
|
+
folds a batch ([Projectors](#projectors)): the total lags behind the source, but writers do not wait for each other.
|
|
731
|
+
|
|
639
732
|
### Protection against concurrent edits: `{ rev }`
|
|
640
733
|
|
|
641
734
|
<!-- run: guide -->
|
|
@@ -666,9 +759,12 @@ const [vip] = await db.Customer(olga).reclass('Vip', { name: 'Olga', email: 'olg
|
|
|
666
759
|
- `rekey(patch)` sets a new key value: a row with a new id, all references moved, the old row deleted. If a referencing
|
|
667
760
|
row includes this end in its own key, its id changes too, and the move goes on. The journal links the old and the new
|
|
668
761
|
id (`moved`).
|
|
669
|
-
- `db.rekeyClass(class, key)` sets a new key for all rows of the family, in one transaction.
|
|
670
|
-
|
|
671
|
-
|
|
762
|
+
- `db.rekeyClass(class, key)` sets a new key for all rows of the family, in one transaction. For a derived class and
|
|
763
|
+
for a class with a derived descendant — `derived_class`: only its projector writes the rows of a derived class.
|
|
764
|
+
- `reclass(class, data)` changes the class; the data is optional, and if given, it is given whole. The row leaves its
|
|
765
|
+
previous class, so it needs the `DELETE` permission on it, and `WRITE` on the new class and on every row whose
|
|
766
|
+
references the move rewrites, at any depth. If the new class has no key, the id is kept (as for `vip` above); if
|
|
767
|
+
it has one, the id is computed anew and the references are moved.
|
|
672
768
|
The roles of referencing rows must accept the new class, otherwise `reclass_denied`. A raw `UPDATE … SET class`
|
|
673
769
|
passes only if neither the old nor the new class has a key (otherwise `use_reclass`).
|
|
674
770
|
- Deleting, changing the class and changing the key work only in `read committed`: library transactions always open at
|
|
@@ -820,7 +916,8 @@ const avgPrice = await db.Shop(shop).Product().avg('data.price')
|
|
|
820
916
|
const byStatus = await db.Order().countBy('data.status') // { new: 1 }
|
|
821
917
|
```
|
|
822
918
|
|
|
823
|
-
The database computes aggregates over the entities of the last step.
|
|
919
|
+
The database computes aggregates over the entities of the last step. `limit(n)` and `offset(n)` before `countBy()`
|
|
920
|
+
apply to the groups: `db.Order().limit(3).countBy('data.status')` gives the three most frequent values.
|
|
824
921
|
|
|
825
922
|
### Paths
|
|
826
923
|
|
|
@@ -887,8 +984,12 @@ await db.Customer(temp).purge({ confirm: true }).rows() // history erased
|
|
|
887
984
|
```
|
|
888
985
|
|
|
889
986
|
`purge` works only for deleted objects (a live one gives `not_deleted`) and erases their whole history; in place of
|
|
890
|
-
version 1, a `purge` record stays with the number of erased versions. The history of system classes is not erased.
|
|
891
|
-
|
|
987
|
+
version 1, a `purge` record stays with the number of erased versions. The history of system classes is not erased.
|
|
988
|
+
While a projector whose `from` sources include the class of the object ([“Projectors”](#projectors)) has not read
|
|
989
|
+
the deletion, `purge` refuses (`projector_pending`): the projector would never see an erased tombstone, and the
|
|
990
|
+
object's contribution would stay in the totals. So a deletion and a `purge` in one transaction do not go through in
|
|
991
|
+
such a class. Likewise, `db.trimHistory` refuses while the projector has not read the first remaining version: it
|
|
992
|
+
would no longer find the previous version for `before`. To delete an account, use `db.auth.purgeAccount` ([section 10](#10-users-sign-in-and-permissions)).
|
|
892
993
|
|
|
893
994
|
## 8. Transactions, batches and money
|
|
894
995
|
|
|
@@ -1031,6 +1132,67 @@ try {
|
|
|
1031
1132
|
- A transfer is two operations in one `db.begin()`, with the wallets edited in id order: then there are no deadlocks.
|
|
1032
1133
|
- Insufficient funds are told apart from other data errors by `e.issues`: the field `balance`, the rule `minimum`.
|
|
1033
1134
|
|
|
1135
|
+
### Recipe: balance as an aggregate
|
|
1136
|
+
|
|
1137
|
+
The same wallet, but the database itself keeps the balance: `balance` is an aggregate, the sum of `amount` over the
|
|
1138
|
+
live operations ([Aggregates in the database](#aggregates-in-the-database)). An operation is one `Movement` record;
|
|
1139
|
+
there is no `inc` increment in the code, so it cannot be forgotten. The balance stays correct even when raw SQL or the
|
|
1140
|
+
loader writes the operations and a cascade deletes them.
|
|
1141
|
+
|
|
1142
|
+
```ts
|
|
1143
|
+
const Wallet = hub('Wallet', { number: z.string(), balance: z.number().int().min(0) }, {
|
|
1144
|
+
ends: { Customer }, key: ['number'],
|
|
1145
|
+
aggregate: { balance: { sum: 'amount', of: 'Movement', via: 'Wallet' } }, // the balance is the sum of live operations
|
|
1146
|
+
})
|
|
1147
|
+
const Movement = link('Movement', { opId: z.string(), amount: z.number().int() },
|
|
1148
|
+
{ ends: { Wallet }, key: ['Wallet', 'opId'] }) // the operation key gives idempotency
|
|
1149
|
+
|
|
1150
|
+
/** Operation: an amount above zero tops up, below zero debits; the database recalculates the balance in the same transaction. */
|
|
1151
|
+
async function operate(db: Db, wallet: string, opId: string, amount: number) {
|
|
1152
|
+
try {
|
|
1153
|
+
await db.Wallet(wallet).Movement().create({ opId, amount }).rows()
|
|
1154
|
+
return 'done'
|
|
1155
|
+
} catch (e) {
|
|
1156
|
+
if (e instanceof LetopisError && e.code === 'exists') return 'repeat' // a repeat after a lost response
|
|
1157
|
+
throw e // invalid_data: the balance would go below zero (min(0))
|
|
1158
|
+
}
|
|
1159
|
+
}
|
|
1160
|
+
|
|
1161
|
+
/** Adjustment "balance = target": an operation for the difference, with the wallet locked until commit. */
|
|
1162
|
+
async function adjust(db: Db, wallet: string, opId: string, target: number) {
|
|
1163
|
+
const tx = await db.begin()
|
|
1164
|
+
try {
|
|
1165
|
+
const w = (await tx.Wallet(wallet).forUpdate().first())!
|
|
1166
|
+
await tx.Wallet(wallet).Movement().create({ opId, amount: target - (w.data.balance as number) }).rows()
|
|
1167
|
+
await tx.commit()
|
|
1168
|
+
return 'done'
|
|
1169
|
+
} catch (e) {
|
|
1170
|
+
await tx.rollback()
|
|
1171
|
+
if (e instanceof LetopisError && e.code === 'exists') return 'repeat' // an adjustment with this opId is already written
|
|
1172
|
+
throw e
|
|
1173
|
+
}
|
|
1174
|
+
}
|
|
1175
|
+
|
|
1176
|
+
const [wallet] = await db.Customer(olga).Wallet().create({ number: 'W-1' }).rows() // balance is 0: the database sets it
|
|
1177
|
+
await operate(db, wallet.id, 'op-1', 10_000) // balance 10,000
|
|
1178
|
+
await operate(db, wallet.id, 'op-2', -3_000) // balance 7,000
|
|
1179
|
+
await adjust(db, wallet.id, 'op-3', 5_000) // an operation of −2,000, balance 5,000
|
|
1180
|
+
```
|
|
1181
|
+
|
|
1182
|
+
- The model here is its own, and the block is not part of the example program: there the balance grows through `inc`,
|
|
1183
|
+
and an aggregate field does not change that way.
|
|
1184
|
+
- `balance` is passed neither to `create` nor to `update` (`aggregate_field`), so an adjustment is an operation too,
|
|
1185
|
+
not an `update` under `{ rev }`. Writing an operation needs no permission to write the wallet: the permission to
|
|
1186
|
+
write `Movement` is enough.
|
|
1187
|
+
- Insufficient funds are `invalid_data`, as with `inc`: `e.issues` holds the field `balance` and the rule `minimum`.
|
|
1188
|
+
Check a repeat as `same()` in the recipe above does: `exists` only says that an operation with this `opId` already
|
|
1189
|
+
exists.
|
|
1190
|
+
- Editing and deleting an operation change the balance too: the database subtracts the previous value. It is still
|
|
1191
|
+
better to cancel an operation with a reverse one, so that the history of the wallet stays complete.
|
|
1192
|
+
- Every operation locks the wallet row and writes its version, so the operations of one wallet go one after another,
|
|
1193
|
+
as with `inc`: about 150 per second on the benchmark machine ([section 17](#17-performance)). If one object gets
|
|
1194
|
+
more, compute the total with a projector instead ([Recipe: daily revenue](#recipe-daily-revenue)).
|
|
1195
|
+
|
|
1034
1196
|
## 9. Bulk loading
|
|
1035
1197
|
|
|
1036
1198
|
### When you need the loader
|
|
@@ -1138,7 +1300,8 @@ problem. Its `detail`: `rows` is the first 100 violations with the row number, c
|
|
|
1138
1300
|
`limit` is the list limit (100), `complete` means the checks that found violations went through all rows. Row codes:
|
|
1139
1301
|
the codes of the database checks, and also `duplicate` (a repeated id), `exists` (the object already exists), `deleted`
|
|
1140
1302
|
(an object with this id existed before) and `class_not_loadable` (the loader does not accept `Class` rows: classes are
|
|
1141
|
-
set by `syncModels`).
|
|
1303
|
+
set by `syncModels`). The loader rejects rows of derived classes and references to them with the code
|
|
1304
|
+
`derived_class`: only a projector writes such rows.
|
|
1142
1305
|
|
|
1143
1306
|
Violations are searched for in stages: parsing the rows (classes, ids, repeats), permissions on classes
|
|
1144
1307
|
(`load({ as })`), data and ends in batches, and at the end of a portion the targets outside the load, owners and the
|
|
@@ -1298,6 +1461,33 @@ cut down to 999, so System rules with a weight of 1000 or more are an administra
|
|
|
1298
1461
|
outweigh. Tenant rules apply to the tenant's rows, System rules apply everywhere. Permissions apply to raw SQL too: rows
|
|
1299
1462
|
without permission are invisible, and an edit changes nothing or fails with `acl_denied`.
|
|
1300
1463
|
|
|
1464
|
+
A class mask in a `READ`, `WRITE` or `DELETE` object matches the descendants of the class too, by the names of the
|
|
1465
|
+
ancestors and their aliases. So an edit of a class that changes `parent`, `alias` or `derivedBy` needs the `READ`,
|
|
1466
|
+
`WRITE` and `DELETE` permissions without conditions on all rows of the class and of each of its descendants,
|
|
1467
|
+
otherwise `acl_denied`: the `WRITE` permission on `Class` alone is not enough to put the rows of a class under other
|
|
1468
|
+
rules. An alias does not match the name of another class (including `Class` and the system ones) or another class's
|
|
1469
|
+
alias in the tenant and System, and the name of a new class does not match another class's alias: otherwise a rule
|
|
1470
|
+
written for one class would act on the rows of another too. A violation gives `invalid_class`.
|
|
1471
|
+
|
|
1472
|
+
By the same names, the rules also find the old versions of rows in the journal. So a class is deleted only when it has
|
|
1473
|
+
no rows, no descendants and no history of rows: `purge()` erases the history of deleted rows first. For a System class,
|
|
1474
|
+
the rows, descendants and history of a tenant that has its own class with the same name do not count: they belong to
|
|
1475
|
+
that tenant's class. A class deleted with history before is created again only with the same `parent`, `alias`,
|
|
1476
|
+
`derivedBy` and `history` — otherwise `invalid_class`: the old versions of its rows would fall under other rules or a
|
|
1477
|
+
different thinning. The rules also find old versions by the aliases of ancestors. So while deleted descendants of a
|
|
1478
|
+
class keep row history, the class does not change its alias (`invalid_class`), and once deleted it is created again
|
|
1479
|
+
only with the same description. If the journal holds rows of a class with this name but not its tombstone (for
|
|
1480
|
+
example, rows of a System class deleted in 1.1), a class with this name is not created until `purge()` erases the
|
|
1481
|
+
history.
|
|
1482
|
+
|
|
1483
|
+
A derived class (`derivedBy`) is created, edited and deleted only by an administrator: an admin connection or an
|
|
1484
|
+
account with the System permission `letopis.projector` (the system administrator has it through the `letopis.*`
|
|
1485
|
+
mask). The projector writes totals over all source rows of the tenant into such a class, without read permissions,
|
|
1486
|
+
so a user who created it on their own would read totals over rows they may not read. A class that has derived
|
|
1487
|
+
descendants is also edited only by an administrator: editing an ancestor changes the description of the descendant
|
|
1488
|
+
(fields, key, `history`). The database itself sets the `aggregatedBy` mark (the class became an aggregate source) on a
|
|
1489
|
+
derived class too: it does not change its fields or key.
|
|
1490
|
+
|
|
1301
1491
|
### Granting a permission
|
|
1302
1492
|
|
|
1303
1493
|
Example: Oleg is a shop employee with the `staff` role, and employees may not delete products.
|
|
@@ -1370,6 +1560,10 @@ Every version has an author, an agent, a time, a number, a kind of operation (`c
|
|
|
1370
1560
|
operation of a block, from `db.begin({ reason })` and `db.batch(name, { reason })`; in raw SQL, from the
|
|
1371
1561
|
`letopis.reason` setting of the transaction. A deletion version has `$deleted`.
|
|
1372
1562
|
|
|
1563
|
+
Without `sort()`, versions go by id and version number (with `follow`, by time). `sort()` sets the order of versions
|
|
1564
|
+
too, and `limit()` limits them in the query itself: `db.Product(kettle).sort('at', 'desc').limit(5).versions()` gives
|
|
1565
|
+
the five latest versions, newest first.
|
|
1566
|
+
|
|
1373
1567
|
### Point-in-time read
|
|
1374
1568
|
|
|
1375
1569
|
<!-- run: guide -->
|
|
@@ -1425,16 +1619,57 @@ const [b] = await db.Order(id).blame() // every field of th
|
|
|
1425
1619
|
await db.Order(id).blame({ at: '2026-10-01', limit: 200 }) // the version at a moment; scan 200 versions
|
|
1426
1620
|
```
|
|
1427
1621
|
|
|
1428
|
-
|
|
1429
|
-
|
|
1430
|
-
|
|
1431
|
-
|
|
1432
|
-
|
|
1433
|
-
|
|
1434
|
-
|
|
1435
|
-
|
|
1436
|
-
|
|
1437
|
-
|
|
1622
|
+
Both terminals return an array, one item per object of the step. This is what the answers look like for a kettle
|
|
1623
|
+
whose price was lowered with a reason:
|
|
1624
|
+
|
|
1625
|
+
```ts
|
|
1626
|
+
import type { BlameResult, DiffResult } from 'letopis'
|
|
1627
|
+
|
|
1628
|
+
// db.Product(kettle).blame(): version 2 changed the price, the other leaves date from the creation (version 1)
|
|
1629
|
+
const blamed: BlameResult = {
|
|
1630
|
+
id: '6f1c…', class: 'Product', rev: 2, complete: true,
|
|
1631
|
+
fields: [ // in path order
|
|
1632
|
+
{ path: ['class'], rev: 1, at: '2026-10-27T09:15:02.118204+00:00', author: '<Anna id>', agent: '<service id>',
|
|
1633
|
+
op: 'create', exact: true },
|
|
1634
|
+
{ path: ['data', 'price'], rev: 2, at: '2026-10-27T09:20:41.502617+00:00', author: '<Anna id>', agent: '<service id>',
|
|
1635
|
+
op: 'update', reason: 'promotion discount', exact: true },
|
|
1636
|
+
// ['data', 'sku'], ['data', 'title'], ['links', 'Shop'], ['owner'] — also version 1, like ['class']
|
|
1637
|
+
],
|
|
1638
|
+
}
|
|
1639
|
+
// db.Product(kettle).diff(): previous version → latest
|
|
1640
|
+
const diffed: DiffResult = {
|
|
1641
|
+
id: '6f1c…', class: 'Product',
|
|
1642
|
+
from: { rev: 1, at: '2026-10-27T09:15:02.118204+00:00' },
|
|
1643
|
+
to: { rev: 2, at: '2026-10-27T09:20:41.502617+00:00' },
|
|
1644
|
+
changes: [{ path: ['data', 'price'], kind: 'changed', before: 2500, after: 2000 }],
|
|
1645
|
+
}
|
|
1646
|
+
```
|
|
1647
|
+
|
|
1648
|
+
`agent` here is the service that writes on behalf of Anna (`db = await svc.as(…)`); a write without impersonation has
|
|
1649
|
+
none.
|
|
1650
|
+
|
|
1651
|
+
- `diff()` returns an array of `DiffResult`: `{ id, class, from, to, changes }`. `from` and `to` are the compared
|
|
1652
|
+
versions `{ rev, at }`. They have `id` if the version belongs to another id (`follow: true` across `reclass` or
|
|
1653
|
+
`rekey`), and `to` has `deleted: true` if the object was deleted at that moment; then `changes` is empty. `from` is
|
|
1654
|
+
`null` if the object did not exist yet: every path is `added`. If there is no version at all, `class` and `to` are
|
|
1655
|
+
`null` too.
|
|
1656
|
+
- A change, `DiffChange`, is `{ path, kind, before, after }`. `path` is an array of segments, not a dotted string;
|
|
1657
|
+
the first segment is the section of the row: `data`, `links`, `tags`, `owner`, `class`. `kind` is `added` (there
|
|
1658
|
+
was no value), `removed` (the value is gone) or `changed`. A missing value is omitted: `added` has no `before`,
|
|
1659
|
+
`removed` has no `after`. Data objects are compared by keys, arrays as a whole. A multi-valued end and tags are
|
|
1660
|
+
sets: one change per id or tag, the path is `['links', role]` or `['tags']`, and the id or tag itself is in
|
|
1661
|
+
`before` or `after`.
|
|
1662
|
+
- `blame()` returns an array of `BlameResult`: `{ id, class, rev, complete, fields }`, plus `deleted: true` if that
|
|
1663
|
+
version is a tombstone. An object with no versions at that moment gives no item. `fields` is an array of
|
|
1664
|
+
`BlameField`, one item per leaf of the version, ordered by path: `{ path, rev, at, author, op, exact }` and, when
|
|
1665
|
+
present, `agent`, `reason`, `id`. A leaf is a scalar or an array in the data, an empty object, the role of a
|
|
1666
|
+
single-valued end, every id of a multi-valued end (`['links', role, id]`), every tag (`['tags', tag]`), `owner` and
|
|
1667
|
+
`class`. `rev`, `at`, `author`, `agent`, `op` and `reason` come from the version that changed the leaf last. Empty
|
|
1668
|
+
`agent` and `reason` are omitted; `id` is present only if it is a version of the former id (the history went
|
|
1669
|
+
through `reclass` or `rekey`).
|
|
1670
|
+
- `exact: false` means there are gaps between the current version and the found one (the history is thinned,
|
|
1671
|
+
trimmed or partly invisible): a missing version may have changed the leaf too. `complete: true` means the whole
|
|
1672
|
+
history was scanned without gaps; by default at most the 1000 latest versions are scanned.
|
|
1438
1673
|
- The database computes both with the caller's rights: raw SQL gets the same, `select "v2.shop".blame('<id>')`,
|
|
1439
1674
|
`select "v2.shop".diff_versions('<id>')`.
|
|
1440
1675
|
- The journal column `changed` (the paths changed by a version, which make `blame` fast) is not granted to the
|
|
@@ -1484,9 +1719,14 @@ for await (const ev of sub) { // ev: { cursor, row, id, class, op, r
|
|
|
1484
1719
|
|
|
1485
1720
|
- `from` is `'now'` (the default: only changes after subscribing), `'start'` (the whole journal) or a saved cursor.
|
|
1486
1721
|
`db.watch()` returns at once, and the subscription takes the "now" moment with its first query to the database: a
|
|
1487
|
-
write made right after `db.watch()` may commit earlier and miss the stream.
|
|
1488
|
-
`'start'`.
|
|
1489
|
-
- `
|
|
1722
|
+
write made right after `db.watch()` may commit earlier and miss the stream. `await sub.ready()` waits for that
|
|
1723
|
+
moment: a write after it comes to the stream. The example above reads from `'start'`.
|
|
1724
|
+
- Calls of `sub.next()` go in turn: parallel calls get different events. A call abandoned by a timeout
|
|
1725
|
+
(`Promise.race`) still gets its event — keep its promise and wait for it again.
|
|
1726
|
+
- `classes`: only these classes with their descendants; `pollMs`: fallback polling (1000 ms). A row that left these
|
|
1727
|
+
classes through a class change without an id change arrives as a deletion event in the previous class (`op:
|
|
1728
|
+
'delete'`, `row` is the previous version with `$deleted`), even if the subscriber cannot see the new class. The
|
|
1729
|
+
event comes if the subscriber could see the previous version.
|
|
1490
1730
|
- Events come without losses or repeats, under the reader's permissions, in the order of transaction numbers, and
|
|
1491
1731
|
within a transaction in the order of writing. This is not the commit order: compare versions of one object by `rev`.
|
|
1492
1732
|
- A long write transaction in any database of the cluster delays the stream: `await sub.lag()` shows the lag in ms.
|
|
@@ -1496,6 +1736,215 @@ for await (const ev of sub) { // ev: { cursor, row, id, class, op, r
|
|
|
1496
1736
|
- A subscription without losses is a saved cursor: store `ev.cursor` of the last processed event and resume from it
|
|
1497
1737
|
after a restart.
|
|
1498
1738
|
|
|
1739
|
+
### Projectors
|
|
1740
|
+
|
|
1741
|
+
A projector builds rows of derived classes from the journal: summaries, denormalized cards and anything else that may
|
|
1742
|
+
be computed with a lag behind the writes. The application writes the source classes (`from`), and only the projector
|
|
1743
|
+
writes the rows of the `into` classes. Each `into` class is marked with the option `derivedBy` holding the projector
|
|
1744
|
+
name ([Class settings](#class-settings)). Only an administrator creates derived classes: sync them with an admin
|
|
1745
|
+
connection holding a tenant session — `adm` from section 3 (`connect({ dsn: ADMIN, schema, token, allowBypassRls:
|
|
1746
|
+
true })`) or `as()` of a service on such a connection; in the CLI, `letopis sync` with the `--dsn` of the admin role.
|
|
1747
|
+
For an application session it is `acl_denied`. In the example below, the projector counts product sales from the
|
|
1748
|
+
order lines `line`.
|
|
1749
|
+
|
|
1750
|
+
```ts
|
|
1751
|
+
const ProductSales = hub('ProductSales', { qty: z.number().int(), revenue: z.number().int() }, {
|
|
1752
|
+
ends: { Product: one(Product).onDelete('cascade') }, key: ['Product'], derivedBy: 'product-sales', history: false,
|
|
1753
|
+
}) // the derived class is in the model; syncModels — on an admin connection
|
|
1754
|
+
|
|
1755
|
+
const sales = adm.projector('product-sales', { // adm is an admin connection
|
|
1756
|
+
from: ['line'], // source classes with descendants; they must keep a journal
|
|
1757
|
+
into: ['ProductSales'], // the derived classes of this projector
|
|
1758
|
+
version: 1, // the handle logic changed: a new version and a rebuild
|
|
1759
|
+
withBefore: true, // c.before is the previous version of the object or null
|
|
1760
|
+
handle(changes, out) {
|
|
1761
|
+
for (const c of changes) {
|
|
1762
|
+
if (c.op === 'purge' || c.op === 'trim' || c.op === 'reset') continue // service changes, not versions
|
|
1763
|
+
const was = c.before // the previous version of the order line: subtract it
|
|
1764
|
+
const now = c.row.$deleted ? null : c.row // the new version: add it; a deletion has none
|
|
1765
|
+
if (was) out.inc('ProductSales', { Product: was.links.Product },
|
|
1766
|
+
{ qty: -Number(was.data.qty), revenue: -Number(was.data.qty) * Number(was.data.price) }, { tenant: c.tenant })
|
|
1767
|
+
if (now) out.inc('ProductSales', { Product: now.links.Product },
|
|
1768
|
+
{ qty: Number(now.data.qty), revenue: Number(now.data.qty) * Number(now.data.price) }, { tenant: c.tenant })
|
|
1769
|
+
}
|
|
1770
|
+
},
|
|
1771
|
+
})
|
|
1772
|
+
const report = await sales.run({ once: true }) // catch up with the journal: { batches, changes, created, updated, … }
|
|
1773
|
+
const state = await sales.status() // { name, state, version, cursor, at, lag, failed, rebuildingTenants }
|
|
1774
|
+
await sales.rebuild() // rebuild from the current state; { chunked: true } works in batches
|
|
1775
|
+
await sales.rebuild({ tenant: tenantId }) // rebuild one tenant, for example a dropped one (failed)
|
|
1776
|
+
await sales.rebuild({ tenant: tenantId, chunked: true }) // the same in batches: meanwhile the journal tail goes on for all
|
|
1777
|
+
sales.run().catch((err) => console.error('the projector stopped', err)) // follow the journal until stop()
|
|
1778
|
+
sales.close() // stop() after the current batch and unsubscribe from signals
|
|
1779
|
+
```
|
|
1780
|
+
|
|
1781
|
+
- A projector needs an admin connection: `connect({ allowBypassRls: true })` with a role that may become the owner of
|
|
1782
|
+
the schema. Otherwise you get `admin_required`, and inside `db.begin()` you get `invalid_query`. The projector reads
|
|
1783
|
+
the journal of all tenants.
|
|
1784
|
+
- The first `run()` rebuilds the projection from the current state: the journal is thinned by the retention policy
|
|
1785
|
+
and cut by `purge`, so the projection cannot be assembled from the start of the journal.
|
|
1786
|
+
- Each batch is one transaction: the journal changes, `handle`, the written results and the cursor move are committed
|
|
1787
|
+
together. A break in the middle of a batch rolls the whole batch back, and the retry starts from the same cursor, so
|
|
1788
|
+
every change is applied exactly once. A projector has one writer: a second process of the same projector skips the
|
|
1789
|
+
batch (the `skipped` counter of the report), and `rebuild()` started while another process writes a batch gives
|
|
1790
|
+
`projector_busy`.
|
|
1791
|
+
- A batch is split by tenant: `handle` gets the changes of one tenant, and its results are written under a savepoint
|
|
1792
|
+
of its own. If the database rejects writing the results of a tenant (the data do not pass the description of its
|
|
1793
|
+
derived class, the target of an end is not found and so on) or `out` rejects a row or a delta built from its data
|
|
1794
|
+
(a delta that is not a number), only its changes are rolled back: the tenant drops out, and the others are written
|
|
1795
|
+
in the same batch. A dropped tenant gets into `status().failed` — `{ [tenant id]: { cursor, code, message } }`, and
|
|
1796
|
+
`message` names the tenant and the row id — and its changes are skipped until it is rebuilt. The derived rows of a
|
|
1797
|
+
dropped tenant no longer change: a tenant dropped in `run()` keeps them as they were before the failed batch, and a
|
|
1798
|
+
tenant dropped in the middle of a rebuild is left without them — the partly built rows are deleted. An exception
|
|
1799
|
+
from `handle`, a breach of the `out` limits and a failure of the environment (a deadlock, a cancelled query, a
|
|
1800
|
+
broken connection) roll the whole batch back, and `run()` is rejected. So catch `run()` without `once`: after an
|
|
1801
|
+
error it stops.
|
|
1802
|
+
- A change (`ProjectorChange`) is `cursor`, `id`, `class`, `tenant`, `op`, `rev`, `at`, `author`, `agent`, `reason`,
|
|
1803
|
+
`moved` and `row`, the version from the journal (for a deletion, a tombstone with `$deleted`). During a rebuild, the
|
|
1804
|
+
changes are rows of the current state: `rebuild: true`, `op` is `create`, `cursor` is empty.
|
|
1805
|
+
- A class change that keeps the id (classes without a key: a raw `UPDATE … SET class` and `reclass()`) is written to
|
|
1806
|
+
the journal as a version of the new class. If it takes the row out of the `from` family, the projector gets a
|
|
1807
|
+
deletion: `op` is `delete`, `row` is the last version of the row in the family with `$deleted`, and `rev`, `at` and
|
|
1808
|
+
the author are those of the class change. A row that enters the family this way comes as a `reclass` change with
|
|
1809
|
+
`before` = `null`.
|
|
1810
|
+
- With `withBefore`, a change has `before`, the previous version of the same id. `before` is `null` if there is no
|
|
1811
|
+
previous version (a creation and the first version of a new id after `reclass` and `rekey`), if the previous
|
|
1812
|
+
version is a deletion (this happens after `restore`) or lies outside the `from` family, and for every change of a
|
|
1813
|
+
rebuild. With `reclass` and `rekey`, the former data are subtracted through the tombstone of the former id: it comes
|
|
1814
|
+
as a change with `$deleted`, its `before` is the last version of the former id, and `moved` of both changes points
|
|
1815
|
+
to the other id. So the fold of the example gives the same sum after `rekey`. The service changes `purge`, `trim`
|
|
1816
|
+
and `reset` have no `before` field at all. So subtract `before` only after checking it for `null`, as in the example
|
|
1817
|
+
above.
|
|
1818
|
+
- The service journal records `purge`, `trim` and `reset` also come as changes, and their `rev` is 1. So check `op`
|
|
1819
|
+
first, and only then compare `rev`.
|
|
1820
|
+
- `out.upsert(class, row, { tenant })` creates a row or replaces it entirely by the class key.
|
|
1821
|
+
`out.inc(class, row, deltas, { tenant })` creates a row or completes it and adds the numbers without reading the row
|
|
1822
|
+
(missing fields count from 0). `out.delete(class, key or id, { tenant })` deletes a row and does nothing if there is
|
|
1823
|
+
none. `await out.get(class, key or id, { tenant })` reads the current row together with the writes of the batch.
|
|
1824
|
+
`out` writes only to the `into` classes and only to the tenant of the changes passed to `handle`; otherwise you get
|
|
1825
|
+
`derived_class` or `tenant_denied`, and the batch is rolled back.
|
|
1826
|
+
- `out` adds up the increments of a row given by key fields only (or by `$id` alone) before sending them to the
|
|
1827
|
+
database, as long as there is no `upsert`, `delete` or `get` between them: one edit of a row per batch, and the class
|
|
1828
|
+
rule is checked on the total of its increments. `inc` does not fold a row with other fields (data, `$tags`,
|
|
1829
|
+
`$owner`): its fields edit the row, and the order of such edits is kept.
|
|
1830
|
+
- A row for `out` is in the loader format: fields and end roles at the top level, the service keys `$tags`, `$owner`
|
|
1831
|
+
and `$id` (`$id` is needed only by a class without a key). The value of a role is the id of the target. An edit that
|
|
1832
|
+
changes nothing gives no version. Versions of derived rows are written by System with the agent
|
|
1833
|
+
`letopis projector <name>`; the journal is kept according to the `history` of the class, and the change signal is
|
|
1834
|
+
always sent.
|
|
1835
|
+
- `rebuild()` rebuilds the projection atomically, in one transaction: until it commits, readers see the previous rows,
|
|
1836
|
+
but the subscription horizon stands still. `rebuild({ chunked: true })` works in batches: the derived rows are
|
|
1837
|
+
deleted at once, the data stays incomplete until the rebuild ends, and `status().state` is `'rebuilding'`. The next
|
|
1838
|
+
start continues an interrupted batch rebuild. `run()` itself, on the first start of a new projection and with
|
|
1839
|
+
`onExpired: 'rebuild'`, rebuilds in batches: the subscription horizon of the cluster stands still no longer than a
|
|
1840
|
+
batch. Meanwhile a second process of the same projector waits or continues the same rebuild instead of starting it
|
|
1841
|
+
anew. A rebuild deletes the rows of the participants only, and only of the classes of their `into` families, and
|
|
1842
|
+
starts all participants anew: the marks of dropped tenants are removed, and a tenant whose data fail again drops
|
|
1843
|
+
out again.
|
|
1844
|
+
- `rebuild({ tenant: id })` rebuilds one tenant in one transaction: it deletes its derived rows, builds them from its
|
|
1845
|
+
current source rows and removes its mark in `status().failed`; the projector does not apply again the changes the
|
|
1846
|
+
rebuild has already counted. Fix the data or the class description of the tenant, then rebuild it. If the database
|
|
1847
|
+
rejects the write again, the rebuild is rolled back with this error, and the tenant stays dropped. No projection yet
|
|
1848
|
+
or a batch rebuild in progress gives `rebuild_required`; a tenant that is not a participant gives `tenant_denied`.
|
|
1849
|
+
With `chunked: true` — in batches, each in a transaction of its own with its own snapshot, while the journal tail goes
|
|
1850
|
+
on for all: the tail skips the tenant's changes above the last batch (a batch will read them) and, in a passed batch,
|
|
1851
|
+
those whose transaction is visible in its snapshot. The tenant's derived rows are deleted at once and are incomplete
|
|
1852
|
+
until the end of the rebuild (`status().rebuildingTenants`); the rebuild waits out a lock held by `run()` of another
|
|
1853
|
+
process. An interrupted rebuild (the process stopped) is continued by `run()` of any process, with or without
|
|
1854
|
+
`onExpired`.
|
|
1855
|
+
- `status().state` describes the projection in the database, not a process: `'new'` means there is no projection yet,
|
|
1856
|
+
and the first `run()` or `rebuild()` builds it; `'running'` means the projection is built and kept up with the
|
|
1857
|
+
journal (it does not mean that some process is executing `run()` right now); `'rebuilding'` means a batch rebuild
|
|
1858
|
+
is in progress or was interrupted.
|
|
1859
|
+
- `version` is the version of the `handle` logic. If the projection was built by another version, `run()` throws
|
|
1860
|
+
`rebuild_required`, and a rebuild with `rebuild()` is needed.
|
|
1861
|
+
- If the cursor is older than the `all` tier of the retention policy of the sources and there are changes after it,
|
|
1862
|
+
some changes may have been thinned out. Each tenant has a tier of its own: it is the description of its classes. If
|
|
1863
|
+
the cursor is older than the tier of every participant, `run()` throws `cursor_expired`, and with
|
|
1864
|
+
`onExpired: 'rebuild'` the projector rebuilds entirely, in batches. If it is older than the tier of only some of
|
|
1865
|
+
them, they drop out (`status().failed`, code `cursor_expired`) and the others go on, and with
|
|
1866
|
+
`onExpired: 'rebuild'` the dropped tenants rebuild themselves, in batches: the rebuild state lies in the checkpoint,
|
|
1867
|
+
and `run()` of any process continues it; a lock held by another process is only a retry on the next pass, and a data
|
|
1868
|
+
error drops the tenant with its code, without retries. The tier of a rebuilt tenant counts from the start of its
|
|
1869
|
+
rebuild until the cursor catches up with it: a lagging projector does not drop it again. A projector without new
|
|
1870
|
+
changes does not expire. `status().lag` is the age of the oldest unprocessed change in ms.
|
|
1871
|
+
- A projector works with the tenants in which every `into` class is marked with its `derivedBy`; it skips the changes
|
|
1872
|
+
of other tenants, so a class with the same name and no mark in another tenant does not get in its way. No such tenant
|
|
1873
|
+
at all — `derived_class`. Only an administrator may mark the classes of a tenant, so it is the administrator who
|
|
1874
|
+
chooses the participants: a tenant user cannot join a projector on their own. The option `tenants: [id, …]`
|
|
1875
|
+
is an explicit list of participants: the projector works only with these tenants (their `into` classes must be
|
|
1876
|
+
marked too) and ignores the marks of the others. The sources are the `from` families of these tenants; a class
|
|
1877
|
+
cannot be a source and a result at the same time. A projector does not see the changes of a class with
|
|
1878
|
+
`history: false`: it has no journal. So a tenant with such a class among the sources drops out with the code
|
|
1879
|
+
`invalid_class`, and the projector does not build its derived rows. Turn the journal of the class on and rebuild the
|
|
1880
|
+
tenant with `rebuild({ tenant })`.
|
|
1881
|
+
- `batch` is how many changes go into a batch (500), `pollMs` is fallback polling (1000 ms): source signals wake the
|
|
1882
|
+
projector earlier.
|
|
1883
|
+
- `lockTimeout` is how long a projector transaction waits for another transaction's row lock (5000 ms; 0 — no limit).
|
|
1884
|
+
Otherwise a transaction of any tenant holding a total row or an end target would hold the batch of all tenants.
|
|
1885
|
+
Longer than that, the batch rolls back, the cursor stays, and `run()` does not reject but skips the pass
|
|
1886
|
+
(`skipped`) and repeats it.
|
|
1887
|
+
- Every edit that `out` sends to the database is a separate edit of a row, and the speed of a projector is set by the
|
|
1888
|
+
number of edits, not by the number of changes. The folding of `inc` brings the edits of a row down to one per batch,
|
|
1889
|
+
so the handle from the example above (`out.inc` for every change) caught up with 6 thousand changes per second with
|
|
1890
|
+
`batch` 500 and 33 thousand with `batch` 2000 on the benchmark machine ([section 17](#17-performance)): the larger
|
|
1891
|
+
the batch, the more increments of one row fold into one edit.
|
|
1892
|
+
|
|
1893
|
+
### Recipe: daily revenue
|
|
1894
|
+
|
|
1895
|
+
An aggregate in the database and a projector solve similar tasks in different ways:
|
|
1896
|
+
|
|
1897
|
+
| | Aggregate in the database | Projector |
|
|
1898
|
+
|---|---|---|
|
|
1899
|
+
| when it is computed | in the same transaction as the write of the source | after the commit, with a lag (`status().lag`) |
|
|
1900
|
+
| what it computes | the sum of a field or the number of source rows for one target row | anything: summaries by day and by group, denormalized cards |
|
|
1901
|
+
| matches the source | always, raw SQL included | when the projector has caught up with the journal |
|
|
1902
|
+
| cost of a write | every operation locks the target row and writes its version, so the writers of one target queue up | writers do not wait for each other: the projector applies changes in batches after the commit |
|
|
1903
|
+
| rule over the total | `min(0)` rejects an operation that would take the total beyond the rule | the total does not block writes: a data error drops the tenant out of the projection |
|
|
1904
|
+
| what to run | nothing: the database keeps the field | a process on an admin connection |
|
|
1905
|
+
|
|
1906
|
+
A wallet balance that must not go below zero is an aggregate
|
|
1907
|
+
([Recipe: balance as an aggregate](#recipe-balance-as-an-aggregate)). The revenue of a shop by day is a total over many
|
|
1908
|
+
sales that may lag a little, which is a job for a projector:
|
|
1909
|
+
|
|
1910
|
+
```ts
|
|
1911
|
+
const Sale = hub('Sale', { day: z.iso.date(), amount: z.number().int().min(0) }, { ends: { Shop } }) // a sale: day and amount
|
|
1912
|
+
const DayRevenue = hub('DayRevenue', { day: z.iso.date(), revenue: z.number().int(), sales: z.number().int() }, {
|
|
1913
|
+
ends: { Shop: one(Shop).onDelete('cascade') }, key: ['Shop', 'day'], derivedBy: 'day-revenue', history: false,
|
|
1914
|
+
})
|
|
1915
|
+
await syncModels(db, [Sale], { apply: true })
|
|
1916
|
+
await syncModels(adm, [DayRevenue], { apply: true }) // the derived class is created by an administrator: adm holds a tenant session
|
|
1917
|
+
|
|
1918
|
+
const revenue = adm.projector('day-revenue', {
|
|
1919
|
+
from: ['Sale'], into: ['DayRevenue'], version: 1, withBefore: true,
|
|
1920
|
+
handle(changes, out) {
|
|
1921
|
+
for (const c of changes) {
|
|
1922
|
+
if (c.op === 'purge' || c.op === 'trim' || c.op === 'reset') continue // service changes
|
|
1923
|
+
const was = c.before // the previous version of the sale: subtract it from its day
|
|
1924
|
+
const now = c.row.$deleted ? null : c.row // the new one: add it to its day; a deletion has none
|
|
1925
|
+
if (was) out.inc('DayRevenue', { Shop: was.links.Shop, day: was.data.day },
|
|
1926
|
+
{ revenue: -Number(was.data.amount), sales: -1 }, { tenant: c.tenant })
|
|
1927
|
+
if (now) out.inc('DayRevenue', { Shop: now.links.Shop, day: now.data.day },
|
|
1928
|
+
{ revenue: Number(now.data.amount), sales: 1 }, { tenant: c.tenant })
|
|
1929
|
+
}
|
|
1930
|
+
},
|
|
1931
|
+
})
|
|
1932
|
+
revenue.run().catch((err) => console.error('the revenue projector stopped', err)) // follows the journal until stop()
|
|
1933
|
+
const week = await db.Shop(shop).DayRevenue().sort('data.day', 'desc').limit(7).rows() // the last seven days
|
|
1934
|
+
```
|
|
1935
|
+
|
|
1936
|
+
- The day is a field of the sale, not the version time `at`: with the version time, an edit, `restore` or `rekey`
|
|
1937
|
+
would move the amount of the sale to its own day.
|
|
1938
|
+
- `out.inc` adds without reading the row, so the total does not depend on the order of changes: the journal gives
|
|
1939
|
+
transactions in the order of their first write, not of their commit. Editing a sale subtracts the previous amount
|
|
1940
|
+
from the previous day and adds the new one to the new day; a deletion only subtracts. A day with no sales left stays
|
|
1941
|
+
as a row of zeros. `out` folds the increments of one day in a batch into one edit.
|
|
1942
|
+
- `DayRevenue` has `history: false`: the derived rows do not double the journal, and a rebuild assembles them without
|
|
1943
|
+
history anyway. `Sale` needs a journal: a projector does not see the changes of a class with `history: false`.
|
|
1944
|
+
- The application reads `DayRevenue` rows like any class, and only the projector writes them: `create` or `update` of
|
|
1945
|
+
such a row gives `derived_class`. The total lags behind the sales until the next batch: write signals wake the
|
|
1946
|
+
projector, and without them it polls every `pollMs`.
|
|
1947
|
+
|
|
1499
1948
|
### Result cache
|
|
1500
1949
|
|
|
1501
1950
|
<!-- run: guide -->
|
|
@@ -1542,11 +1991,11 @@ const report = await adm.maintain() // { sessions, codes, thinned, batches,
|
|
|
1542
1991
|
PostgreSQL signal queue: if it is more than half full, you get a warning. The system administrator, a role with the
|
|
1543
1992
|
`letopis.maintain` permission or an admin connection can call it. An external scheduler runs it (`letopis maintain`),
|
|
1544
1993
|
or the library itself does: `connect({ maintain: '1h' })`. Of several processes, only one works: each batch is its own
|
|
1545
|
-
transaction, and the first thing it does is take a
|
|
1546
|
-
|
|
1547
|
-
the first batch, and between batches it stops (that
|
|
1548
|
-
`more: true`. There is no session-level lock, so `maintain()`
|
|
1549
|
-
transaction mode.
|
|
1994
|
+
transaction, and the first thing it does is take the maintenance lock — a row of the `secret` table that neither the
|
|
1995
|
+
application role nor any other database role can take; the lock is released when the batch commits. If another
|
|
1996
|
+
process holds the lock, the call answers `skipped: true` before the first batch, and between batches it stops (that
|
|
1997
|
+
process continues) and reports what it has done with `more: true`. There is no session-level lock, so `maintain()`
|
|
1998
|
+
works behind any pool, including PgBouncer in transaction mode.
|
|
1550
1999
|
|
|
1551
2000
|
### Reset: `reset()`
|
|
1552
2001
|
|
|
@@ -1571,8 +2020,157 @@ const modelText = describeText(model) // the same as text: for the application's
|
|
|
1571
2020
|
```
|
|
1572
2021
|
|
|
1573
2022
|
`db.describe()` returns a description of the classes available to the session: fields, ends, keys, policies and
|
|
1574
|
-
metadata, plus the multi-field conditions `anyOf` and `oneOf` with inheritance applied,
|
|
1575
|
-
the
|
|
2023
|
+
metadata, plus the multi-field conditions `anyOf` and `oneOf` with inheritance applied, the derived class mark
|
|
2024
|
+
`derivedBy` and the aggregates `aggregate`, when a class has them. From the command line:
|
|
2025
|
+
`letopis describe --format text --out llms.txt`.
|
|
2026
|
+
|
|
2027
|
+
### Agents and MCP
|
|
2028
|
+
|
|
2029
|
+
An agent — Claude, Cursor or your own — works with the data through MCP tools: it learns the model, reads with
|
|
2030
|
+
chains, looks at the history and, if allowed, writes. The MCP server acts on behalf of one user and only with that
|
|
2031
|
+
user's rights: every call is a chain under the user's session, and the database checks rights and tenants. The agent
|
|
2032
|
+
has no raw SQL.
|
|
2033
|
+
|
|
2034
|
+
| Tool | What it does |
|
|
2035
|
+
|---|---|
|
|
2036
|
+
| `letopis_model` | the model: classes, fields (JSON Schema), the `anyOf` and `oneOf` conditions, ends, keys, aliases, metadata; with `class`, one class in detail |
|
|
2037
|
+
| `letopis_query` | reading with a JSON chain: the start is `class` and `filter`, `id` or `ids`; `steps` are hops by roles and classes; `asOf`, `withDeleted`, `sort`, `limit`, `cursor`; the terminal is `rows`, `first`, `ids`, `count`, `sum`, `avg`, `min`, `max` or `countBy` |
|
|
2038
|
+
| `letopis_history` | `versions`, `diff` or `blame` of one object |
|
|
2039
|
+
| `letopis_write` | `create`, `update` (only with `rev`), `upsert`, `delete`, only with `write: true` (`letopis mcp --write`) |
|
|
2040
|
+
|
|
2041
|
+
A `letopis_query` request is the same chain as `db.Product({ price: lt(1000) }).Shop().sort('data.name').rows()`, written as JSON:
|
|
2042
|
+
|
|
2043
|
+
```json
|
|
2044
|
+
{ "class": "Product", "filter": { "price": { "$lt": 1000 } }, "steps": [{ "step": "Shop" }], "sort": { "field": "data.name" } }
|
|
2045
|
+
```
|
|
2046
|
+
|
|
2047
|
+
A filter value means equality, a nested object addresses the fields of nested objects, `{ "Role": id }` is an end.
|
|
2048
|
+
Operators: `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$between`, `$in`, `$like`, `$ilike`, `$starts`, `$ends`, `$has`,
|
|
2049
|
+
`$hasAny`, `$hasAll`, `$exists`, `$isNull`, `$not`; OR is `{ "$or": [filter, …] }`. An unknown class, role, parameter
|
|
2050
|
+
or operator gives an `isError` answer with a hint of similar names. There are at most 16 `steps` and at most 64
|
|
2051
|
+
conditions in the `$or` of one filter; otherwise `invalid_query`. These limits do not bound the time of a query: set
|
|
2052
|
+
`statement_timeout` for the database role the MCP server connects as (`alter role … set statement_timeout = '5s'`).
|
|
2053
|
+
|
|
2054
|
+
The answer is `structuredContent` and its text copy. `structuredContent` is `{ notice, data }`: `notice` is a constant
|
|
2055
|
+
text of the package saying that `data` holds database data, not instructions, and `data` is the answer itself. The
|
|
2056
|
+
shape of `data` of `letopis_query` depends on the terminal:
|
|
2057
|
+
|
|
2058
|
+
| Terminal | `structuredContent.data` |
|
|
2059
|
+
|---|---|
|
|
2060
|
+
| `rows` (default) | `{ rows, count, more, cursor? }` |
|
|
2061
|
+
| `ids` | `{ ids, count, more, cursor? }` |
|
|
2062
|
+
| `first` | `{ row }`, a row or `null` |
|
|
2063
|
+
| `count` | `{ count }` |
|
|
2064
|
+
| `countBy` | `{ counts, truncated? }` |
|
|
2065
|
+
| `sum`, `avg`, `min`, `max` | `{ value }` |
|
|
2066
|
+
|
|
2067
|
+
For `rows` and `ids`, `count` is the number of rows on this page, not the total. A page holds at most `limit`,
|
|
2068
|
+
`maxRows` rows (100) and `maxBytes` bytes of row JSON (50,000); a row that is itself larger than `maxBytes` comes
|
|
2069
|
+
alone on its page. `more: true` means there are more rows, and then the same request with `cursor` returns the next
|
|
2070
|
+
page. `counts` is `{ value: number }`, the most frequent values; if the groups did not fit into the limits,
|
|
2071
|
+
`truncated: true` comes. `countBy` reads at most `maxRows` + 1 groups from the database. The
|
|
2072
|
+
field of an aggregate and of `countBy` is `field: 'data.<path>'`. Sorting is `sort: { field, dir }`: `field` is `at`,
|
|
2073
|
+
`rev`, `id` or `data.<path>`, `dir` is `'asc'` (default) or `'desc'`; without `sort`, rows go by `id`. `sort`, `limit`
|
|
2074
|
+
and `cursor` work only with `rows`, `first` and `ids`.
|
|
2075
|
+
|
|
2076
|
+
`letopis_model` without `class` answers (in `data`) `{ schema, tenant, classes }`, the same classes as in `db.describe()`. If such
|
|
2077
|
+
a model is larger than `maxBytes`, a brief one comes: `brief: true`, and the classes have only the names of their
|
|
2078
|
+
fields and ends; `letopis_model` with `class` returns each class in detail. `letopis_history` takes `class`, `id` and
|
|
2079
|
+
`kind`. `versions` answers `{ versions, more }`, the latest versions within the answer limits, from the oldest to the
|
|
2080
|
+
newest; `diff` with `from` and `to` (a version number or a moment) answers `{ diff }`; `blame` with `at` and `limit`
|
|
2081
|
+
answers `{ blame }`. If the changes or the leaves did not fit, the answer is cut, and it has `truncated: true` and
|
|
2082
|
+
`total`. `follow` works with `versions` and `diff`.
|
|
2083
|
+
|
|
2084
|
+
A tool refusal is `isError: true` with a `structuredContent` of the form `{ notice, error }`, where `error` is `{ code, message }`, plus `similar` (similar names),
|
|
2085
|
+
`detail` (details) and `sqlstate` (the PostgreSQL code of an error not from letopis) when present. A parameter that
|
|
2086
|
+
does not match the tool input schema (a wrong type, a value outside the list, a missing required one) is rejected by
|
|
2087
|
+
the SDK before letopis: then it is `isError: true` with text only, without `structuredContent`.
|
|
2088
|
+
|
|
2089
|
+
In the text of the answer, everything that came from the database (rows, names and descriptions of classes, error
|
|
2090
|
+
texts, hints) sits inside `<untrusted-data-…>` with a random suffix and a note that this is data, not instructions.
|
|
2091
|
+
Tool descriptions and the server's `instructions` are fixed text of the package: tenants write the `meta` of classes,
|
|
2092
|
+
so it only appears in results.
|
|
2093
|
+
|
|
2094
|
+
Writing. Without `write`, the `letopis_write` tool is not in the tool list, and a call to it is an SDK protocol error
|
|
2095
|
+
(`Tool letopis_write not found`), not a tool answer with `isError`. The ends of a write are set by
|
|
2096
|
+
`links: { Role: id | [id] | null }`, and `create`, `update` and `upsert` answer `{ rows }`. `update` takes only the
|
|
2097
|
+
`rev` of the version that was read: someone else's version gives `conflict`, and no update is lost. `delete` without
|
|
2098
|
+
`confirm` returns a preview of the cascade and the operation code: `{ preview, operation }`. With `confirm: true`, a
|
|
2099
|
+
human confirms the deletion in the client through elicitation (in protocol revision 2025 as a request to the client,
|
|
2100
|
+
in 2026-07-28 as an `input_required` answer), the confirmation is bound to the hash of the operation, and the answer
|
|
2101
|
+
is `{ deleted, operation }`. In revision 2026-07-28, the `requestState` signature with which the client repeats the
|
|
2102
|
+
call is also bound to the user of the server and is valid for 10 minutes: a signature of another user's server and an
|
|
2103
|
+
expired one give the refusal `confirmation_required`. The confirmation window shows the canonical class name and the ids of the targets (the
|
|
2104
|
+
first 10, then «и ещё K», that is "and K more"); a tenant writes the alias and the `meta` of a class, so they do not
|
|
2105
|
+
get into the window. An operation code sent by the agent is not a confirmation; a client without elicitation gets the
|
|
2106
|
+
refusal `confirmation_required`. Every write gets the reason `mcp: letopis_write`, and `blame()` shows it. By default,
|
|
2107
|
+
`letopis_write` does not create, edit or delete the permission classes, `rule`, `Resource`, `member`, `Account`,
|
|
2108
|
+
`Class` and their descendants (rules, memberships, class descriptions): the refusal is `acl_denied`. An agent under an
|
|
2109
|
+
injection in the data would grant a stranger a membership or a rule without a human confirmation. Only the option
|
|
2110
|
+
`allowAclWrite: true` allows it.
|
|
2111
|
+
|
|
2112
|
+
**stdio.** `letopis mcp` is a stdio server for MCP clients. It needs the SDK package: `npm i @modelcontextprotocol/server`
|
|
2113
|
+
(an optional peer dependency). The client passes the connection string, the schema and the token as flags or as the
|
|
2114
|
+
variables `LETOPIS_DSN`, `LETOPIS_SCHEMA`, `LETOPIS_TOKEN`; `--write` turns writing on. Only a user session token is
|
|
2115
|
+
accepted, and the application issues it: `svc.auth.sessionFor(user, { ttl })`, while `auth.switch(tenant)` sets the
|
|
2116
|
+
working tenant of the session. The command rejects the service key (`--api-key`): it grants all `auth.*` rights and
|
|
2117
|
+
would lie on the agent's machine. Without a token and with `--api-key`, the command exits with code 2 and writes the
|
|
2118
|
+
reason to stderr. The command also rejects the token of a session in the System tenant (this is how a system
|
|
2119
|
+
administrator works): it has no `allowSystemAdmin` and `allowAclWrite` options. Client configuration (Claude Desktop,
|
|
2120
|
+
Claude Code, Cursor):
|
|
2121
|
+
|
|
2122
|
+
```json
|
|
2123
|
+
{
|
|
2124
|
+
"mcpServers": {
|
|
2125
|
+
"letopis": {
|
|
2126
|
+
"command": "npx",
|
|
2127
|
+
"args": ["letopis", "mcp"],
|
|
2128
|
+
"env": {
|
|
2129
|
+
"LETOPIS_DSN": "postgres://shop_app:app-secret@localhost:5432/shop",
|
|
2130
|
+
"LETOPIS_SCHEMA": "v2.shop",
|
|
2131
|
+
"LETOPIS_TOKEN": "<user session token>"
|
|
2132
|
+
}
|
|
2133
|
+
}
|
|
2134
|
+
}
|
|
2135
|
+
}
|
|
2136
|
+
```
|
|
2137
|
+
|
|
2138
|
+
**HTTP.** There is no built-in HTTP server. `createMcpServer(db, { write, maxRows, maxBytes })` from `letopis/mcp`
|
|
2139
|
+
returns an MCP server, and the application plugs it into its own `createMcpHandler` handler from the SDK with its own
|
|
2140
|
+
OAuth 2.1 authorization: the application checks the request token and finds the letopis user by it. The connection
|
|
2141
|
+
is a user session (`connect({ token })`) or the facade `svc.as(user)`: the service key stays in the application
|
|
2142
|
+
process. A connection that bypasses RLS is refused with `bypass_rls`; the service root without `as()` and a connection
|
|
2143
|
+
without a session, with `user_session_required`; the facade of an explicit transaction `db.begin()`, with
|
|
2144
|
+
`invalid_query`. The server also refuses a session in the System tenant with `user_session_required` unless the option
|
|
2145
|
+
`allowSystemAdmin: true` is passed: this is how a system administrator works, and their `admin` role grants rights to
|
|
2146
|
+
the accounts, memberships and rules of the installation.
|
|
2147
|
+
|
|
2148
|
+
<!-- run: mcp -->
|
|
2149
|
+
```ts
|
|
2150
|
+
import { up, connect } from 'letopis'
|
|
2151
|
+
import { createMcpServer } from 'letopis/mcp'
|
|
2152
|
+
import { createMcpHandler } from '@modelcontextprotocol/server'
|
|
2153
|
+
|
|
2154
|
+
const ADMIN = 'postgres://postgres:secret@localhost:5432/shop'
|
|
2155
|
+
const APP = 'postgres://shop_app:app-secret@localhost:5432/shop'
|
|
2156
|
+
const r = await up({ dsn: ADMIN, schema: 'shop', quiet: true,
|
|
2157
|
+
tenant: { name: 'Chamomile', user: { name: 'Anna', login: 'anna@shop.ru', password: 'Chamomile-2026!' } } })
|
|
2158
|
+
const svc = await connect({ dsn: APP, schema: r.schema, apiKey: r.serviceKey! })
|
|
2159
|
+
|
|
2160
|
+
// authInfo is the result of the application's check of the request token; it puts the letopis account id into extra
|
|
2161
|
+
const mcp = createMcpHandler(async ({ authInfo }) =>
|
|
2162
|
+
createMcpServer(await svc.as(String(authInfo?.extra?.account), { tenant: r.tenant!.tenant })))
|
|
2163
|
+
// in the application's HTTP handler, after checking the token: return mcp.fetch(request, { authInfo })
|
|
2164
|
+
```
|
|
2165
|
+
|
|
2166
|
+
The handler calls the factory on every HTTP request, so `createMcpServer` checks the connection every time (one query
|
|
2167
|
+
to the database).
|
|
2168
|
+
|
|
2169
|
+
The handler can be checked without a network: the SDK client from the `@modelcontextprotocol/client` package reaches
|
|
2170
|
+
it through the application's `fetch`: `new StreamableHTTPClientTransport(new URL('http://localhost/mcp'), { fetch:
|
|
2171
|
+
(url, init) => mcp.fetch(new Request(url, init), { authInfo }) })`. This is how the `docs` test checks the example
|
|
2172
|
+
above (the `mcp` scenario in `lib/test/docs.test.ts` of the repository): without `write` the client sees three tools,
|
|
2173
|
+
and `letopis_model` answers on behalf of Anna in the shop.
|
|
1576
2174
|
|
|
1577
2175
|
### Raw SQL
|
|
1578
2176
|
|
|
@@ -1599,6 +2197,8 @@ letopis maintain --dsn <dsn> --schema <v2.name> [--token <token> | --api-key <ke
|
|
|
1599
2197
|
letopis describe --dsn <dsn> --schema <v2.name> (--token <token> | --api-key <key>) [--format json|text] [--out <file>]
|
|
1600
2198
|
letopis import --from <0.21 dsn> --from-schema <v1.name> --dsn <dsn> --schema <v2.name> [--tenant <name>] [--keyless <Class>]…
|
|
1601
2199
|
[--rename <Old>=<New>]… [--update] [--commit-every <N>] [--partition <name>] [--initial]
|
|
2200
|
+
letopis mcp --dsn <dsn> --schema <v2.name> --token <user token> [--write]
|
|
2201
|
+
(or LETOPIS_DSN, LETOPIS_SCHEMA, LETOPIS_TOKEN; the service key --api-key is not accepted)
|
|
1602
2202
|
```
|
|
1603
2203
|
|
|
1604
2204
|
### Monitoring
|
|
@@ -1668,9 +2268,10 @@ The full machine-readable list of exports, methods and error texts is `docs/api-
|
|
|
1668
2268
|
|
|
1669
2269
|
| Entry point | What it exports |
|
|
1670
2270
|
|---|---|
|
|
1671
|
-
| `letopis` | `connect`, `up`, `createTenant`; operators `ne`, `gt`, `gte`, `lt`, `lte`, `between`, `inList`, `like`, `ilike`, `starts`, `ends`, `has`, `hasAny`, `hasAll`, `exists`, `isNull`, `not`, `or`, `isOp`; `inc`, `isInc`, `cursorOf`; `idOf`, `uuidv5`, `v5Name`, `v5Id`; `LetopisError`, `ValidationError`, `fromDbError`; `syncModels`, `generateTypes`, `schemaToTs`; `compileValidator`, `applyDefaults`, `Registry`; `totpCode`, `hashPassword`, `verifyHash`; `describeText`; constants `DDL_REVISION`, `RESERVED_CLASS_NAMES`, `CHAIN_METHODS`, `DB_METHODS`; types `Db`, `DbTx`, `TypedDb`, `Chain`, `Row`, `Filter`, `Cursor`, `ConnectOptions`, `UpOptions`, `LoadOptions`, `LoadReport`, `WatchOptions`, `WatchEvent`, `AuthApi`, `AclApi` and others |
|
|
1672
|
-
| `letopis/model` | `hub`, `link`, `one`, `many`, `End`, types `Infer`, `InferLinks`, `Model`, `ClassDef`, `ClassMeta`, `HistoryPolicy`; `normalizeSchema`, `translatePattern`, type `JsonSchema` |
|
|
1673
|
-
|
|
|
2271
|
+
| `letopis` | `connect`, `up`, `createTenant`; operators `ne`, `gt`, `gte`, `lt`, `lte`, `between`, `inList`, `like`, `ilike`, `starts`, `ends`, `has`, `hasAny`, `hasAll`, `exists`, `isNull`, `not`, `or`, `isOp`; `inc`, `isInc`, `cursorOf`; `idOf`, `uuidv5`, `v5Name`, `v5Id`; `LetopisError`, `ValidationError`, `fromDbError`; `syncModels`, `generateTypes`, `schemaToTs`; `compileValidator`, `applyDefaults`, `Registry`; `totpCode`, `hashPassword`, `verifyHash`; `describeText`; constants `DDL_REVISION`, `RESERVED_CLASS_NAMES`, `CHAIN_METHODS`, `DB_METHODS`; types `Db`, `DbTx`, `TypedDb`, `Chain`, `Row`, `Filter`, `Cursor`, `ConnectOptions`, `UpOptions`, `LoadOptions`, `LoadReport`, `WatchOptions`, `WatchEvent`, `AuthApi`, `AclApi`, `DiffResult`, `DiffChange`, `BlameResult`, `BlameField`, `Projector`, `ProjectorOptions`, `ProjectorChange`, `ProjectorOut`, `ProjectorRow`, `ProjectorReport`, `ProjectorStatus`, `ProjectorFailure` ([Projector](#projector)), `AggregateInfo` and others |
|
|
2272
|
+
| `letopis/model` | `hub`, `link`, `one`, `many`, `End`, types `Infer`, `InferLinks`, `Model`, `ClassDef`, `ClassMeta`, `HistoryPolicy`, `AggregateDef`; `normalizeSchema`, `translatePattern`, type `JsonSchema` |
|
|
2273
|
+
| `letopis/mcp` | `createMcpServer(db, options)`: an MCP SDK server with the letopis tools; the type `McpOptions`: `write` adds the `letopis_write` tool (absent by default), `maxRows` is the rows in an answer (100, from 1 to 10,000), `maxBytes` is the bytes in an answer (50,000, from 1,000 to 50,000,000); `allowAclWrite` lets `letopis_write` write the permission classes too (by default `acl_denied`); `allowSystemAdmin` accepts a session in the System tenant (by default `user_session_required`); a value out of range gives `invalid_data`. Needs the `@modelcontextprotocol/server` package ([section 13](#agents-and-mcp)) |
|
|
2274
|
+
| CLI `letopis` | `sync`, `types`, `load`, `verify`, `maintain`, `describe`, `import`, `mcp` ([section 13](#13-operations)); `mcp` takes `--dsn`, `--schema`, `--token` (or `LETOPIS_DSN`, `LETOPIS_SCHEMA`, `LETOPIS_TOKEN`) and `--write` |
|
|
1674
2275
|
|
|
1675
2276
|
Removed 0.21 names (`uuidv7`, `LETOPIS_NS` and the methods listed below) throw `removed` with the replacement.
|
|
1676
2277
|
|
|
@@ -1685,6 +2286,8 @@ Removed 0.21 names (`uuidv7`, `LETOPIS_NS` and the methods listed below) throw `
|
|
|
1685
2286
|
| `begin()`, `commit(tr)`, `rollback(tr)`, `lock(…keys)` | an explicit transaction ([section 8](#8-transactions-batches-and-money)) |
|
|
1686
2287
|
| `batch(name)` | a queue of plans: `run()`, `discard()`, `size()` |
|
|
1687
2288
|
| `load(input, options)` | the loader ([section 9](#9-bulk-loading)) |
|
|
2289
|
+
| `projector(name, options)` | a projector from the journal: `run()`, `rebuild({ chunked, tenant })`, `status()`, `stop()`, `close()` ([Projectors](#projectors)) |
|
|
2290
|
+
| `rebuildAggregates(class)` | repair of the aggregates of a class and its descendants in all tenants: the field anew from live sources, a version only for corrected rows; → their number ([Aggregates in the database](#aggregates-in-the-database)) |
|
|
1688
2291
|
| `watch(options)`, `cache` | subscription; result cache (`clear()`, `stats()`) |
|
|
1689
2292
|
| `` sql`…` ``, `transaction(fn)` | raw SQL under the session and RLS |
|
|
1690
2293
|
| `idOf(class, key)` | the id of an object of a class with a key, in the session tenant |
|
|
@@ -1702,7 +2305,7 @@ Removed 0.21 names (`uuidv7`, `LETOPIS_NS` and the methods listed below) throw `
|
|
|
1702
2305
|
| modifiers | `limit(n)`, `offset(n)`, `sort(field, 'asc' \| 'desc')`, `after(cursor)`, `asOf(moment)`, `withDeleted()`, `deep(max)`, `exact()`, `alias(name)`, `tags(…)`, `owner(…)`, `entity(…)`, `forUpdate()`, `cache({ ttl })` |
|
|
1703
2306
|
| verbs | `create(data)`, `update(patch, { rev })`, `upsert(data)`, `delete({ confirm })`, `anonymize(fields)`, `reclass(class, data)`, `rekey(patch)`, `restore()`, `purge({ confirm })` |
|
|
1704
2307
|
| slots | after a verb: `.RoleName.set(value)`, `.RoleName.unset()`, `.RoleName.add(value)`, `.RoleName.remove(value)` |
|
|
1705
|
-
| terminals | `rows()`, `first()`, `ids()`, `count({ paths })`, `paths()`, `versions({ follow })`, `sum(field)`, `avg(field)`, `min(field)`, `max(field)`, `countBy(field)` |
|
|
2308
|
+
| terminals | `rows()`, `first()`, `ids()`, `count({ paths })`, `paths()`, `versions({ follow })`, `diff({ from, to, follow })`, `blame({ at, limit })`, `sum(field)`, `avg(field)`, `min(field)`, `max(field)`, `countBy(field)` |
|
|
1706
2309
|
|
|
1707
2310
|
### `db.auth`
|
|
1708
2311
|
|
|
@@ -1729,6 +2332,23 @@ its own key, but not its other credentials; nobody can revoke the last active ke
|
|
|
1729
2332
|
`TOTP` method. The credential kind `kind` is `^[A-Z][A-Z0-9_]{1,31}$` (`PASSWORD`, `EMAIL`,
|
|
1730
2333
|
`TELEGRAM` …); `SESSION`, `APIKEY`, `KEYSECRET`, `OTP`, `TOTP` and `KEY` are taken.
|
|
1731
2334
|
|
|
2335
|
+
### Projector
|
|
2336
|
+
|
|
2337
|
+
`db.projector(name, options)` works only on an admin connection ([Projectors](#projectors)). The name is lowercase
|
|
2338
|
+
Latin letters, digits and `_.:-`, up to 64 characters; it is also the `derivedBy` of the derived classes. Wrong options
|
|
2339
|
+
give `invalid_data`.
|
|
2340
|
+
|
|
2341
|
+
| Member | What it does |
|
|
2342
|
+
|---|---|
|
|
2343
|
+
| options `ProjectorOptions` | `from`, `into`: classes with descendants; `version`: an integer from 1; `handle(changes, out)`; `withBefore`; `onExpired: 'rebuild' \| 'error'` (`'rebuild'` rebuilds in batches); `batch`: changes in a batch (500, from 1 to 10,000); `pollMs`: fallback polling (1000 ms); `lockTimeout`: waiting for another transaction's row lock (5000 ms, 0 — no limit); `tenants`: ids of the participant tenants |
|
|
2344
|
+
| `run({ once })` | catch up with the journal (`once`) or follow it until `stop()` → `ProjectorReport` |
|
|
2345
|
+
| `rebuild({ chunked, tenant })` | rebuild from the current state: atomically, in batches or for one tenant → `ProjectorReport` |
|
|
2346
|
+
| `status()` | → `ProjectorStatus`: `{ name, state, version, cursor, at, lag, failed, rebuildingTenants }`; `failed` is `{ [tenant id]: ProjectorFailure }`, where `ProjectorFailure` is `{ cursor, code, message }` |
|
|
2347
|
+
| `stop()`, `close()` | stop `run()` after the current batch; the same plus unsubscribing from signals |
|
|
2348
|
+
| `ProjectorReport` | `{ batches, changes, created, updated, deleted, unchanged, skipped, rebuilds }` |
|
|
2349
|
+
| `ProjectorChange` | `{ cursor, id, class, tenant, op, rev, at, author, row }` and, when present, `agent`, `reason`, `moved`, `before`, `rebuild` |
|
|
2350
|
+
| `ProjectorOut` | `upsert(class, row, { tenant })`, `inc(class, row, deltas, { tenant })`, `delete(class, key or id, { tenant })`, `get(class, key or id, { tenant })` → `Row \| null`; a row is a `ProjectorRow` in the loader format |
|
|
2351
|
+
|
|
1732
2352
|
### Result row
|
|
1733
2353
|
|
|
1734
2354
|
| Field | Meaning |
|
|
@@ -1754,8 +2374,8 @@ database does not accept a link with such a name (`reserved_names()`; the instal
|
|
|
1754
2374
|
`owner`, `forUpdate`, `cache`, `rekey`, `diff`, `blame`; guards of the removed API: `run`, `execute`, `account`;
|
|
1755
2375
|
- the `db` root: `as`, `begin`, `commit`, `rollback`, `lock`, `batch`, `watch`, `close`, `registry`, `sql`, `load`,
|
|
1756
2376
|
`verify`, `maintain`, `reset`, `describe`, `auth`, `acl`, `idOf`, `refresh`, `transaction`, `flushSignals`,
|
|
1757
|
-
`schema`, `token`, `tenant`, `trimHistory`, `rekeyClass`, `cache`; guards: `reloadSchema`,
|
|
1758
|
-
`credentials`, `resources`, `rules`, `compact`;
|
|
2377
|
+
`schema`, `token`, `tenant`, `trimHistory`, `rekeyClass`, `cache`, `projector`, `rebuildAggregates`; guards: `reloadSchema`,
|
|
2378
|
+
`accounts`, `credentials`, `resources`, `rules`, `compact`;
|
|
1759
2379
|
- batch: `discard`, `size`.
|
|
1760
2380
|
|
|
1761
2381
|
## 16. Errors
|
|
@@ -1775,16 +2395,19 @@ Database codes. Raw SQL gets them too: SQLSTATE class `LT`, the details are JSON
|
|
|
1775
2395
|
| `acl_denied` | LT006 | denied by the access rules |
|
|
1776
2396
|
| `id_mismatch`, `id_from_db` | LT007, LT008 | the id is not v5 of the key; the id of a class without a key is assigned by the database |
|
|
1777
2397
|
| `immutable_key` | LT009 | a key field, the id, the tenant and the owner do not change (a new key is `rekey`) |
|
|
1778
|
-
| `invalid_class` | LT010 | the class description is rejected, the class is not found or is abstract |
|
|
2398
|
+
| `invalid_class` | LT010 | the class description is rejected, the class is not found or is abstract; a class with rows, descendants or a history of rows is not deleted |
|
|
1779
2399
|
| `use_reclass`, `reclass_denied` | LT011, LT012 | changing the class with `update` when the old or the new class has a key: the id changes, so use `reclass()`; the roles of referencing rows do not accept the new class |
|
|
1780
2400
|
| `tightening_conflict` | LT013 | rows prevent tightening the description |
|
|
1781
2401
|
| `isolation_level` | LT014 | deleting, changing the class and changing the key work only in `read committed` |
|
|
1782
2402
|
| `recreated_in_statement` | LT015 | an id was deleted and created again in one statement |
|
|
1783
|
-
| `cursor_expired` | LT016 | the subscription
|
|
2403
|
+
| `cursor_expired` | LT016 | the cursor of a subscription or a projector is older than the `all` tier of the retention policy |
|
|
1784
2404
|
| `delete_restricted` | LT017 | referencing rows prevent the deletion (`onDelete: 'restrict'`) |
|
|
1785
2405
|
| `not_deleted` | LT018 | `restore` and `purge` work only for deleted objects |
|
|
1786
2406
|
| `reset_disabled` | LT019 | reset is disabled in the installation (`allowReset`) |
|
|
1787
2407
|
| `confirm_mismatch` | LT020 | the reset confirmation does not match the schema name |
|
|
2408
|
+
| `derived_class` | LT021 | a row of a derived class (`derivedBy`) is written, edited or deleted by someone other than its projector; an end leads to such a row or to a derived class; a projector writes to a class outside its `into` or has found no tenant in which every `into` class is marked with its `derivedBy` |
|
|
2409
|
+
| `aggregate_field` | LT022 | the application passed or changed an aggregate field (`aggregate`): the database calculates it |
|
|
2410
|
+
| `projector_pending` | LT023 | `purge` of a deleted object whose deletion a projector of its class has not read yet, or `trimHistory` whose first remaining version it has not read yet: run the projector first |
|
|
1788
2411
|
|
|
1789
2412
|
Library codes:
|
|
1790
2413
|
|
|
@@ -1801,10 +2424,14 @@ Library codes:
|
|
|
1801
2424
|
| `no_key` | `upsert`, `rekey` or `idOf` on a class without a key |
|
|
1802
2425
|
| `no_path` | there is no hop between the classes of neighboring steps |
|
|
1803
2426
|
| `invalid_query` | the chain or the API is used incorrectly (a step modifier `deep`, `exact`, `alias`, `tags` or `owner` after a verb, a slot not after a verb, an `after()` cursor with a value of a different type than the `sort` field, without `v` or with an `id` that is not a uuid, and so on) |
|
|
1804
|
-
| `admin_required` | the loader
|
|
2427
|
+
| `admin_required` | the loader, a projector and `rebuildAggregates` need an admin connection |
|
|
2428
|
+
| `rebuild_required` | the projection was built by another version of the projector (`version`): a rebuild with `rebuild()` is needed; `rebuild({ tenant })` while there is no projection or a batch rebuild is in progress |
|
|
2429
|
+
| `projector_busy` | a rebuild cannot start: the projector is running in another process right now |
|
|
1805
2430
|
| `not_empty` | the initial load (`initial`) goes only into an empty schema |
|
|
1806
2431
|
| `class_not_loadable` | the loader does not accept `Class` rows |
|
|
1807
|
-
| `bypass_rls` | `connect()` with a role that bypasses RLS, without `allowBypassRls` |
|
|
2432
|
+
| `bypass_rls` | `connect()` with a role that bypasses RLS, without `allowBypassRls`; `createMcpServer` with a connection that bypasses RLS |
|
|
2433
|
+
| `user_session_required` | `createMcpServer` and `letopis mcp`: a user session is needed — the connection has no session, it is the service root without `as()`, a session in the System tenant without `allowSystemAdmin`, or the command line got the service key or no token |
|
|
2434
|
+
| `confirmation_required` | `letopis_write` (MCP): a deletion without a human confirmation — the client has no elicitation, the human declined, the answer belongs to another operation, or the data changed after the preview |
|
|
1808
2435
|
| `removed` | a removed 0.21 API; the text gives the replacement |
|
|
1809
2436
|
|
|
1810
2437
|
A PostgreSQL permission denial (`42501`) comes, for example, on an `upsert` over someone else's row that you may not
|
|
@@ -1847,9 +2474,26 @@ load in 77 s (13,902 rows/s, 1.4 GB of WAL). Most of the difference comes from t
|
|
|
1847
2474
|
the load costs 36 s, building them at the end about 5 s. Preparing rows in the loader itself (checks, ends, ids) runs
|
|
1848
2475
|
alongside the `COPY` of the previous batch and now sets the pace: about 60 s of the 77.
|
|
1849
2476
|
|
|
2477
|
+
Derived data in 1.2 (`cd lib && npx tsx bench/derived.bench.mjs`, 2026-10-27, PostgreSQL 18.6, one machine; for
|
|
2478
|
+
aggregates, the median of three rounds; the projector rows are a repeat of series 2 and 3 after the folding of `inc` in
|
|
2479
|
+
`out`, `BENCH_SERIES=2,3`):
|
|
2480
|
+
|
|
2481
|
+
| Measurement | 1.2 | For comparison |
|
|
2482
|
+
|---|---|---|
|
|
2483
|
+
| an operation with an aggregate, one writer | 94 operations/s, p50 10.7 ms | without an aggregate 137 and 6.9 ms; the `inc` recipe 69 |
|
|
2484
|
+
| operations on one wallet with an aggregate, 20 writers | 152 operations/s, p95 0.47 s | without an aggregate 1,987; the `inc` recipe 190 |
|
|
2485
|
+
| operations on 20 wallets with an aggregate, 20 writers | 1,379 operations/s | without an aggregate 2,023 |
|
|
2486
|
+
| projector catch-up, `out.inc` for every change | 6,076 and 33,395 changes/s (`batch` 500 and 2000) | before the folding of `inc` in `out`: 992 and 1,203 |
|
|
2487
|
+
| rebuild of 75 thousand rows | atomic 13.3–13.5 s, in batches 13.0–13.2 s | before the folding of `inc` in `out`: 101 s and 58–63 s |
|
|
2488
|
+
| delay of a subscription event during a rebuild | atomic up to 13.4 s, in batches up to 0.21 s | without a rebuild up to 0.12 s |
|
|
2489
|
+
|
|
2490
|
+
The aggregate triggers fire on every write statement, but for classes without aggregates the measurement does not see
|
|
2491
|
+
their cost: the p50 difference is from −0.28 to +0.21 ms. The benchmark against 1.0.1 (`bench/vs-101.bench.mjs`, four
|
|
2492
|
+
runs) showed no write or read regressions against 1.1: two outliers did not repeat in separate checks.
|
|
2493
|
+
|
|
1850
2494
|
## 18. Limitations
|
|
1851
2495
|
|
|
1852
|
-
This section lists what letopis 1.
|
|
2496
|
+
This section lists what letopis 1.3 deliberately does not do or does with caveats, and where you need to watch the
|
|
1853
2497
|
behavior.
|
|
1854
2498
|
|
|
1855
2499
|
**Platform**
|
|
@@ -1935,13 +2579,22 @@ behavior.
|
|
|
1935
2579
|
attempts.
|
|
1936
2580
|
- When users link a taken identifier themselves, the answer is `acl_denied`, so a user with a recent sign-in learns
|
|
1937
2581
|
this way that the identifier is taken. Permission changes in all tenants are serialized through one counter row.
|
|
1938
|
-
-
|
|
1939
|
-
|
|
2582
|
+
- Refusal texts name the class, role and id of a referencing row only if the actor can see it: a cascade delete,
|
|
2583
|
+
unsetting an end, rewriting references on a class or key change, and a `parent` change conflict
|
|
2584
|
+
(`tightening_conflict`). The number of invisible rows that block a deletion or a
|
|
2585
|
+
class change (`reclass_denied`), and the existence of an account by id (as a reference target), are not hidden.
|
|
2586
|
+
- Tightening a class description and `rekeyClass` check all rows of the family, so they need the `READ` permission
|
|
2587
|
+
without conditions on all rows of the class and its descendants: otherwise a trial tightening (`maximum`, `enum`,
|
|
2588
|
+
`pattern`) would answer with the number of invisible rows that do not match it. A system administrator who tightens
|
|
2589
|
+
a System class gets the number of violating rows of other tenants (`others`): this is the installation level.
|
|
1940
2590
|
- Class metadata (`meta`) is visible to anyone who has a session: anyone can read `Class` rows. Secrets do not belong
|
|
1941
2591
|
there.
|
|
1942
|
-
- An application role can slow the installation down for everyone: `LOCK TABLE entity`, advisory locks with
|
|
1943
|
-
|
|
1944
|
-
installation). The global journal number reveals the overall write activity.
|
|
2592
|
+
- An application role can slow the installation down for everyone: `LOCK TABLE entity`, advisory locks with the keys
|
|
2593
|
+
of the application's `lock()`, fake `NOTIFY` signals (they only clear the cache), frequent permission changes (one
|
|
2594
|
+
counter row per installation). The global journal number reveals the overall write activity. Internal locks — class
|
|
2595
|
+
families, projectors and maintenance — are rows of the `secret` table, not advisory locks: neither the application
|
|
2596
|
+
role nor any other database role can take them. A family writer passes the tightening gate in a subtransaction — one
|
|
2597
|
+
per transaction for the first write to the family.
|
|
1945
2598
|
|
|
1946
2599
|
**History, signals, cache**
|
|
1947
2600
|
|
|
@@ -1969,6 +2622,12 @@ behavior.
|
|
|
1969
2622
|
- The `NOTIFY` queue (8 GB) is shared by the whole cluster: a stuck listener fills it up, and then transactions with
|
|
1970
2623
|
`NOTIFY` fail on commit. The library reads signals right away, and `maintain()` warns when the queue is more than
|
|
1971
2624
|
half full.
|
|
2625
|
+
- PostgreSQL commits transactions with `NOTIFY` one at a time: a commit holds the cluster-wide lock of the signal queue
|
|
2626
|
+
together with the disk write. A raw SQL write on its own connection sends the signal from a trigger in the same
|
|
2627
|
+
transaction, so concurrent raw SQL writers commit one after another: on the benchmark machine, 20 such writers did
|
|
2628
|
+
about 520 inserts per second. Library transactions, `db.sql` and `db.transaction()` included, commit the data
|
|
2629
|
+
without `NOTIFY` and send the signals later, separately, so their writers do not wait in this queue: 20 library
|
|
2630
|
+
writers did about 2,000 operations per second.
|
|
1972
2631
|
- The subscription horizon is shared by the whole cluster: a long write transaction delays the events of all
|
|
1973
2632
|
subscribers (`sub.lag()`); long operations are better done in batches.
|
|
1974
2633
|
- Between processes, the result cache goes stale for the signal sending interval (100 ms) plus delivery, and for the
|
|
@@ -1977,6 +2636,67 @@ behavior.
|
|
|
1977
2636
|
- A "from now" subscription cursor carries a snapshot of the subscription moment until it passes that boundary: such a
|
|
1978
2637
|
cursor is longer than a regular one.
|
|
1979
2638
|
|
|
2639
|
+
**Projectors**
|
|
2640
|
+
|
|
2641
|
+
- A projector is an admin process: it works only on an admin connection, reads the journal of all tenants bypassing
|
|
2642
|
+
RLS and writes its results as the owner role. Keep it in a separate process, like the loader. A participant of a
|
|
2643
|
+
projector is a tenant in which all `into` classes are marked with its `derivedBy`; only an administrator may mark a
|
|
2644
|
+
class (the `letopis.projector` permission).
|
|
2645
|
+
- The totals of a derived class are seen by everyone whom the tenant rules give `READ` on it — even by those who do not
|
|
2646
|
+
read the source rows: decide who reads it with rules on the derived class itself. With `history: true`, its journal
|
|
2647
|
+
shows the change of every batch, that is, the contribution of single source rows too.
|
|
2648
|
+
- Only their projector writes the rows of derived classes: the application, raw SQL and the loader get
|
|
2649
|
+
`derived_class`, and ends may not refer to such rows.
|
|
2650
|
+
- An atomic rebuild is one long write transaction: while it runs, the subscription horizon stands still, and events
|
|
2651
|
+
are delayed for the subscribers and projectors of all processes. On the benchmark machine, a rebuild of 75 thousand
|
|
2652
|
+
rows took 13 s, and the subscriber got no event until it ended. That is why `run()` itself rebuilds in batches, and
|
|
2653
|
+
only an explicit `rebuild()` runs an atomic rebuild.
|
|
2654
|
+
- A batch rebuild holds the horizon only for the time of a batch (in the same measurements, events were delayed by up
|
|
2655
|
+
to 0.2 s), but the derived data is incomplete until it ends: the rows are deleted at once and reappear as the
|
|
2656
|
+
rebuild goes on.
|
|
2657
|
+
- A projector sees only the changes from the journal, so a class with `history: false` cannot be a source: a tenant
|
|
2658
|
+
with such a source drops out of the projector (`invalid_class`) until the journal of the class is turned on and the
|
|
2659
|
+
tenant is rebuilt.
|
|
2660
|
+
- Thinning the journal older than the `all` tier leads to a rebuild: a projector that lags behind by more than that
|
|
2661
|
+
tier gets `cursor_expired` or, with `onExpired: 'rebuild'`, rebuilds itself. The tier is per tenant: if it has
|
|
2662
|
+
expired only for some participants, only they drop out and rebuild.
|
|
2663
|
+
- A projector works only on a schema with the engine revision the library expects; on a schema of another revision it
|
|
2664
|
+
answers `invalid_data`.
|
|
2665
|
+
- `purge()` waits for projectors: while a projector whose sources include the class of the object, and which has a
|
|
2666
|
+
derived class in the object's tenant, has not read the deletion, `purge` answers `projector_pending`. A projector
|
|
2667
|
+
that is no longer run holds `purge` of its sources while the tenant has its derived classes: run it or delete them.
|
|
2668
|
+
`trimHistory` waits the same way: while the projector has not read the first remaining version, trimming answers
|
|
2669
|
+
`projector_pending`.
|
|
2670
|
+
|
|
2671
|
+
**Aggregates in the database**
|
|
2672
|
+
|
|
2673
|
+
- An aggregate queues the writers of one object: every operation locks the target row and writes its version. On the
|
|
2674
|
+
benchmark machine, 20 writers of one wallet with an aggregate did 152 operations per second (p95 is 0.47 s), without
|
|
2675
|
+
an aggregate 1,987. For hot counters, use a projector that folds a batch.
|
|
2676
|
+
- `reclass` and `rekey` accumulate contributions until the end of the move, while a delete cascade and the loader apply
|
|
2677
|
+
them per statement: the class rule is checked on the total of each cascade statement and of each loader batch. So a
|
|
2678
|
+
load in which debits come in a batch before top-ups gets `invalid_data`, although the total of the whole load is
|
|
2679
|
+
valid. Every batch that touches a target writes its version.
|
|
2680
|
+
- A tenant aggregate does not count rows of a System class, and the field of a System row counts only System sources.
|
|
2681
|
+
|
|
2682
|
+
**Agents and MCP**
|
|
2683
|
+
|
|
2684
|
+
- The `<untrusted-data-…>` wrapper is a mitigation, not a protection: the model may follow the text inside it. The
|
|
2685
|
+
protection is the rights in the database: the agent reads and writes only what its user is allowed to. In
|
|
2686
|
+
`structuredContent`, the database data lies in `data` and `error` next to the `notice`, without marks: a client that
|
|
2687
|
+
gives the model that instead of the text passes the data with the note but without a random mark.
|
|
2688
|
+
- By default the server only reads; writing is `write: true` (`--write`); deletion is only with a human confirmation.
|
|
2689
|
+
The server writes the permission classes only with `allowAclWrite` and accepts a session in the System tenant only
|
|
2690
|
+
with `allowSystemAdmin`; `create`, `update` and `upsert` of the other classes need no human confirmation.
|
|
2691
|
+
- The HTTP handler serves revision 2025 statelessly: the per-request instance does not know the client's
|
|
2692
|
+
capabilities, so deletion through it is refused (`confirmation_required`). Delete over stdio or in revision
|
|
2693
|
+
2026-07-28.
|
|
2694
|
+
- A confirmation in revision 2026-07-28 is signed with a process key: a retried call that lands in another process of
|
|
2695
|
+
the application (several instances behind a load balancer) is refused with `confirmation_required`.
|
|
2696
|
+
- `versions` returns the latest versions within `maxRows` and `maxBytes` (at most `maxRows` + 1 versions are read
|
|
2697
|
+
from the database) and `more`; earlier ones, through `diff` and `blame` with `from`, `to`, `at`. A mode where the
|
|
2698
|
+
agent writes code against the API (code mode) is not part of the package.
|
|
2699
|
+
|
|
1980
2700
|
**Import from 0.21**
|
|
1981
2701
|
|
|
1982
2702
|
- 1.0 is a new major version, not a continuation of 0.21: `count()`, `paths()`, `create`, roles, multiple ends and
|
|
@@ -1994,7 +2714,8 @@ This section is for those who develop letopis itself. Run: `cd lib && npm test`.
|
|
|
1994
2714
|
# connect — connecting under a session, refusing roles that bypass RLS, the registry, the deferred signal, error translation (stage 2)
|
|
1995
2715
|
# context — sessions, service sign-in by key, the RLS policies of stage 1 (stage 1)
|
|
1996
2716
|
# access-rest — the remaining rows of stage 5: loading on behalf of a user, balance No. 20, 22, 29, upsert and { rev } under ACL, inlining of acl_ok (stage 5)
|
|
1997
|
-
#
|
|
2717
|
+
# aggregate — aggregates in the database: 20 writers of one wallet, raw SQL, a cascade, reclass, restore, the loader, account deletion, restore and rekey of the target, a description change under writes and race No. 18, aggregate_field, repair and verify, a System row (plan 1.1, stage 4b)
|
|
2718
|
+
# acl — ACL permissions: the 0.21 suite ported, row conditions, weight and default deny, cascade, deny on a descendant, deep, an invisible target, raw SQL versus the library; a change of parent and alias of a class only with permissions on all its rows, an alias is not the name of another class, a name is not another class's alias (stage 5; security review No. 2)
|
|
1998
2719
|
# acl-parity — 1000 random rule sets: the database permission plan and endpoint decisions versus decide from 0.21 (stage 5)
|
|
1999
2720
|
# auth — credentials (password, key, key + secret, OTP, TOTP, identity), timing equalization, case, session revocation, recent sign-in (stage 5)
|
|
2000
2721
|
# balance — top-up and debit: idempotency by opId, races, deadlock, connection drop (TCP proxy), adjustment under { rev } and forUpdate (stage 4)
|
|
@@ -2004,16 +2725,16 @@ This section is for those who develop letopis itself. Run: `cd lib && npm test`.
|
|
|
2004
2725
|
# chain-write — writing with chains: create, update, upsert, delete with preview, anonymize, reclass, slots, inc, { rev }, fan-out, plan rollback (stage 4)
|
|
2005
2726
|
# demo — the booking demo domain on Zod: seed from models, writing data, id by key, agreement between Zod and the database (stage 2)
|
|
2006
2727
|
# docs — documentation examples are executed: the marked blocks of the README and the cheatsheet (Russian and English) run as scenarios against PostgreSQL 18 (stage 8)
|
|
2007
|
-
# engine-attacks — raw SQL attacks from the application role: the rows of §7.3.1 for stages 1 and 3 with specific outcomes; the signal key without permission, flooding with fake signals, Class included (stages 1, 3; plan 1.1,
|
|
2008
|
-
# engine-class — classes as entities: metaschema, inheritance, recomputing descendants, tightening, metadata (stage 1)
|
|
2728
|
+
# engine-attacks — raw SQL attacks from the application role: the rows of §7.3.1 for stages 1 and 3 with specific outcomes; the signal key without permission, flooding with fake signals, Class included; writing to a derived class and a reference to it, a projector checkpoint through the credential functions; an agent through MCP without --write and without a human confirmation; a change of parent, alias and derivedBy of a class with the WRITE permission on Class alone, deleting and re-creating a class with a history of rows, a trial tightening as an oracle, reclass without DELETE and rewriting references deeper than the first level, a retention policy with the WRITE permission on Class alone; an agent through MCP grants a membership or a rule, a system administrator session; a tenant user creates a derived class on their own, `rekeyClass` of an ancestor of a derived class; moving a row in a way that would rewrite references of derived class rows; the application role holds advisory locks with the former keys of internal locks, while lock rows are out of its reach; a forged note of the family locks already taken (stages 1, 3; plan 1.1, stages 2, 4a, 5, checkpoint A, security review No. 2, checkpoint B and the work after it, security review No. 3)
|
|
2729
|
+
# engine-class — classes as entities: metaschema, inheritance, recomputing descendants, tightening, metadata; a class with a history of rows is deleted only after purge() (stage 1; the work after checkpoint B)
|
|
2009
2730
|
# engine-write — the write pipeline in raw SQL: v5 ids, defaults, merge, inc, deletion, resurrection, journal, signals (stage 1)
|
|
2010
|
-
# engine-races — races on two connections in both orders, without timed pauses (stage 1)
|
|
2011
|
-
# explain — chain indexes from the application role: outside, the primary key and (class, tenant); inside find_*(), GIN (stage 3)
|
|
2012
|
-
# find-ids-parity — 30 chains through find_*() and through a direct read under RLS give the same answers (stage 3)
|
|
2731
|
+
# engine-races — races on two connections in both orders, without timed pauses; tightening during continuous writes to the family does not starve; locking families whose roots differ only in case, and a planted row of kind LOCK (stage 1; security review No. 3)
|
|
2732
|
+
# explain — chain indexes from the application role: outside, the primary key and (class, tenant); inside find_*(), GIN; an asOf slice by class does not read another tenant's journal (stage 3; the work after checkpoint B)
|
|
2733
|
+
# find-ids-parity — 30 chains through find_*() and through a direct read under RLS give the same answers; own memberships in other tenants and the Class rows of System — now, asOf and withDeleted (stage 3; the work after checkpoint B)
|
|
2013
2734
|
# hash — the version hash chain, journal tampering and gaps, verify() (stage 1)
|
|
2014
2735
|
# harness — the test harness: starting the database, a schema per file, two connection strings (stage 0)
|
|
2015
2736
|
# jsonschema — compiling JSON Schema to jsonpath and checking data in the database (stage 1)
|
|
2016
|
-
# impersonation — impersonation: letopis.account only for a service with the permission, target restrictions, author and agent in the journal, sessionFor (stage 5)
|
|
2737
|
+
# impersonation — impersonation: letopis.account only for a service with the permission, target restrictions, author and agent in the journal, sessionFor; the db.as() registry is one per tenant, does not empty, is re-read on a signal of its tenant (stage 5; the work after checkpoint B); the facade registry is not shared with the root, an empty registry is not cached (security review No. 3 leftovers)
|
|
2017
2738
|
# install — the installer and roles: a repeated installation, two installations in one database, three tables (stage 1)
|
|
2018
2739
|
# member — memberships: working in a tenant by roles, auth.switch, leaving with ownership transfer, system rows, tenant roles without System permissions (stage 5)
|
|
2019
2740
|
# model — Zod models, the JSON Schema normalizer, pattern translation, agreement with the database metaschema (stage 2)
|
|
@@ -2037,11 +2758,14 @@ This section is for those who develop letopis itself. Run: `cd lib && npm test`.
|
|
|
2037
2758
|
# cache — result cache: hits, own and others' writes, the key with the user, permissions, revoked sessions, listener drop, ttl, class marking, without LISTEN, a race, db.sql, the limit of signal counters (stage 6; plan 1.1, checkpoint A)
|
|
2038
2759
|
# maintain — maintenance: expired sessions and codes, permissions, one process under a lock, schedule (stage 6)
|
|
2039
2760
|
# up — the full installer: fresh installation, re-apply, custom seeds, automatic database creation, UTF8, version 17, upgrade with migrations, PGlite, the first tenant and user (up({ tenant }), createTenant) (stages 6, 8)
|
|
2761
|
+
# upgrade — schema upgrade from 1.1.0 (re-apply) and from 1.0.1 (upgrade: true, migration 0006) with the letopis-11 and letopis-1 packages: versions and hashes written by the old engine, verify(), the catalog against a clean installation, lock rows, the application, a subscription and a projector over old data, a repeated up() (work after 1.2.0, stage 1)
|
|
2040
2762
|
# scenario-salon — end-to-end salon scenario: 21 acts from 0.21 (salon) with a coverage matrix of the 1.0 API (stage 6)
|
|
2041
2763
|
# telemetry — onQuery and slowMs, describe() and class metadata with inheritance, describeText (stage 6)
|
|
2042
2764
|
# history — diff and blame over the log, the reason of a change for verbs, db.begin() and batches; a version hidden by an ACL condition, the changed column, a missing version, row_changes over 20 thousand targets (plan 1.1, stage 1 and checkpoint A)
|
|
2043
2765
|
# signals — signal modes: the payload seen by a foreign role, cache precision per tenant, System, permission classes, a process without the key, the loader, subscriptions, the key, none, coalescing, the server version, a class change, a resend, the mode after up() and reset({ all }) (plan 1.1, stage 2)
|
|
2044
|
-
# initial-load — the initial load: equality with the regular loader, structure and trigger modes, system rows, a non-empty schema, an insert during the start, rollback, MULTI_INSERT in WAL, commitEvery, import --initial (plan 1.1,
|
|
2766
|
+
# initial-load — the initial load: equality with the regular loader, structure and trigger modes, system rows, a non-empty schema, an insert during the start, rollback, MULTI_INSERT in WAL, commitEvery, import --initial, aggregates (agg_fill) (plan 1.1, stages 3 and 4b)
|
|
2767
|
+
# projector — projectors from the journal: catching up, exactly once on a break, one writer, a long transaction, a transaction on the boundary of the rebuild snapshot, no writes to derived classes and no references to them, the bounds of out, atomic and batch rebuilds during source writes, purge of a source, lag and cursor_expired, the credential functions; the version, the name, an admin connection; a race of two processes; a data error of one tenant drops out only this tenant (failed, rebuild({ tenant }), the tenants option), the checkpoint is only a System row, a rebuild touches only the rows of participants and into families, before after rekey, the all tier and the journal of sources per tenant; folding of out.inc by key in a batch, batch rebuilds on the automatic paths of run() and a second process during them; rebuilding one tenant in batches while the tail goes on; onExpired in batches in another process, a data error without retries, the tier from the start of the rebuild; an empty last pass of a batch rebuild, a tenant that drops out in the middle of its rebuild keeps no partial rows, a class change that keeps the id out of the source family and into it, purge of an unread tombstone (plan 1.1, stage 4a; security review No. 2; checkpoint B and the work after it; security review No. 3)
|
|
2768
|
+
# mcp — MCP tools: the model and meta only in results, tenant rights, the JSON chain and hints of similar names, row and byte limits with a cursor, the untrusted-data wrapper, writing only with write, update only with rev, deletion with a human confirmation (elicitation in revisions 2025 and 2026-07-28), the reason in blame, credentials, tool schemas, letopis without the SDK; letopis mcp over stdio, the HTTP handler; the confirmation window without the alias and meta, countBy and versions limited in the query itself, limits of steps and $or conditions; the permission classes only with allowAclWrite, a session in the System tenant only with allowSystemAdmin; structuredContent with a note, a requestState signature of the server user with an expiry (plan 1.1, stage 5; security review No. 2; checkpoint B and the work after it)
|
|
2045
2769
|
# import — import from 0.21 with a fixture of a real 0.21: key violation, --keyless, per-class counters, verify, self-reference, repeat, ids without a key, a dangling reference, $$strict and fields starting with $, update, credentials, CLI (stage 7)
|
|
2046
2770
|
# security-final — the final security review (stage 8, §7.10 No. 8.4): a family lock by the tenant of the root, account_purge without permission and restore/purge of a foreign id give neither a leak nor a lock; the functions of stage 6, log_refs_at, journal-based find_* reads and watch_read stay within their own tenant; NOTIFY carries only names or mode tokens; the cache key distinguishes the token, the actor and the tenant (stage 8)
|
|
2047
2771
|
```
|