@jarenjs/db 0.46.5 → 0.56.0

package/docs/MODEL-FORMAT.md

@@ -71,6 +71,24 @@ The store runs over SQLite — on Node (`@jarenjs/db/node`), on Bun
 An invalid model document is `JD0005` with a `docPath` pointing at the
 offending member. Model checking happens before any database work.
 
+**A time series needs no new declaration.** A composite index whose
+LAST path is a finite numeric epoch member is the whole physical
+feature a temporal plan reads:
+
+```json
+{ "indexes": [{ "name": "by_series_at", "path": ["$.series", "$.at"] }] }
+```
+
+There is no `derive` kind for it, no column type and no host function.
+The columns before the instant are the prefix an equality has to pin
+for the index to seek, exactly as for any other composite index, and
+the schema is the type source as always: type the instant `integer` and
+a bucket ladder is pushed as integer arithmetic; type it `number` and
+the ladder stays a core refinement, because a truncating division over
+a real would put an instant in the wrong bucket. Which shapes are
+pushed, and the reason each refinement is one, are in ARCHITECTURE.md's
+"The temporal plan".
+
 ### 2.1 Derived indexes (spatial and vector storage)
 
 A generated column must be a scalar (§3), and a GeoJSON position is an
@@ -145,9 +163,9 @@ Three things follow, and none of them is optional:
   per-mapping cell.
 
 **What it costs, both halves** (`benchmark/spatial.js`, the store's own
-rows over 50 000 points): the same `$within` measures <!--bm:spatial.rtreeStore-->0.46 ms against 2 ms — 4.3× in the R\*Tree's favour<!--/bm-->, and loading them
-costs <!--bm:spatial.rtreeLoad-->718 ms against 399 ms for 50,000 documents in one transaction — 1.8× the write cost<!--/bm-->. Isolated from the store on a raw
-connection, the same probe is <!--bm:spatial.rtree-->0.3 ms against 1.9 ms — 6.4× in the R\*Tree's favour<!--/bm-->. Read speed bought with write cost and a
+rows over 50 000 points): the same `$within` measures <!--fact:spatial.rtreeStore-->0.46 ms against 2 ms — 4.3× in the R\*Tree's favour<!--/fact-->, and loading them
+costs <!--fact:spatial.rtreeLoad-->718 ms against 399 ms for 50,000 documents in one transaction — 1.8× the write cost<!--/fact-->. Isolated from the store on a raw
+connection, the same probe is <!--fact:spatial.rtree-->0.3 ms against 1.9 ms — 6.4× in the R\*Tree's favour<!--/fact-->. Read speed bought with write cost and a
 second table: choose it deliberately, per index, which is why it is
 neither automatic nor a store-wide option.
 
@@ -180,11 +198,11 @@ Three things follow:
 
 **What it buys, and what it costs.** Measured against the same query
 over a collection with no such column — the whole embedding parsed out
-of the stored JSON per row — the column is worth <!--bm:vector.jsonDoc-->15.0× the plan at 10,000 × 768<!--/bm-->
-on the read, and costs <!--bm:vector.write-->10.6 s against 5.0 s for 50,000 documents in one transaction — 2.1× the write cost<!--/bm-->
+of the stored JSON per row — the column is worth <!--fact:vector.jsonDoc-->15.0× the plan at 10,000 × 768<!--/fact-->
+on the read, and costs <!--fact:vector.write-->10.6 s against 5.0 s for 50,000 documents in one transaction — 2.1× the write cost<!--/fact-->
 on the way in, because every write pays a JSON round trip of the member
 plus the normalize and the pack. On
-disk it is <!--bm:vector.storage-->3,072 B packed against 16,141 B as a JSON number array inside the document — 5.3× smaller<!--/bm-->
+disk it is <!--fact:vector.storage-->3,072 B packed against 16,141 B as a JSON number array inside the document — 5.3× smaller<!--/fact-->
 per vector — smaller than the member, and *added* to it, since the
 document still carries what the column is derived from. Choose it the
 way `physical: 'rtree'` is chosen: per index, with both halves in view
@@ -599,6 +617,15 @@ returns a validation function; the function returns `true`/`false` or
 and `store.capabilities.validated === false` — a declared downgrade.
 The cost of running without one: the database constraints only see the
 key and the indexed members; everything else is stored as given.
+Stated more sharply, because it changes ANSWERS and not only what is
+stored: a typed generated column carries SQLite affinity, so a value
+that violates the collection schema — the string `'020'` under an
+`integer` path — reads as the integer `20` in the column while the
+document still holds text, and a pushed `$eq: ['$it.n', '20']` finds
+a row the engine, which compares the JSON value, does not. On an
+unvalidated store the native and residual paths agree only over
+documents that conform to the schema; with the hook injected no other
+document is ever stored.
 `@jarenjs/db` never runs a validator of its own; what it imports from
 `@jarenjs/validate` is only the pure same-document `$ref`/`$anchor`
 resolution in `@jarenjs/validate/normalize`, for model compilation.
@@ -622,6 +649,13 @@ throw rolls back exactly its own level and rethrows — an outer
 transaction that catches the error continues and its own work
 commits. There is no implicit retry.
 
+**On an asynchronous driver a refused write rejects.** Every write —
+`insert`, `put`, `patch`, `delete`, the entity set's `create`/`update`/
+`delete` and the queue's `enqueue` — answers its coded error (`JD2001`,
+`JD2003`, `JD2005`, …) through the promise it returned: never a
+synchronous throw, and never an unhandled rejection beside a result
+that looks like success.
+
 ### 5.1 Transaction ownership
 
 A SQLite connection holds ONE savepoint stack, so two transactions that
@@ -709,10 +743,12 @@ error.
 | `JD0030` | an unknown x-entity member was declared |
 | `JD0031` | relation declarations contradict each other |
 | `JD0032` | the include specification is invalid |
+| `JD0033` | an entity query names no entity array |
 | `JD0040` | the save spans a relation cycle |
 | `JD0050` | live queries require change capture |
 | `JD0051` | the demanded live mode is unavailable |
 | `JD0052` | the live-query bound was reached |
+| `JD0053` | the live event-time declaration is invalid |
 | `JD2001` | insert found the key already present |
 | `JD2002` | a usable key could not be resolved for the write |
 | `JD2003` | the write failed schema validation |
@@ -726,6 +762,7 @@ error.
 | `JD2060` | the maintained live state exceeded its bound |
 | `JD2061` | another context owns the database |
 | `JD2062` | the store closed with job handlers still in flight |
+| `JD2063` | the store is closed |
 
 The table above is proven in sync with the runtime `DB_CODES` table by
 a test.
@@ -744,7 +781,11 @@ collection.query(doc, { profile: { maxRows: 200, externals: ['min'] } });
 ```
 
 `'safe'` is the default table; a profile object overrides members over
-it. The defaults: engine limits
+it. A per-call `profile` REPLACES the store's for that call — it is
+normalized over the `'safe'` defaults, not over the store's profile —
+so a store-level mandatory predicate or allow-list does not carry into
+`execute(doc, { profile: { maxRows: 50 } })`: spell the whole profile
+per call, or set it once on the store and pass none. The defaults: engine limits
 `{ sequenceItems: 100000, resultItems: 10000, steps: 1000000, depth: 32 }`,
 `maxRows: 1000`, no externals, no host functions, no collations, all
 of the store's collections, no mandatory predicates, no scan refusal.
@@ -926,15 +967,15 @@ changing state. Measured on the same terms as the `$sqrt` rows
 mean of 20 executions after a warm one; the residual comparator
 pushes everything BUT the spatial conjunct):
 
-<!--bm:spatial.udfTable-->
+<!--fact:spatial.udfTable-->
 | shape | pushed (ms) | residual (ms) | verdict |
 |---|---|---|---|
 | solo `$within` over a full scan | 94 | 80 | ~even |
 | indexed `$eq` **and** `$within` (~5 % pass the index) | 5.7 | 52 | push **9.1×** |
 | `$within` with `LIMIT 10` | 1.9 | 77 | push **41.0×** |
-<!--/bm-->
+<!--/fact-->
 
-So the spatial hatch <!--bm:spatial.udfVerdict-->earns its row: 9.1× beside the selective conjunct and 41.0× under the LIMIT<!--/bm-->,
+So the spatial hatch <!--fact:spatial.udfVerdict-->earns its row: 9.1× beside the selective conjunct and 41.0× under the LIMIT<!--/fact-->,
 by the same rule as `$sqrt`: a sole `$within` over a full scan is a
 loss (the UDF re-parses every row in the callback, and the exact
 containment test is dearer than a square root), a `$within` beside
@@ -994,7 +1035,10 @@ names:
 The schema stays a valid JSON Schema throughout: strip every
 `x-entity` member and it accepts and rejects exactly the same values
 (test-asserted over a corpus). The vocabulary is invisible to the
-validator by the same argument as `x-form`.
+validator by the same argument as `x-form`. A model document may also
+be written by code — `@jarenjs/linq/model`'s `defineModel()` emits
+exactly this document ([MODEL-PEN.md](../../linq/docs/MODEL-PEN.md)), and this section stays its
+one specification.
 
 ### 9.2 The `x-entity` vocabulary (a closed set)
 
@@ -1014,6 +1058,13 @@ happen, so this vocabulary is deliberately stricter than the
 validator's ignore-unknown posture — the strictness is local to the
 one namespace this package owns.
 
+The vocabulary is closed in POSITION as well as in name: `x-entity` is
+read on an entity's top-level properties (and through their `allOf`,
+`$defs` and `definitions` blocks) only. A block nested anywhere else —
+under a property's `properties`, `items`, `anyOf`, … — is never walked
+for mapping directives, so one found there is `JD0030` at its `docPath`
+rather than a key, an index or a relation that silently never existed.
+
 ### 9.3 The hybrid mapping
 
 Stated once, mechanically applied, and returned as data by
@@ -1023,7 +1074,8 @@ Stated once, mechanically applied, and returned as data by
 |---|---|
 | scalar (`string`/`number`/`integer`/`boolean`) at the top level | a real typed column |
 | `format: date-time`/`date` with `column: "integer"` | an epoch-milliseconds `INTEGER` column; the document keeps the RFC 3339 string, the column carries the derived epoch |
-| `enum` of scalars | a column plus a `CHECK (column IN (…))` |
+| `enum` of scalars | a column plus a `CHECK (column IN (…))` — a `null` member of the enum is left to the column's nullability, never written into the list |
+| a union of several scalar types (`['string', 'integer']`) | the JSONB document — a column has one affinity and a union has several; `['integer', 'null']` is that scalar, nullable |
 | nested object / array, or `column: "json"` | the JSONB document column, queryable by path exactly as in phase A |
 | relation | a foreign-key column, or a join table for many-to-many (§9.4) |
 
@@ -1033,13 +1085,39 @@ everything else; a read merges them back. **The absent-versus-null
 rule, plainly**: for a column-mapped scalar, JSON `null` and absence
 both store as SQL `NULL` and read back as ABSENT. A property that
 needs present-`null` semantics declares `column: "json"` and stays in
-the document.
+the document. The rule reaches the WRITE's return value, not only the
+read: `create()`, `update()` and `add()` answer the document a read
+would answer, so a `null` handed in for a column-mapped scalar is
+absent from what comes back — a returned object never names a member
+no read will show. An epoch column (§9.3's `column: "integer"` row)
+keeps its string in the document, present-`null` included.
 
 ### 9.4 Relations and referential integrity
 
 Declared on one side, inferred on the other; when both sides declare,
 the inverses MUST agree (`JD0031` on any contradiction).
 
+What counts as "the same edge" is decided by `via`, not by the pair of
+entity names: two declarations pair up when they name the same
+foreign-key property, and a pair is one `many` side and one `one`
+side. Two `many` sides, or two `one` sides, on one `via` is `JD0031`,
+and a paired edge must agree on `onDelete`.
+
+Between one pair of entities, a `many` declaration facing a `one`
+declaration with a DIFFERENT `via` is `JD0031` as well (the pair
+disagrees on its key) — the check is per entity pair, so **two
+one-to-many edges between the same two entities declare their inverse
+on ONE side only**. The author-and-editor model is written with both
+edges on `User` (`authored` by `authorId`, `edited` by `editorId`) and
+the foreign-key properties on `Post`, or with both edges on `Post` and
+none on `User`; declaring `Post.author` AND `User.edited` makes
+`User.authored` face `Post.editor` and is refused. Two declarations of
+the SAME kind with different `via`s are two independent edges — which
+is how a legitimate cycle is written (`Post.author` by `authorId` and
+`User.featured` by `featuredPostId`, one-to-one each way).
+Many-to-many pairs must agree on the join table, and a many-to-many
+facing a foreign-key relation is `JD0031`.
+
 - **one-to-many** — `{ to, many: true, via, onDelete }`: `via` names
   the foreign-key property on the TARGET entity (`authorId` on
   `Post`). If the target declares that property it MUST be a
@@ -1077,19 +1155,34 @@ reads and deletes take `{ prop: value, … }`.
 Applied on write in JavaScript, never by SQL `DEFAULT`, so the value
 the application sees and the value stored are the same — and the
 behaviour is identical on every driver. `"now"` stamps an RFC 3339
-UTC string on insert when the property is absent; `"updated"` stamps
+UTC string on insert when the property is absent — the calendar date
+alone (`YYYY-MM-DD`) on a `format: date` property, so the stamp
+validates against its own format; `"updated"` stamps
 on insert AND on every update, always; `{ "value": … }` fills a
 literal when absent; `{ "query": … }` evaluates a query document over
 the document being written. Defaults run BEFORE validation, so the
-injected hook sees the completed document.
+injected hook sees the completed document — with one member exempt: a
+store-allocated key (`default: "auto"`) is allocated by the database
+AFTER validation, so a write validates against the schema with that
+key dropped from `required` (the generated input type marks it
+optional for the same reason); the document the store answers carries
+it, and the read shape keeps it required. A `version` property is
+engine-owned and never defaulted by the caller: an insert without one
+writes `0` — not SQL `NULL`, which no `WHERE version = ?` guard could
+match — and every successful write bumps it (§11.5).
 
 ### 9.7 Error-code additions
 
-The entity engine adds two codes to the package's single table (§7):
-`JD0030` — an unknown `x-entity` member; `JD0031` — relation
-declarations whose inverses contradict. Everything else raises the
-existing codes (`JD0005` for structural model defects, `JD2005` for
-database-refused writes including foreign-key violations).
+The entity engine adds three codes to the package's single table (§7):
+`JD0030` — an unknown or unread `x-entity` member; `JD0031` — relation
+declarations whose inverses contradict; `JD0033` — an entity query
+document that binds no entity array (§10.1). Everything else raises the
+existing codes: `JD0005` for structural model defects (a key, index or
+version property without a column of its own, a default on a relation,
+`default: "auto"` off the key, a self-referencing many-to-many, a
+foreign key onto a composite key), `JD2001` for a duplicate key on
+`create()` exactly as on a collection `insert`, and `JD2005` for
+database-refused writes including foreign-key violations.
 
 ## 10. Relational translation
 
@@ -1103,13 +1196,65 @@ translatable runs the set residual over the fetched entity root,
 
 `store.execute(document)` queries the **multi-entity root**: the
 engine-side value is `{ <EntityName>: [documents…], … }` and bindings
-range over `$.<Entity>[*]`. This is the shape the differential oracle
+range over `$.<Entity>[*]`. A document that binds no entity array at
+all — a `$for` over a member the model does not name, or over a
+scalar — is `JD0033` at compile, never an empty answer. The
+array-constructor spelling `["$.<Entity>[*]"]`, which `@jarenjs/linq`
+emits so that an item that is itself an array stays one item
+(QUERY-PEN §3), names the same whole-entity source; the planner
+reads through it for collections (`["$[*]"]`) and entities alike.
+
+An entity set is a provider (QUERY-PEN §8): `store.entity(name)` and
+`store.sync.entity(name)` carry `execute(document, options)` and
+`explain(document, options)` — the document is over the multi-entity
+root and goes to the entity translator whole, exactly as
+`store.execute` runs it — plus `root` (`"$.<Entity>[*]"`, the one
+spelling `collectEntityRoots` reads, so a chain binds its items through
+it) and `scope` (one identity per store, shared by every set, so two
+sets may be joined in one document). The store itself carries `roots`,
+the entity names, because it has no single root: a chain over it is
+refused by name (`JL0007`) rather than answering the mixed rows or
+counting the sets, and a document naming no entity array stays
+`JD0033`. A chain's element terminal hands over the one-item window
+`[<phrase>]` (QUERY-PEN §6); the planner reads through the window as
+it reads through the packed source — the phrase inside plans as if
+bare, and the store answers its rows as the ONE array item the engine
+would (`[]` for none, `[row]` for one, never singleton-unwrapped;
+`explain().wrapped` says so) — so `toArray()` over an entity set runs
+the translator, and a two-root equijoin returning a bare binding runs
+as one statement.
+This is the shape the differential oracle
 can actually prove — the in-memory engine sees exactly the documents
 the entity sets return (`test/db/oracle/relations/`). Relation-NAME
 navigation (`$.author.name`) is deliberately not query-document sugar:
 the engine has no embedded `author` member to walk, so no oracle could
 vouch for it. Name-based navigation lives on the `load` surface
-(§10.4), where results and statement counts are the proof.
+(§10.4), where results and statement counts are the proof — and, by
+code, on a query producer that lowers a hop to what the engine CAN walk.
+
+For that, every entity set carries **`relations`**, the relation table
+a producer may read: a frozen plain record, one entry per declared
+relation member of the entity, `{ to, kind, via?, fkEntity?, fkTargets?,
+joinTable?, targetKey }` — `to` the target entity, `kind` one of
+`oneToOne`/`oneToMany`/`manyToMany`; for a foreign-key relation `via`
+the key property, `fkEntity` the entity holding it, `fkTargets` the
+entity it references and `targetKey` the key property it references
+there (the column a hop's equality compares `via` with; `kind` says
+which side holds the key: `oneToOne` the declaring entity, `oneToMany`
+the target); for a many-to-many `joinTable` and the target's `targetKey`.
+The store (`store.relations`) and every set's `scope` (`scope.relations`)
+carry the same tables keyed by entity name, so a producer holding one
+set can follow a hop into another root. `relationTables(entities)` is
+the one function that derives them. The document format is unchanged:
+`@jarenjs/linq`'s chain reads the table and lowers `p.author.email` to
+`{ $for: { r1: '$.User[*]' }, $where: { $eq: ['$r1.id', '$it.authorId'] },
+$return: '$r1.email' }` (QUERY-PEN §4, relation navigation) — a
+document a hand could have written, which this translator and the
+engine both run and the oracle proves (`15-linq-hops.json`). Every such
+lowered shape is a residual here (§10.6 — a projection, a comparison or
+an existence test over a phrase rather than a member path, an ordering
+by a phrase), named by `explain()` and refused under `strict`; none
+pushes natively in this version.
 
 Per binding, predicates resolve through three reference flavors:
 
@@ -1193,8 +1338,11 @@ store.entity('User').load({
 })
 ```
 
-Per-relation `where`/`orderBy`/`take` apply INSIDE the subquery — the
-point where naive loaders fall back to N+1. Clauses compile against
+Per-relation `where`/`orderBy`/`take`/`skip` apply INSIDE the subquery
+(`LIMIT`/`OFFSET` per parent row) — the point where naive loaders fall
+back to N+1; `after` (§10.5) paginates the ROOT alone, since a keyset
+cursor is one position in one ordered set, and on an include it is
+`JD0032`. Clauses compile against
 the child's own reference flavors; an untranslatable clause is a
 refusal (`JD0032`) naming the include path, never a silent residual.
 Include depth is bounded (default 3, override with `maxDepth`);
@@ -1223,14 +1371,33 @@ a number appears only where `capabilities.rowEstimates` is filled):
 
 - three or more bindings;
 - non-equality join predicates, and disjunctions spanning bindings;
-- `$groupby` (the engine's post-group cardinality rebinding deserves
-  its own order; the count-of-related-rows case ORMs are bad at is
-  already native via `count: true` includes);
+- `$groupby`, except the `$time-bucket` ladder the series plan pushes
+  (README, *Time series*) — the engine's post-group cardinality
+  rebinding deserves its own order; the count-of-related-rows case
+  ORMs are bad at is already native via `count: true` includes;
 - projections (`$return` objects) — over one binding or across a join;
 - externals against document paths; booleans and `null` at bind time;
 - everything phase A already listed (§8 of `QUERY-FORMAT.md`
   notwithstanding, the truth table is the contract).
 
+Two deviations between a pushed answer and the engine's are DECLARED
+rather than refused, because in both the database is right by its own
+arithmetic:
+
+- `$sum`/`$avg` over `number` paths — SQLite sums with Kahan–Babuška
+  compensation and the engine sums naively, so over `0.1, 0.2, 0.3`
+  the store answers `0.6` and the residual `0.6000000000000001`: equal
+  to within an ulp, never equal by `===` (the differential oracle
+  draws dyadic fractions, which are exact on both sides).
+- a `$time-bucket` `$groupby` whose aggregated path admits `null`
+  (`['number', 'null']`): the pushed `AVG` skips a `null` reading
+  exactly as the series kernel — and therefore the `$resample`
+  spelling — does, while the engine's `$avg` over a sequence holding
+  `null` raises `JQ2001`; under this one spelling the store answers
+  where the in-memory engine refuses. Everywhere else an ordering or
+  an aggregate over a path that admits `null`, or over a boolean path,
+  is a named residual, so the two paths keep answering alike.
+
 
 ## 11. The unit of work
 
@@ -1274,8 +1441,19 @@ const report = await store.saveChanges();     // one transaction
 
 ### 11.2 Explicit updates (the other mode)
 
-`set.update(key, changes)` and `set.delete(key)` skip tracking: one
-immediate statement, last-write-wins by contract. An explicit update
+`set.create(doc)`, `set.update(key, changes)` and `set.delete(key)`
+skip tracking: one immediate statement, last-write-wins by contract.
+`create` applies defaults, validates, inserts, and — for a
+many-to-many relation whose member the document carries as an array
+of target keys — writes the join rows in the same transaction, which
+rolls back whole when a membership names a row that does not exist; a
+duplicate key is `JD2001`. `update` takes the document's own members
+only: a relation member (`posts`) is `JD2003` exactly as on `create`,
+and a membership array (`labels`) — which `create` writes — is
+`JD2003` on `update` as well, because an explicit update never writes
+join rows or child rows and never echoes back a member it did not
+store; a change to a key member is `JD2003` too (the key column would
+go stale — delete and create). An explicit update
 still bumps a declared version property, so optimistic savers observe
 the row changed. This is the path reactive layers and job runners use.
 
@@ -1318,7 +1496,10 @@ the 100× statement reduction is the point for anything remote.
 
 ### 11.5 Optimistic concurrency
 
-Declare a token with `version: true` (§9.2). Every `saveChanges()`
+Declare a token with `version: true` (§9.2). A row inserted without
+one starts at `0` — never SQL `NULL`, which no guard could match
+(§9.6) — so the first tracked save after `create()`/`add()` carries
+`WHERE version = 0`. Every `saveChanges()`
 update and guarded delete carries `WHERE version = ?` (the SNAPSHOT
 version) and writes snapshot + 1; a zero-row result is **`JD2040`**
 carrying the entity and key, and the whole save rolls back. Without a
@@ -1348,3 +1529,43 @@ The return value is data, not a boolean:
   elapsedMs,
 }
 ```
+
+### 11.7 Membership: `link` and `unlink`
+
+A many-to-many membership is attached or detached one row at a time
+through the unit of work, without carrying the whole membership array:
+
+```js
+users.link('u1', 'labels', 'admin');           // a key, or a document carrying the key
+users.unlink(ada, 'labels', { name: 'dev' });  // the own side is a key or a document too
+const report = await store.saveChanges();      // { joinInserted: 1, joinDeleted: 1, … }
+```
+
+- `link`/`unlink` are local, synchronous bookkeeping, like `add`/`put`/
+  `remove`; nothing reaches the database until `saveChanges()`.
+- The member MUST be a many-to-many relation of the entity — `JD2003`
+  otherwise, naming the relation's kind or the missing declaration. The
+  target is read exactly as a membership array's element is: a key, or
+  a document carrying the target's key (`JD2003` when it carries none).
+- The own side needs the entity's key. A pending insert whose `auto`
+  key the save allocates has none to attach to and is refused (`JD2003`:
+  "save the entity first, then attach"); a pending insert with a
+  caller-supplied key may be linked in the same save, since join rows
+  run after the inserts (§11.4).
+- The baseline is the join table **as read at save time**, not a
+  snapshot: linking a member that already exists and unlinking one that
+  does not are no-ops, so a save repeated with the same calls changes
+  nothing (asserted). The last word on one target wins — `unlink` after
+  `link` means unlink.
+- A `link`/`unlink` beside a `put` carrying the SAME member's membership
+  array folds into that array's key-set difference (§11.3): one intent
+  per entity, own key and member, never two statements racing for one
+  row.
+- A tracked snapshot's loaded membership array is not rewritten by a
+  saved `link`/`unlink` — a projection stays what it was read as;
+  re-read (`load({ include })`) to see the membership. `discard(key)`
+  drops the key's pending membership changes together with its
+  tracking, and a failed save leaves them pending, as §11.6 promises.
+- The report counts the rows written under `joinInserted`/`joinDeleted`
+  and `stats().tracker.pendingMemberships` counts the pending
+  (entity, key, member) records.

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@jarenjs/db",
   "private": false,
-  "version": "0.46.5",
+  "version": "0.56.0",
   "type": "module",
   "main": "./src/index.js",
   "types": "./types/index.d.ts",
@@ -36,7 +36,6 @@
   },
   "files": [
     "types/",
-    "dist/types/",
     "src/",
     "schemas/",
     "docs/",
@@ -71,9 +70,9 @@
     "prepack": "npm run build:types"
   },
   "dependencies": {
-    "@jarenjs/core": "^0.46.5",
-    "@jarenjs/json": "^0.46.5",
-    "@jarenjs/validate": "^0.46.5"
+    "@jarenjs/core": "^0.56.0",
+    "@jarenjs/json": "^0.56.0",
+    "@jarenjs/validate": "^0.56.0"
   },
   "bin": {
     "jaren-db": "./src/cli.js"

package/schemas/jaren-migration.draft-07.schema.json

@@ -58,6 +58,12 @@
         },
         {
           "$ref": "#/definitions/deriveStep"
+        },
+        {
+          "$ref": "#/definitions/sqlStep"
+        },
+        {
+          "$ref": "#/definitions/rebuildStep"
         }
       ]
     },
@@ -216,6 +222,73 @@
         "columns"
       ],
       "additionalProperties": false
+    },
+    "sqlStep": {
+      "description": "A DATA step spelled directly as one SQL statement (MIGRATION-FORMAT §9.4): the planner's fold of a dropped column into the document, the move of a document member into its new column, an R*Tree backfill. Executed exactly as a ddl step; distinct in kind so a reviewer reads intent and a dry run shows it as data.",
+      "type": "object",
+      "properties": {
+        "kind": {
+          "const": "sql"
+        },
+        "sql": {
+          "type": "string",
+          "minLength": 1
+        },
+        "note": {
+          "type": "string"
+        }
+      },
+      "required": [
+        "kind",
+        "sql"
+      ],
+      "additionalProperties": false
+    },
+    "rebuildStep": {
+      "description": "The ALTER TABLE procedure SQLite lacks (MIGRATION-FORMAT §10), self-contained: create the target shape under `<table>__rebuild`, copy the rows through the rendered column mapping, drop the old table, rename, recreate the indexes, then `PRAGMA foreign_key_check` inside the transaction.",
+      "type": "object",
+      "properties": {
+        "kind": {
+          "const": "rebuild"
+        },
+        "table": {
+          "type": "string",
+          "pattern": "^[A-Za-z_][A-Za-z0-9_]*$"
+        },
+        "create": {
+          "description": "The CREATE statements of the temporary table.",
+          "type": "array",
+          "minItems": 1,
+          "items": {
+            "type": "string",
+            "minLength": 1
+          }
+        },
+        "copy": {
+          "description": "The INSERT … SELECT that carries every row across the column mapping.",
+          "type": "string",
+          "minLength": 1
+        },
+        "indexes": {
+          "description": "The index statements recreated under the final name.",
+          "type": "array",
+          "items": {
+            "type": "string",
+            "minLength": 1
+          }
+        },
+        "note": {
+          "type": "string"
+        }
+      },
+      "required": [
+        "kind",
+        "table",
+        "create",
+        "copy",
+        "indexes"
+      ],
+      "additionalProperties": false
     }
   }
 }

package/schemas/jaren-migration.schema.json

@@ -58,6 +58,12 @@
     },
     {
      "$ref": "#/$defs/deriveStep"
+    },
+    {
+     "$ref": "#/$defs/sqlStep"
+    },
+    {
+     "$ref": "#/$defs/rebuildStep"
     }
    ]
   },
@@ -216,6 +222,73 @@
     "columns"
    ],
    "additionalProperties": false
+  },
+  "sqlStep": {
+   "description": "A DATA step spelled directly as one SQL statement (MIGRATION-FORMAT §9.4): the planner's fold of a dropped column into the document, the move of a document member into its new column, an R*Tree backfill. Executed exactly as a ddl step; distinct in kind so a reviewer reads intent and a dry run shows it as data.",
+   "type": "object",
+   "properties": {
+    "kind": {
+     "const": "sql"
+    },
+    "sql": {
+     "type": "string",
+     "minLength": 1
+    },
+    "note": {
+     "type": "string"
+    }
+   },
+   "required": [
+    "kind",
+    "sql"
+   ],
+   "additionalProperties": false
+  },
+  "rebuildStep": {
+   "description": "The ALTER TABLE procedure SQLite lacks (MIGRATION-FORMAT §10), self-contained: create the target shape under `<table>__rebuild`, copy the rows through the rendered column mapping, drop the old table, rename, recreate the indexes, then `PRAGMA foreign_key_check` inside the transaction.",
+   "type": "object",
+   "properties": {
+    "kind": {
+     "const": "rebuild"
+    },
+    "table": {
+     "type": "string",
+     "pattern": "^[A-Za-z_][A-Za-z0-9_]*$"
+    },
+    "create": {
+     "description": "The CREATE statements of the temporary table.",
+     "type": "array",
+     "minItems": 1,
+     "items": {
+      "type": "string",
+      "minLength": 1
+     }
+    },
+    "copy": {
+     "description": "The INSERT … SELECT that carries every row across the column mapping.",
+     "type": "string",
+     "minLength": 1
+    },
+    "indexes": {
+     "description": "The index statements recreated under the final name.",
+     "type": "array",
+     "items": {
+      "type": "string",
+      "minLength": 1
+     }
+    },
+    "note": {
+     "type": "string"
+    }
+   },
+   "required": [
+    "kind",
+    "table",
+    "create",
+    "copy",
+    "indexes"
+   ],
+   "additionalProperties": false
   }
  }
 }