@jarenjs/db 0.34.0

package/docs/LIVE-FORMAT.md

@@ -0,0 +1,348 @@
+# The Jaren live format — change capture and live queries
+
+This document is normative. The key words MUST, MUST NOT, SHOULD and
+MAY are to be interpreted as described in RFC 2119.
+
+Change capture and live queries are ONE story — patches out — and
+share this document. This order writes §§1–6 (capture); live queries
+continue at §7. Error codes join the package's single runtime table
+(MODEL-FORMAT §7), never a per-document list.
+
+## 1. Scope
+
+Every committed write to a store opened with `capture` produces an
+observable, ordered stream of **RFC 6902 patches** describing what
+changed — derived from SQLite's own session changesets where the
+binding has them, from a write-path journal where it does not. One
+diff format then runs end to end: store → patch → live query → patch
+→ O(k) render. The patches MUST be consumable by
+`applyJSONPatch` from `@jarenjs/json/patch`, unmodified.
+
+```js
+const store = await openStore(model, {
+  driver: nodeDriver(),
+  capture: { mode: 'auto', log: { retention: 1000 } },
+});
+const stop = store.observe(({ seq, at, source, collections, patch }) => {
+  // one record per committed transaction, in commit order
+});
+```
+
+Capture is **opt-in per store** — it costs a transaction wrapper and
+(in session mode) a session per commit; the measured overhead is
+published with the benchmarks, not waved away.
+
+## 2. The pointer contract
+
+Every op's `path` is `/<table>/<key>/<path…>`, each token escaped per
+RFC 6901 (`~` → `~0`, `/` → `~1`):
+
+- `<table>` is the collection, entity or join-table name.
+- `<key>`: a SINGLE key renders as its scalar text (integers in
+  decimal); a COMPOSITE key renders as the JSON text of its parts
+  array (`["ada","math"]`), then escaped as one token. The encoding is
+  deterministic: parts appear in declared key order.
+- `<path…>` is the document-relative remainder, RFC 6901 as always.
+
+The contract is stable — consumers depend on it. Join-table rows are
+tiny documents under the join table's name (`/Label_User/["admin","u1"]`
+→ `{ "Label_key": "admin", "User_key": "u1" }`, columns in the sorted
+pair order): membership changes are part of the stream, not a blind
+spot.
+
+**Op order within a record is UNSPECIFIED.** A commit carries one net
+op per row (SQLite sessions coalesce insert+update, drop
+insert+delete, omit no-op updates — and the journal mirrors that
+discipline), every op targets a distinct pointer, and the record
+applies correctly in any order. The two capture modes MAY order the
+same ops differently; they MUST agree as sets.
+
+## 3. The op mapping
+
+| Row change | Patch |
+|---|---|
+| INSERT | `add` at `/<table>/<key>` with the full document (mapped columns folded back: booleans as `true`/`false`, SQL `NULL` absent, derived epoch columns skipped — the document string is authoritative) |
+| DELETE | `remove` at `/<table>/<key>` |
+| UPDATE, document column changed | old JSONB diffed against new (`createJSONPatch`) → minimal nested ops under `/<table>/<key>` |
+| UPDATE, mapped scalar/foreign-key column changed | property-level op at `/<table>/<key>/<property>`: `replace` (both present), `add` (was `NULL`), `remove` (now `NULL` — absent per MODEL-FORMAT §9.3) |
+| join-table INSERT / DELETE | `add` / `remove` of the membership row document |
+
+Changesets (not patchsets) are used because they carry OLD values —
+that is what makes the minimal nested diff and the add/remove/replace
+discrimination possible; the price is a larger in-memory blob per
+transaction, stated here. Integer values beyond ±2⁵³ convert lossily
+to JS numbers (a raw-SQL concern only; JS documents cannot produce
+them).
+
+## 4. Capture modes and their limitations
+
+`capture.mode` is `'auto'` (default), `'session'` or `'journal'`;
+`store.capabilities.capture` reports what actually runs
+(`'session'`, `'journal'` or `'none'`).
+
+- **`session`** — SQLite's session extension records row changes at
+  the storage layer: every write through this connection is seen,
+  including future raw-SQL surfaces. Requires the binding to expose
+  `createSession` (node:sqlite does; **bun:sqlite does not**; a wasm
+  build may not). Sessions are per COMMIT, never long-lived (an
+  unbounded session is a memory leak); a rolled-back transaction —
+  and a rolled-back savepoint inside a committed one — contributes
+  nothing (probed and pinned by test).
+- **`journal`** — the write path itself emits records: the store
+  already knows what it wrote. **Less complete, stated plainly**: it
+  cannot see writes made through raw SQL, triggers, or another
+  connection; a journal-mode delete of a row the store never read
+  emits its `remove` without having seen the old document; and of the
+  database's own `ON DELETE` side effects it reconstructs exactly ONE
+  — join-table membership dying with its entity (read before the
+  delete) — while cascades into CHILD rows (`onDelete: 'cascade'` /
+  `'setNull'` on one-to-many relations) stay invisible. For writes
+  made through the store API within those bounds the two modes MUST
+  produce the same op sets — proven by a differential test.
+- Requesting `mode: 'session'` on a driver without sessions is a
+  `TypeError` at open — an application that needs completeness can
+  refuse to start rather than silently miss changes.
+
+## 5. The persisted log and retention
+
+With `capture.log`, each record is appended to `_jaren_changes`
+**inside the same transaction** as the writes it describes — an
+observer crash cannot lose a committed record, and a late joiner
+reads forward:
+
+```js
+const records = await store.changesSince(lastSeq);  // JD2051 when no log
+```
+
+`seq` is monotonic; with the log enabled it continues across reopens
+(seeded from `MAX(seq)`), without it it is per-process. Retention is
+a bounded count (`retention`, default 1000): older rows are pruned in
+the same transaction. The log is an ordered, replayable stream —
+which is what makes a late-joining consumer possible. **Replication
+is not built here**, and this log alone does not make it safe: there
+is no conflict resolution, no site identity, no causal ordering
+across writers. That sentence is the whole claim.
+
+## 6. Cross-connection behaviour and non-claims
+
+Another process (or another connection) writing to the same file
+produces NO local patches — sessions and journals are per-connection
+facts. The honest mitigation is a coarse signal, not a pretend
+fine-grained one: `store.dataVersion()` reads `PRAGMA data_version`,
+which changes when ANOTHER connection commits; poll it and treat a
+change as "re-read what you care about". Cross-tab delivery is §7's
+story (the live-query layer).
+
+Non-claims, in one place: no replication, no conflict resolution, no
+capture of writes made by other connections, no capture on stores
+opened without `capture`, and no statement-level ordering within a
+commit (§2).
+
+## 7. Live queries: the maintenance table
+
+A live query is a registered query document whose result is
+**maintained** as committed writes arrive, emitting RFC 6902 patches
+against its own result document (§9). Registration:
+
+```js
+const live = await store.collection('users').live(document, {
+  externals: {},          // fixed at registration (§8)
+  mode: 'auto',           // 'auto' | 'incremental' | 'rerun'
+});
+live.result;              // the maintained result document
+live.mode;                // { strategy, mode: 'incremental'|'rerun', reason }
+const stop = live.subscribe(({ patch, seq, error }) => { /* … */ });
+live.close();
+```
+
+`store.live(document, options)` registers an entity-root document (the
+multi-entity shape of MODEL-FORMAT §10) the same way. Live queries
+REQUIRE change capture — the patch stream is the invalidation source —
+and registering on a store opened without `capture` is `JD0050`.
+
+**This table is normative.** Every row is implemented and tested;
+nothing outside it is attempted. Classification reads the compiled
+PLAN (never the raw document), so "extractable" below means exactly
+what the pushdown planner already means by it.
+
+| Construct (as planned) | Strategy | Maintained state |
+|---|---|---|
+| `where` whose predicates translate (the plan's filter), no order, no aggregate | **incremental rows**: per-row re-evaluation; insert / remove / replace in the result | the result rows |
+| the same with a per-row `select` projection (row-mode plan) | **incremental rows**: the affected row alone is recomputed; a source row may project to several items | result rows, grouped by source key |
+| `orderBy` over extractable paths, optional `limit`, offset 0 | **maintained window**: a sorted structure; ties broken by the collection key, appended as the final sort term; an insert sorting beyond a full window is a no-op | the window rows and their sort keys |
+| whole-query `count` / `sum` / `avg` / `min` / `max` (the plan's aggregate), optional `where` | **running accumulator** plus a per-row contribution map — a delete can only be answered from retained contributions (§3: a `remove` carries no old value). `min`/`max` removal of the last extremum holder FALLS BACK to a recompute over the retained contributions; the accumulator alone cannot answer, and this fallback is the documented cost | one contribution per matching row |
+| single-level `groupBy` with aggregate returns, in the canonical form below | **per-group deltas**: the accumulator machinery, one instance per group; groups appear in first-appearance order, exactly the engine's order | per-group, per-row contributions |
+| joins, multi-entity roots, graph loads, every entity query | **re-run on invalidation — declared, not attempted** in this version | the previous result, for diffing |
+| anything else: non-translatable predicates, `limit` without `orderBy`, `offset` > 0, windowed aggregates, `@jarenjs/linq`'s nested two-level `groupBy` emission, non-canonical group returns | **re-run on invalidation**, the reason named | the previous result, for diffing |
+
+Re-run is a first-class, documented outcome, not a failure. What is
+forbidden is *silently* re-running while the reader believes the query
+is incremental: `live.mode` reports `'incremental'` or `'rerun'`, the
+strategy, and — for re-run — the reason. `mode: 'incremental'` in the
+options DEMANDS incrementality: a query that classifies as re-run then
+refuses at registration (`JD0051`), the same shape as capture's
+demanded session — an application that needs the property can refuse
+to start.
+
+The **canonical group form** the classifier recognises (and the only
+one — the linq chain's nested emission re-runs, stated plainly):
+
+```json
+{ "$for": { "it": "$[*]" },
+  "$where": { "…optional, translatable…": [] },
+  "$groupby": { "g": "$it.dept" },
+  "$return": { "key": { "$default": ["$g", null] },
+               "n": { "$count": "$it" }, "total": { "$sum": "$it.pay" } } }
+```
+
+After `$groupby`, `$it` is the group's item sequence and `$g` its key;
+return members are the group key or an aggregate over `$it` (a path
+below it selects the aggregated member). Anything else in the return
+is not canonical and re-runs.
+
+## 8. Invalidation
+
+Each live query derives its dependencies from the plan at
+registration:
+
+- the collection (or, for entity documents, every entity the plan
+  binds), matched first against a record's `collections` — a
+  non-matching record costs ONE array scan;
+- for accumulator and group strategies, the top-level members the
+  plan actually reads (filter refs, the aggregated path, the group
+  key): an update record touching only other members is skipped;
+- row and window strategies depend on their WHOLE collection — the
+  result carries the row documents, so any member change changes an
+  emitted row.
+
+Matching is by table plus pointer prefix — cheap, sound, and
+**over-approximate in exactly one direction**: an unnecessary
+re-evaluation is a performance bug; a missed one would be a
+correctness bug, so the approximation always leans toward
+re-evaluating. Row inserts arrive with their full document in the
+patch; row UPDATES are minimal (§3), so maintenance issues a
+point-read of the touched row (one indexed lookup per touched key per
+record) to re-evaluate; deletes are answered entirely from maintained
+state. `externals` are fixed at registration — a query whose inputs
+change is a new registration.
+
+Maintenance runs synchronously inside patch delivery, in commit
+order, on the store's own connection. Writes from ANOTHER connection
+are invisible to capture (§6) and therefore to live queries; the
+coarse `dataVersion()` signal and the §11 topology are the honest
+answers, and re-registering re-reads.
+
+## 9. The emitted patch contract
+
+The result document is `{ "rows": [...] }` — always. Row and window
+strategies fill `rows` with result items; group strategies with one
+row per group; a whole-query aggregate is a ZERO-OR-ONE row result
+(`rows: [42]`; an empty `min()` is `rows: []`, the engine's
+undefined-as-absent mapping made visible). Consumers hold the result
+document and apply patches to it; ops are `add`, `remove` and
+`replace` only.
+
+- **One record, one emission.** A committed transaction touching any
+  number of rows produces at most ONE `{ patch, seq }` event per live
+  query — the record's changes coalesce into one patch array, and a
+  record that ends up changing nothing emits nothing.
+- **Structural sharing is the contract, not an optimisation.** After
+  an emission, every unaffected row in `live.result` is
+  REFERENCE-IDENTICAL to before; the `rows` array and the result
+  object are fresh per emission (a held previous result is never
+  mutated). Applying the emitted patch with `applyJSONPatch`'s
+  copy-on-write preserves the same sharing on the consumer's side —
+  which is what keeps the O(k) renderer's fast paths alive end to
+  end.
+- **Order.** An ordered (window) query's row order is the engine's,
+  with ties broken by the collection key — the key is appended to the
+  declared terms at registration, so the order is total and stable by
+  construction. An UNORDERED query's initial order is the engine's;
+  maintenance then appends newly matching rows and splices removed
+  ones, which is deterministic given the write history but is NOT
+  re-derived rowid order — a consumer that needs a specific order
+  declares an `orderBy`. Group rows keep first-appearance order.
+- `seq` is the capture record's `seq`; re-run emissions carry it too.
+
+## 10. The app binding
+
+A live query reaches an app as a SUBSCRIPTION (APP-FORMAT §5.3) whose
+handler dispatches a patch-carrying action — generated documents, no
+import in either direction (the `fsmToApp` precedent):
+
+```js
+import { liveAppBinding, createLiveSubscription } from '@jarenjs/db/app';
+
+const { subscription, actions } = liveAppBinding({
+  run: 'db/live', action: 'db/liveChanged', statePath: '/live/users' });
+// subscription → { run: 'db/live', with: { …, statePath, action } }
+// actions      → { 'db/liveChanged': { patch: '$payload' } }
+
+createApp({ …doc, subs: [subscription], actions: { …doc.actions, ...actions } },
+  { subs: { 'db/live': createLiveSubscription(store) } });
+```
+
+The handler registers the live query, dispatches ONE initializing
+patch (`replace` of the whole `statePath` slot with the initial
+result), then forwards each emission with every op's path prefixed by
+`statePath` — the prefixing happens in the handler, so the action
+document stays the two-line literal above and the app loop needs
+nothing new: `@jarenjs/app` already applies patches copy-on-write and
+derives changed paths, which is the payoff of one diff format end to
+end. Closing is the subscription's cleanup; a handler props change
+restarts it through the app's own key rule.
+
+## 11. Cross-tab: the owner topology
+
+Decided by a platform fact: OPFS synchronous access handles are
+**exclusive** — a second tab cannot open the same database files at
+all, so "one connection per tab" is not available and never will be.
+Therefore:
+
+- ONE owning context holds the sole connection — a `SharedWorker`
+  where available, else a leader tab elected via `navigator.locks` —
+  and every other tab is a client;
+- queries, writes and the patch stream travel between clients and the
+  owner over `BroadcastChannel` / `MessagePort`; a client's live query
+  is a remote registration whose emissions arrive as messages;
+- a second context attempting to OPEN the database is refused with
+  `JD2061` — a coded refusal, never a mysterious storage failure;
+- conflict handling stays out of scope: there is one writer by
+  construction.
+
+Node and Bun present the same API with no channel at all — the store
+is its own owner, and application code is identical everywhere. The
+in-browser proof of this topology (delivery across real tabs, the
+refusal, reload survival) belongs to the browser-driver order and its
+Playwright suite; this section is the decided contract it implements.
+
+## 12. Lifecycle, bounds, and non-goals
+
+A live query holds resources: dependency registrations, its
+maintained state, possibly a sorted window. `close()` releases all of
+them and is MANDATORY; closing the store closes every live query
+first; after close, `subscribe` and re-registration refuse, `result`
+stays readable (the last value), and a leak test asserts the live set
+after a forced GC.
+
+Bounds, both configurable at `openStore({ live: { … } })`, both
+ERRORING rather than degrading (the D14 rule — the bound is printed):
+
+- `maxQueries` (default 64): registrations beyond it are `JD0052`;
+- `maxMaintained` (default 10 000): the per-query ceiling on
+  maintained ENTRIES — result rows, window rows, accumulator and
+  per-group contributions all count, because the state is the cost.
+  Crossing it mid-maintenance is `JD2060`: the live query delivers the
+  error to its subscribers and CLOSES — degraded silence is the one
+  outcome this format forbids. An unbounded live query over a growing
+  table is the classic memory leak of this category; the accumulator
+  strategies trade exactly one contribution entry per matching row for
+  delete-correctness, and a count over a table larger than the bound
+  is a conscious `maxMaintained` raise, not a silent one.
+
+Non-claims, in one place: no incremental joins (re-run is the declared
+strategy), no cross-connection invalidation (§6's `data_version` is
+the signal), no maintenance over asynchronous connections in this
+version (every current driver is synchronous; the browser driver's
+order owns that story), no replication, and no ordering guarantee for
+unordered queries beyond §9's determinism.

package/docs/MIGRATION-FORMAT.md

@@ -0,0 +1,302 @@
+# The Jaren migration format (`jaren-migration`)
+
+This document is normative. The key words MUST, MUST NOT, SHOULD and
+MAY are to be interpreted as described in RFC 2119.
+
+Canonical schema: [`schemas/jaren-migration.schema.json`](../schemas/jaren-migration.schema.json)
+(draft 2020-12), with the mechanically derived draft-07 twin beside it.
+
+## 1. Scope
+
+A store evolves without hand-written SQL: two model documents diff into
+a **migration document** whose ordered steps are rendered DDL, JSLT
+data transforms and query assertions. The migration replays on a
+shadow database before the real store is touched; a history table
+records what ran, in what order, with a checksum; rolling forward is
+deterministic and inspectable, and the generated SQL is always shown
+before it is executed.
+
+This is the phase-A payoff for storing documents rather than rows: a
+shape change is a **transformation of values**, not a table rebuild.
+
+## 2. The migration document
+
+```json
+{
+  "$migration": "0.1",
+  "id": "0002-split-name",
+  "from": "1m60qna…", "to": "8kf22bd…",
+  "steps": [
+    { "kind": "ddl",   "sql": "DROP INDEX \"users_by_first\"" },
+    { "kind": "jslt",  "collection": "users", "stylesheet": [ { "match": "$", "body": { } } ] },
+    { "kind": "query", "collection": "users",
+      "assert": { "$for": { "it": "$[*]" }, "$where": { "$empty": "$it.name" }, "$return": "$it.id" } }
+  ]
+}
+```
+
+- `from`/`to` are `hashContent(canonicalizeJson(model))` — the
+  identity of a **shape**, not a version number a human must remember
+  to bump. A migration whose `from` does not match the database's
+  recorded shape MUST refuse to run (`JD0020`).
+- `kind: "ddl"` executes one rendered statement. The planner produces
+  these through the dialect; they are ordinary text in the document so
+  a reviewer reads exactly what will run.
+- `kind: "jslt"` rewrites every document of a collection through a
+  compiled JSLT stylesheet, in batches, inside the migration's
+  transaction. The empty stylesheet (`[]`) is the identity transform.
+  A step carrying `"draft": true` is a planner placeholder and MUST
+  refuse to run (`JD0021`) until the author fills it in.
+- `kind: "query"` is an assertion: the query runs over the
+  collection's documents and MUST answer an empty sequence (`expect:
+  "empty"`, the default) or an EBV-true value (`expect: "ebv"`) for
+  the migration to proceed. This is how a migration states its own
+  precondition — "no user has a null email before the NOT NULL
+  index" — and it is checked on the shadow first.
+- Steps are ordered, and the order is the contract.
+
+## 3. Planning and the widening/narrowing rule
+
+`planMigration(fromModel, toModel, { dialect, id })` produces
+`{ migration, report }` by diffing the two models' PHYSICAL plans:
+
+- An added collection becomes its full CREATE DDL; a removed
+  collection becomes a `DROP TABLE` step whose note says
+  **DESTRUCTIVE** plainly, and `report.destructive` is `true` — the
+  planner never guesses at data loss.
+- A rename cannot be inferred from a diff. It is DECLARED with
+  `"x-rename": "oldName"` on the target collection; the planner emits
+  the rename first and rebuilds the indexes (a renamed SQLite table
+  keeps its old index names — probed). Without the hint, a rename is
+  a drop plus a create and the report says so.
+- Added, removed and changed indexes become index DDL — reusing the
+  store's own DDL generator, never a second implementation. A changed
+  generated column (type or path) is a drop plus an add, with its
+  dependent indexes dropped first and recreated after.
+- A changed schema gets a `jslt` step with the identity stylesheet and
+  `"draft": true`. The planner **cannot** infer a data transform and
+  MUST NOT pretend to — a silent identity transform is how data gets
+  quietly lost. The author fills in the stylesheet, or deletes the
+  step when the change is a pure widening.
+- **The widening/narrowing rule runs against real data, not schema
+  comparison**: at the end of the migration run (inside its
+  transaction) every stored document is validated against the target
+  schema through the injected `compileSchema` hook. A document that no
+  longer validates is `JD0021` and the whole migration rolls back — a
+  narrowing without an adequate transform cannot land. A widening
+  needs no transform, and passes this check by fact.
+- Changing a collection's key declaration is not planned (a rebuild);
+  the planner refuses with a `TypeError` naming the non-goal.
+
+## 4. The shadow database
+
+Before the real store is touched, the WHOLE chain — the baseline
+shape, every applied migration, every pending migration — replays on a
+shadow database (`:memory:` by default; free with SQLite, no server).
+The shadow proves **structure**: every DDL statement runs, every
+stylesheet and assertion compiles and executes, and the end shape is
+verified against the target model. A failure there leaves the real
+store untouched.
+
+The shadow runs over an empty data set; the real-data facts (the
+widening check, key consistency, the assertions over real rows) run on
+the real store inside its transaction. The model format declares no
+UDF-expression indexes, so there is no function set to re-register on
+the shadow — stated here because a dialect that allowed such indexes
+would make the shadow fail on a schema the real store accepts.
+
+## 5. History and checksums
+
+```
+_jaren_migrations(id TEXT PRIMARY KEY, applied_at INTEGER,
+                  from_hash TEXT, to_hash TEXT, checksum TEXT, steps INTEGER)
+```
+
+`checksum` is `hashContent(canonicalizeJson(migration))` —
+signature-grade (D12), never the memo-grade `contentKey`. On every
+run, the supplied migration list MUST contain every applied migration,
+in order, with matching checksums; a migration whose recorded checksum
+differs from the document on disk is `JD0022` — someone edited an
+applied migration, which is always a bug and always worth failing on.
+The database's current shape is the last applied `to_hash`, or the
+hash of the `baseline` model when no migration has run.
+
+## 6. Running and batching
+
+`migrate({ driver, path }, migrations, options)`:
+
+- `options.baseline` (REQUIRED) — the model the store was FIRST
+  created with: the chain's anchor and the shadow's starting shape.
+- `options.model` (RECOMMENDED) — the target model. When present, the
+  last pending migration's `to` MUST equal its shape hash (`JD0020`
+  otherwise), the physical end shape is verified, and the real-data
+  validation of §3 runs.
+- `dryRun: true` prints every statement and the affected document
+  counts, validates the chain on the shadow, and writes NOTHING. The
+  API default is to run; a CLI SHOULD default to the dry run.
+- Each pending migration runs in ONE exclusive transaction
+  (`BEGIN IMMEDIATE` on SQLite — concurrent writers wait or time out
+  under the busy timeout) with a savepoint per step; any failure rolls
+  back the whole migration including its earlier steps. Where a driver
+  cannot open exclusively, the transaction still isolates; the busy
+  policy of MODEL-FORMAT §4 governs contention.
+- JSLT steps walk the collection in bounded batches
+  (`options.batchSize`, default 500) ordered by row identity, report
+  progress through `options.onProgress`, and never hold the whole
+  collection in memory. Assertion steps read the whole collection into
+  one array — a documented cost; keep assertions early, before the
+  data grows.
+- A transform MUST NOT change a caller-keyed document's key member —
+  the key column would go stale; the run refuses (`JD0023`).
+
+## 7. Non-goals
+
+- **Down migrations are not shipped in 0.1.** A JSLT transform is not
+  generally invertible, and a reverse step that silently loses data is
+  worse than a restore from backup. The recommended path: branch the
+  shape (a new collection or a new store), migrate forward, drop the
+  old collection once verified. `down` is not planned for this format
+  version; if it ever arrives it will be an explicit author-written
+  document, never an inferred inverse.
+- **Inferred renames.** A diff cannot distinguish a rename from a drop
+  plus a create; guessing risks silent data loss. Renames are declared
+  with `x-rename`, or they are what they look like.
+- **Key-declaration changes** (see §3) — a rebuild, not a migration
+  step.
+
+## 8. Error codes
+
+| code | raised when |
+|---|---|
+| `JD0020` | the migration's from-shape does not match the database |
+| `JD0021` | the migration is missing a required data transform |
+| `JD0022` | an applied migration disagrees with the history record |
+| `JD0023` | a migration step failed |
+
+These live in the same runtime `DB_CODES` table as the storage codes
+(MODEL-FORMAT §7); the union of both documents is proven in sync with
+the runtime table by a test.
+
+## 9. Relational changes (entities)
+
+`planModelMigration(fromModel, toModel, { dialect })` extends the §3
+planner to models with `entities`. The strategy table is the design;
+every row has a shadow-verified test that migrates seeded data:
+
+| Change | Strategy |
+|---|---|
+| add mapped column (property added, or moved out of the document) | `ALTER TABLE ADD COLUMN` — always nullable (absent reads back absent, MODEL-FORMAT §9.3) — plus a `sql` data step when the property's values already live in the document |
+| drop mapped column (property removed, or moved into the document) | fold the column back into the document first (`sql` step) when the property survives; drop its index, then `DROP COLUMN` where SQLite's conditions hold, else rebuild |
+| change type / enum CHECK / key / epoch flavor | **rebuild** (§10) |
+| add or drop an index (`unique`/`index`/version) | plain DDL |
+| add or drop a relation (foreign-key column, join table) | foreign keys **rebuild** the holder; join tables create/drop directly |
+| entity added / dropped | create / `DROP TABLE` (destructive, named) |
+| entity renamed | declared with `x-rename` on the target entity — never inferred; join tables renamed mechanically with their endpoints |
+| scalar ⇄ JSONB move (`column: "json"` toggled, shape change) | rebuild + a data step |
+
+Two rules keep the diff honest:
+
+- **Document changes are compared with `x-entity` stripped.** A pure
+  mapping change (an index added, a column toggle) is NOT a document
+  schema change and demands no transform; a real document change
+  yields the §3 draft-`jslt` step over the entity's table.
+- **Epoch columns populate in SQL** via
+  `(julianday(value) − 2440587.5) × 86 400 000`, rounded to the
+  millisecond — fractional seconds beyond that are the write
+  contract's business (MODEL-FORMAT §10.3), not the migration's.
+
+### 9.4 The `sql` step
+
+```json
+{ "kind": "sql", "sql": "UPDATE …", "note": "why" }
+```
+
+A data step spelled directly — for row-shuffling that is more honest
+as SQL than as a stylesheet. Runs like `ddl` (inside the migration's
+transaction, its own savepoint), but a dry run ALWAYS prints it with
+its note, and reviewers read intent from the kind.
+
+## 10. The rebuild procedure
+
+SQLite's `ALTER TABLE` cannot drop a constraint, change a type or
+reorder columns; the documented procedure for "making other kinds of
+table schema changes" (sqlite.org/lang_altertable.html §7) is followed
+literally, as one implementation used by every rebuilding strategy:
+
+```json
+{ "kind": "rebuild", "table": "User",
+  "create": ["CREATE TABLE \"User__rebuild\" (…)"],
+  "copy": "INSERT INTO \"User__rebuild\" (…) SELECT … FROM \"User\"",
+  "indexes": ["CREATE INDEX …"], "note": "…" }
+```
+
+The step is SELF-CONTAINED rendered SQL — reviewable in the migration
+document, mechanical to run: create the new shape under the temporary
+name, copy (the planner renders the column mapping: surviving columns
+verbatim, new ones from the document, dropped ones already folded),
+`DROP` the old table, `RENAME` the new one into place, recreate every
+index from the target model, then **`PRAGMA foreign_key_check` inside
+the transaction** — a broken reference fails the migration rather
+than shipping.
+
+Two deviations from the cited twelve steps, recorded: (1) the
+procedure brackets itself with `PRAGMA foreign_keys=OFF/ON`, which is
+a no-op inside a transaction — the migration connection never enables
+the pragma (SQLite's default is off; `openStore` enables AND verifies
+it per connection), so enforcement during the rebuild is off exactly
+as the procedure wants, and `foreign_key_check` provides the
+guarantee; (2) triggers and views are not re-created because this
+store creates none — a hand-added trigger is outside the model and
+outside the diff, which drift (§12) will name.
+
+**Shape equality is the acceptance criterion.** After a rebuild —
+after ANY relational migration — the database's declared schema
+(`schemaShapeOf`) must equal what a fresh `createModelShape(toModel)`
+produces, indexes, foreign keys and constraints included. The shadow
+asserts it before the real database is touched, and the real run
+asserts it again after the last migration.
+
+**UDF-expression indexes.** An index over a registered deterministic
+function is invisible to any connection that has not registered the
+function (probed): `migrate(…, { registerFunctions })` re-registers
+every declared function on the real, shadow AND reference connections
+before any DDL runs — without it, a rebuild would fail (or silently
+drop the index) on a schema the store accepts.
+
+## 11. The CLI
+
+`jaren-db` drives the workflow (mirroring `jaren-emit`):
+
+```
+jaren-db plan   --from <model> --to <model> [--store <db>] [--id x] [--out file]
+jaren-db status --model <model> --store <db> --baseline <model> [--migrations <dir>]
+jaren-db apply  --store <db> --baseline <model> --migrations <dir> [--model <m>] [--dry-run] [--yes]
+jaren-db check  --model <model> --store <db> --baseline <model> [--migrations <dir>]
+jaren-db shape  --model <model>
+```
+
+- `plan` diffs two model FILES (a database stores shape hashes, not
+  models — the from-model is the previous model file); with `--store`
+  it first verifies the from-model's hash matches the database's
+  recorded shape.
+- `check` is the CI command: exit 1 when migrations are pending OR the
+  database drifted; 0 in sync.
+- `apply` prints every statement before running; destructive steps
+  (drop table/column, rebuild) require `--yes` or an interactive
+  confirmation that NAMES what is lost. Default is dry-run + ask.
+- `status` lists applied/pending and reports drift (§12).
+- `shape` prints the physical mapping a model produces.
+
+## 12. Drift
+
+Drift is the database not matching what its history says it should
+be: someone changed it by hand. `status`/`check` detect it by
+verifying the current model's physical shape against the actual
+database (`schemaShapeOf` against `createModelShape`, when the chain
+is fully applied) — a hand-added index, a dropped column or a foreign
+key edited outside a migration is named early, which is the
+difference between a puzzled afternoon and a five-minute fix.
+
+Down migrations REMAIN a non-goal (§7's reasoning is unchanged): a
+down migration is a data-loss generator wearing a seatbelt; recovery
+is a backup restored plus the forward chain.