@jarenjs/db 0.49.2 → 0.56.0

package/docs/MIGRATION-FORMAT.md

@@ -45,18 +45,29 @@ shape change is a **transformation of values**, not a table rebuild.
 - `kind: "jslt"` rewrites every document of a collection through a
   compiled JSLT stylesheet, in batches, inside the migration's
   transaction. The empty stylesheet (`[]`) is the identity transform.
-  A step carrying `"draft": true` is a planner placeholder and MUST
-  refuse to run (`JD0021`) until the author fills it in.
+  Over an ENTITY table the stylesheet sees the whole row — the mapped
+  columns merged into the document under the TARGET model's mapping —
+  and what it returns is split back into columns and document by that
+  mapping; the key member is kept from the row (a stylesheet that
+  omits it loses nothing) and a stylesheet that changes it is
+  `JD0023`. A step carrying `"draft": true` is a planner placeholder
+  and MUST refuse to run (`JD0021`) until the author fills it in.
 - `kind: "query"` is an assertion: the query runs over the
   collection's documents and MUST answer an empty sequence (`expect:
   "empty"`, the default) or an EBV-true value (`expect: "ebv"`) for
   the migration to proceed. This is how a migration states its own
   precondition — "no user has a null email before the NOT NULL
-  index" — and it is checked on the shadow first.
+  index" — and it is checked on the shadow first. Over an entity table
+  the assertion reads the same merged rows a `jslt` step sees.
 - `kind: "derive"` recomputes named STORED derived index columns from
   the documents already in a collection — the backfill described in
   §2.1. It is idempotent: a derived value is a pure function of the
   document, so a replay writes what the first run wrote.
+- `kind: "sql"` executes one rendered DATA statement — a fold of a
+  column into the document, a backfill, an `INSERT … SELECT` — §9.4; a
+  dry run always prints it with its note.
+- `kind: "rebuild"` is the entity restructure of §10, self-contained:
+  the `CREATE` of the new shape, the copy and the index DDL.
 - Steps are ordered, and the order is the contract.
 
 ### 2.1 Derived spatial columns and the backfill
@@ -97,7 +108,11 @@ and the column entry carries the width the value is packed to:
 ## 3. Planning and the widening/narrowing rule
 
 `planMigration(fromModel, toModel, { dialect, id, derived })` produces
-`{ migration, report }` by diffing the two models' PHYSICAL plans:
+`{ migration, report }` by diffing the two models' PHYSICAL plans. The
+from-model is the previous model — the previous model FILE, or, under
+the CLI's snapshot discipline (§11), the committed `model.snapshot.json`
+the last `plan` advanced: a database stores shape hashes, never models,
+so the previous shape lives beside the code, where a diff can read it.
 
 - An added collection becomes its full CREATE DDL; a removed
   collection becomes a `DROP TABLE` step whose note says
@@ -107,7 +122,16 @@ and the column entry carries the width the value is packed to:
   `"x-rename": "oldName"` on the target collection; the planner emits
   the rename first and rebuilds the indexes (a renamed SQLite table
   keeps its old index names — probed). Without the hint, a rename is
-  a drop plus a create and the report says so.
+  a drop plus a create and the report says so. The hint is not part of
+  the shape: `x-rename` is stripped before the shape hash is computed,
+  so a model that keeps carrying a satisfied hint hashes the same as
+  one without it and plans nothing — a rename is idempotent across
+  `plan` runs. An entity rename carries its join tables with it, by
+  the endpoints the mapping records rather than by splitting the
+  table's name (an entity name may itself contain `_`); where the
+  rename flips the sorted endpoint order the join table is rebuilt
+  (create, `INSERT … SELECT`, drop) under its new name and no
+  membership is lost.
 - Added, removed and changed indexes become index DDL — reusing the
   store's own DDL generator, never a second implementation. A changed
   generated column (type or path) is a drop plus an add, with its
@@ -123,7 +147,10 @@ and the column entry carries the width the value is packed to:
   schema through the injected `compileSchema` hook. A document that no
   longer validates is `JD0021` and the whole migration rolls back — a
   narrowing without an adequate transform cannot land. A widening
-  needs no transform, and passes this check by fact.
+  needs no transform, and passes this check by fact. For an entity
+  the validated document is the whole row — columns merged back under
+  the target mapping — so a pure widening of a column-mapped member
+  passes and a narrowing of one is caught.
 - Changing a collection's key declaration is not planned (a rebuild);
   the planner refuses with a `TypeError` naming the non-goal.
 
@@ -139,10 +166,11 @@ store untouched.
 
 The shadow runs over an empty data set; the real-data facts (the
 widening check, key consistency, the assertions over real rows) run on
-the real store inside its transaction. The model format declares no
-UDF-expression indexes, so there is no function set to re-register on
-the shadow — stated here because a dialect that allowed such indexes
-would make the shadow fail on a schema the real store accepts.
+the real store inside its transaction. The shadow registers the same
+functions as the real run: `migrate(…, { registerFunctions })` runs on
+the shadow, the real and the reference connections before any DDL
+(§10), so a hand-created index over a registered deterministic
+function neither fails the shadow nor is silently dropped by it.
 
 ## 5. History and checksums
 
@@ -171,8 +199,16 @@ hash of the `baseline` model when no migration has run.
   otherwise), the physical end shape is verified, and the real-data
   validation of §3 runs.
 - `dryRun: true` prints every statement and the affected document
-  counts, validates the chain on the shadow, and writes NOTHING. The
-  API default is to run; a CLI SHOULD default to the dry run.
+  counts, validates the chain on the shadow, and writes NOTHING — not
+  even the history table: it PROBES for one and reads an absent one as
+  an empty history, so a dry run may be pointed at a production
+  database and leave its file byte-identical. The API default is to
+  run; a CLI SHOULD default to the dry run.
+- `migrationStatus` (and the CLI's `status`/`check`) create the empty
+  history table on a database that has none — the one write a reading
+  command makes, so a fresh file answers `applied: (none)` rather than
+  a missing-table error. This is the one place the two differ: a dry
+  run reports the same state and writes nothing at all.
 - Each pending migration runs in ONE exclusive transaction
   (`BEGIN IMMEDIATE` on SQLite — concurrent writers wait or time out
   under the busy timeout) with a savepoint per step; any failure rolls
@@ -225,13 +261,13 @@ every row has a shadow-verified test that migrates seeded data:
 | Change | Strategy |
 |---|---|
 | add mapped column (property added, or moved out of the document) | `ALTER TABLE ADD COLUMN` — always nullable (absent reads back absent, MODEL-FORMAT §9.3) — plus a `sql` data step when the property's values already live in the document |
-| drop mapped column (property removed, or moved into the document) | fold the column back into the document first (`sql` step) when the property survives; drop its index, then `DROP COLUMN` where SQLite's conditions hold, else rebuild |
+| drop mapped column (property removed, or moved into the document) | fold the column back into the document first (`sql` step) when the property survives — a `NULL` column folds to ABSENT, never to JSON `null`, so §9.3's rule survives the fold, in the rebuild copy too; drop its index, then `DROP COLUMN` where SQLite's conditions hold, else rebuild |
 | change type / enum CHECK / key / epoch flavor | **rebuild** (§10) |
 | add or drop an index (`unique`/`index`/version) | plain DDL |
-| add or drop a relation (foreign-key column, join table) | foreign keys **rebuild** the holder; join tables create/drop directly |
-| entity added / dropped | create / `DROP TABLE` (destructive, named) |
+| add or drop a relation (foreign-key column, join table) | foreign keys **rebuild** the holder — an inferred foreign-key column the target model no longer declares is folded into the document when the target declares the property, else named in `report.lost` and the plan is destructive; join tables create/drop directly |
+| entity added / dropped | create / `DROP TABLE` (destructive, named), children before parents so no foreign key dangles mid-migration |
 | entity renamed | declared with `x-rename` on the target entity — never inferred; join tables renamed mechanically with their endpoints |
-| scalar ⇄ JSONB move (`column: "json"` toggled, shape change) | rebuild + a data step |
+| scalar ⇄ JSONB move (`column: "json"` toggled) | the first two rows: `ADD COLUMN` plus a `sql` lift out of the document, or a `sql` fold plus `DROP COLUMN` — a rebuild only where SQLite cannot drop the column in place |
 
 Two rules keep the diff honest:
 
@@ -279,12 +315,13 @@ the transaction** — a broken reference fails the migration rather
 than shipping.
 
 Two deviations from the cited twelve steps, recorded: (1) the
-procedure brackets itself with `PRAGMA foreign_keys=OFF/ON`, which is
-a no-op inside a transaction — the migration connection never enables
-the pragma (SQLite's default is off; `openStore` enables AND verifies
-it per connection), so enforcement during the rebuild is off exactly
-as the procedure wants, and `foreign_key_check` provides the
-guarantee; (2) triggers and views are not re-created because this
+procedure's `PRAGMA foreign_keys=OFF/ON` bracket is honoured
+literally, OUTSIDE the transaction (inside one the pragma is a no-op):
+`node:sqlite` enables enforcement by default, and with it on a parent
+table could not even be dropped, so a migration holding a rebuild
+step turns enforcement off before `BEGIN IMMEDIATE` and back on after
+it settles, and `foreign_key_check` inside the transaction provides
+the guarantee the bracket suspended; (2) triggers and views are not re-created because this
 store creates none — a hand-added trigger is outside the model and
 outside the diff, which drift (§12) will name.
 
@@ -307,23 +344,68 @@ drop the index) on a schema the store accepts.
 `jaren-db` drives the workflow (mirroring `jaren-emit`):
 
 ```
-jaren-db plan   --from <model> --to <model> [--store <db>] [--id x] [--out file]
-jaren-db status --model <model> --store <db> --baseline <model> [--migrations <dir>]
-jaren-db apply  --store <db> --baseline <model> --migrations <dir> [--model <m>] [--dry-run] [--yes]
-jaren-db check  --model <model> --store <db> --baseline <model> [--migrations <dir>]
-jaren-db shape  --model <model>
+jaren-db plan     --from <model> --to <model> [--store <db>] [--id x] [--out file]
+jaren-db plan     --model <model> [--snapshot <file>] [--store <db>] [--id x] --out file
+jaren-db snapshot --model <model> [--snapshot <file>] [--types <file>]
+jaren-db status   --model <model> --store <db> --baseline <model> [--migrations <dir>] [--snapshot <file>]
+jaren-db apply    --store <db> --baseline <model> --migrations <dir> [--model <m>] [--dry-run] [--yes]
+jaren-db check    --model <model> --store <db> --baseline <model> [--migrations <dir>] [--snapshot <file>]
+jaren-db shape    --model <model>
 ```
 
+- **A model or a migration is a `.json` file or a MODULE.** `--model`,
+  `--from`, `--to` and `--baseline` accept a `.json` file or a module
+  (`.js`, `.mjs`, `.cjs` — and `.ts` where the host strips types: Node
+  ≥ 24 does by default, and `--no-strip-types` is refused by name)
+  loaded with `import()` and read as its `default` export or its `model`
+  export — the model pen's document, or any object whose `toJSON()`
+  emits one; `--migrations <dir>` reads `.json` files and modules
+  (`default` or `migration` — the migration pen's builder), sorted by
+  file name. A module whose export is not a document, or whose emission
+  is not JSON, fails with the module named. **Modules are pure:** the
+  CLI loads every module TWICE (two `import()`s under distinct
+  cache-busting queries) and refuses one whose two emissions differ —
+  no clock, no env, no randomness — because a migration that hashes
+  differently per load can never match its own history.
 - `plan` diffs two model FILES (a database stores shape hashes, not
-  models — the from-model is the previous model file); with `--store`
-  it first verifies the from-model's hash matches the database's
-  recorded shape.
-- `check` is the CI command: exit 1 when migrations are pending OR the
-  database drifted; 0 in sync.
-- `apply` prints every statement before running; destructive steps
-  (drop table/column, rebuild) require `--yes` or an interactive
-  confirmation that NAMES what is lost. Default is dry-run + ask.
-- `status` lists applied/pending and reports drift (§12).
+  models — the from-model is the previous model file), or, with
+  `--model`, the committed SNAPSHOT against the model: `--snapshot`
+  names it and defaults to `model.snapshot.json` beside the model; a
+  model whose shape equals the snapshot's plans nothing and exits 0;
+  otherwise the migration is written (`--out`) and the snapshot is
+  advanced to the model — without `--out` the plan is printed and the
+  snapshot stays, and the CLI says so. With `--store` it first compares
+  the from-model's physical shape with the database itself — never
+  with the history, which would refuse every database that has applied
+  a migration.
+- `snapshot` writes the model's snapshot (`--snapshot`, the same
+  default) — from the model the store was created with, before the
+  first `plan --model`; with `--types <file>` it also writes emit's
+  TypeScript declaration for the model (`entityEmitModel` rendered by
+  `@jarenjs/emit`, loaded lazily — `@jarenjs/db` does not depend on emit,
+  and a host without it is told exactly what `--types` needs), so a
+  transform over a JSON snapshot can be typed by annotation. Two runs on
+  one input write nothing the second time.
+- `check` is the CI command: exit 1 on an UNPLANNED MODEL CHANGE (a
+  snapshot in use whose shape is not the model's — the model moved and
+  nobody planned; named as such, never as the database's drift), when
+  migrations are pending, OR when the database drifted; 0 in sync.
+  `--model` is required — without it drift cannot be measured, and
+  `check` refuses rather than print `in sync`. `status` reports the same
+  verdict on its `model:` line.
+- `apply` prints every statement, then asks; destructive steps (drop
+  table/column, rebuild) print what is lost and ask for that
+  separately. `--yes` answers both, `--dry-run` stops after the
+  printout. Without an interactive terminal there is nobody to ask, so
+  `apply` without `--yes` exits 1 after the printout with nothing
+  applied — a CI job passes `--yes` deliberately, never by default.
+  `apply --dry-run` is the CLI's printout, not §6's `dryRun: true`: it
+  reads the history the way `status` does — creating the empty table on
+  a database that has none — and does NOT replay the chain on the
+  shadow, so a draft step still prints instead of refusing. The
+  shadow's verdict comes with the real `apply`.
+- `status` lists applied/pending and reports drift (§12); on a
+  database without a history table it creates the empty one (§6).
 - `shape` prints the physical mapping a model produces.
 
 ## 12. Drift