@jarenjs/db 0.49.2 → 0.66.1

package/docs/LIVE-FORMAT.md

@@ -92,7 +92,10 @@ them).
   already knows what it wrote. **Less complete, stated plainly**: it
   cannot see writes made through raw SQL, triggers, or another
   connection; a journal-mode delete of a row the store never read
-  emits its `remove` without having seen the old document; and of the
+  emits its `remove` without having seen the old document; a keyed
+  `put` reads the stored document first, so it emits the `replace`
+  session mode emits and a `put` that changes nothing emits nothing;
+  and of the
   database's own `ON DELETE` side effects it reconstructs exactly ONE
   — join-table membership dying with its entity (read before the
   delete) — while cascades into CHILD rows (`onDelete: 'cascade'` /
@@ -114,14 +117,86 @@ reads forward:
 const records = await store.changesSince(lastSeq);  // JD2051 when no log
 ```
 
-`seq` is monotonic; with the log enabled it continues across reopens
-(seeded from `MAX(seq)`), without it it is per-process. Retention is
+`seq` is monotonic; with the log enabled the DATABASE allocates it —
+one statement, inside the write's own transaction, advances a durable
+singleton row (`_jaren_changes_state`, the highest sequence this file
+ever allocated) and reads the new value back through `RETURNING`, and
+the record is then inserted under it — so two stores over one file
+never collide on the log's key, each sees the other's sequence continue,
+and a write that rolls back takes its allocation back with its row. The
+state row is engine metadata: it is created beside the log, seeded once
+from an existing file's surviving `MAX(seq)` (0 for a file that never
+held a row; a later open changes nothing), never lowered, never pruned,
+and never a capture, live, model or migration subject. Without the log,
+`seq` is per-process. `changesSince`
+answers records in the shape observers receive, `collections`
+included; a cursor that is not a number is a `TypeError`, as is a
+`retention` that is not a positive integer. Retention is
 a bounded count (`retention`, default 1000): older rows are pruned in
-the same transaction. The log is an ordered, replayable stream —
-which is what makes a late-joining consumer possible. **Replication
-is not built here**, and this log alone does not make it safe: there
-is no conflict resolution, no site identity, no causal ordering
-across writers. That sentence is the whole claim.
+the same transaction — every write deletes the records more than
+`retention` behind the one it just appended, and nothing else prunes
+the log. The log is an ordered, replayable stream — which is what
+makes a late-joining consumer possible.
+
+**`changesSince` is unbounded, and unsafe for a reconnecting
+consumer.** It answers every surviving record in one array, with no
+limit, no byte bound and no watermark: a consumer whose last `seq` fell
+below the retention floor receives the suffix that happens to survive
+and cannot distinguish "everything you missed" from "some of what you
+missed, and the rest is gone" — it believes itself caught up with a
+hole in its state. It stays as a published member, and it is not the
+supported path for a consumer that reconnects.
+
+**The bounded reader: `store.changes`.** Present exactly when the log
+is enabled (`JD2051` otherwise, as `changesSince`).
+
+- `changes.bounds()` answers the two watermarks: `earliestAvailable`,
+  the earliest surviving sequence (`MIN(seq)` over the log; `null` when
+  nothing survives), and `highWatermark`, the highest sequence the FILE
+  ever allocated, read from the durable state row — so a log that
+  retention emptied, reopened in a new process, still answers
+  `{ earliestAvailable: null, highWatermark: N }` and a cursor below `N`
+  meets `resetRequired` rather than a plausible empty history. Both are
+  the file's facts, never a process counter or a clock; cheap, and what a
+  consumer needs before it decides whether its cursor is usable.
+- `changes.page({ after, limit, maxBytes, signal })` answers
+  `{ items, next, earliestAvailable, highWatermark, hasMore,
+  resetRequired }`. `after` is the last sequence seen and is required:
+  there is no legitimate "give me everything" for a change log. The
+  page never holds more than `limit` records (default 100, applied as
+  SQL `LIMIT`) nor more than `maxBytes` serialised patch bytes,
+  accumulated at record boundaries; a single record larger than
+  `maxBytes` is the refusal `JD2074` without advancing `next` — the
+  same rule and the same implementation an entity page uses
+  (MODEL-FORMAT §10.5). `hasMore` is decided by one peek past the
+  page; `next` is the sequence to continue from (`after` itself when
+  nothing was delivered); `signal` cancels at a record boundary
+  (`JD2072`).
+- **`resetRequired: true`** when the record after `after` no longer
+  survives — `after + 1 < earliestAvailable` (or the log is empty
+  above `after`'s successor). Then `items` is **empty** and `next` is
+  **absent**: the refusal is total, because a partial suffix beside a
+  reset flag would invite a consumer to use both. The watermarks are
+  read after the rows, so a floor that rose during the read can only
+  make the verdict stricter, never let a pruned gap pass as a
+  continuation.
+
+**The consumer's recovery procedure**, in words: keep the last `seq`
+you applied; on reconnect, call `changes.page({ after: lastSeq })` and
+apply pages while `hasMore`, storing `next` as you go; when a page
+answers `resetRequired: true`, stop applying — your state has a hole —
+re-seed it from a full snapshot of the collections you follow, and
+resume paging from that page's `highWatermark`, because every record
+at or below it is already reflected in the snapshot you just took.
+Choosing how much history to keep is the host's decision
+(`retention`); what the reader owes is that when rows go, it reports
+the gap instead of hiding it.
+
+**Replication is not built here**, and this log alone does not make
+it safe: there is no conflict resolution, no site identity, no causal
+ordering across writers. The bounded reader with its watermarks and
+its explicit gap is the precondition a replication protocol would be
+built on — not the protocol. That sentence is the whole claim.
 
 ## 6. Cross-connection behaviour and non-claims
 
@@ -159,7 +234,25 @@ live.close();
 `store.live(document, options)` registers an entity-root document (the
 multi-entity shape of MODEL-FORMAT §10) the same way. Live queries
 REQUIRE change capture — the patch stream is the invalidation source —
-and registering on a store opened without `capture` is `JD0050`.
+and registering on a store opened without `capture` is `JD0050`. Both
+registrations run under the store gate through their **initial query**
+(MODEL-FORMAT §5.1): a registration made while another transaction is
+open waits for it to settle and initializes from committed rows only, a
+rolled-back row never reaches `result`, and a refused registration
+leaves `stats().liveQueries` unchanged. A registration made from INSIDE
+a transaction view initializes from that transaction's rows and shares
+its fate — kept and maintained on commit, closed on rollback. Once
+registered, the handle is maintained by committed writes alone and takes
+no gate of its own. `changes.page()` takes the same `deadline` every
+other page does (`JD2075` at a record boundary).
+
+A producer may hand a registration a CHAIN instead of a document: the
+`@jarenjs/linq/db` client's `live(chain, options)` passes the chain's
+`toDocument()` and its `explain().bindings` as the externals to exactly
+these two registrations (`store.live` for an entity-root chain,
+`collection.live` for a collection's), so the strategy, the reason and
+the maintenance are this table's — an entity chain re-runs, declared —
+and this document stays the only place they are decided.
 
 **This table is normative.** Every row is implemented and tested;
 nothing outside it is attempted. Classification reads the compiled
@@ -260,7 +353,11 @@ state. `externals` are fixed at registration — a query whose inputs
 change is a new registration.
 
 Maintenance runs synchronously inside patch delivery, in commit
-order, on the store's own connection. Writes from ANOTHER connection
+order, on the store's own connection. Delivery is never re-entered: a
+write made from inside an observer or a subscriber commits at once,
+but its record is queued and delivered after the current record has
+reached every consumer, so sibling live views see commits in commit
+order rather than in call-stack order. Writes from ANOTHER connection
 are invisible to capture (§6) and therefore to live queries; the
 coarse `dataVersion()` signal and the §11 topology are the honest
 answers, and re-registering re-reads.
@@ -332,8 +429,11 @@ Decided by a platform fact: OPFS synchronous access handles are
 all, so "one connection per tab" is not available and never will be.
 Therefore:
 
-- ONE owning context holds the sole connection — a `SharedWorker`
-  where available, else a leader tab elected via `navigator.locks` —
+- ONE owning context holds the sole connection — the first tab's
+  dedicated worker to install the OPFS access-handle pool owns it (the
+  pool is exclusive by construction) and holds a `navigator.locks` lock
+  for its lifetime so a later tab can tell a busy owner from no owner
+  (a `BroadcastChannel` ping is the fallback where locks are absent) —
   and every other tab is a client;
 - queries, writes and the patch stream travel between clients and the
   owner over `BroadcastChannel` / `MessagePort`; a client's live query
@@ -346,8 +446,37 @@ Therefore:
 Node and Bun present the same API with no channel at all — the store
 is its own owner, and application code is identical everywhere. The
 in-browser proof of this topology (delivery across real tabs, the
-refusal, reload survival) belongs to the browser-driver order and its
-Playwright suite; this section is the decided contract it implements.
+refusal, reload survival) is the website's Playwright suite over the
+`#/data` studio; this section is the decided contract it implements.
+
+The wasm session adapter performs an actual disposable create/attach/changeset/delete
+probe. It declares sessions only after success; `sessionReason` explains journal
+fallback. Changeset bytes are detached from wasm-owned memory before transfer, and
+capture cleanup deletes every session even on rollback or connection close.
+
+The studio probes isolated SharedArrayBuffer OPFS, header-free SAH-pool OPFS,
+atomic IndexedDB snapshots, then visibly non-durable memory. IndexedDB snapshots
+require exclusive ownership and acknowledge writes after atomic version replacement.
+They expose no synchronous/live surface; the Store pane explicitly refreshes after
+writes. Failed snapshot persistence invalidates the connection without publishing
+partial state. [Execution hosts](HOSTS.md) specifies bounds and the observed matrix.
+
+**The browser boot is a closed protocol.** Reaching an owner, a client
+or a standalone memory store passes through five named stages —
+`worker-start`, `sqlite-init`, `vfs-acquire`, `topology`,
+`store-open` — and every attempt ends in exactly one of two states:
+ready, or a stable failure record `{ code: 'DATA_BOOT', stage, message }`
+naming the stage that failed. Each stage carries its own budget, so a
+stage that never settles fails under its own name rather than under an
+outer deadline that cannot say which resource to release; an OPFS pool
+that is absent advances to the next persistence probe; an existing owner
+produces the `client` answer above, while a pool install that hangs is a `vfs-acquire` failure
+and never masquerades as absence. A failed attempt releases everything it
+created — worker, port client, channel, listeners, timers — before the
+page hears of it, so a retry (or a reload) starts clean, and the page
+shows the stage and offers the retry. The stage runner is
+`packages/website/src/lib/boot-stages.js`; the site's transport and its
+owner worker are the two halves that run it.
 
 ## 12. Lifecycle, bounds, and non-goals
 
@@ -375,9 +504,11 @@ ERRORING rather than degrading (the D14 rule — the bound is printed):
 
 Non-claims, in one place: no incremental joins (re-run is the declared
 strategy), no cross-connection invalidation (§6's `data_version` is
-the signal), no maintenance over asynchronous connections in this
-version (every current driver is synchronous; the browser driver's
-order owns that story), no replication, and no ordering guarantee for
+the signal), no maintenance over asynchronous connections —
+`capabilities.live` is `false` there and a registration is `JD0051`
+naming the reason, because maintenance point-reads rows synchronously
+inside delivery (the wasm driver's oo1 API is synchronous, which is
+why the browser has live queries at all) — no replication, and no ordering guarantee for
 unordered queries beyond §9's determinism.
 
 ## 13. Event time
@@ -404,7 +535,8 @@ live.stats().watermark;          // what it is now
 `eventTime` is a **closed** member set: `path`, `watermark`,
 `allowedLateness` (default 0) and `retention`. Anything else — a
 misspelling, a non-finite epoch, a negative lateness, a `path` that is
-not a singular row selector — is `JD0053` at registration, not a member
+not a singular row selector — is `JD0053` at registration (its
+`docPath` names the collection, `/collections/<name>`), not a member
 quietly ignored. `advance()` refuses a value that is not finite or that
 goes backwards (a `TypeError`), and it is absent on every view
 registered without an `eventTime`. An entity document has no collection
@@ -415,7 +547,12 @@ to place rows in and re-runs, so an `eventTime` on `store.live` is
 
 Two documents, and only these two shapes: `$resample` and `$rolling`
 whose series operand is the collection (`"$[*]"`, or a FLWOR over it
-whose `$where` narrows and whose `$return` is the bare binding).
+whose `$where` narrows and whose `$return` is the bare binding). A
+spec that spells `at` and `value` explicitly — the spelling the query
+language accepts — is maintained: the view folds with the kernel
+reading the declared instant member and the `value` member the spec
+names, and a `value` selector the view cannot follow re-runs with the
+reason named, never a maintained view that dies on its first fold.
 
 - **A bucket view keeps its rows by bucket.** A write touches one bucket
   — two, when it moves a reading across a boundary — and exactly those

package/docs/MIGRATION-FORMAT.md

@@ -45,18 +45,29 @@ shape change is a **transformation of values**, not a table rebuild.
 - `kind: "jslt"` rewrites every document of a collection through a
   compiled JSLT stylesheet, in batches, inside the migration's
   transaction. The empty stylesheet (`[]`) is the identity transform.
-  A step carrying `"draft": true` is a planner placeholder and MUST
-  refuse to run (`JD0021`) until the author fills it in.
+  Over an ENTITY table the stylesheet sees the whole row — the mapped
+  columns merged into the document under the TARGET model's mapping —
+  and what it returns is split back into columns and document by that
+  mapping; the key member is kept from the row (a stylesheet that
+  omits it loses nothing) and a stylesheet that changes it is
+  `JD0023`. A step carrying `"draft": true` is a planner placeholder
+  and MUST refuse to run (`JD0021`) until the author fills it in.
 - `kind: "query"` is an assertion: the query runs over the
   collection's documents and MUST answer an empty sequence (`expect:
   "empty"`, the default) or an EBV-true value (`expect: "ebv"`) for
   the migration to proceed. This is how a migration states its own
   precondition — "no user has a null email before the NOT NULL
-  index" — and it is checked on the shadow first.
+  index" — and it is checked on the shadow first. Over an entity table
+  the assertion reads the same merged rows a `jslt` step sees.
 - `kind: "derive"` recomputes named STORED derived index columns from
   the documents already in a collection — the backfill described in
   §2.1. It is idempotent: a derived value is a pure function of the
   document, so a replay writes what the first run wrote.
+- `kind: "sql"` executes one rendered DATA statement — a fold of a
+  column into the document, a backfill, an `INSERT … SELECT` — §9.4; a
+  dry run always prints it with its note.
+- `kind: "rebuild"` is the entity restructure of §10, self-contained:
+  the `CREATE` of the new shape, the copy and the index DDL.
 - Steps are ordered, and the order is the contract.
 
 ### 2.1 Derived spatial columns and the backfill
@@ -97,7 +108,11 @@ and the column entry carries the width the value is packed to:
 ## 3. Planning and the widening/narrowing rule
 
 `planMigration(fromModel, toModel, { dialect, id, derived })` produces
-`{ migration, report }` by diffing the two models' PHYSICAL plans:
+`{ migration, report }` by diffing the two models' PHYSICAL plans. The
+from-model is the previous model — the previous model FILE, or, under
+the CLI's snapshot discipline (§11), the committed `model.snapshot.json`
+the last `plan` advanced: a database stores shape hashes, never models,
+so the previous shape lives beside the code, where a diff can read it.
 
 - An added collection becomes its full CREATE DDL; a removed
   collection becomes a `DROP TABLE` step whose note says
@@ -107,7 +122,16 @@ and the column entry carries the width the value is packed to:
   `"x-rename": "oldName"` on the target collection; the planner emits
   the rename first and rebuilds the indexes (a renamed SQLite table
   keeps its old index names — probed). Without the hint, a rename is
-  a drop plus a create and the report says so.
+  a drop plus a create and the report says so. The hint is not part of
+  the shape: `x-rename` is stripped before the shape hash is computed,
+  so a model that keeps carrying a satisfied hint hashes the same as
+  one without it and plans nothing — a rename is idempotent across
+  `plan` runs. An entity rename carries its join tables with it, by
+  the endpoints the mapping records rather than by splitting the
+  table's name (an entity name may itself contain `_`); where the
+  rename flips the sorted endpoint order the join table is rebuilt
+  (create, `INSERT … SELECT`, drop) under its new name and no
+  membership is lost.
 - Added, removed and changed indexes become index DDL — reusing the
   store's own DDL generator, never a second implementation. A changed
   generated column (type or path) is a drop plus an add, with its
@@ -123,7 +147,10 @@ and the column entry carries the width the value is packed to:
   schema through the injected `compileSchema` hook. A document that no
   longer validates is `JD0021` and the whole migration rolls back — a
   narrowing without an adequate transform cannot land. A widening
-  needs no transform, and passes this check by fact.
+  needs no transform, and passes this check by fact. For an entity
+  the validated document is the whole row — columns merged back under
+  the target mapping — so a pure widening of a column-mapped member
+  passes and a narrowing of one is caught.
 - Changing a collection's key declaration is not planned (a rebuild);
   the planner refuses with a `TypeError` naming the non-goal.
 
@@ -139,10 +166,15 @@ store untouched.
 
 The shadow runs over an empty data set; the real-data facts (the
 widening check, key consistency, the assertions over real rows) run on
-the real store inside its transaction. The model format declares no
-UDF-expression indexes, so there is no function set to re-register on
-the shadow — stated here because a dialect that allowed such indexes
-would make the shadow fail on a schema the real store accepts.
+the real store inside its transaction. Batched transforms and validation
+include every row identity, including negative integer primary keys;
+target-schema validation visits every declared entity, even after an
+empty entity table.
+The shadow registers the same functions as the real run:
+`migrate(…, { registerFunctions })` runs on
+the shadow, the real and the reference connections before any DDL
+(§10), so a hand-created index over a registered deterministic
+function neither fails the shadow nor is silently dropped by it.
 
 ## 5. History and checksums
 
@@ -171,23 +203,120 @@ hash of the `baseline` model when no migration has run.
   otherwise), the physical end shape is verified, and the real-data
   validation of §3 runs.
 - `dryRun: true` prints every statement and the affected document
-  counts, validates the chain on the shadow, and writes NOTHING. The
-  API default is to run; a CLI SHOULD default to the dry run.
+  counts, validates the chain on the shadow, and writes NOTHING — not
+  even the history table: it PROBES for one and reads an absent one as
+  an empty history, so a dry run may be pointed at a production
+  database and leave its file byte-identical. The API default is to
+  run; a CLI SHOULD default to the dry run.
+- `migrationStatus` (and the CLI's `status`/`check`) write nothing
+  either: the history table is probed, never created, and an absent one
+  reads as an empty history, so a fresh file answers `applied: (none)`
+  and stays byte for byte what it was. The history table is created by
+  the real run alone, before its first migration is recorded.
 - Each pending migration runs in ONE exclusive transaction
   (`BEGIN IMMEDIATE` on SQLite — concurrent writers wait or time out
   under the busy timeout) with a savepoint per step; any failure rolls
   back the whole migration including its earlier steps. Where a driver
   cannot open exclusively, the transaction still isolates; the busy
   policy of MODEL-FORMAT §4 governs contention.
+- A run is cancellable: `migrate(target, migrations, { signal,
+  deadline })` checks both BETWEEN migrations, between steps and
+  between the batches of a data step — never inside a statement, which
+  runs to its end — with the deadline read against `options.runtime`'s
+  clock. An abort is `JD2080` and a passed deadline `JD2075`; the
+  migration in flight rolls back whole (its savepoints, its history
+  row), the migrations already committed stand, and a rerun resumes
+  from the recorded position. The shadow replay is cancellable at the
+  same boundaries. `migrationStatus` refuses a call already cancelled
+  or past its deadline before it opens anything.
+- **An assertion is classified before it runs, and the classification
+  decides what it costs.** One classifier answers for every host — a
+  Store, an array, a file — so they cannot disagree about the price:
+
+  | strategy | which assertions | what it costs |
+  |---|---|---|
+  | per-document | a FLWOR over `$[*]` whose `$where`/`$return` read only the binding | one keyset batch at a time; fails fast at the first batch that violates |
+  | fold | exactly one of `$count`, `$sum`, `$min`, `$max` over the root | one batch at a time; each batch is answered by the ENGINE and the partial answers combine |
+  | materialize | everything else (`$let`, `$distinct`, a nested `$for`, two aggregates) | every document at once, under `assertionBounds` |
+
+  A fold is sound because the operator is associative: the answer over a
+  collection is the combination of the answers over any partition of it.
+  Nothing reimplements an operator — each batch is evaluated by the same
+  compiled query the whole-collection path would use, and only the
+  COMBINE step is written here, so null handling, empty-sequence answers
+  and type coercions are the engine's. A suite runs every fold shape both
+  ways, over ten corpora and six partitions, and requires the value and
+  the verdict to be indistinguishable; a shape that cannot pass it is not
+  in the set.
+
+  **A materializing assertion is bounded.** `options.assertionBounds`
+  defaults to `{ maxRows: 100000, maxBytes: 67108864 }` and is crossed
+  BEFORE the excess is held — the walk stops at the row that would break
+  it, refusing `JD2007` (rows) or `JD2076` (bytes) and naming the two
+  assertion shapes that are answered in batches instead. `null` on either
+  member removes that bound, which a caller must ask for: an unbounded
+  read nobody declared is exactly what this classification removes. This
+  is a deliberate behavior change — a migration that used to read a very
+  large collection whole now refuses until its bound is raised or its
+  assertion is rewritten.
 - JSLT steps walk the collection in bounded batches
   (`options.batchSize`, default 500) ordered by row identity, report
-  progress through `options.onProgress`, and never hold the whole
-  collection in memory. Assertion steps read the whole collection into
-  one array — a documented cost; keep assertions early, before the
-  data grows.
+  progress through `options.onProgress` (`{ migration, collection,
+  transformed | derived | asserted }`, one event per batch), and never
+  hold the whole collection in memory. A PER-DOCUMENT assertion — a
+  FLWOR over `$[*]` whose `$where` and `$return` read only the binding
+  — walks the same batches and fails fast at the first batch that
+  violates, because its answer over each batch is its answer over the
+  whole. A cross-document assertion (one that reads the root: `$count:
+  '$[*]'`, a `$let`, a `$distinct`, a nested `$for`) reads the whole
+  collection into one array — a stated cost; keep such assertions
+  early, before the data grows. A cross-document assertion that is one
+  associative aggregate no longer costs that read at all — see the
+  classification table above.
 - A transform MUST NOT change a caller-keyed document's key member —
   the key column would go stale; the run refuses (`JD0023`).
 
+### 6.1 Running without a database
+
+A migration's `jslt` and `query` steps act on DOCUMENTS, so they do not
+need tables. Two surfaces run them against documents a caller already
+holds, sharing one implementation of what a step means with the Store —
+the same transform rule, the same key rule, the same classification of
+an assertion, the same refusals in the same words.
+
+- `migrateDocuments({ collection: [...] }, migrations, options)` answers
+  `{ documents, report }`. The source is REWINDABLE, so every step runs
+  over the whole collection before the next begins, exactly as a Store
+  runs it. This is what makes its answer — and its refusal, on the same
+  step — identical to the Store's for the same documents.
+- `streamDocuments({ collection: iterable }, migrations, { write })`
+  walks a source that can be read only once, writing each document out
+  as it finishes. The input is consumed exactly once and nothing beyond
+  one batch is held, so a collection larger than memory still migrates.
+
+Both refuse, BEFORE asking for the first document, any step this host
+cannot honour (`JD0023`):
+
+| Step kind | Without a database |
+|---|---|
+| `jslt`, `query` | runs |
+| `ddl`, `sql`, `rebuild`, `derive` | refused by name — no tables to change |
+
+A step naming a collection the caller did not supply is refused the same
+way. Nothing is half-applied: a runner without a transaction cannot take
+a partial write back, so the whole refusal happens before the first read.
+
+Two limits are the single pass's, and are stated rather than hidden:
+
+- A CROSS-DOCUMENT assertion needs every document at once, which one
+  pass does not hold. `streamDocuments` refuses it by name; run that
+  collection through `migrateDocuments`, whose source it can re-read.
+- When two different steps would each refuse, `migrateDocuments` and the
+  Store name the EARLIER step, because each step finishes before the
+  next begins. `streamDocuments` carries a batch through every step, so
+  it can name the later one. Both refuse, with the same code and the
+  same words; only which step is named can differ.
+
 ## 7. Non-goals
 
 - **Down migrations are not shipped in 0.1.** A JSLT transform is not
@@ -211,6 +340,7 @@ hash of the `baseline` model when no migration has run.
 | `JD0021` | the migration is missing a required data transform |
 | `JD0022` | an applied migration disagrees with the history record |
 | `JD0023` | a migration step failed |
+| `JD0024` | a document source or target could not be read or written |
 
 These live in the same runtime `DB_CODES` table as the storage codes
 (MODEL-FORMAT §7); the union of both documents is proven in sync with
@@ -225,13 +355,13 @@ every row has a shadow-verified test that migrates seeded data:
 | Change | Strategy |
 |---|---|
 | add mapped column (property added, or moved out of the document) | `ALTER TABLE ADD COLUMN` — always nullable (absent reads back absent, MODEL-FORMAT §9.3) — plus a `sql` data step when the property's values already live in the document |
-| drop mapped column (property removed, or moved into the document) | fold the column back into the document first (`sql` step) when the property survives; drop its index, then `DROP COLUMN` where SQLite's conditions hold, else rebuild |
+| drop mapped column (property removed, or moved into the document) | fold the column back into the document first (`sql` step) when the property survives — a `NULL` column folds to ABSENT, never to JSON `null`, so §9.3's rule survives the fold, in the rebuild copy too; drop its index, then `DROP COLUMN` where SQLite's conditions hold, else rebuild |
 | change type / enum CHECK / key / epoch flavor | **rebuild** (§10) |
 | add or drop an index (`unique`/`index`/version) | plain DDL |
-| add or drop a relation (foreign-key column, join table) | foreign keys **rebuild** the holder; join tables create/drop directly |
-| entity added / dropped | create / `DROP TABLE` (destructive, named) |
+| add or drop a relation (foreign-key column, join table) | foreign keys **rebuild** the holder — an inferred foreign-key column the target model no longer declares is folded into the document when the target declares the property, else named in `report.lost` and the plan is destructive; join tables create/drop directly |
+| entity added / dropped | create / `DROP TABLE` (destructive, named), children before parents so no foreign key dangles mid-migration |
 | entity renamed | declared with `x-rename` on the target entity — never inferred; join tables renamed mechanically with their endpoints |
-| scalar ⇄ JSONB move (`column: "json"` toggled, shape change) | rebuild + a data step |
+| scalar ⇄ JSONB move (`column: "json"` toggled) | the first two rows: `ADD COLUMN` plus a `sql` lift out of the document, or a `sql` fold plus `DROP COLUMN` — a rebuild only where SQLite cannot drop the column in place |
 
 Two rules keep the diff honest:
 
@@ -279,12 +409,13 @@ the transaction** — a broken reference fails the migration rather
 than shipping.
 
 Two deviations from the cited twelve steps, recorded: (1) the
-procedure brackets itself with `PRAGMA foreign_keys=OFF/ON`, which is
-a no-op inside a transaction — the migration connection never enables
-the pragma (SQLite's default is off; `openStore` enables AND verifies
-it per connection), so enforcement during the rebuild is off exactly
-as the procedure wants, and `foreign_key_check` provides the
-guarantee; (2) triggers and views are not re-created because this
+procedure's `PRAGMA foreign_keys=OFF/ON` bracket is honoured
+literally, OUTSIDE the transaction (inside one the pragma is a no-op):
+`node:sqlite` enables enforcement by default, and with it on a parent
+table could not even be dropped, so a migration holding a rebuild
+step turns enforcement off before `BEGIN IMMEDIATE` and back on after
+it settles, and `foreign_key_check` inside the transaction provides
+the guarantee the bracket suspended; (2) triggers and views are not re-created because this
 store creates none — a hand-added trigger is outside the model and
 outside the diff, which drift (§12) will name.
 
@@ -307,24 +438,100 @@ drop the index) on a schema the store accepts.
 `jaren-db` drives the workflow (mirroring `jaren-emit`):
 
 ```
-jaren-db plan   --from <model> --to <model> [--store <db>] [--id x] [--out file]
-jaren-db status --model <model> --store <db> --baseline <model> [--migrations <dir>]
-jaren-db apply  --store <db> --baseline <model> --migrations <dir> [--model <m>] [--dry-run] [--yes]
-jaren-db check  --model <model> --store <db> --baseline <model> [--migrations <dir>]
-jaren-db shape  --model <model>
+jaren-db plan     --from <model> --to <model> [--store <db>] [--id x] [--out file]
+jaren-db plan     --model <model> [--snapshot <file>] [--store <db>] [--id x] --out file
+jaren-db snapshot --model <model> [--snapshot <file>] [--types <file>]
+jaren-db status   --model <model> --store <db> --baseline <model> [--migrations <dir>] [--snapshot <file>]
+jaren-db apply    --store <db> --baseline <model> --migrations <dir> [--model <m>] [--dry-run] [--yes]
+jaren-db check    --model <model> --store <db> --baseline <model> [--migrations <dir>] [--snapshot <file>]
+jaren-db shape    --model <model>
+jaren-db documents --migrations <dir> --in <file|-> (--out <file|-> | --in-place --yes | --check)
+                   [--format json|jsonl] [--out-format json|jsonl] [--collection <name>]
+                   [--batch-size <n>]
 ```
 
+- **A model or a migration is a `.json` file or a MODULE.** `--model`,
+  `--from`, `--to` and `--baseline` accept a `.json` file or a module
+  (`.js`, `.mjs`, `.cjs` — and `.ts` where the host strips types: Node
+  ≥ 24 does by default, and `--no-strip-types` is refused by name)
+  loaded through `@jarenjs/json/node` — the suite's one document loader,
+  shared with `jaren-contract` — and read as its `default` export or its `model`
+  export — the model pen's document, or any object whose `toJSON()`
+  emits one; `--migrations <dir>` reads `.json` files and modules
+  (`default` or `migration` — the migration pen's builder), sorted by
+  file name. A module whose export is not a document, or whose emission
+  is not JSON, fails with the module named. **Modules are pure:** the
+  CLI loads every module TWICE (two `import()`s under distinct
+  cache-busting queries) and refuses one whose two emissions differ —
+  no clock, no env, no randomness — because a migration that hashes
+  differently per load can never match its own history.
 - `plan` diffs two model FILES (a database stores shape hashes, not
-  models — the from-model is the previous model file); with `--store`
-  it first verifies the from-model's hash matches the database's
-  recorded shape.
-- `check` is the CI command: exit 1 when migrations are pending OR the
-  database drifted; 0 in sync.
-- `apply` prints every statement before running; destructive steps
-  (drop table/column, rebuild) require `--yes` or an interactive
-  confirmation that NAMES what is lost. Default is dry-run + ask.
-- `status` lists applied/pending and reports drift (§12).
+  models — the from-model is the previous model file), or, with
+  `--model`, the committed SNAPSHOT against the model: `--snapshot`
+  names it and defaults to `model.snapshot.json` beside the model; a
+  model whose shape equals the snapshot's plans nothing and exits 0;
+  otherwise the migration is written (`--out`) and the snapshot is
+  advanced to the model — without `--out` the plan is printed and the
+  snapshot stays, and the CLI says so. With `--store` it first compares
+  the from-model's physical shape with the database itself — never
+  with the history, which would refuse every database that has applied
+  a migration.
+- `snapshot` writes the model's snapshot (`--snapshot`, the same
+  default) — from the model the store was created with, before the
+  first `plan --model`; with `--types <file>` it also writes emit's
+  TypeScript declaration for the model (`entityEmitModel` rendered by
+  `@jarenjs/emit`, loaded lazily — `@jarenjs/db` does not depend on emit,
+  and a host without it is told exactly what `--types` needs), so a
+  transform over a JSON snapshot can be typed by annotation. Two runs on
+  one input write nothing the second time.
+- `check` is the CI command: exit 1 on an UNPLANNED MODEL CHANGE (a
+  snapshot in use whose shape is not the model's — the model moved and
+  nobody planned; named as such, never as the database's drift), when
+  migrations are pending, OR when the database drifted; 0 in sync.
+  `--model` is required — without it drift cannot be measured, and
+  `check` refuses rather than print `in sync`. `status` reports the same
+  verdict on its `model:` line.
+- `apply` prints every statement, then asks; destructive steps (drop
+  table/column, rebuild) print what is lost and ask for that
+  separately. `--yes` answers both, `--dry-run` stops after the
+  printout. Without an interactive terminal there is nobody to ask, so
+  `apply` without `--yes` exits 1 after the printout with nothing
+  applied — a CI job passes `--yes` deliberately, never by default.
+  `apply --dry-run` is the CLI's printout, not §6's `dryRun: true`: it
+  reads the history the way `status` does — probed, never created — and
+  does NOT replay the chain on the shadow, so a draft step still prints
+  instead of refusing. The shadow's verdict comes with the real `apply`.
+- `status` lists applied/pending and reports drift (§12); on a
+  database without a history table it creates nothing (§6).
 - `shape` prints the physical mapping a model produces.
+- `documents` runs a migration's DOCUMENT steps over a file instead of a
+  database — §6.1's runners, given a path or stdio. `--in`/`--out` take
+  a file or `-`; the encoding follows the extension (`.jsonl`/`.ndjson`
+  line-delimited, everything else one JSON array) unless `--format` /
+  `--out-format` says otherwise, and stdio defaults to JSONL. Input and
+  output encodings are independent, so this is also the converter.
+  - **A file holds ONE collection.** The migrations name it; a chain
+    whose document steps touch more than one cannot be applied to a
+    file, and is refused rather than partly run. `--collection` asserts
+    which collection the file holds and refuses a mismatch.
+  - **`--out` writes elsewhere; `--in-place` replaces the input and
+    needs `--yes`.** Either way the documents land in a sibling
+    temporary that is renamed over the target only once every document
+    has survived every step. A failure — a step, a malformed source, a
+    cancelled run — removes the temporary and leaves the target byte for
+    byte as it was.
+  - **`--check` transforms and validates everything and writes nothing**,
+    which is the CI shape: it answers whether this chain still applies
+    to this data.
+  - **Three exit codes, three meanings:** `0` the chain applies and every
+    assertion holds; `1` the run failed (a step refused, the source was
+    malformed, the file was missing, a step needs a database); `2` the
+    command line itself was wrong (a missing or contradictory flag, an
+    unknown format). A script can tell "you asked for the wrong thing"
+    from "what you asked for does not hold".
+  - The report names the strategy §6.1 chose — `streamed`, or
+    `materialized` when a cross-document assertion needs the collection
+    at once.
 
 ## 12. Drift