@jarenjs/db 0.86.0 → 0.89.0

package/docs/SQLITE-RELATIONAL.md

@@ -49,9 +49,16 @@ Selections support table and subquery sources, inner/left/cross joins, correlate
 `sql.scalar`/`sql.exists`, CASE, IN/NOT IN, DISTINCT, grouped and distinct
 aggregates, HAVING, UNION/UNION ALL and ordered windows. The declaration file
 lists the closed operators and functions, including trim/coalesce, LIKE, JSON
-extraction, casts and SQLite date functions. Neither arbitrary function names
+extraction/type inspection, casts and SQLite date functions. Neither arbitrary function names
 nor a raw-expression escape hatch is accepted.
 
+`sql.call('json_type', [document, path])` distinguishes a missing path (SQL NULL)
+from JSON null (text `'null'`) and reports SQLite's native scalar/container type
+names. Omitting the path inspects the whole document. SQL-null input stays null;
+malformed JSON raises SQLite's error. It composes with CASE and synchronous
+cursors without decoding or rewriting the original column. See
+[SQLite JSON type inspection](https://www.sqlite.org/json1.html#the_json_type_function).
+
 ## Exact writes and bytes
 
 ```js
@@ -114,7 +121,9 @@ applyTableMigration(connection, migration);
 Columns retain declaration order and exact INTEGER/REAL/TEXT/BLOB/NUMERIC/ANY
 types. Definitions support ordered primary keys, rowid or AUTOINCREMENT identity,
 nullability, database defaults, generated columns, STRICT/WITHOUT ROWID, named
-UNIQUE/CHECK/foreign-key constraints and delete/update actions. Index terms
+UNIQUE/CHECK/foreign-key constraints and delete/update actions. A column can also
+declare `references: { table, columns: [name], onDelete, onUpdate, deferred }`.
+Index terms
 support expressions, direction and collation, with an optional partial predicate.
 Triggers support BEFORE/AFTER, INSERT/UPDATE/DELETE, UPDATE OF, OLD/NEW conditions,
 mutation steps and RAISE. Schema expressions use the same structural emitter,
@@ -126,14 +135,17 @@ Entity `.physical(...)` metadata can also supply column `type`, `defaultValue`,
 `planEntity(...).createSql` now emits explicit table DDL for complete writable
 declarations; views and incomplete generated/default definitions remain adoption
 metadata. `openStore` still verifies physical tables without creating them.
-Identical physical models produce an empty `planModelMigration`; changed
-physical tables use the live-schema `planTableMigration` API.
+Identical physical models produce an empty `planModelMigration`. Changed
+physical declarations remain a specific `JD0021` policy refusal; use the
+live-schema `planTableMigration` API and review the resulting structural plan.
+The model diff does not infer application-owned DDL or business backfills.
 
 ## Guarded upgrades
 
 Planning inspects the existing schema without modifying it. A differing existing
-table requires `allowRebuild: true`; additions currently use the same guarded
-rebuild. Every removed column needs `dropColumns`; removing an existing explicit
+table requires `allowRebuild: true` when using `planTableMigration`; use the narrow
+schema operations below to append a column without rebuilding. Every removed
+column needs `dropColumns`; removing an existing explicit
 index/trigger requires `dropObjects`. Unmentioned indexes and triggers survive.
 An optional `copy` maps writable non-key target columns to structural expressions;
 new columns otherwise use their defaults. The plan retains the source schema and
@@ -145,6 +157,8 @@ unchanged values/storage classes, drops/renames atomically, restores indexes and
 triggers, and checks foreign keys and the target schema. Primary-key columns,
 unshadowed hidden rowids, raw text, bytes and AUTOINCREMENT high-water marks are
 preserved. A key change, rowid-ownership change or ambiguous rowid alias refuses.
+This includes converting an `INTEGER PRIMARY KEY DESC` with an independent
+hidden rowid into an ordinary INTEGER primary key that owns the rowid.
 No migration history table or model adoption is needed.
 
 Rebuilds require SQLite's foreign-key transition outside a transaction. For a
@@ -157,6 +171,115 @@ refuses before DDL. Node/Bun regressions cover populated history and references,
 failed copies, repeat reopening, nested rollback, and process death after DROP
 with WAL recovery. These tests do not establish power-loss durability.
 
+### Composing a guarded plan with migration history
+
+Pass the complete saved table artifact as `{ kind: 'table', plan }` in
+`planPhysicalMigration` or `.step({ kind: 'table', plan })` in the migration
+pen. It retains `version`, `id`, `table`, `source`, `after`, `rebuild`,
+`temporary`, `unchanged`, `statements`, `finish` and `checksum`; incomplete
+plans refuse. Do not flatten its SQL arrays into separate DDL steps: the
+guarded executor also preserves storage classes and allocated identities.
+
+`migrate` supplies the existing history/receipt transaction around that
+executor. It admits the writer with IMMEDIATE semantics before source reads,
+uses savepoints for nested work and restores FK/legacy settings exactly.
+`{ connection }` borrows the same driver handle; `{ driver, path? }` owns its
+resource. Both compose with optional historical transforms/assertions and a
+complete `physicalTarget` copied from a disposable target fixture. With
+`shadowFixture`, replay uses the same migration and history executor.
+The [runnable example](MIGRATION-FORMAT.md#runnable-physical-lifecycle) proves
+this composition and a checked second run.
+
+Complete physical targets retain physical column and constraint order,
+quoted text, index predicates/collations and trigger programs. Only token
+whitespace/comments and CREATE-prefix `IF NOT EXISTS` are formatting;
+quote styles and case are not inferred equivalent. The separate exact
+source snapshot/checksum remains the stale-plan guard.
+
+## Additive and object operations
+
+`planSchemaChange(connection, operation)` returns a reviewable
+`{ version, operation, sql, source, settings, checksum }`. It reads the main schema
+and relevant connection settings without executing DDL. `applySchemaChange`
+checks that source and settings again under an IMMEDIATE transaction before
+executing the single statement. It returns `{ changed }`, the number of schema
+operations that changed the catalog, rather than the number of affected rows.
+Both methods require the same available synchronous SQLite ownership as rebuilds.
+
+```js
+import { planSchemaChange, applySchemaChange, sql } from '@jarenjs/db/relational';
+const addition = planSchemaChange(connection, {
+  op: 'addColumn', table: 'entries', column: {
+    name: 'revision', type: 'INTEGER', nullable: false, default: 1,
+    check: sql.binary('>=', sql.column('revision'), 1),
+  },
+});
+// Review addition.sql and addition.source before execution.
+applySchemaChange(connection, addition);
+applySchemaChange(connection, planSchemaChange(connection, {
+  op: 'dropIndex', name: 'obsolete_index', ifExists: true,
+}));
+```
+
+The closed operations are `addColumn` (`table`, `column`), `dropIndex` (`name`,
+optional `ifExists`), `renameTable` (`table`, `to`) and `dropTable` (`table`,
+optional `ifExists`). Identifiers address **main** explicitly; a temporary table
+with the same spelling cannot redirect an operation. A dot inside a name is a
+literal character. Attached database operations are not part of this surface.
+The column definition and expression renderer are shared with `planTable`.
+
+ADD COLUMN appends a declaration and preserves existing rowids, values, storage
+classes, column order, indexes and triggers. It does not derive a complete table
+definition, normalize unknown constraints or copy the table. Literal defaults
+are supported; identity columns, STORED generated columns and expression defaults
+refuse. Ordinary NOT NULL additions require a non-null default. REFERENCES
+additions require a NULL default (or no default), and a single referenced column.
+SQLite checks existing rows for a new CHECK or generated NOT NULL constraint;
+those checks can scan the table even though the operation does not copy it.
+See [SQLite ADD COLUMN restrictions](https://www.sqlite.org/lang_altertable.html#alter_table_add_column).
+
+Drop operations fail on absence unless `ifExists: true` is supplied. A missing
+object then returns `changed: 0`. DROP INDEX affects the named index, not its
+table or triggers. DROP TABLE removes the table and its owned indexes/triggers;
+SQLite's active foreign-key actions still apply. RENAME follows SQLite's current
+dependency rewriting behavior and the reviewed `legacy_alter_table` setting.
+See [DROP TABLE](https://www.sqlite.org/lang_droptable.html) and
+[RENAME TABLE](https://www.sqlite.org/lang_altertable.html#alter_table_rename).
+
+These are **single-source plans**, not durable migration receipts. Replan after
+any schema change. Reapplying a successful ADD/RENAME plan refuses as stale;
+it does not infer completion from a same-named object. An ordered migration must
+check its trusted schema/version receipt before planning the next operation and
+record completion in the same transaction. Unknown members or unsupported column
+declarations refuse with `JD0005`; modified/stale plans refuse with `JD0021`.
+Native object, data-constraint and dependency errors are left to SQLite. A
+checksum detects accidental plan edits; it does not authorize untrusted plans.
+
+## Explicit identity-changing upgrades
+
+The ordinary rebuild planner continues to refuse key reassignment or row loss.
+A reviewed upgrade can use the primitive operations under
+`withForeignKeysSuspended`, with its own explicit data policy:
+
+1. Check the durable migration receipt or the exact supported legacy schema.
+2. Create a distinct replacement table with `planTable`.
+3. Use relational insert-select with a scoped correlated selection, deterministic
+   identity choice and explicit exclusion predicate. Assert selected, inserted
+   and excluded counts, and any business-specific preservation requirements.
+4. Plan/apply `dropTable` for the original and then `renameTable` for the
+   replacement. Create the required indexes/triggers and validate dependents.
+5. Record completion inside the same transaction so another opening does no work.
+
+Create the replacement before dropping the original; renaming the original first
+can redirect dependent references. Plan each primitive inside the FK scope, after
+the preceding schema operation. The helper checks references before commit and
+restores connection settings. The caller must review incoming key references,
+views, triggers, immutable neighbors and each row disposition. Conflicts fail and
+roll back; there is no inferred permission to discard rows. The installed native
+[qualification fixture](../../../test/db/fixtures/schema-upgrade.mjs) demonstrates
+scoped minimum-ID selection, explicit exclusions, rollback, repeated reopening
+and recovery after actual process death at DROP.
+
 ## Read-only inspection and disk snapshots
 
 `readSchema` from `@jarenjs/db/model` inventories tables without adopting them,
@@ -188,3 +311,15 @@ calls. No global strong model cache is introduced. Keep connection/query caches
 bounded and reuse compiled metadata. Smaller import graphs alone do not prove
 the complete application meets its RSS budget; measure application memory with
 representative workloads.
+
+Entity mutation engines retain prepared statements by their complete emitted SQL
+in the bounded core LRU cache. Bound values, output projections and row/byte limits
+belong to each execution; changing a payload does not retain another document and
+another copy of the same statement. Distinct SQL stays isolated. This reduces
+retained payload memory, at the cost of rebinding/compiling an identical mutation
+document on each call. The synthetic retention probe at
+`test/db/fixtures/mutation-memory.mjs` reports both varying-payload and identical
+mutation timings, heap and RSS on Node (`--expose-gc`) and Bun. Such samples do not
+replace a complete application's resource gate. Metadata remains caller-owned;
+keep one compiled mapping per used model, one query state per connection, and
+release facade/cache references on close. Driver cursors remain ephemeral.

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@jarenjs/db",
   "private": false,
-  "version": "0.86.0",
+  "version": "0.89.0",
   "type": "module",
   "main": "./src/index.js",
   "types": "./types/index.d.ts",
@@ -108,9 +108,9 @@
     "prepack": "npm run build:types"
   },
   "dependencies": {
-    "@jarenjs/core": "^0.86.0",
-    "@jarenjs/json": "^0.86.0",
-    "@jarenjs/validate": "^0.86.0"
+    "@jarenjs/core": "^0.89.0",
+    "@jarenjs/json": "^0.89.0",
+    "@jarenjs/validate": "^0.89.0"
   },
   "bin": {
     "jaren-db": "./src/cli.js"

package/schemas/jaren-migration.draft-07.schema.json

@@ -29,7 +29,7 @@
       "type": "string"
     },
     "steps": {
-      "description": "Ordered steps; the order is the contract. A pure widening may carry NO steps \u2014 the migration then only moves the recorded shape.",
+      "description": "Ordered steps; the order is the contract. A pure widening may carry NO steps — the migration then only moves the recorded shape.",
       "type": "array",
       "items": {
         "$ref": "#/definitions/step"
@@ -80,6 +80,53 @@
             },
             "additionalProperties": false
           }
+        },
+        "target": {
+          "type": "object",
+          "properties": {
+            "objects": {
+              "type": "array",
+              "items": {
+                "type": "object",
+                "required": [
+                  "type",
+                  "name",
+                  "owner",
+                  "sql"
+                ],
+                "properties": {
+                  "type": {
+                    "enum": [
+                      "table",
+                      "view",
+                      "index",
+                      "trigger"
+                    ]
+                  },
+                  "name": {
+                    "type": "string"
+                  },
+                  "owner": {
+                    "type": "string"
+                  },
+                  "sql": {
+                    "type": "string"
+                  }
+                }
+              }
+            },
+            "tables": {
+              "type": "array",
+              "uniqueItems": true,
+              "items": {
+                "type": "string"
+              }
+            }
+          },
+          "required": [
+            "objects"
+          ],
+          "additionalProperties": false
         }
       },
       "additionalProperties": false
@@ -113,6 +160,9 @@
         },
         {
           "$ref": "#/definitions/rebuildStep"
+        },
+        {
+          "$ref": "#/definitions/tableStep"
         }
       ]
     },
@@ -158,6 +208,18 @@
         },
         "note": {
           "type": "string"
+        },
+        "model": {
+          "description": "The immutable model describing this step’s current physical layout.",
+          "type": "object",
+          "properties": {
+            "$model": {
+              "const": "0.1"
+            }
+          },
+          "required": [
+            "$model"
+          ]
         }
       },
       "required": [
@@ -189,6 +251,18 @@
         },
         "note": {
           "type": "string"
+        },
+        "model": {
+          "description": "The immutable model describing this step’s current physical layout.",
+          "type": "object",
+          "properties": {
+            "$model": {
+              "const": "0.1"
+            }
+          },
+          "required": [
+            "$model"
+          ]
         }
       },
       "required": [
@@ -199,7 +273,7 @@
       "additionalProperties": false
     },
     "deriveStep": {
-      "description": "Recompute named STORED derived index columns from the documents already in a collection. A spatial column is stored only under the physical mapping that cannot index a registered deterministic function (the generated mapping populates itself); a vector column is stored under both. The step is idempotent \u2014 a derived value is a pure function of the document \u2014 so a replay writes what the first run wrote.",
+      "description": "Recompute named STORED derived index columns from the documents already in a collection. A spatial column is stored only under the physical mapping that cannot index a registered deterministic function (the generated mapping populates itself); a vector column is stored under both. The step is idempotent — a derived value is a pure function of the document — so a replay writes what the first run wrote.",
       "type": "object",
       "properties": {
         "kind": {
@@ -273,7 +347,7 @@
       "additionalProperties": false
     },
     "sqlStep": {
-      "description": "A DATA step spelled directly as one SQL statement (MIGRATION-FORMAT \u00a79.4): the planner's fold of a dropped column into the document, the move of a document member into its new column, an R*Tree backfill. Executed exactly as a ddl step; distinct in kind so a reviewer reads intent and a dry run shows it as data.",
+      "description": "A DATA step spelled directly as one SQL statement (MIGRATION-FORMAT §9.4): the planner's fold of a dropped column into the document, the move of a document member into its new column, an R*Tree backfill. Executed exactly as a ddl step; distinct in kind so a reviewer reads intent and a dry run shows it as data.",
       "type": "object",
       "properties": {
         "kind": {
@@ -294,7 +368,7 @@
       "additionalProperties": false
     },
     "rebuildStep": {
-      "description": "The ALTER TABLE procedure SQLite lacks (MIGRATION-FORMAT \u00a710), self-contained: create the target shape under `<table>__rebuild`, copy the rows through the rendered column mapping, drop the old table, rename, recreate the indexes, then `PRAGMA foreign_key_check` inside the transaction.",
+      "description": "The ALTER TABLE procedure SQLite lacks (MIGRATION-FORMAT §10), self-contained: create the target shape under `<table>__rebuild`, copy the rows through the rendered column mapping, drop the old table, rename, recreate the indexes, then `PRAGMA foreign_key_check` inside the transaction.",
       "type": "object",
       "properties": {
         "kind": {
@@ -314,7 +388,7 @@
           }
         },
         "copy": {
-          "description": "The INSERT \u2026 SELECT that carries every row across the column mapping.",
+          "description": "The INSERT … SELECT that carries every row across the column mapping.",
           "type": "string",
           "minLength": 1
         },
@@ -338,6 +412,96 @@
         "indexes"
       ],
       "additionalProperties": false
+    },
+    "tableStep": {
+      "description": "A complete reviewed planTableMigration artifact, executed with its source, row, storage and identity guards.",
+      "type": "object",
+      "properties": {
+        "kind": {
+          "const": "table"
+        },
+        "plan": {
+          "type": "object",
+          "required": [
+            "version",
+            "id",
+            "table",
+            "checksum",
+            "source",
+            "after",
+            "rebuild",
+            "temporary",
+            "unchanged",
+            "statements",
+            "finish"
+          ],
+          "properties": {
+            "version": {
+              "const": 1
+            },
+            "id": {
+              "type": "string",
+              "minLength": 1
+            },
+            "table": {
+              "type": "string",
+              "minLength": 1
+            },
+            "checksum": {
+              "type": "string",
+              "minLength": 1
+            },
+            "source": {
+              "type": "array",
+              "items": {
+                "type": "object"
+              }
+            },
+            "after": {
+              "type": "array",
+              "items": {
+                "type": "object"
+              }
+            },
+            "rebuild": {
+              "type": "boolean"
+            },
+            "temporary": {
+              "type": "string",
+              "minLength": 1
+            },
+            "unchanged": {
+              "type": "array",
+              "items": {
+                "type": "string",
+                "minLength": 1
+              }
+            },
+            "statements": {
+              "type": "array",
+              "items": {
+                "type": "string",
+                "minLength": 1
+              }
+            },
+            "finish": {
+              "type": "array",
+              "items": {
+                "type": "string",
+                "minLength": 1
+              }
+            }
+          }
+        },
+        "note": {
+          "type": "string"
+        }
+      },
+      "required": [
+        "kind",
+        "plan"
+      ],
+      "additionalProperties": false
     }
   }
 }

package/schemas/jaren-migration.schema.json

@@ -80,6 +80,53 @@
       },
       "additionalProperties": false
      }
+    },
+    "target": {
+     "type": "object",
+     "properties": {
+      "objects": {
+       "type": "array",
+       "items": {
+        "type": "object",
+        "required": [
+         "type",
+         "name",
+         "owner",
+         "sql"
+        ],
+        "properties": {
+         "type": {
+          "enum": [
+           "table",
+           "view",
+           "index",
+           "trigger"
+          ]
+         },
+         "name": {
+          "type": "string"
+         },
+         "owner": {
+          "type": "string"
+         },
+         "sql": {
+          "type": "string"
+         }
+        }
+       }
+      },
+      "tables": {
+       "type": "array",
+       "uniqueItems": true,
+       "items": {
+        "type": "string"
+       }
+      }
+     },
+     "required": [
+      "objects"
+     ],
+     "additionalProperties": false
     }
    },
    "additionalProperties": false
@@ -113,6 +160,9 @@
     },
     {
      "$ref": "#/$defs/rebuildStep"
+    },
+    {
+     "$ref": "#/$defs/tableStep"
     }
    ]
   },
@@ -158,6 +208,18 @@
     },
     "note": {
      "type": "string"
+    },
+    "model": {
+     "description": "The immutable model describing this step’s current physical layout.",
+     "type": "object",
+     "properties": {
+      "$model": {
+       "const": "0.1"
+      }
+     },
+     "required": [
+      "$model"
+     ]
     }
    },
    "required": [
@@ -189,6 +251,18 @@
     },
     "note": {
      "type": "string"
+    },
+    "model": {
+     "description": "The immutable model describing this step’s current physical layout.",
+     "type": "object",
+     "properties": {
+      "$model": {
+       "const": "0.1"
+      }
+     },
+     "required": [
+      "$model"
+     ]
     }
    },
    "required": [
@@ -338,6 +412,96 @@
     "indexes"
    ],
    "additionalProperties": false
+  },
+  "tableStep": {
+   "description": "A complete reviewed planTableMigration artifact, executed with its source, row, storage and identity guards.",
+   "type": "object",
+   "properties": {
+    "kind": {
+     "const": "table"
+    },
+    "plan": {
+     "type": "object",
+     "required": [
+      "version",
+      "id",
+      "table",
+      "checksum",
+      "source",
+      "after",
+      "rebuild",
+      "temporary",
+      "unchanged",
+      "statements",
+      "finish"
+     ],
+     "properties": {
+      "version": {
+       "const": 1
+      },
+      "id": {
+       "type": "string",
+       "minLength": 1
+      },
+      "table": {
+       "type": "string",
+       "minLength": 1
+      },
+      "checksum": {
+       "type": "string",
+       "minLength": 1
+      },
+      "source": {
+       "type": "array",
+       "items": {
+        "type": "object"
+       }
+      },
+      "after": {
+       "type": "array",
+       "items": {
+        "type": "object"
+       }
+      },
+      "rebuild": {
+       "type": "boolean"
+      },
+      "temporary": {
+       "type": "string",
+       "minLength": 1
+      },
+      "unchanged": {
+       "type": "array",
+       "items": {
+        "type": "string",
+        "minLength": 1
+       }
+      },
+      "statements": {
+       "type": "array",
+       "items": {
+        "type": "string",
+        "minLength": 1
+       }
+      },
+      "finish": {
+       "type": "array",
+       "items": {
+        "type": "string",
+        "minLength": 1
+       }
+      }
+     }
+    },
+    "note": {
+     "type": "string"
+    }
+   },
+   "required": [
+    "kind",
+    "plan"
+   ],
+   "additionalProperties": false
   }
  }
 }