@simtlix/simfinity-postgres 3.6.0 → 3.8.0

package/README.md

@@ -31,9 +31,9 @@ import { postgresPlugin } from '@simtlix/simfinity-postgres';
 const simfinity = createSQL({ plugin: postgresPlugin({ pool, schema: 'library' }) });
 ```
 
-Install `@simtlix/simfinity-sql@3.6.0` as a direct dependency when importing its factory. PostgreSQL 15 or later is the only supported SQL plugin. The [SQL package guide](https://github.com/simtlix/simfinity.js/tree/master/packages/sql) documents relational planning and the versioned plugin contract.
+Install `@simtlix/simfinity-sql@3.8.0` as a direct dependency when importing its factory. PostgreSQL 15 or later is the only supported SQL plugin. The [SQL package guide](https://github.com/simtlix/simfinity.js/tree/master/packages/sql) documents relational planning and the versioned plugin contract.
 
-Both factories permanently bind one pool and schema, and four optional settings: `multiRowInserts: false` keeps one `INSERT` per owned row instead of multi-row statements; `explicitNullsOrder: true` keeps the `NULLS` clause on sorts of `id` and required fields, as 3.5.9 did, instead of letting indexes serve them; the opt-in `refreshTriggers: 'perRoot'` refreshes each changed root's owned shape and unique keys once per transaction, while its guard trigger stays deferred, instead of once per changed row; and the opt-in `ownerIndexes: 'minimal'` leaves out the `(__owner_id)` indexes that the primary key or a unique index covers on new owned and guard tables. An unknown `refreshTriggers` or `ownerIndexes` value throws `INVALID_DATABASE_SCHEMA` (400). Read the [upgrade notes](https://simtlix.github.io/simfinity.js/resources/compatibility.html#upgrade-to-3-6-0) before enabling the opt-ins: versions up to 3.5.9 cannot validate databases created with them. The returned runtime supports registration, generated queries and mutations, middleware and scopes, controllers and validators, custom mutations, state machines, nested relationship writes, embedded reconstruction, and native PostgreSQL model handles. Native handles expose `findById`, `find`, `create`, `update`, and `delete`; they are not Mongoose models. `find(args, { session })` accepts the generated GraphQL list query arguments. Existing `pg.Pool` configuration and `pg.PoolClient` query interfaces remain compatible across both factories. A custom pool or client wrapper should pass `pg`'s `command` through in query results: the plugin reads it from the `COMMIT` result to detect a transaction that PostgreSQL rolled back because one of its statements failed, and without it such a transaction reports success with nothing stored.
+Both factories permanently bind one pool and schema, and five optional settings: `multiRowInserts: false` keeps one `INSERT` per owned row instead of multi-row statements; `explicitNullsOrder: true` keeps the `NULLS` clause on sorts of `id` and required fields, as 3.5.9 did, instead of letting indexes serve them; `batchedCollections: true` reads the generated collection fields of the parents that share their arguments with one statement per 100 parents, in a transaction that turns JIT off for that transaction only, instead of one statement per field as 3.6.0 did and as the default still does, and can return unsorted and tied rows in another order; the opt-in `refreshTriggers: 'perRoot'` refreshes each changed root's owned shape and unique keys once per transaction, while its guard trigger stays deferred, instead of once per changed row; and the opt-in `ownerIndexes: 'minimal'` leaves out the `(__owner_id)` indexes that the primary key or a unique index covers on new owned and guard tables. An unknown `refreshTriggers` or `ownerIndexes` value throws `INVALID_DATABASE_SCHEMA` (400). Read the [upgrade notes](https://simtlix.github.io/simfinity.js/resources/compatibility.html#upgrade-to-3-6-0) before enabling the opt-ins: versions up to 3.5.9 cannot validate databases created with them. The returned runtime supports registration, generated queries and mutations, middleware and scopes, controllers and validators, custom mutations, state machines, nested relationship writes, embedded reconstruction, and native PostgreSQL model handles. Native handles expose `findById`, `find`, `create`, `update`, and `delete`; they are not Mongoose models. `find(args, { session })` accepts the generated GraphQL list query arguments. Existing `pg.Pool` configuration and `pg.PoolClient` query interfaces remain compatible across both factories. A custom pool or client wrapper should pass `pg`'s `command` through in query results: the plugin reads it from the `COMMIT` result to detect a transaction that PostgreSQL rolled back because one of its statements failed, and without it such a transaction reports success with nothing stored.
 
 Operations attempted before successful initialization fail with `DATABASE_NOT_INITIALIZED`. Call `createSchema` before `initializeDatabase`. Configuration and registrations become immutable once schema preparation begins. `preventCreatingCollection(true)` makes instance initialization validate existing storage without issuing creation DDL. A rejected initialization request leaves readiness as it was; a ready instance keeps serving while it is initialized again, and becomes unavailable if that initialization fails. `create` mode reads the catalog without table locks and locks nothing when nothing is missing. When it creates objects, writers of the tables that it changes, of existing tables that a new foreign key references, or of roots whose keys it rebuilds wait until it commits, so use `mode: 'validate'` to check a live instance. `simfinity.compileDatabaseSchema()` returns the instance's DDL for review or export.
 
@@ -55,14 +55,14 @@ await initializeDatabase({ mode: 'validate' });
 
 Called without arguments after `createSchema()`, the module's `describeDatabase()` and `compileDatabaseSchema()` describe and compile this default instance in its configured schema.
 
-Transactions use one borrowed `pg` client and repeatable-read isolation. Reads without a session of types without owned embedded tables run as a single statement on a pooled connection instead. A supplied Simfinity PostgreSQL session joins the active transaction. Serialization failures and deadlocks are retried up to five times, each after a short random delay that starts once the client is released; mutation input is reset for each attempt. Unknown commit outcomes are not replayed. The caller owns the pool lifecycle.
+Transactions use one borrowed `pg` client and repeatable-read isolation. Reads without a session of types without owned embedded tables run as a single statement on a pooled connection instead. With `batchedCollections: true`, the runtime reads the generated collection fields of up to 100 parents that share their arguments in one batch, with one statement, a `UNION ALL` of each parent's own statement with its parent ID as a parameter of its own, so PostgreSQL plans each parent's rows as it plans that parent's own statement; more parents run as several batches, each in its own transaction. Each batch runs in one repeatable-read transaction: `BEGIN`, `SET LOCAL jit = off`, the statement, the owned-row reads, then `COMMIT`. Only a direct `findChildrenBatch` call on the SQL adapter with more than 100 IDs repeats `SET LOCAL` and the statement for each further 100 within its transaction. The batch's plan can cost more than `jit_above_cost` when no parent's own plan does, and JIT compilation of its branches can take longer than the statement itself; `SET LOCAL` ends with the transaction, so pooled connections keep the server's setting. A batch takes these 4 round trips, or more with owned rows, one after another on one connection and runs its parents one after another, where per-parent statements take one round trip each, concurrently on the pool's free connections. It pays off when the pool is saturated or much smaller than the number of parents per level, as with many parents with small child lists; with free connections it can be slower, even on a remote database, so measure before enabling it. Transaction-mode PgBouncer keeps each batch on one server connection, while Amazon RDS Proxy pins the pooled connections that run batches, for their `SET LOCAL` and their statements over 16 KB. See [batched collection reads](https://simtlix.github.io/simfinity.js/guide/relationships.html#batched-collection-reads). A supplied Simfinity PostgreSQL session joins the active transaction. Serialization failures and deadlocks are retried up to five times, each after a short random delay that starts once the client is released; mutation input is reset for each attempt. Unknown commit outcomes are not replayed. The caller owns the pool lifecycle.
 
 The module and every `createPostgres` instance expose the shared `auth`, `validators`, `scalars`, and `plugins` helpers. These are the same helper objects exported by the MongoDB package, so rules, scalar identities, and errors can be shared safely between backends.
 
 MCP integration is an independent opt-in and does not add MCP dependencies to PostgreSQL applications:
 
 ```sh
-npm install @simtlix/simfinity-mcp@3.6.0 @modelcontextprotocol/sdk@^1.31.0
+npm install @simtlix/simfinity-mcp@3.8.0 @modelcontextprotocol/sdk@^1.31.0
 ```
 
 ```javascript
@@ -81,9 +81,11 @@ Errors that PostgreSQL reports become `SimfinityError`s without SQL details, wit
 
 PostgreSQL enforces required values, real FKs, JSONB/owned embedded shapes, root and embedded scalar/reference uniqueness, and scalar-list multikey uniqueness. Root unique indexes use `NULLS NOT DISTINCT`; private typed owner-key tables allow repeated keys within one owner but reject the same key across owners. Presence markers preserve missing versus explicit null fields. This is stronger and broader than generated Mongoose storage.
 
+Virtual fields (`extensions.virtual`), on entities and on members of embedded values, have no storage: no column, presence marker, check, JSON validation function, owned table, index or foreign key, and any output type works as their result. Adding one to a deployed type is therefore not a schema change. A field that a database already stores and that becomes virtual leaves its storage behind, so initialization keeps failing with `SCHEMA_MISMATCH` until the database is migrated. When the leftover is a column named after that field, its presence marker or owned-records state, its owned table, a validation or maintenance function of its own column or root, or the owned table of an embedded value whose last member with its own table it was, the error names the field and lists the migration. Storage named otherwise is not attributed: the `connectionField` column of a former reference, and the private back-reference column of a former collection in its child's table.
+
 Scalar-list leaves inside nested embedded lists support filters, sorts, ragged group projections, and array-valued facts. Array `MIN`/`MAX` compares complete arrays; result sorting uses immediate extrema. Date parameters use per-query UTC encoding without changing global `pg` behavior.
 
-The following mappings remain unsupported and fail explicitly: implicit many-to-many reciprocal lists, non-embedded collections inside embedded objects, embedded cycles, nested list wrappers, whole embedded-object uniqueness, conflicting relationship metadata, enum values that collide after text conversion, and custom scalars without a recognized base scalar. Query paths cannot sort or group whole embedded objects. Arbitrary MongoDB pipelines and Mongoose-native methods are outside the PostgreSQL contract.
+The following mappings remain unsupported and fail explicitly: implicit many-to-many reciprocal lists, non-embedded collections inside embedded objects, embedded cycles, nested list wrappers, whole embedded-object uniqueness, conflicting relationship metadata, enum values that collide after text conversion, and stored custom scalars without a recognized base scalar. Query paths cannot sort or group whole embedded objects. Arbitrary MongoDB pipelines and Mongoose-native methods are outside the PostgreSQL contract.
 
 ## Low-level schema API
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@simtlix/simfinity-postgres",
-  "version": "3.6.0",
+  "version": "3.8.0",
   "description": "PostgreSQL GraphQL runtime and schema generation with foreign keys for Simfinity",
   "type": "module",
   "main": "./src/index.js",
@@ -27,8 +27,8 @@
     "directory": "packages/postgres"
   },
   "dependencies": {
-    "@simtlix/simfinity-core": "3.6.0",
-    "@simtlix/simfinity-sql": "3.6.0",
+    "@simtlix/simfinity-core": "3.8.0",
+    "@simtlix/simfinity-sql": "3.8.0",
     "pg": "^8.16.3"
   },
   "peerDependencies": {

package/src/plugin.js

@@ -4,7 +4,7 @@ import { identifier, generatedName } from './schema/sql.js';
 import { describeRelationalSchema } from './schema/describe.js';
 import { initializeDatabase } from './schema/initialize.js';
 import { compileDatabaseSchema } from './schema/ddl.js';
-import { compileQuery } from './query/compiler.js';
+import { compileChildrenQuery, compileQuery } from './query/compiler.js';
 import { compileRecord, compileRecordBatch, MAX_RECORD_BATCH_VALUES } from './record-statements.js';
 import { castId, encodeScalar, decodeScalar, encodeParameter, isRetryableDatabaseError, normalizeDatabaseError } from './codecs.js';
 
@@ -20,9 +20,15 @@ export const postgresPlugin = (options) => ({
   naming: { validateIdentifier: identifier, generatedName },
   // The runtime passes its bound configuration: createPostgres(options) or configure(value).
   describeSchema: (plan, configuration) => describeRelationalSchema(plan, { refreshTriggers: configuration?.refreshTriggers, ownerIndexes: configuration?.ownerIndexes }),
-  initialize: (configuration, description, initialization) => initializeDatabase(configuration.pool, description, initialization),
+  // `context.virtualFields` locates virtual fields, so that SCHEMA_MISMATCH can name one whose earlier storage is left.
+  initialize: (configuration, description, initialization, context) => initializeDatabase(configuration.pool, description, initialization, context),
   compileSchema: compileDatabaseSchema,
   compileQuery,
+  // Optional member: the find statements of many parents in one statement, for findChildrenBatch, which the SQL
+  // adapter defines only with `batchedCollections: true`. Each statement carries setup that turns JIT off for its
+  // transaction, and the compileQuery it reproduces, so a copy of this plugin with another compileQuery reads each
+  // parent alone.
+  compileChildrenQuery,
   compileRecord,
   compileRecordBatch,
   maxRecordBatchValues: MAX_RECORD_BATCH_VALUES,

package/src/query/compiler.js

@@ -4,11 +4,65 @@ import { encodeScalar, storageTypes, invalidValue } from '../codecs.js';
 
 const unsupported = (message) => { throw new SimfinityError(message, 'UNSUPPORTED_QUERY', 400); };
 
+// Alias of each per-parent statement in the batched children statement. The per-parent statement only uses t<n>,
+// f<n> and short subquery aliases, so it never shadows this one.
+const CHILDREN = 'simfinity_children';
+// Stands for the parent's parameter in the per-parent statement that compileChildrenQuery repeats once per parent.
+// Compiled statements carry no NUL character, which PostgreSQL rejects in statement text: caller values are
+// parameters and identifiers and literals come from GraphQL names.
+const PARENT = '\u0000';
+// Runs before each batched children statement, in its transaction; SET LOCAL ends with that transaction, so the
+// connection goes back to the pool with the server's JIT setting.
+const CHILDREN_SETUP = 'SET LOCAL jit = off';
+
 /**
  * Compiles validated paths; all caller values are parameters, identifiers come from metadata.
  * `configuration.explicitNullsOrder: true` keeps the NULLS clause on every sort term, as 3.5.9 did.
  */
-export const compileQuery = (models, database, plan, extra = null, configuration = null) => {
+export const compileQuery = (models, database, plan, extra = null, configuration = null) => compileSelect(models, database, plan, extra, configuration, false);
+
+// A result column for the batched statement that no column of the child table has. Planning rejects storage names
+// that start with `__`, and generated root columns end in `_state` or `_present`, so the loop only guards a
+// description built by hand.
+const freeColumn = (table, name, taken = []) => {
+  let column = name;
+  while (taken.includes(column) || table.columns.some((item) => item.name === column)) column = `_${column}`;
+  return column;
+};
+
+/**
+ * The find statement of each parent in `extra.ids`, in one statement: a UNION ALL with one branch per ID, in
+ * order. Each branch is the statement compileQuery returns for `{ column, id }`, with the same filters, joins,
+ * ORDER BY, LIMIT and OFFSET, and with that ID as a parameter of its own, so PostgreSQL plans every branch for its
+ * own parent, with the statistics of that value, as it plans the per-parent statement. The other parameters, such
+ * as filter values and the page, are shared by every branch. Each branch tags its rows with the 1-based position of
+ * its parent ID, in `parentColumn`, and with their row number in the order its statement returns them: the
+ * statement always has a LIMIT, so PostgreSQL keeps it a subquery and never merges it into the branch. The outer
+ * ORDER BY on both returns each parent's rows together, in position order, and in its statement's order, which a
+ * Parallel Append would not keep.
+ * Values are the per-parent statement's values without the parent ID, then one uuid per parent ID.
+ * Its `setup` turns JIT compilation off for the transaction the SQL adapter runs it in: the plan of up to 100
+ * branches costs about as much as the per-parent plans together, so it can cross jit_above_cost when none of them
+ * does, and JIT then compiles every branch's expressions, which can take longer than running the statement.
+ */
+export const compileChildrenQuery = (models, database, plan, extra, configuration = null) => {
+  if (plan?.mode !== 'find' || !plan.pagination) unsupported('Only paginated find queries read children in batches');
+  if (!Array.isArray(extra?.ids) || !extra.ids.length) unsupported('Batched children reads require parent IDs');
+  const { text, values } = compileSelect(models, database, plan, { column: extra.column, id: null }, configuration, true);
+  const [before, after, ...rest] = text.split(PARENT);
+  if (after === undefined || rest.length) unsupported('Invalid batched children statement');
+  const root = database.tables.find((item) => item.name === plan.entity);
+  const parentColumn = freeColumn(root, '__simfinity_parent');
+  const ordinalColumn = freeColumn(root, '__simfinity_ordinal', [parentColumn]);
+  const branches = extra.ids.map((id, index) => `SELECT ${index + 1}::integer AS ${q(parentColumn)}, row_number() OVER () AS ${q(ordinalColumn)}, ${CHILDREN}.* FROM (${before}$${values.length + index + 1}::uuid${after}) ${CHILDREN}`);
+  return { text: `${branches.join(' UNION ALL ')} ORDER BY ${q(parentColumn)}, ${q(ordinalColumn)}`, values: [...values, ...extra.ids], parentColumn, setup: [{ text: CHILDREN_SETUP, values: [] }] };
+};
+// The compileQuery whose per-parent statements compileChildrenQuery reproduces. bindPlugin binds the children
+// compiler only for a plugin whose compileQuery is this one, so a plugin copy that overrides compileQuery reads
+// each parent with its own compileQuery.
+Object.defineProperty(compileChildrenQuery, 'compileQuery', { value: compileQuery });
+
+const compileSelect = (models, database, plan, extra, configuration, batched) => {
   const explicitNullsOrder = configuration?.explicitNullsOrder === true;
   const values = [];
   const joins = new Map();
@@ -232,7 +286,7 @@ export const compileQuery = (models, database, plan, extra = null, configuration
   if (extra) {
     const column = root.columns.find((item) => item.name === extra.column);
     if (!column || column.type !== 'uuid') unsupported('Invalid inverse connection field');
-    condition += ` AND ${col('t0', column.name)} = ${bind(extra.id, 'uuid')}`;
+    condition += ` AND ${col('t0', column.name)} = ${batched ? PARENT : bind(extra.id, 'uuid')}`;
   }
   let select = 't0.*';
   // List values that MIN and MAX facts read, each evaluated once per row in a lateral subquery.

package/src/schema/initialize.js

@@ -1,5 +1,5 @@
 import { SimfinityError } from '@simtlix/simfinity-core';
-import { identifier, qualified, generatedName } from './sql.js';
+import { identifier, qualified, generatedName, literal } from './sql.js';
 import { createTableSQL, createForeignKeySQL, createIndexSQL, createFunctionSQL, createTriggerSQL, triggerEvents } from './ddl.js';
 import { maintenanceLayouts, layoutOf, withLayouts } from './constraints.js';
 
@@ -342,8 +342,18 @@ const traverse = async (client, description, catalog, pass, missing, created) =>
   return legacy;
 };
 
-// Names to read from the catalog: the description's, plus the refresh-trigger objects of both layouts.
-const catalogNames = (description, layouts) => {
+// The table that an embedded value holding a virtual member owned while that member was stored, when the description
+// no longer has it: the value then has no member with its own table, so it is stored as JSON.
+const formerOwnedTable = (description, location) => {
+  if (!location.path.length) return null;
+  const name = generatedName(location.table, location.path[0]);
+  return description.tables.some((table) => table.name === name) ? null : name;
+};
+
+// Names to read from the catalog: the description's, plus the refresh-trigger objects of both layouts. With virtual
+// fields, also the touch trigger of each root that owns nothing, and the former owned table of each embedded value
+// that holds one (see formerOwnedTable), which only storage no longer described can leave.
+const catalogNames = (description, layouts, virtualFields) => {
   const functions = new Set((description.functions || []).map((fn) => fn.name));
   const triggers = new Set((description.triggers || []).map((trigger) => trigger.name));
   for (const layout of layouts.values()) {
@@ -352,7 +362,18 @@ const catalogNames = (description, layouts) => {
       for (const trigger of objects.triggers) triggers.add(trigger.name);
     }
   }
-  return { tables: description.tables.map((table) => table.name), functions: [...functions], triggers: [...triggers] };
+  const tables = new Set(description.tables.map((table) => table.name));
+  if (virtualFields.length) {
+    const maintained = new Set((description.maintenance || []).map((item) => item.rootTable));
+    for (const table of description.tables) {
+      if (!table.ownership && !table.auxiliary && !maintained.has(table.name)) triggers.add(generatedName(table.name, 'touch'));
+    }
+    for (const location of virtualFields) {
+      const former = formerOwnedTable(description, location);
+      if (former) tables.add(former);
+    }
+  }
+  return { tables: [...tables], functions: [...functions], triggers: [...triggers] };
 };
 
 // Each maintained root keeps the refresh-trigger layout its database already has: a function or trigger
@@ -412,6 +433,148 @@ const lockRoots = async (client, description, catalog, roots) => {
   if (names.length) await client.query(`LOCK TABLE ${names.map((table) => qualified(description.schema, table.name)).join(', ')} IN SHARE ROW EXCLUSIVE MODE`);
 };
 
+const UPGRADE_NOTES = 'https://simtlix.github.io/simfinity.js/resources/compatibility.html';
+const series = (items) => (items.length < 3 ? items.join(' and ') : `${items.slice(0, -1).join(', ')} and ${items.at(-1)}`);
+// A name longer than PostgreSQL identifiers was never stored, so nothing can refer to it.
+const quotedName = (name) => (Buffer.byteLength(name) <= 63 ? identifier(name) : null);
+
+// The described functions whose bodies could refer to the storage of a virtual field's location: for a member of a
+// JSON value, the validation functions of its column, which the column's shape check calls; for a column, the
+// maintenance functions of its table's root and the functions of that root's triggers. Each set includes the
+// described functions that its functions call.
+const functionOwners = (description) => {
+  const functions = description.functions || [];
+  const called = (text) => functions.filter((fn) => text.includes(qualified(description.schema, fn.name)));
+  const closure = (seeds) => {
+    const names = new Set();
+    const queue = [...seeds];
+    while (queue.length) {
+      const fn = queue.pop();
+      if (names.has(fn.name)) continue;
+      names.add(fn.name);
+      queue.push(...called(fn.body));
+    }
+    return names;
+  };
+  const ofRoot = (table) => {
+    const root = rootOf(description, table);
+    const item = (description.maintenance || []).find((entry) => entry.rootTable === root);
+    const triggered = new Set((description.triggers || []).filter((trigger) => rootOf(description, trigger.table) === root).map((trigger) => trigger.function));
+    return closure(functions.filter((fn) => triggered.has(fn.name) || (item && [item.validateFunction, item.refreshFunction].includes(fn.name))));
+  };
+  const ofColumn = (tableName, column) => {
+    const table = description.tables.find((item) => item.name === tableName);
+    const check = table?.checks.find((item) => item.name === generatedName(tableName, column, 'shape'));
+    return closure(check ? called(check.expression) : []);
+  };
+  return { ofRoot, ofColumn };
+};
+
+// Storage that virtual fields (`extensions.virtual`) left in a database created while they were stored, found in
+// the catalog this run has read, with no further statement: a column of a described table named as such a field's
+// column, presence flag or owned-records state, a described function whose stored body still refers to such a
+// field while its description does not, and the owned table of an embedded value that such a field, as its last
+// member with its own table, made the value own. A state column means the field owned tables, which are dropped. An
+// embedded value that no longer owns a table is stored as JSON instead, which needs a data migration. A root that
+// now owns nothing but still has its refresh triggers loses them and its guard table, since the functions they call
+// refer to the storage being dropped. Only the functions of a field's own column or root are compared (see
+// functionOwners). Returns the explanation, with the migration, that a SCHEMA_MISMATCH appends, or null when
+// nothing is left.
+const virtualLeftovers = (description, catalog, virtualFields) => {
+  const { schema } = description;
+  const maintained = new Set((description.maintenance || []).map((item) => item.rootTable));
+  const owners = functionOwners(description);
+  const left = new Set();
+  const columns = new Map();
+  const owned = [];
+  const moved = new Map();
+  const has = (actual, name) => actual.columns.some((column) => column.name === name);
+  for (const location of virtualFields) {
+    const former = formerOwnedTable(description, location);
+    const table = former && description.tables.find((item) => item.name === location.table);
+    const actual = table && catalog.tables.get(table.name);
+    const ownedActual = former && catalog.tables.get(former);
+    const [value, member = location.name] = location.path;
+    const state = `__${value}_state`;
+    if (!actual || !ownedActual || table.columns.some((column) => column.name === state) || !has(actual, state)) continue;
+    // The member, or the value inside the owned table that holds it, had storage there.
+    if (![member, generatedName('__field', member, 'present'), `__${member}_state`].some((name) => has(ownedActual, name))) continue;
+    left.add(location.field);
+    moved.set(former, { table: table.name, value, state, location });
+  }
+  for (const location of virtualFields.filter((item) => !item.path.length)) {
+    const table = description.tables.find((item) => item.name === location.table);
+    const actual = table && catalog.tables.get(table.name);
+    if (!actual) continue;
+    const state = `__${location.name}_state`;
+    const names = [location.name, generatedName('__field', location.name, 'present'), state].filter((name) => !table.columns.some((column) => column.name === name)
+      && actual.columns.some((column) => column.name === name));
+    if (!names.length) continue;
+    left.add(location.field);
+    columns.set(table.name, [...(columns.get(table.name) || []), ...names]);
+    if (names.includes(state)) owned.push(location);
+  }
+  const statements = [...columns].map(([table, names]) => `ALTER TABLE ${qualified(schema, table)} ${names.map((name) => `DROP COLUMN ${identifier(name)}`).join(', ')};`);
+  const add = (statement) => { if (!statements.includes(statement)) statements.push(statement); };
+  for (const location of owned) add(`DROP TABLE IF EXISTS ${qualified(schema, generatedName(location.table, location.name))} CASCADE;`);
+  for (const [former, { table, value, state }] of moved) {
+    const json = [value, generatedName('__field', value, 'present')].map(identifier).join(' and ');
+    add(`The embedded value ${value} of ${qualified(schema, table)} now has no member with its own table, so it is stored as JSON in the columns ${json} `
+      + `of that table, with the checks and functions that compileDatabaseSchema() gives them, instead of in the table ${qualified(schema, former)}: `
+      + `add them, fill them from the rows of ${qualified(schema, former)}, then drop the column ${identifier(state)} and run DROP TABLE IF EXISTS ${qualified(schema, former)} CASCADE;`);
+  }
+  for (const root of new Set([...columns.keys(), ...[...moved.values()].map((item) => item.table)].map((table) => rootOf(description, table)))) {
+    if (maintained.has(root) || !catalog.triggers(root, generatedName(root, 'touch')).length) continue;
+    for (const suffix of ['touch', 'deferred']) add(`DROP TRIGGER IF EXISTS ${identifier(generatedName(root, suffix))} ON ${qualified(schema, root)};`);
+    add(`DROP TABLE IF EXISTS ${qualified(schema, generatedName(root, 'guard'))};`);
+  }
+  // A column is referred to as `alias."name"`, a member of a JSON value as `($1 -> 'name')`, and a former owned table
+  // by its qualified name.
+  const references = [
+    ...virtualFields.map((location) => ({
+      field: location.field,
+      functions: location.path.length ? owners.ofColumn(location.table, location.path[0]) : owners.ofRoot(location.table),
+      tokens: location.path.length
+        ? [`($1 -> ${literal(location.name)})`]
+        : [location.name, generatedName('__field', location.name, 'present'), `__${location.name}_state`].map(quotedName).filter(Boolean).map((name) => `.${name}`),
+    })),
+    ...[...moved].map(([former, { table, location }]) => ({ field: location.field, functions: owners.ofRoot(table), tokens: [qualified(schema, former)] })),
+  ];
+  const functions = [];
+  for (const fn of description.functions || []) {
+    const rows = catalog.functions(fn.name);
+    if (rows.length !== 1 || rows[0].body === fn.body) continue;
+    const matched = references.filter(({ functions: owned, tokens }) => owned.has(fn.name)
+      && tokens.some((token) => rows[0].body.includes(token) && !fn.body.includes(token)));
+    if (!matched.length) continue;
+    functions.push(qualified(schema, fn.name));
+    for (const { field } of matched) left.add(field);
+  }
+  if (!left.size) return null;
+  const fields = [...new Set(virtualFields.map((item) => item.field))].filter((field) => left.has(field));
+  const many = fields.length > 1;
+  if (functions.length) {
+    statements.push(`Replace ${functions.length > 1 ? 'the functions' : 'the function'} ${series(functions)} with ${functions.length > 1 ? 'their CREATE FUNCTION statements' : 'its CREATE FUNCTION statement'} `
+      + 'from compileDatabaseSchema(), run as CREATE OR REPLACE FUNCTION.');
+  }
+  return `${series(fields)} ${many ? 'are' : 'is'} virtual (extensions.virtual), so ${many ? 'they are' : 'it is'} never stored, but this database still holds ${many ? 'their' : 'its'} `
+    + `earlier storage. Once nothing reads that storage, migrate it: ${statements.join(' ')} See the upgrade notes at ${UPGRADE_NOTES}`;
+};
+// A SCHEMA_MISMATCH of a database that still holds storage of fields that are now virtual also names them.
+const explainMismatch = (error, description, catalog, layouts, virtualFields) => {
+  if (!virtualFields.length || !catalog || !(error instanceof SimfinityError) || error.extensions?.code !== 'SCHEMA_MISMATCH') return error;
+  let resolved = description;
+  // Refresh-trigger objects of two layouts in one root are themselves the mismatch; compare with the description.
+  try { resolved = resolveLayouts(description, catalog, layouts); } catch { /* keep the description */ }
+  try {
+    const explanation = virtualLeftovers(resolved, catalog, virtualFields);
+    return explanation ? new SimfinityError(`${error.message} ${explanation}`, 'SCHEMA_MISMATCH', 409) : error;
+  } catch {
+    // An explanation never replaces the mismatch it explains.
+    return error;
+  }
+};
+
 const backfill = async (client, description, roots) => {
   // The keys of these roots can be stale because maintenance objects were recreated. Their tables are
   // locked; rebuild their complete derived key set before testing owners against one another. Rollback
@@ -430,18 +593,23 @@ const backfill = async (client, description, roots) => {
  * Create missing objects or validate an existing schema; never alter/drop existing storage. The catalog is
  * read in a fixed number of statements without table locks. Create mode changes nothing, and locks nothing,
  * when nothing is missing; otherwise it locks only the tables of roots whose keys it must rebuild.
+ * `context.virtualFields`, which the SQL runtime passes when its types declare virtual fields, lets a
+ * SCHEMA_MISMATCH name such a field whose earlier storage the database still holds, with its migration.
  */
-export const initializeDatabase = async (pool, description, { mode = 'create' } = {}) => {
+export const initializeDatabase = async (pool, description, { mode = 'create' } = {}, context) => {
   if (!['create', 'validate'].includes(mode)) throw new SimfinityError(`Unknown initialization mode: ${mode}`, 'INVALID_INITIALIZATION_MODE', 400);
   const { schema } = description;
   identifier(schema);
   if (schema.startsWith('pg_') || schema === 'information_schema') throw new SimfinityError('System schemas cannot be managed', 'INVALID_DATABASE_SCHEMA', 400);
   const layouts = maintenanceLayouts(description);
-  const names = catalogNames(description, layouts);
+  const virtualFields = Array.isArray(context?.virtualFields) ? context.virtualFields : [];
+  const names = catalogNames(description, layouts, virtualFields);
   const client = await pool.connect();
   const created = [];
   let legacy = [];
   let rollbackError;
+  // The catalog most recently read, which explains a mismatch.
+  let latest = null;
   try {
     await client.query(mode === 'validate' ? 'BEGIN READ ONLY' : 'BEGIN');
     await client.query('SET LOCAL search_path = pg_catalog');
@@ -451,6 +619,7 @@ export const initializeDatabase = async (pool, description, { mode = 'create' }
     // Serializes initializers of this schema before anything is read.
     await client.query('SELECT pg_advisory_xact_lock(hashtextextended($1, 0))', [`simfinity:${schema}`]);
     const inspected = await loadCatalog(client, schema, names);
+    latest = inspected;
     if (mode === 'validate') legacy = await traverse(client, resolveLayouts(description, inspected, layouts), inspected, 'validate', [], created);
     else {
       // Inspect first, without table locks or DDL. Drift before the first missing object is reported as
@@ -468,6 +637,7 @@ export const initializeDatabase = async (pool, description, { mode = 'create' }
         const roots = staleRoots(planned, missing);
         await lockRoots(client, planned, inspected, roots);
         const catalog = await loadCatalog(client, schema, names);
+        latest = catalog;
         const effective = resolveLayouts(description, catalog, layouts);
         const objects = [];
         legacy = await traverse(client, effective, catalog, 'create', objects, created);
@@ -487,7 +657,7 @@ export const initializeDatabase = async (pool, description, { mode = 'create' }
     return { mode, created };
   } catch (error) {
     try { await client.query('ROLLBACK'); } catch (failure) { rollbackError = failure; }
-    throw error;
+    throw explainMismatch(error, description, latest, layouts, virtualFields);
   } finally {
     // A failed ROLLBACK can leave the transaction open; the error makes pg-pool destroy the connection.
     client.release(rollbackError);

package/types/index.d.ts

@@ -16,6 +16,7 @@ export type {
   AuthPluginOptions,
   AuthRule,
   AuthRuleFunction,
+  AuthSchemaPlugin,
   EntityController,
   EnvelopSchemaPlugin,
   FieldValidations,
@@ -31,6 +32,8 @@ export type {
   StateMachine,
   TypePermissions,
   TypeScopes,
+  VirtualFieldDeclaration,
+  VirtualFieldOptions,
 } from '@simtlix/simfinity-core';
 
 export interface DatabaseColumn {
@@ -178,6 +181,18 @@ export interface PostgresConfiguration {
    * indexes can return them in order. `true` keeps the explicit NULLS clause on every sort term, as 3.5.9 did.
    */
   explicitNullsOrder?: boolean;
+  /**
+   * Defaults to false: each generated collection field is read with its own statement, as 3.6.0 did. `true` reads the
+   * collection fields of many parents that share their arguments with one statement per 100 parents, a UNION ALL of
+   * each parent's own statement with its parent ID as a parameter of its own, so PostgreSQL plans each parent's rows
+   * for that parent's ID. A batch runs in one repeatable-read transaction, each of its statements right after
+   * `SET LOCAL jit = off`, because the batch's plan can cross jit_above_cost when no per-parent plan does.
+   * Each parent gets the same rows as before when its order is total, such as a sort that ends in `id`; PostgreSQL
+   * does not promise an order for unsorted rows and rows with equal sort keys, which can then come in another order,
+   * and on a page that does not hold all of them, other such rows can fill it. Any value other than `true` keeps the
+   * default.
+   */
+  batchedCollections?: boolean | undefined;
   /**
    * Defaults to 'perRow', as 3.5.9: the deferred triggers of a root with owned tables or unique lists refresh its owned
    * shape and unique keys once per changed row. 'perRoot' refreshes each changed root once per transaction while its
@@ -209,7 +224,8 @@ export interface PostgresRuntime extends Runtime<PostgresModel, PostgresSession>
    * Must be awaited before operations execute. Does not synchronize incompatible schemas. A request rejected before
    * storage is touched, such as an unknown mode (INVALID_INITIALIZATION_MODE, 400) or, on a validation-only runtime,
    * a non-empty mode other than validate (DATABASE_CREATION_DISABLED, 409), keeps the current readiness; a ready runtime keeps serving while it is initialized again, and any failure of the latest
-   * initialization makes it unavailable.
+   * initialization makes it unavailable. When the database still holds storage of a field that is now virtual (`extensions.virtual`), such as its column,
+   * the SCHEMA_MISMATCH (409) message also names the field and gives the migration.
    */
   initializeDatabase(options?: InitializationOptions): Promise<InitializationResult>;
   withTransaction<T>(session: PostgresSession | null | undefined, callback: (session: PostgresSession) => Promise<T> | T): Promise<T>;