@simtlix/simfinity-postgres 3.9.0 → 3.10.0

package/README.md

@@ -31,9 +31,9 @@ import { postgresPlugin } from '@simtlix/simfinity-postgres';
 const simfinity = createSQL({ plugin: postgresPlugin({ pool, schema: 'library' }) });
 ```
 
-Install `@simtlix/simfinity-sql@3.9.0` as a direct dependency when importing its factory. PostgreSQL 15 or later is the only supported SQL plugin. The [SQL package guide](https://github.com/simtlix/simfinity.js/tree/master/packages/sql) documents relational planning and the versioned plugin contract.
+Install `@simtlix/simfinity-sql@3.10.0` as a direct dependency when importing its factory. PostgreSQL 15 or later is the only supported SQL plugin. The [SQL package guide](https://github.com/simtlix/simfinity.js/tree/master/packages/sql) documents relational planning and the versioned plugin contract.
 
-Both factories permanently bind one pool and schema, and five optional settings: `multiRowInserts: false` keeps one `INSERT` per owned row instead of multi-row statements; `explicitNullsOrder: true` keeps the `NULLS` clause on sorts of `id` and required fields, as 3.5.9 did, instead of letting indexes serve them; `batchedCollections: true` reads the generated collection fields of the parents that share their arguments with one statement per 100 parents, in a transaction that turns JIT off for that transaction only, instead of one statement per field as 3.6.0 did and as the default still does, and can return unsorted and tied rows in another order; the opt-in `refreshTriggers: 'perRoot'` refreshes each changed root's owned shape and unique keys once per transaction, while its guard trigger stays deferred, instead of once per changed row; and the opt-in `ownerIndexes: 'minimal'` leaves out the `(__owner_id)` indexes that the primary key or a unique index covers on new owned and guard tables. An unknown `refreshTriggers` or `ownerIndexes` value throws `INVALID_DATABASE_SCHEMA` (400). Read the [upgrade notes](https://simtlix.github.io/simfinity.js/resources/compatibility.html#upgrade-to-3-6-0) before enabling the opt-ins: versions up to 3.5.9 cannot validate databases created with them. The returned runtime supports registration, generated queries and mutations, middleware and scopes, controllers and validators, custom mutations, state machines, nested relationship writes, embedded reconstruction, and native PostgreSQL model handles. Native handles expose `findById`, `find`, `create`, `update`, and `delete`; they are not Mongoose models. `find(args, { session })` accepts the generated GraphQL list query arguments. Existing `pg.Pool` configuration and `pg.PoolClient` query interfaces remain compatible across both factories. A custom pool or client wrapper should pass `pg`'s `command` through in query results: the plugin reads it from the `COMMIT` result to detect a transaction that PostgreSQL rolled back because one of its statements failed, and without it such a transaction reports success with nothing stored.
+Both factories permanently bind one pool and schema, and five optional settings: `multiRowInserts: false` keeps one `INSERT` per owned row instead of multi-row statements; `explicitNullsOrder: true` keeps the `NULLS` clause on sorts of `id` and required fields, as 3.5.9 did, instead of letting indexes serve them; `batchedCollections: true` reads the generated collection fields of the parents that share their arguments with one statement per 100 parents, in a transaction that turns JIT off for that transaction only, instead of one statement per field as 3.6.0 did and as the default still does, and can return unsorted and tied rows in another order; the opt-in `refreshTriggers: 'perRoot'` refreshes each changed root's owned shape and unique keys once per transaction, while its guard trigger stays deferred, instead of once per changed row; and the opt-in `ownerIndexes: 'minimal'` leaves out the `(__owner_id)` indexes that the primary key or a unique index covers on new owned and guard tables. An unknown `refreshTriggers` or `ownerIndexes` value throws `INVALID_DATABASE_SCHEMA` (400). Read the [upgrade notes](https://simtlix.github.io/simfinity.js/resources/compatibility.html#upgrade-to-3-6-0) before enabling the opt-ins: versions up to 3.5.9 cannot validate databases created with them. The returned runtime supports registration, generated queries and mutations, middleware and scopes, controllers and validators, custom queries and mutations, state machines, nested relationship writes, embedded reconstruction, and native PostgreSQL model handles. Native handles expose `findById`, `find`, `create`, `update`, and `delete`; they are not Mongoose models. `find(args, { session })` accepts the generated GraphQL list query arguments. Existing `pg.Pool` configuration and `pg.PoolClient` query interfaces remain compatible across both factories. A custom pool or client wrapper should pass `pg`'s `command` through in query results: the plugin reads it from the `COMMIT` result to detect a transaction that PostgreSQL rolled back because one of its statements failed, and without it such a transaction reports success with nothing stored.
 
 Operations attempted before successful initialization fail with `DATABASE_NOT_INITIALIZED`. Call `createSchema` before `initializeDatabase`. Configuration and registrations become immutable once schema preparation begins. `preventCreatingCollection(true)` makes instance initialization validate existing storage without issuing creation DDL. A rejected initialization request leaves readiness as it was; a ready instance keeps serving while it is initialized again, and becomes unavailable if that initialization fails. `create` mode reads the catalog without table locks and locks nothing when nothing is missing. When it creates objects, writers of the tables that it changes, of existing tables that a new foreign key references, or of roots whose keys it rebuilds wait until it commits, so use `mode: 'validate'` to check a live instance. `simfinity.compileDatabaseSchema()` returns the instance's DDL for review or export.
 
@@ -62,7 +62,7 @@ The module and every `createPostgres` instance expose the shared `auth`, `valida
 MCP integration is an independent opt-in and does not add MCP dependencies to PostgreSQL applications:
 
 ```sh
-npm install @simtlix/simfinity-mcp@3.9.0 @modelcontextprotocol/sdk@^1.31.0
+npm install @simtlix/simfinity-mcp@3.10.0 @modelcontextprotocol/sdk@^1.31.0
 ```
 
 ```javascript
@@ -79,6 +79,8 @@ Queries support nested scalar/reference paths, logical filter groups, scopes, so
 
 The text operators take literal values as bound parameters. `LIKE`, `STARTS_WITH` and `ENDS_WITH` compare code points, and `STARTS_WITH` can use a column's btree index. `ILIKE`, `ISTARTS_WITH`, `IENDS_WITH` and `IEQ` fold both sides with the Unicode simple case foldings that MongoDB's case-insensitive matching applies, through one expression, `translate(lower((x) COLLATE "pg_catalog"."und-x-icu"), from, to)`, where `translate()` maps the 22 lowercase variants that ICU `lower()` keeps, such as `ſ`, `µ` and the final sigma `ς`, to their usual letters; so each returns every row that its case-sensitive form returns, and `ISTARTS_WITH "ΚΑΣ"` finds `ΚΑΣΤΡΟ`. The first three match a `LIKE` pattern whose `%`, `_` and `\` are escaped. The two backends still differ on `İ`, which ICU lowers to `i` followed by U+0307, and on letters newer than one side's Unicode tables. The fold needs a `UTF8` database on a server built with ICU: on another, such as one in `LATIN1`, these four operators fail with `UNSUPPORTED_QUERY` (400), and every other operation works. Indexes serve them only on exactly the compiled expression, which the exported `caseFoldExpression(sqlExpression)` returns, such as `caseFoldExpression('"name"')`: a btree with `text_pattern_ops` on it serves `IEQ` and `ISTARTS_WITH`, and a `pg_trgm` GIN index with `gin_trgm_ops` serves `ILIKE`, `ISTARTS_WITH` and `IENDS_WITH`. The [queries guide](https://simtlix.github.io/simfinity.js/guide/queries.html#case-folding) gives the details and the index statements.
 
+`IS_EMPTY` and `IS_NOT_EMPTY` take no value and test the queried record's own fields: a missing, null or `""` String, a missing or null value of any other scalar, a missing, null or `[]` list, and an embedded object that is missing, null or has every stored member empty. They compile from where the path is stored and bind no parameters: `(c IS NULL OR c = '')` for a String column, so that a btree serves it through a `BitmapOr`, `c IS NULL` for any other column, including the uuid identity of a record whose `id` is declared `String`, and `COALESCE(cardinality(c), 0) = 0` for a native array, also on owned rows; a jsonb member is read as text with `->>` and never cast, a jsonb list through a `jsonb_typeof` guard, and a jsonb object as SQL `NULL`, JSON `null` or all of its members empty; an owned single object is empty when its state is not `present` or no owned row has a member that is not empty, and an owned list when its state is not `present` or it has no row, null items included, with a subquery alias of its own at each level of owned storage. `IS_NOT_EMPTY` is the positive form of a leaf and the `NOT` of an object or owned form, so every record matches exactly one of the two. A path that continues inside an embedded list or a collection, such as `lines.qty` or `seasons.year`, fails with the permanent `INVALID_FILTER_PATH` (400), since emptiness of a value inside a list is not defined. In this release, any other path that crosses or ends at a reference, a path that ends at a collection, such as `seasons`, and an embedded object with a reference member fail with `UNSUPPORTED_QUERY` (400). The [queries guide](https://simtlix.github.io/simfinity.js/guide/queries.html) describes the rules.
+
 Errors that PostgreSQL reports become `SimfinityError`s without SQL details, with the driver error as `getCause()`; errors that application code throws propagate unchanged. Text cannot contain NUL and native date-times start at 4714-11-24 BC: such input fails with `INVALID_VALUE` (400).
 
 PostgreSQL enforces required values, real FKs, JSONB/owned embedded shapes, root and embedded scalar/reference uniqueness, and scalar-list multikey uniqueness. Root unique indexes use `NULLS NOT DISTINCT`; private typed owner-key tables allow repeated keys within one owner but reject the same key across owners. Presence markers preserve missing versus explicit null fields. This is stronger and broader than generated Mongoose storage.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@simtlix/simfinity-postgres",
-  "version": "3.9.0",
+  "version": "3.10.0",
   "description": "PostgreSQL GraphQL runtime and schema generation with foreign keys for Simfinity",
   "type": "module",
   "main": "./src/index.js",
@@ -27,8 +27,8 @@
     "directory": "packages/postgres"
   },
   "dependencies": {
-    "@simtlix/simfinity-core": "3.9.0",
-    "@simtlix/simfinity-sql": "3.9.0",
+    "@simtlix/simfinity-core": "3.10.0",
+    "@simtlix/simfinity-sql": "3.10.0",
     "pg": "^8.16.3"
   },
   "peerDependencies": {

package/src/index.js

@@ -20,7 +20,7 @@ export {
 /** Each instance permanently binds one pool/schema; getModel exposes PostgreSQL native helpers. */
 export const createPostgres = (options) => createSQL({ plugin: postgresPlugin(options) });
 const defaultInstance = createPostgres();
-export const { configureQueryLimits, configureMutationLimits, configure, connect, addNoEndpointType, createSchema, getModel, getType, getInputType, getRegistrations, use, registerMutation, saveObject, preventCreatingCollection, withTransaction } = defaultInstance;
+export const { configureQueryLimits, configureMutationLimits, configure, connect, addNoEndpointType, createSchema, getModel, getType, getInputType, getRegistrations, use, registerMutation, registerCustomQuery, saveObject, preventCreatingCollection, withTransaction } = defaultInstance;
 
 /** Preserve the foundation low-level API while allowing initialization of the configured default instance. */
 export const initializeDatabase = (poolOrOptions, description, options) => description

package/src/plugin.js

@@ -22,7 +22,7 @@ export const postgresPlugin = (options) => ({
   // QLOperator value is rejected until the compiler supports it. The declaration names the compileQuery it describes,
   // so a copy of this plugin with another compileQuery compiles the operators of 3.7.0 until it declares its own.
   queryFeatures: {
-    operators: ['EQ', 'NE', 'LT', 'LTE', 'GT', 'GTE', 'BTW', 'IN', 'NIN', 'LIKE', 'ILIKE', 'STARTS_WITH', 'ISTARTS_WITH', 'ENDS_WITH', 'IENDS_WITH', 'IEQ'],
+    operators: ['EQ', 'NE', 'LT', 'LTE', 'GT', 'GTE', 'BTW', 'IN', 'NIN', 'LIKE', 'ILIKE', 'STARTS_WITH', 'ISTARTS_WITH', 'ENDS_WITH', 'IENDS_WITH', 'IEQ', 'IS_EMPTY', 'IS_NOT_EMPTY'],
     compileQuery,
   },
   naming: { validateIdentifier: identifier, generatedName },

package/src/query/compiler.js

@@ -1,5 +1,7 @@
 import { resolveModelPath, SimfinityError } from '@simtlix/simfinity-core';
-import { textQueryOperator, validateTextFilterValue } from '@simtlix/simfinity-core/internal/query-operators';
+import {
+  describeEmptiness, describeModelEmptiness, isEmptinessOperator, textQueryOperator, validateTextFilterValue,
+} from '@simtlix/simfinity-core/internal/query-operators';
 import { identifier as q, literal, qualified } from '../schema/sql.js';
 import { encodeScalar, storageTypes, invalidValue } from '../codecs.js';
 import { caseFoldExpression as fold } from './case-fold.js';
@@ -88,6 +90,12 @@ const compileSelect = (models, database, plan, extra, configuration, batched) =>
     }
     return joins.get(key).alias;
   };
+  // The owned table that stores `field` of the row `context`, or undefined when the row stores it; values inside
+  // jsonb never own tables.
+  const ownedOf = (context, field) => context.table && database.tables.find((item) => item.ownership?.ownerTable === context.table.name && item.ownership.field === field.name);
+  // The LEFT JOIN of an owned single object, one per path, which filters and the emptiness forms share. A missed
+  // join, and an object in another state, read as NULL.
+  const joinOwned = (key, context, owned) => join(key, owned.name, (name) => `${col(name, '__owner_id')} = ${col(context.alias, context.table.primaryKey.columns[0])} AND ${col(context.alias, owned.ownership.stateColumn)} = 'present'`);
   const arrayFrom = (vector, expression) => `ARRAY(SELECT ${expression} FROM ${vector.from} WHERE ${vector.where})`;
   const jsonItems = (expression, missing = '[]') => `jsonb_array_elements(CASE WHEN jsonb_typeof(${expression}) = 'array' THEN ${expression} ELSE '${missing}'::jsonb END)`;
   const helper = (name, ...args) => `${qualified(database.schema, `__simfinity_${name}`)}(${args.join(', ')})`;
@@ -146,8 +154,7 @@ const compileSelect = (models, database, plan, extra, configuration, batched) =>
           const state = `${col(ownerAlias, owned.ownership.stateColumn)} = 'present'`;
           const presence = (name) => owned.ownership.nullableItems && usage !== 'sort' ? ` AND ${col(name, '__item_present')}` : '';
           if (!field.list && !context.vector) {
-            const joined = join(key, owned.name, (name) => `${col(name, '__owner_id')} = ${ownerId} AND ${state}`);
-            context = { table: owned, alias: joined, json: null, vector: null };
+            context = { table: owned, alias: joinOwned(key, context, owned), json: null, vector: null };
           } else {
             const name = alias();
             const stored = alias();
@@ -319,9 +326,84 @@ const compileSelect = (models, database, plan, extra, configuration, batched) =>
     const left = ignoreCase ? uncollated : sql;
     return negated ? `COALESCE(${compare(left)}, FALSE)` : compare(left);
   };
-  // A negation reaches every predicate below it, including those nested in AND/OR groups.
+  /**
+   * The IS_EMPTY form of `field`, stored on the row or jsonb value `context`, by its emptiness shape, or with `empty`
+   * false the IS_NOT_EMPTY form: the positive form of a leaf, which a btree can serve, and the NOT of an object or an
+   * owned form. Every form is two-valued and binds no parameters. A leaf is a column of any row, or the text of a
+   * jsonb member, never cast, since a cast can fail; a list a native array column, a jsonb list, or an owned list,
+   * whose null items are rows too, so `[null]` is not empty; an object a jsonb value or an owned single object whose
+   * stored members are all empty. An owned table is read in a subquery with an alias of its own, correlated to the
+   * key of the row that owns it: the root's id, or the `__id` of a joined or enclosing owned row.
+   */
+  const emptyForm = (context, field, shape, empty = true) => {
+    const compare = empty ? '=' : '>';
+    const negate = (form) => (empty ? form : `NOT ${form}`);
+    const leaf = (value) => {
+      if (shape.rule === 'string') return empty ? `(${value} IS NULL OR ${value} = '')` : `(${value} IS NOT NULL AND ${value} <> '')`;
+      return `${value} IS ${empty ? '' : 'NOT '}NULL`;
+    };
+    // The CASE keeps jsonb_array_length from failing (22023) on a value that is not an array.
+    const jsonList = (json) => `COALESCE(jsonb_array_length(CASE WHEN jsonb_typeof(${json}) = 'array' THEN ${json} END), 0) ${compare} 0`;
+    // An embedded type without stored members is always empty.
+    const members = (row) => {
+      const forms = shape.members.map((member) => {
+        const stored = field.fields?.find((item) => item.name === member.name);
+        if (!stored) unsupported('Invalid query path');
+        return emptyForm(row, stored, member.shape);
+      });
+      return forms.length ? `(${forms.join(' AND ')})` : 'TRUE';
+    };
+    const jsonObject = (json) => negate(`(${json} IS NULL OR ${json} = 'null'::jsonb OR ${members({ table: null, alias: null, json })})`);
+    if (context.json) {
+      const json = `(${context.json} -> ${literal(field.storageName)})`;
+      if (shape.rule === 'object') return jsonObject(json);
+      if (shape.rule === 'list') return jsonList(json);
+      return leaf(`(${context.json} ->> ${literal(field.storageName)})`);
+    }
+    const owned = ownedOf(context, field);
+    if (owned) {
+      // A missed join leaves the state NULL, which is not 'present'.
+      const state = `${col(context.alias, owned.ownership.stateColumn)} IS DISTINCT FROM 'present'`;
+      const name = alias();
+      const rows = `SELECT 1 FROM ${qualified(database.schema, owned.name)} ${name} WHERE ${col(name, '__owner_id')} = ${col(context.alias, context.table.primaryKey.columns[0])}`;
+      if (shape.rule === 'list') return negate(`(${state} OR NOT EXISTS (${rows}))`);
+      // An object marked present without a row, which the owned-shape check rejects, is empty too.
+      return negate(`(${state} OR NOT EXISTS (${rows} AND NOT ${members({ table: owned, alias: name, json: null })}))`);
+    }
+    // The record identity is the uuid primary key, whatever scalar it is declared with.
+    const identity = !context.table.ownership && ['id', '_id'].includes(field.name);
+    const column = col(context.alias, identity ? 'id' : field.storageName);
+    if (field.kind === 'embedded') return shape.rule === 'object' ? jsonObject(column) : jsonList(column);
+    if (shape.rule === 'list') return `COALESCE(cardinality(${column}), 0) ${compare} 0`;
+    return leaf(column);
+  };
+  /**
+   * IS_EMPTY and IS_NOT_EMPTY (#108, part 2a). Every emptiness predicate, leaves included, compiles here from where
+   * its path is stored, not from path(), whose values cast jsonb members and materialize jsonb lists, and which
+   * rejects paths that end at an embedded field. The checks of createQueryPlan run again, with its codes, on the
+   * entity's GraphQL type or on the field descriptions, so a plan built by hand gets the same answers; they leave
+   * only single embedded objects before the last field, reached through the LEFT JOIN of each owned single object.
+   */
+  const emptiness = ({ path: parts, operator, value }) => {
+    const steps = resolveModelPath(models, plan.entity, parts);
+    const options = { operator, value, path: parts.join('.') };
+    const gqltype = models.entities.find((item) => item.name === plan.entity)?.gqltype;
+    const shape = gqltype ? describeEmptiness(gqltype, parts, options) : describeModelEmptiness(steps, options);
+    let context = { table: root, alias: 't0', json: null };
+    steps.slice(0, -1).forEach((field, index) => {
+      if (field.kind !== 'embedded' || field.list) unsupported('Invalid query path');
+      const owned = ownedOf(context, field);
+      context = owned
+        ? { table: owned, alias: joinOwned(parts.slice(0, index + 1).join('.'), context, owned), json: null }
+        : { table: null, alias: null, json: context.json ? `(${context.json} -> ${literal(field.storageName)})` : col(context.alias, field.storageName) };
+    });
+    return emptyForm(context, steps.at(-1), shape, operator === 'IS_EMPTY');
+  };
+  // A negation reaches every predicate below it, including those nested in AND/OR groups; the emptiness forms are
+  // two-valued, so they need no null-safe form under it.
   const where = (node, negated = false) => {
     if (!node) return 'TRUE';
+    if (node.kind === 'predicate' && isEmptinessOperator(node.operator)) return `(${emptiness(node)})`;
     if (node.kind === 'predicate') return `(${predicate(path(node.path), node.operator, node.value, negated)})`;
     return `(${node.terms.map((term) => where(term, negated)).join(node.kind === 'and' ? ' AND ' : ' OR ')})`;
   };

package/types/index.d.ts

@@ -17,6 +17,7 @@ export type {
   AuthRule,
   AuthRuleFunction,
   AuthSchemaPlugin,
+  CreateSchemaOptions,
   EntityController,
   EnvelopSchemaPlugin,
   FieldValidations,
@@ -235,6 +236,14 @@ export interface PostgresRuntime extends Runtime<PostgresModel, PostgresSession>
  * `QLOperator` value. The case-insensitive text operators fold case with {@link caseFoldExpression}, which needs a UTF8
  * database with the ICU collation `und-x-icu`: on another database, such as one in LATIN1 or on a server built without
  * ICU, they fail with UNSUPPORTED_QUERY (400), and every other operation works.
+ *
+ * `IS_EMPTY` and `IS_NOT_EMPTY` compile from where a path is stored, binding no parameters: `(c IS NULL OR c = '')`
+ * for a String column and `c IS NULL` for any other, the uuid identity included; `COALESCE(cardinality(c), 0) = 0`
+ * for a native array; jsonb members as text without a cast, and jsonb lists through a `jsonb_typeof` guard; an
+ * embedded object as missing, JSON `null`, or every stored member empty; and owned objects and lists through their
+ * state column and a `NOT EXISTS` subquery on the owned table, null list items counting as items. `IS_NOT_EMPTY` is
+ * the positive form of a leaf, which a btree can serve, and the `NOT` of an object or owned form. Declaring the two
+ * operators does not lift the UNSUPPORTED_QUERY (400) that emptiness through references and collections gets.
  */
 export function postgresPlugin(options?: PostgresConfiguration): SQLPlugin<
   PostgresConfiguration,
@@ -269,6 +278,7 @@ export const getInputType: PostgresRuntime['getInputType'];
 export const getRegistrations: PostgresRuntime['getRegistrations'];
 export const use: PostgresRuntime['use'];
 export const registerMutation: PostgresRuntime['registerMutation'];
+export const registerCustomQuery: PostgresRuntime['registerCustomQuery'];
 export const saveObject: PostgresRuntime['saveObject'];
 export const preventCreatingCollection: PostgresRuntime['preventCreatingCollection'];
 export const withTransaction: PostgresRuntime['withTransaction'];