@simtlix/simfinity-js 3.7.0 → 3.9.0

package/README.md

@@ -24,6 +24,10 @@ Generated single-reference reads can be batched within a request. Models with an
 
 Generated models store enum internal values with their own type (numbers, booleans or strings; other combinations as given) and support embedded fields named `type`. Embedded types that contain each other are rejected with `INVALID_MODEL` (`Embedded cycle at …`) when a generated model reaches them, and always in transactional mode; in the default mode, a cycle that only supplied models reach is not checked. Range filters, sorts and aggregates on numeric and boolean enums follow that type; PostgreSQL uses their text form. Earlier versions stored numeric and boolean enum values as strings; convert existing documents once as described in the [schema guide](https://simtlix.github.io/simfinity.js/guide/schema.html#what-gets-generated).
 
+Virtual fields (`extensions: { virtual: true }`, with the field's own `resolve`) get no path, no `[]` default and no index in generated models, on entities and embedded types, and no entry in the transactional integrity registry, so they need no `relation` in either mode and documents look the same with or without them. Generated writes never set or unset them, and Mongoose's strict mode drops a value that a controller assigns to a generated document for one. An embedded type whose members are all virtual becomes a strict subdocument without paths rather than a Mixed path, so nothing is stored for it but the `_id` of list items. With a generated model, values stored before a field became virtual stay in the documents: Mongoose drops an `$unset` of a path the model does not declare, so remove them with a driver-level update, such as `getModel(Type).collection.updateMany({}, { $unset: { busyRanges: '' } })`, or with `updateMany(filter, update, { strict: false })`; a generated update that sets an embedded parent rewrites it, which drops the stored keys of its virtual members. A supplied model keeps the paths it declares: documents stored earlier keep their values for the field, which nothing reads or writes until `$unset` removes them, and an index on such a path stays until you drop it. `buildQuery()` rejects a path through a virtual field with `INVALID_FILTER_FIELD` (400).
+
+Text filters compile to escaped regular expressions, so a value never becomes regex syntax: `LIKE` and `ILIKE` match anywhere, `STARTS_WITH` and `ISTARTS_WITH` are anchored with `^`, `ENDS_WITH` and `IENDS_WITH` with `\z` (so a trailing newline is part of the value), and `IEQ` with both. The `I` operators add the `i` option, which applies the Unicode simple case folding of the server's PCRE2 build (`É`/`é`, `ẞ`/`ß`, `σ`/`ς`, `µ`/`μ`, `ſ`/`s`), never a length-changing or Turkic folding (`ß`/`ss`, `İ`/`i`), and does not fold characters newer than that build, such as U+A7DC; the [queries guide](https://simtlix.github.io/simfinity.js/guide/queries.html) lists where PostgreSQL differs. Only `STARTS_WITH` reads a prefix range of an index on the field, a String list or an embedded path; the other text operators test every key of such an index, or every document without one, and `$regex` ignores collations, so a case-insensitive collation index does not serve them. A value whose escaped pattern is longer than MongoDB's 32,764-byte pattern limit fails with `INVALID_FILTER_VALUE` (400) before the query runs, also for `LIKE`. `createMongoAdapter()` declares every operator it compiles in `queryFeatures`; a copy of the adapter without it supports only the operators of 3.7.0.
+
 The facade registers generated models on your application's default Mongoose instance and does not change global Mongoose options such as `strictQuery`. Supplied models may use another connection, such as one from `mongoose.createConnection()`: generated mutations and `saveObject()` use the model's connection, and, in the default `referentialIntegrity: 'off'` mode, custom mutations use the default connection unless it is unused, never opened and without compiled models, also on its `useDb()` descendants, and the registered models share one MongoDB client; with transactional reference integrity they use the first protected model's connection. `createMongoAdapter().castId` accepts only an ObjectId or its 24-character hexadecimal string and throws `NOT_VALID_ID` (400) otherwise; create new identifiers with `new mongoose.Types.ObjectId()`. Owned transactions retry transient failures after a short random delay, and a write conflict that outlasts the retries fails with `TRANSACTION_RETRY_EXCEEDED` (409).
 
 Mongoose 8.24.2 or later within version 8 is required to exclude the update-casting prototype-pollution vulnerability. Existing applications should update Mongoose and their dependency lockfile, then run `npm audit`; upgrading Simfinity does not refresh every transitive dependency already locked by the application.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@simtlix/simfinity-js",
-  "version": "3.7.0",
+  "version": "3.9.0",
   "description": "MongoDB/Mongoose adapter and compatibility facade for Simfinity GraphQL APIs",
   "main": "src/index.js",
   "types": "./types/index.d.ts",
@@ -29,8 +29,8 @@
     "owner": "simtlix"
   },
   "dependencies": {
-    "@simtlix/simfinity-core": "3.7.0",
-    "@simtlix/simfinity-mcp": "3.7.0"
+    "@simtlix/simfinity-core": "3.9.0",
+    "@simtlix/simfinity-mcp": "3.9.0"
   },
   "files": [
     "src",

package/src/mongo/adapter.js

@@ -3,7 +3,7 @@ import { SimfinityError } from '@simtlix/simfinity-core';
 import { getListShape, normalizeConnectionField } from '@simtlix/simfinity-core/internal/relation-storage';
 
 import { createMongoModel } from './models.js';
-import { createMongoQueries } from './queries.js';
+import { MONGO_QUERY_OPERATORS, createMongoQueries } from './queries.js';
 import { withMongoTransaction } from './transactions.js';
 import { createMongoIntegrity } from './integrity.js';
 import {
@@ -12,6 +12,9 @@ import {
 
 const withSession = (query, session) => (session ? query.session(session) : query);
 
+// Every filter operator the query compiler supports; the runtime rejects any other one.
+const QUERY_FEATURES = Object.freeze({ operators: MONGO_QUERY_OPERATORS });
+
 // A write that fails only on malformed ObjectId values reports NOT_VALID_ID, not a driver error.
 const mapIdCastErrors = async (write) => {
   try {
@@ -216,6 +219,7 @@ export const createMongoAdapter = (options) => {
   };
 
   const adapter = {
+    queryFeatures: QUERY_FEATURES,
     bind(binding) {
       integrity.bind(binding);
       getRegistrations = binding.getRegistrations;

package/src/mongo/models.js

@@ -12,7 +12,7 @@ import {
 } from 'graphql';
 import mongoose from 'mongoose';
 import { SimfinityError } from '@simtlix/simfinity-core';
-import { getFieldStorageName } from '@simtlix/simfinity-core/internal/relation-storage';
+import { getFieldStorageName, isVirtualField } from '@simtlix/simfinity-core/internal/relation-storage';
 
 const isNonNullOfType = (fieldEntryType, graphQLType) => (
   fieldEntryType instanceof GraphQLNonNull && fieldEntryType.ofType instanceof graphQLType
@@ -97,11 +97,24 @@ const assertNoEmbeddedCycle = (gqlType, entryType, ancestors, fieldPath) => {
   if (ancestors.includes(entryType)) throw new SimfinityError(`Embedded cycle at ${fieldPath}`, 'INVALID_MODEL', 400);
 };
 
+// Mongoose reads an empty definition, `{}` or `[{}]`, as Mixed, which stores any keys. An embedded
+// type whose members are all virtual has such a definition, so it gets a strict subdocument without
+// paths instead: no value of its virtual members, or any other key, is stored. A list item keeps its
+// `_id`, as other embedded list items do. Embedded types without virtual members keep their
+// definition, even an empty one.
+const storedEmbeddedDefinition = (entryType, definition, list) => {
+  if (Object.keys(definition).length || !Object.values(entryType.getFields()).some(isVirtualField)) return definition;
+  return list ? new mongoose.Schema({}) : new mongoose.Schema({}, { _id: false });
+};
+
 const generateSchemaDefinition = (gqlType, nested = false, ancestors = [], path = gqlType.name) => {
   const argTypes = gqlType.getFields();
   const schemaArg = {};
 
   for (const [fieldEntryName, fieldEntry] of Object.entries(argTypes)) {
+    // A virtual field is resolved by its own resolver and stored nowhere, on entities and embedded
+    // types alike: it gets no path, so no `[]` default and no ObjectId index either.
+    if (isVirtualField(fieldEntry)) continue;
     const { type } = fieldEntry;
 
     if (matchesScalar(type, GraphQLID)) {
@@ -123,9 +136,10 @@ const generateSchemaDefinition = (gqlType, nested = false, ancestors = [], path
           const entryType = unwrapNonNull(type);
           const fieldPath = `${path}.${fieldEntryName}`;
           assertNoEmbeddedCycle(gqlType, entryType, ancestors, fieldPath);
-          const definition = generateSchemaDefinition(entryType, true, [...ancestors, gqlType], fieldPath);
+          const definition = storedEmbeddedDefinition(entryType,
+            generateSchemaDefinition(entryType, true, [...ancestors, gqlType], fieldPath), false);
           // A nested object under a `type` key can only be a subdocument; like other embedded objects, it has no _id.
-          schemaArg[fieldEntryName] = nested && fieldEntryName === 'type'
+          schemaArg[fieldEntryName] = nested && fieldEntryName === 'type' && !(definition instanceof mongoose.Schema)
             ? new mongoose.Schema(definition, { _id: false })
             : definition;
         }
@@ -137,7 +151,8 @@ const generateSchemaDefinition = (gqlType, nested = false, ancestors = [], path
           assertEmbeddedFieldName(gqlType, fieldEntryName);
           const fieldPath = `${path}.${fieldEntryName}`;
           assertNoEmbeddedCycle(gqlType, itemType, ancestors, fieldPath);
-          schemaArg[fieldEntryName] = [generateSchemaDefinition(itemType, true, [...ancestors, gqlType], fieldPath)];
+          schemaArg[fieldEntryName] = [storedEmbeddedDefinition(itemType,
+            generateSchemaDefinition(itemType, true, [...ancestors, gqlType], fieldPath), true)];
         }
       } else if (listItemMatchesScalar(type, GraphQLID)) {
         schemaArg[fieldEntryName] = [mongoose.Schema.Types.ObjectId];
@@ -186,7 +201,7 @@ const findObjectIdFields = (schemaDefinition, parentPath = '') => {
     if (definition === mongoose.Schema.Types.ObjectId) {
       objectIdFields.push(currentPath);
     } else if (Array.isArray(definition)) {
-      const arrayElement = definition[0];
+      const arrayElement = definition[0] instanceof mongoose.Schema ? definition[0].obj : definition[0];
       if (typeof arrayElement === 'object' && arrayElement !== null) {
         objectIdFields.push(...findObjectIdFields(arrayElement, currentPath));
       }

package/src/mongo/queries.js

@@ -9,7 +9,10 @@ import mongoose from 'mongoose';
 
 import { SimfinityError, QLOperator, paginationStages } from '@simtlix/simfinity-core';
 import {
-  getFieldStorageName, getListShape, hasStoredIdentity, normalizeConnectionField,
+  TEXT_QUERY_OPERATORS, textQueryOperator, validateTextFilterValue,
+} from '@simtlix/simfinity-core/internal/query-operators';
+import {
+  getFieldStorageName, getListShape, hasStoredIdentity, isVirtualField, normalizeConnectionField,
 } from '@simtlix/simfinity-core/internal/relation-storage';
 
 import { castObjectId } from './ids.js';
@@ -26,6 +29,15 @@ const isGraphQLisoDate = (typeName) => (
 // GraphQL reserves `__` names, so these aliases never replace a stored field.
 const LOOKUP_ALIAS_PREFIX = '__sf_l';
 
+/**
+ * The filter operators this compiler supports, which the adapter declares as its queryFeatures. They
+ * are listed explicitly, so that a QLOperator value added later stays unsupported until it compiles.
+ */
+export const MONGO_QUERY_OPERATORS = Object.freeze([
+  'EQ', 'NE', 'LT', 'LTE', 'GT', 'GTE', 'BTW', 'IN', 'NIN', 'LIKE',
+  'ILIKE', 'STARTS_WITH', 'ISTARTS_WITH', 'ENDS_WITH', 'IENDS_WITH', 'IEQ',
+]);
+
 export const createMongoQueries = ({ getModel, getRegistrations }) => {
 // One alias per relation path for each query build, keyed by the build's `included` accumulator.
 const lookupAliases = new WeakMap();
@@ -62,6 +74,34 @@ const buildRelationLookup = ({
 
 const escapeRegex = (s) => String(s).replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
 
+// MongoDB rejects a $regex pattern longer than 32,764 UTF-8 bytes on the server (Location51091),
+// which would report a client's long value as an internal error. Escaping adds a byte to each
+// metacharacter and the operators add their anchors, so the limit applies to the compiled pattern.
+const MAX_REGEX_PATTERN_BYTES = 32764;
+// A $regex match, rejected with INVALID_FILTER_VALUE before the query runs when MongoDB would reject
+// its pattern. `path` is the condition's GraphQL path, for the message.
+const regexMatch = (operator, pattern, ignoreCase, path) => {
+  const bytes = Buffer.byteLength(pattern, 'utf8');
+  if (bytes > MAX_REGEX_PATTERN_BYTES) {
+    const where = path == null ? '' : ` on ${path}`;
+    throw filterError(`${operator} value${where} is too long: MongoDB accepts regular expressions of at most ${MAX_REGEX_PATTERN_BYTES} bytes, and this one has ${bytes}`);
+  }
+  return ignoreCase ? { $regex: pattern, $options: 'i' } : { $regex: pattern };
+};
+
+// The literal text operators: the escaped value, anchored with `^` at the start of the string and
+// `\z` at its very end (`$` would also match before a final newline), with the `i` option to ignore
+// case. Without the `m` option, `^` never matches after a newline.
+const TEXT_PATTERNS = {
+  contains: (escaped) => escaped,
+  prefix: (escaped) => `^${escaped}`,
+  suffix: (escaped) => `${escaped}\\z`,
+  exact: (escaped) => `^${escaped}\\z`,
+};
+const textRegex = (operator, { match, ignoreCase }) => (value, path) => (
+  regexMatch(operator, TEXT_PATTERNS[match](escapeRegex(value)), ignoreCase, path)
+);
+
 const PATH_SEGMENT_RE = /^[A-Za-z_][A-Za-z0-9_]*$/;
 
 const assertValidFilterPath = (path) => {
@@ -101,7 +141,8 @@ const OP_TO_MONGO = {
   GTE: (v) => ({ $gte: v }),
   NE: (v) => ({ $ne: v }),
   BTW: (v) => ({ $gte: v[0], $lte: v[1] }),
-  LIKE: (v) => ({ $regex: `.*${escapeRegex(v)}.*` }),
+  LIKE: (v, path) => regexMatch('LIKE', `.*${escapeRegex(v)}.*`, false, path),
+  ...Object.fromEntries(Object.entries(TEXT_QUERY_OPERATORS).map(([op, text]) => [op, textRegex(op, text)])),
 };
 
 const AGG_OP_TO_MONGO = {
@@ -140,6 +181,12 @@ const resolveQueryPath = (gqltype, path, aggregationsIncluded) => {
   for (const [index, part] of parts.entries()) {
     const field = Object.hasOwn(currentType.getFields(), part) ? currentType.getFields()[part] : null;
     if (!field) throw filterError(`Unknown query field: ${path}`, 'INVALID_FILTER_FIELD');
+    // Core rejects generated and trusted paths through a virtual field before the adapter runs. A
+    // native buildQuery() or buildFilterGroupMatch() call fails here, instead of matching storage
+    // that does not exist.
+    if (isVirtualField(field)) {
+      throw filterError(`Query path ${path} uses the virtual field ${currentType.name}.${part}, which is not stored`, 'INVALID_FILTER_FIELD');
+    }
     const fieldType = getNamedType(field.type);
     const relation = field.extensions?.relation;
     if (fieldType instanceof GraphQLObjectType) {
@@ -219,11 +266,18 @@ const castFilterValue = (value, resolved) => {
   }
 };
 
-const buildMatchesClause = (resolved, operator, value) => {
+// `path` is the condition's GraphQL path, for messages.
+const buildMatchesClause = (resolved, operator, value, path) => {
   const op = operator ?? 'EQ';
   if (!QLOperator.getValues().some((entry) => entry.value === op)) {
     throw filterError(`Unsupported filter operator: ${op}`, 'INVALID_FILTER_OPERATOR');
   }
+  // Text operators take the value as is, once core's shared rule accepts it: a String field (a
+  // validated scalar by its base scalar) and a string without U+0000.
+  if (textQueryOperator(op)) {
+    const text = validateTextFilterValue(op, value, getEffectiveTypeName(resolved.fieldType), path);
+    return { [resolved.mongoPath]: OP_TO_MONGO[op](text, path) };
+  }
   const collectionOperator = op === 'IN' || op === 'NIN' || op === 'BTW';
   if (collectionOperator && (!Array.isArray(value) || (op === 'BTW' && value.length !== 2))) {
     throw filterError(`${op} requires ${op === 'BTW' ? 'exactly two values' : 'an array value'}`);
@@ -239,7 +293,7 @@ const buildMatchesClause = (resolved, operator, value) => {
   const coerced = collectionOperator ? value.map(cast) : op === 'LIKE' ? value : cast(value);
   if (op === 'EQ') return { [resolved.mongoPath]: coerced };
   if (op === 'IN' || op === 'NIN') return { [resolved.mongoPath]: { [op === 'IN' ? '$in' : '$nin']: coerced } };
-  return { [resolved.mongoPath]: OP_TO_MONGO[op](coerced) };
+  return { [resolved.mongoPath]: OP_TO_MONGO[op](coerced, path) };
 };
 
 const buildQueryTerms = async (filterField, qlField, fieldName, gqltype, aggregationsIncluded) => {
@@ -256,9 +310,10 @@ const buildQueryTerms = async (filterField, qlField, fieldName, gqltype, aggrega
   for (const term of terms) {
     if (!term || typeof term !== 'object' || Array.isArray(term)) throw filterError(`Invalid term for ${fieldName}`);
     if (isObject) assertValidFilterPath(term.path);
-    const resolved = resolveQueryPath(gqltype, isObject ? `${fieldName}.${term.path}` : fieldName, aggregationsIncluded);
+    const path = isObject ? `${fieldName}.${term.path}` : fieldName;
+    const resolved = resolveQueryPath(gqltype, path, aggregationsIncluded);
     Object.assign(aggregateClauses, resolved.aggregateClauses);
-    matchesClauses.push(buildMatchesClause(resolved, term.operator, term.value));
+    matchesClauses.push(buildMatchesClause(resolved, term.operator, term.value, path));
   }
   return { aggregateClauses, matchesClauses };
 };

package/types/index.d.ts

@@ -15,6 +15,7 @@ import type {
 import type {
   DatabaseAdapter,
   MutationLimitsOptions,
+  QueryFeatures,
   RuntimeRegistration,
   SimfinityError,
 } from '@simtlix/simfinity-core';
@@ -46,6 +47,8 @@ export type {
   ScopeParams,
   TypePermissions,
   TypeScopes,
+  VirtualFieldDeclaration,
+  VirtualFieldOptions,
 } from '@simtlix/simfinity-core';
 export * from '@simtlix/simfinity-mcp';
 export { default as mcp } from '@simtlix/simfinity-mcp';
@@ -86,6 +89,13 @@ export interface MongoAdapterOptions {
  */
 export interface MongoAdapter extends DatabaseAdapter {
   readonly referentialIntegrity: 'off' | 'transactional';
+  /**
+   * Frozen; lists every `QLOperator` value. The literal text operators compile to escaped regular
+   * expressions: `ILIKE` unanchored, `STARTS_WITH` and `ISTARTS_WITH` anchored with `^`, `ENDS_WITH` and
+   * `IENDS_WITH` with `\z`, and `IEQ` with both, with the `i` option for the case-insensitive ones. A
+   * copy of the adapter without this member supports only the operators of 3.7.0.
+   */
+  readonly queryFeatures: QueryFeatures;
   /**
    * Accepts only an ObjectId or a 24-character hexadecimal string, normalized to lowercase, and
    * throws `NOT_VALID_ID` (400) for anything else, including undefined, null and numbers. It never
@@ -163,6 +173,13 @@ export function addNoEndpointType(gqltype: GraphQLObjectType): void;
  * Embedded types that contain each other throw INVALID_MODEL (400, `Embedded cycle at …`) when a
  * generated model's embedded fields reach them, and always in transactional mode; in default mode a
  * cycle that only supplied models reach is not checked.
+ * Invalid virtual field declarations (`extensions.virtual`, see `VirtualFieldOptions`) throw
+ * INVALID_MODEL (400) before models are created, in both referentialIntegrity modes. A virtual field
+ * gets no path, `[]` default or index in generated models, on entities and embedded types, and no
+ * entry in the transactional integrity registry, so it needs no `relation` in either mode; generated
+ * writes never set or unset it. A supplied model keeps the paths it declares. A readOnly field with
+ * arguments and its own resolver that is not virtual, and is not a reference or collection, logs a
+ * `Configuration issue` warning once.
  * A generated input that would have no fields is left out with a warning, together with the
  * collection operation, embedded field or add/update/state-action mutation that would take it; a
  * non-null embedded field whose embedded type has no writable fields throws INVALID_MODEL (400)
@@ -296,6 +313,7 @@ export interface FilterGroupInput {
  * pipeline. `input.aggregation` filters a field named `aggregation`; the aggregate stage is never
  * built here. An embedded `.id` path uses the declared `id` member, except for entity types embedded
  * as lists and supplied subdocument schemas without an `id` path, which use the subdocument `_id`.
+ * A filter, condition or sort path through a virtual field rejects with `INVALID_FILTER_FIELD` (400).
  */
 export function buildQuery(
   input: Record<string, any>,
@@ -303,7 +321,10 @@ export function buildQuery(
   isCount?: boolean,
 ): Promise<Array<Record<string, any>>>;
 
-/** Translate a QLFilterGroup into a MongoDB `$match` expression, collecting required lookups. */
+/**
+ * Translate a QLFilterGroup into a MongoDB `$match` expression, collecting required lookups. A
+ * condition path through a virtual field rejects with `INVALID_FILTER_FIELD` (400).
+ */
 export function buildFilterGroupMatch(
   filterGroup: FilterGroupInput,
   gqltype: GraphQLObjectType,