@simtlix/simfinity-js 3.6.0 → 3.8.0

package/README.md

@@ -24,10 +24,25 @@ Generated single-reference reads can be batched within a request. Models with an
 
 Generated models store enum internal values with their own type (numbers, booleans or strings; other combinations as given) and support embedded fields named `type`. Embedded types that contain each other are rejected with `INVALID_MODEL` (`Embedded cycle at …`) when a generated model reaches them, and always in transactional mode; in the default mode, a cycle that only supplied models reach is not checked. Range filters, sorts and aggregates on numeric and boolean enums follow that type; PostgreSQL uses their text form. Earlier versions stored numeric and boolean enum values as strings; convert existing documents once as described in the [schema guide](https://simtlix.github.io/simfinity.js/guide/schema.html#what-gets-generated).
 
+Virtual fields (`extensions: { virtual: true }`, with the field's own `resolve`) get no path, no `[]` default and no index in generated models, on entities and embedded types, and no entry in the transactional integrity registry, so they need no `relation` in either mode and documents look the same with or without them. Generated writes never set or unset them, and Mongoose's strict mode drops a value that a controller assigns to a generated document for one. An embedded type whose members are all virtual becomes a strict subdocument without paths rather than a Mixed path, so nothing is stored for it but the `_id` of list items. With a generated model, values stored before a field became virtual stay in the documents: Mongoose drops an `$unset` of a path the model does not declare, so remove them with a driver-level update, such as `getModel(Type).collection.updateMany({}, { $unset: { busyRanges: '' } })`, or with `updateMany(filter, update, { strict: false })`; a generated update that sets an embedded parent rewrites it, which drops the stored keys of its virtual members. A supplied model keeps the paths it declares: documents stored earlier keep their values for the field, which nothing reads or writes until `$unset` removes them, and an index on such a path stays until you drop it. `buildQuery()` rejects a path through a virtual field with `INVALID_FILTER_FIELD` (400).
+
 The facade registers generated models on your application's default Mongoose instance and does not change global Mongoose options such as `strictQuery`. Supplied models may use another connection, such as one from `mongoose.createConnection()`: generated mutations and `saveObject()` use the model's connection, and, in the default `referentialIntegrity: 'off'` mode, custom mutations use the default connection unless it is unused, never opened and without compiled models, also on its `useDb()` descendants, and the registered models share one MongoDB client; with transactional reference integrity they use the first protected model's connection. `createMongoAdapter().castId` accepts only an ObjectId or its 24-character hexadecimal string and throws `NOT_VALID_ID` (400) otherwise; create new identifiers with `new mongoose.Types.ObjectId()`. Owned transactions retry transient failures after a short random delay, and a write conflict that outlasts the retries fails with `TRANSACTION_RETRY_EXCEEDED` (409).
 
 Mongoose 8.24.2 or later within version 8 is required to exclude the update-casting prototype-pollution vulnerability. Existing applications should update Mongoose and their dependency lockfile, then run `npm audit`; upgrading Simfinity does not refresh every transitive dependency already locked by the application.
 
+## Optional batched collection reads
+
+Generated collection fields read one parent at a time by default. With `batchedCollections: true`, the adapter reads the collection fields of many parents that share their arguments with aggregate commands of up to `collectionBatchSize` parents each (an integer from 1 to 100, default 5), sent concurrently:
+
+```javascript
+import { createMongoAdapter, createRuntime } from '@simtlix/simfinity-js';
+
+const adapter = createMongoAdapter({ batchedCollections: true }); // collectionBatchSize: 5
+const simfinity = createRuntime(adapter);
+```
+
+Each parent's own pipeline (filters, sort and page) runs as one sub-pipeline: the first is the command's pipeline and the others are `$unionWith` stages. Like `referentialIntegrity`, the option needs your own adapter and runtime; the package-root runtime (`simfinity.connect()` and the rest) keeps one read per parent. Children of models with `aggregate` middleware, their own `aggregate` static, a discriminator child model or a wrapped `Model.collection.aggregate`, reads inside a `transactionAsyncLocalStorage` transaction, and adapter copies or overrides of `findChildren` keep one read per parent. A command runs its sub-pipelines one after another, so fewer commands can come with a slower single request, and command monitoring, query settings and `$queryStats` see the new command shape: read the [upgrade notes](https://simtlix.github.io/simfinity.js/resources/compatibility.html#upgrade-to-3-7-0) before enabling it. Invalid values throw `INVALID_MONGO_ADAPTER_OPTIONS` (400).
+
 ## Optional transactional reference integrity
 
 Choose the mode once when creating an adapter; the default remains `'off'`:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@simtlix/simfinity-js",
-  "version": "3.6.0",
+  "version": "3.8.0",
   "description": "MongoDB/Mongoose adapter and compatibility facade for Simfinity GraphQL APIs",
   "main": "src/index.js",
   "types": "./types/index.d.ts",
@@ -29,8 +29,8 @@
     "owner": "simtlix"
   },
   "dependencies": {
-    "@simtlix/simfinity-core": "3.6.0",
-    "@simtlix/simfinity-mcp": "3.6.0"
+    "@simtlix/simfinity-core": "3.8.0",
+    "@simtlix/simfinity-mcp": "3.8.0"
   },
   "files": [
     "src",

package/src/mongo/adapter.js

@@ -53,6 +53,37 @@ const hasNoQueryHooks = (Model, operations) => hasNoHooks(Model.schema?.s?.hooks
 // getById reads through `findOne` and getByIds through `find`.
 const batchesFindOne = (Model) => hasNoQueryHooks(Model, ['find', 'findOne']);
 
+// Batched collection reads (findChildrenBatch), opt in with `batchedCollections`. Each parent's own
+// findChildren pipeline runs as one sub-pipeline of an aggregate command, the first as the command's
+// pipeline and the others in $unionWith stages, at most `collectionBatchSize` parents per command;
+// the commands of one call run concurrently. A command runs its sub-pipelines one after another.
+const DEFAULT_COLLECTION_BATCH_SIZE = 5;
+const MAX_COLLECTION_BATCH_SIZE = 100;
+// Sub-pipeline bytes per command, well below MongoDB's 16 MiB command limit.
+const COLLECTION_COMMAND_BYTES = 8 * 1024 * 1024;
+const invalidAdapterOption = (message) => new SimfinityError(message, 'INVALID_MONGO_ADAPTER_OPTIONS', 400);
+const readCollectionBatching = (options) => {
+  const { batchedCollections = false, collectionBatchSize = DEFAULT_COLLECTION_BATCH_SIZE } = options ?? {};
+  if (typeof batchedCollections !== 'boolean') throw invalidAdapterOption('batchedCollections must be true or false');
+  if (!Number.isSafeInteger(collectionBatchSize) || collectionBatchSize < 1 || collectionBatchSize > MAX_COLLECTION_BATCH_SIZE) {
+    throw invalidAdapterOption(`collectionBatchSize must be an integer from 1 to ${MAX_COLLECTION_BATCH_SIZE}`);
+  }
+  return { enabled: batchedCollections, parentsPerCommand: collectionBatchSize };
+};
+
+// Mongoose applies aggregate middleware, a model's own aggregate static (schema.static, loadClass, or
+// a model from another Mongoose copy), a discriminator's key and a wrapper of the model's
+// collection.aggregate to the command's pipeline only, never to $unionWith sub-pipelines, so such
+// models read their children one parent at a time.
+const batchesAggregate = (Model) => Model?.aggregate === mongoose.Model.aggregate
+  && hasNoHooks(Model.hooks, ['aggregate']) && hasNoHooks(Model.schema?.s?.hooks, ['aggregate'])
+  && !(Model.schema?.discriminatorMapping && !Model.schema.discriminatorMapping.isRoot)
+  && Model.collection != null && !Object.hasOwn(Model.collection, 'aggregate');
+// MongoDB rejects $unionWith in a multi-document transaction, and the rejection aborts the
+// transaction. Inside connection.transaction() with Mongoose's transactionAsyncLocalStorage, an
+// aggregate without a session joins the transaction.
+const inTransactionStore = (Model) => Model?.db?.base?.transactionAsyncLocalStorage?.getStore()?.session != null;
+
 // The statics that getById, update and delete call, directly or through each other.
 const BY_ID_STATICS = ['findById', 'findOne', 'findByIdAndUpdate', 'findOneAndUpdate', 'findByIdAndDelete', 'findOneAndDelete'];
 // Mongoose's automatic ObjectId setter only changes document assignments, never query filters.
@@ -128,6 +159,7 @@ const nestedPathOf = (nested) => {
 
 export const createMongoAdapter = (options) => {
   const integrity = createMongoIntegrity(options);
+  const collectionBatching = readCollectionBatching(options);
   let queries;
   let getRegistrations = () => [];
   const privateConnectionFields = new Map();
@@ -347,6 +379,71 @@ export const createMongoAdapter = (options) => {
     },
   };
 
+  // Without the option the adapter has neither batching member, so the runtime reads each
+  // collection field with findChildren, as 3.6.0 did.
+  if (collectionBatching.enabled) {
+    const ownFindChildren = adapter.findChildren;
+    // Why `self` (the adapter the runtime called) cannot batch the children of `Model`, or null. A
+    // batch reproduces only this adapter's own findChildren, so a copy or an override of findChildren
+    // reads one parent at a time with that findChildren.
+    const declineReason = (self, Model, session) => {
+      if (self?.findChildren !== ownFindChildren) return 'its findChildren is not the adapter\'s own';
+      if (!batchesAggregate(Model)) {
+        return 'its aggregate reads have middleware, an own aggregate static, a discriminator or a wrapped collection.aggregate';
+      }
+      if (session != null || inTransactionStore(Model)) return 'it reads in a session or a transaction';
+      return null;
+    };
+    adapter.batchesChildren = function batchesChildren(Model) {
+      return declineReason(this, Model, null) === null;
+    };
+    // One array per parent ID, in order, each equal to findChildren's result for that parent. It
+    // declines by throwing before any command, and fails as a whole, never with partial results.
+    adapter.findChildrenBatch = async function findChildrenBatch(Model, gqltype, connectionField, parentIds, args, batchOptions) {
+      integrity.assertReady();
+      const reason = declineReason(this, Model, batchOptions?.session);
+      if (reason) throw new Error(`${Model?.modelName} reads children one parent at a time: ${reason}`);
+      const pipeline = await queries.buildQuery(args, gqltype);
+      const schemaType = Model.schema.path(connectionField);
+      const lists = parentIds.map(() => []);
+      const subPipelines = [];
+      parentIds.forEach((parentId, index) => {
+        let parentKey;
+        try {
+          parentKey = castPathValue(schemaType, parentId);
+        } catch (error) {
+          // As in findChildren: a parent key that the connection path cannot cast has no children.
+          if (error?.extensions?.code === 'NOT_VALID_ID') return;
+          throw error;
+        }
+        // Wrapping each row, instead of adding a field, cannot collide with a stored field.
+        subPipelines.push([{ $match: { [connectionField]: parentKey } }, ...pipeline, { $replaceWith: { i: index, d: '$$ROOT' } }]);
+      });
+      if (subPipelines.length === 0) return lists;
+      const parentsPerCommand = Math.max(1, Math.min(collectionBatching.parentsPerCommand,
+        Math.floor(COLLECTION_COMMAND_BYTES / mongoose.mongo.BSON.calculateObjectSize({ pipeline: subPipelines[0] }))));
+      const limit = pipeline.findLast((stage) => Object.hasOwn(stage, '$limit'))?.$limit;
+      const coll = Model.collection.collectionName;
+      const commands = [];
+      for (let start = 0; start < subPipelines.length; start += parentsPerCommand) {
+        const [first, ...others] = subPipelines.slice(start, start + parentsPerCommand);
+        const command = Model.aggregate([...first, ...others.map((subPipeline) => ({ $unionWith: { coll, pipeline: subPipeline } }))]);
+        // One more than the rows the command can return, so that its first batch closes the cursor.
+        if (Number.isSafeInteger(limit)) command.option({ batchSize: limit * (others.length + 1) + 1 });
+        commands.push(command);
+      }
+      // Every command settles before a failure is reported, so per-parent reads that follow it do not
+      // overlap the remaining commands.
+      const outcomes = await Promise.allSettled(commands);
+      const failure = outcomes.find(({ status }) => status === 'rejected');
+      if (failure) throw failure.reason;
+      for (const { value: rows } of outcomes) {
+        for (const { i, d } of rows) lists[i].push(d);
+      }
+      return lists;
+    };
+  }
+
   Object.defineProperty(adapter, 'referentialIntegrity', { value: integrity.mode, enumerable: true });
   return adapter;
 };

package/src/mongo/integrity.js

@@ -26,10 +26,13 @@ const valuesAt = (value, path) => {
   return valuesAt(value[path[0]], path.slice(1));
 };
 
+// createMongoAdapter's options: the adapter reads the collection batching ones.
+const ADAPTER_OPTIONS = ['referentialIntegrity', 'batchedCollections', 'collectionBatchSize'];
+
 export const createMongoIntegrity = (options = {}) => {
   if (!options || typeof options !== 'object' || Array.isArray(options)
-    || Object.keys(options).some((key) => key !== 'referentialIntegrity')) {
-    throw invalid('Expected Mongo adapter options with referentialIntegrity');
+    || Object.keys(options).some((key) => !ADAPTER_OPTIONS.includes(key))) {
+    throw invalid('Expected Mongo adapter options among referentialIntegrity, batchedCollections and collectionBatchSize');
   }
   const mode = options.referentialIntegrity === undefined ? 'off' : options.referentialIntegrity;
   if (!['off', 'transactional'].includes(mode)) {

package/src/mongo/models.js

@@ -12,7 +12,7 @@ import {
 } from 'graphql';
 import mongoose from 'mongoose';
 import { SimfinityError } from '@simtlix/simfinity-core';
-import { getFieldStorageName } from '@simtlix/simfinity-core/internal/relation-storage';
+import { getFieldStorageName, isVirtualField } from '@simtlix/simfinity-core/internal/relation-storage';
 
 const isNonNullOfType = (fieldEntryType, graphQLType) => (
   fieldEntryType instanceof GraphQLNonNull && fieldEntryType.ofType instanceof graphQLType
@@ -97,11 +97,24 @@ const assertNoEmbeddedCycle = (gqlType, entryType, ancestors, fieldPath) => {
   if (ancestors.includes(entryType)) throw new SimfinityError(`Embedded cycle at ${fieldPath}`, 'INVALID_MODEL', 400);
 };
 
+// Mongoose reads an empty definition, `{}` or `[{}]`, as Mixed, which stores any keys. An embedded
+// type whose members are all virtual has such a definition, so it gets a strict subdocument without
+// paths instead: no value of its virtual members, or any other key, is stored. A list item keeps its
+// `_id`, as other embedded list items do. Embedded types without virtual members keep their
+// definition, even an empty one.
+const storedEmbeddedDefinition = (entryType, definition, list) => {
+  if (Object.keys(definition).length || !Object.values(entryType.getFields()).some(isVirtualField)) return definition;
+  return list ? new mongoose.Schema({}) : new mongoose.Schema({}, { _id: false });
+};
+
 const generateSchemaDefinition = (gqlType, nested = false, ancestors = [], path = gqlType.name) => {
   const argTypes = gqlType.getFields();
   const schemaArg = {};
 
   for (const [fieldEntryName, fieldEntry] of Object.entries(argTypes)) {
+    // A virtual field is resolved by its own resolver and stored nowhere, on entities and embedded
+    // types alike: it gets no path, so no `[]` default and no ObjectId index either.
+    if (isVirtualField(fieldEntry)) continue;
     const { type } = fieldEntry;
 
     if (matchesScalar(type, GraphQLID)) {
@@ -123,9 +136,10 @@ const generateSchemaDefinition = (gqlType, nested = false, ancestors = [], path
           const entryType = unwrapNonNull(type);
           const fieldPath = `${path}.${fieldEntryName}`;
           assertNoEmbeddedCycle(gqlType, entryType, ancestors, fieldPath);
-          const definition = generateSchemaDefinition(entryType, true, [...ancestors, gqlType], fieldPath);
+          const definition = storedEmbeddedDefinition(entryType,
+            generateSchemaDefinition(entryType, true, [...ancestors, gqlType], fieldPath), false);
           // A nested object under a `type` key can only be a subdocument; like other embedded objects, it has no _id.
-          schemaArg[fieldEntryName] = nested && fieldEntryName === 'type'
+          schemaArg[fieldEntryName] = nested && fieldEntryName === 'type' && !(definition instanceof mongoose.Schema)
             ? new mongoose.Schema(definition, { _id: false })
             : definition;
         }
@@ -137,7 +151,8 @@ const generateSchemaDefinition = (gqlType, nested = false, ancestors = [], path
           assertEmbeddedFieldName(gqlType, fieldEntryName);
           const fieldPath = `${path}.${fieldEntryName}`;
           assertNoEmbeddedCycle(gqlType, itemType, ancestors, fieldPath);
-          schemaArg[fieldEntryName] = [generateSchemaDefinition(itemType, true, [...ancestors, gqlType], fieldPath)];
+          schemaArg[fieldEntryName] = [storedEmbeddedDefinition(itemType,
+            generateSchemaDefinition(itemType, true, [...ancestors, gqlType], fieldPath), true)];
         }
       } else if (listItemMatchesScalar(type, GraphQLID)) {
         schemaArg[fieldEntryName] = [mongoose.Schema.Types.ObjectId];
@@ -186,7 +201,7 @@ const findObjectIdFields = (schemaDefinition, parentPath = '') => {
     if (definition === mongoose.Schema.Types.ObjectId) {
       objectIdFields.push(currentPath);
     } else if (Array.isArray(definition)) {
-      const arrayElement = definition[0];
+      const arrayElement = definition[0] instanceof mongoose.Schema ? definition[0].obj : definition[0];
       if (typeof arrayElement === 'object' && arrayElement !== null) {
         objectIdFields.push(...findObjectIdFields(arrayElement, currentPath));
       }

package/src/mongo/queries.js

@@ -9,7 +9,7 @@ import mongoose from 'mongoose';
 
 import { SimfinityError, QLOperator, paginationStages } from '@simtlix/simfinity-core';
 import {
-  getFieldStorageName, getListShape, hasStoredIdentity, normalizeConnectionField,
+  getFieldStorageName, getListShape, hasStoredIdentity, isVirtualField, normalizeConnectionField,
 } from '@simtlix/simfinity-core/internal/relation-storage';
 
 import { castObjectId } from './ids.js';
@@ -140,6 +140,12 @@ const resolveQueryPath = (gqltype, path, aggregationsIncluded) => {
   for (const [index, part] of parts.entries()) {
     const field = Object.hasOwn(currentType.getFields(), part) ? currentType.getFields()[part] : null;
     if (!field) throw filterError(`Unknown query field: ${path}`, 'INVALID_FILTER_FIELD');
+    // Core rejects generated and trusted paths through a virtual field before the adapter runs. A
+    // native buildQuery() or buildFilterGroupMatch() call fails here, instead of matching storage
+    // that does not exist.
+    if (isVirtualField(field)) {
+      throw filterError(`Query path ${path} uses the virtual field ${currentType.name}.${part}, which is not stored`, 'INVALID_FILTER_FIELD');
+    }
     const fieldType = getNamedType(field.type);
     const relation = field.extensions?.relation;
     if (fieldType instanceof GraphQLObjectType) {

package/types/index.d.ts

@@ -33,6 +33,7 @@ export type {
   AuthPluginOptions,
   AuthRule,
   AuthRuleFunction,
+  AuthSchemaPlugin,
   EnvelopSchemaPlugin,
   FieldValidations,
   FieldValidator,
@@ -45,12 +46,36 @@ export type {
   ScopeParams,
   TypePermissions,
   TypeScopes,
+  VirtualFieldDeclaration,
+  VirtualFieldOptions,
 } from '@simtlix/simfinity-core';
 export * from '@simtlix/simfinity-mcp';
 export { default as mcp } from '@simtlix/simfinity-mcp';
 export interface MongoAdapterOptions {
   /** Fixed at adapter creation. Defaults to off; transactional mode requires initialize(). */
   referentialIntegrity?: 'off' | 'transactional';
+  /**
+   * Opt in to batched collection reads; defaults to false. When true, the adapter defines
+   * `findChildrenBatch` and `batchesChildren`, so the runtime reads the generated collection fields
+   * of one request that share a related type, a connection field and their arguments with aggregate
+   * commands of up to `collectionBatchSize` parents each, run concurrently. Each parent's own
+   * pipeline (filters, sort and page) runs as one sub-pipeline of a command, in a `$unionWith` stage
+   * after the first. Children of models with aggregate middleware, their own `aggregate` static, a
+   * discriminator key or a wrapped `collection.aggregate`, reads inside a Mongoose
+   * `transactionAsyncLocalStorage` transaction, and adapter copies or overrides of `findChildren`
+   * keep one read per parent. When false, the adapter has neither member and reads one parent at a
+   * time, as 3.6.0 did. Fixed at adapter creation; a value other than a boolean throws
+   * `INVALID_MONGO_ADAPTER_OPTIONS` (400).
+   */
+  batchedCollections?: boolean | undefined;
+  /**
+   * Parents per batched aggregate command, an integer from 1 to 100; defaults to 5. A command runs
+   * its parents' sub-pipelines one after another, and the commands of one batch run concurrently, so
+   * a larger value sends fewer commands at the cost of a longer command. Only used with
+   * `batchedCollections: true`. Fixed at adapter creation; another value throws
+   * `INVALID_MONGO_ADAPTER_OPTIONS` (400).
+   */
+  collectionBatchSize?: number | undefined;
 }
 /**
  * In the default 'off' mode, withTransaction(null, callback) without a model, as custom mutations
@@ -58,7 +83,8 @@ export interface MongoAdapterOptions {
  * client, no model is compiled on it or on its useDb() descendants, and every model of the bound
  * runtime's registrations uses one MongoDB client: then it opens it on that client. A supplied
  * session or model is used as is. In transactional mode it uses the first protected model's
- * connection.
+ * connection. With `batchedCollections: true` the adapter also has the optional `DatabaseAdapter`
+ * members `findChildrenBatch` and `batchesChildren`; without it, it has neither.
  */
 export interface MongoAdapter extends DatabaseAdapter {
   readonly referentialIntegrity: 'off' | 'transactional';
@@ -139,6 +165,13 @@ export function addNoEndpointType(gqltype: GraphQLObjectType): void;
  * Embedded types that contain each other throw INVALID_MODEL (400, `Embedded cycle at …`) when a
  * generated model's embedded fields reach them, and always in transactional mode; in default mode a
  * cycle that only supplied models reach is not checked.
+ * Invalid virtual field declarations (`extensions.virtual`, see `VirtualFieldOptions`) throw
+ * INVALID_MODEL (400) before models are created, in both referentialIntegrity modes. A virtual field
+ * gets no path, `[]` default or index in generated models, on entities and embedded types, and no
+ * entry in the transactional integrity registry, so it needs no `relation` in either mode; generated
+ * writes never set or unset it. A supplied model keeps the paths it declares. A readOnly field with
+ * arguments and its own resolver that is not virtual, and is not a reference or collection, logs a
+ * `Configuration issue` warning once.
  * A generated input that would have no fields is left out with a warning, together with the
  * collection operation, embedded field or add/update/state-action mutation that would take it; a
  * non-null embedded field whose embedded type has no writable fields throws INVALID_MODEL (400)
@@ -272,6 +305,7 @@ export interface FilterGroupInput {
  * pipeline. `input.aggregation` filters a field named `aggregation`; the aggregate stage is never
  * built here. An embedded `.id` path uses the declared `id` member, except for entity types embedded
  * as lists and supplied subdocument schemas without an `id` path, which use the subdocument `_id`.
+ * A filter, condition or sort path through a virtual field rejects with `INVALID_FILTER_FIELD` (400).
  */
 export function buildQuery(
   input: Record<string, any>,
@@ -279,7 +313,10 @@ export function buildQuery(
   isCount?: boolean,
 ): Promise<Array<Record<string, any>>>;
 
-/** Translate a QLFilterGroup into a MongoDB `$match` expression, collecting required lookups. */
+/**
+ * Translate a QLFilterGroup into a MongoDB `$match` expression, collecting required lookups. A
+ * condition path through a virtual field rejects with `INVALID_FILTER_FIELD` (400).
+ */
 export function buildFilterGroupMatch(
   filterGroup: FilterGroupInput,
   gqltype: GraphQLObjectType,