@simtlix/simfinity-js 3.10.0 → 3.11.0

package/README.md

@@ -28,7 +28,9 @@ Virtual fields (`extensions: { virtual: true }`, with the field's own `resolve`)
 
 Text filters compile to escaped regular expressions, so a value never becomes regex syntax: `LIKE` and `ILIKE` match anywhere, `STARTS_WITH` and `ISTARTS_WITH` are anchored with `^`, `ENDS_WITH` and `IENDS_WITH` with `\z` (so a trailing newline is part of the value), and `IEQ` with both. The `I` operators add the `i` option, which applies the Unicode simple case folding of the server's PCRE2 build (`É`/`é`, `ẞ`/`ß`, `σ`/`ς`, `µ`/`μ`, `ſ`/`s`), never a length-changing or Turkic folding (`ß`/`ss`, `İ`/`i`), and does not fold characters newer than that build, such as U+A7DC; the [queries guide](https://simtlix.github.io/simfinity.js/guide/queries.html) lists where PostgreSQL differs. Only `STARTS_WITH` reads a prefix range of an index on the field, a String list or an embedded path; the other text operators test every key of such an index, or every document without one, and `$regex` ignores collations, so a case-insensitive collation index does not serve them. A value whose escaped pattern is longer than MongoDB's 32,764-byte pattern limit fails with `INVALID_FILTER_VALUE` (400) before the query runs, also for `LIKE`. `createMongoAdapter()` declares every operator it compiles in `queryFeatures`; a copy of the adapter without it supports only the operators of 3.7.0.
 
-`IS_EMPTY` and `IS_NOT_EMPTY` take no value and compile to plain query operators inside the same `$match`, so they also filter counts, aggregates and batched collection reads (`$unionWith`): `$in: [null, ""]` and `$nin` on `String` fields, which an index on the field serves; `null` and `$ne: null` on other scalars, enums and the record identity `_id`; on an application scalar without a supported base, such as a `JSON` scalar, which MongoDB stores as written, `null` without arrays, `{ $and: [{ p: null }, { p: { $not: { $type: "array" } } }] }`, and its `$nor`, so that an array, also `[null]`, is a value, as reads return it; `"p.0": { $exists: false }` and `$exists: true` on scalar lists and embedded lists, which never look inside the items; and, on an embedded object, an `$or` of `null` and the `IS_EMPTY` form of every stored member, with its `$nor` for `IS_NOT_EMPTY`, so an object whose stored members are all empty, such as `{ w: null, label: "" }`, is empty. A term or group condition without a `path` tests the object or list field itself. Embedded `id` members are read where other filters read them, so an object of a supplied subdocument schema with an automatic `_id` is never empty, unlike on PostgreSQL. A path that continues inside an embedded list or a collection, such as `lines.qty` or `seasons.year`, fails with the permanent `INVALID_FILTER_PATH` (400), since emptiness of a value inside a list is not defined. In this release, any other path that crosses or ends at a reference, a path that ends at a collection, such as `seasons`, and an embedded object with a reference or collection member fail with `UNSUPPORTED_QUERY` (400); the [queries guide](https://simtlix.github.io/simfinity.js/guide/queries.html#empty-values) has the rules and the other codes.
+`IS_EMPTY` and `IS_NOT_EMPTY` take no value and compile to plain query operators inside the same `$match`, so they also filter counts, aggregates and batched collection reads (`$unionWith`): `$in: [null, ""]` and `$nin` on `String` fields, which an index on the field serves; `null` and `$ne: null` on other scalars, enums and the record identity `_id`; on an application scalar without a supported base, such as a `JSON` scalar, which MongoDB stores as written, `null` without arrays, `{ $and: [{ p: null }, { p: { $not: { $type: "array" } } }] }`, and its `$nor`, so that an array, also `[null]`, is a value, as reads return it; `"p.0": { $exists: false }` and `$exists: true` on scalar lists and embedded lists, which never look inside the items; and, on an embedded object, an `$or` of `null` and the `IS_EMPTY` form of every stored member, with its `$nor` for `IS_NOT_EMPTY`, so an object whose stored members are all empty, such as `{ w: null, label: "" }`, is empty. A term or group condition without a `path` tests the object or list field itself. Embedded `id` members are read where other filters read them, so an object of a supplied subdocument schema with an automatic `_id` is never empty, unlike on PostgreSQL. A path that continues inside an embedded list or a collection, such as `lines.qty` or `seasons.year`, fails with the permanent `INVALID_FILTER_PATH` (400), since emptiness of a value inside a list is not defined; the [queries guide](https://simtlix.github.io/simfinity.js/guide/queries.html#empty-values) has the rules and the other codes.
+
+The adapter also declares `queryFeatures.referenceEmptiness`, so these operators cross single references and end at references and collections, such as `director IS_EMPTY`, `director.name IS_EMPTY` or `seasons IS_EMPTY`, and an embedded object with a reference or collection member follows the member rule. A crossing joins the target with `$lookup` and `$unwind`, and the target's `id` right after it is its record identity, which is empty exactly when the reference is. A reference or collection is a frame: one `$lookup` that keeps at most one record id, so it never repeats a document, matched with `"alias.0": { $exists: false }` and its `$nor`; a collection frame reads the child's connection field and also tests its local key for null, because `$lookup` matches a missing local key with every child whose connection field is null. The joins and frames run before the `$match`, so they also filter counts, aggregates and batched collection reads, and they probe the `_id` index or the index on the connection field, which generated models create and a supplied model needs to declare. A client's condition reads each target through its `find` scope, inside the `$lookup` pipeline, so a record the scope hides reads as missing, also on the way to a later hop. Conditions that middleware and scopes add, and `buildQuery()` called directly, read stored data, so a dangling reference, to a record that no longer exists, is empty. A related type without a model fails with `INVALID_FILTER_PATH` (400), as for the other operators, and supplied models on another database or connection and discriminator targets are not supported, as for every relation path. A collection inside an embedded type, which only a supplied model can declare, is read through the embedded object's `_id`, as relation paths read it. The [query scope guide](https://simtlix.github.io/simfinity.js/guide/query-scope.html) describes client and trusted conditions, and the [middleware guide](https://simtlix.github.io/simfinity.js/guide/middleware.html) how to edit client filters without detaching them.
 
 The facade registers generated models on your application's default Mongoose instance and does not change global Mongoose options such as `strictQuery`. Supplied models may use another connection, such as one from `mongoose.createConnection()`: generated mutations and `saveObject()` use the model's connection, and, in the default `referentialIntegrity: 'off'` mode, custom mutations use the default connection unless it is unused, never opened and without compiled models, also on its `useDb()` descendants, and the registered models share one MongoDB client; with transactional reference integrity they use the first protected model's connection. Custom queries registered with `registerCustomQuery()` receive no session; a callback that needs a transaction opens it with the `transaction()` of the Mongoose connection whose models it reads. `createMongoAdapter().castId` accepts only an ObjectId or its 24-character hexadecimal string and throws `NOT_VALID_ID` (400) otherwise; create new identifiers with `new mongoose.Types.ObjectId()`. Owned transactions retry transient failures after a short random delay, and a write conflict that outlasts the retries fails with `TRANSACTION_RETRY_EXCEEDED` (409).
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@simtlix/simfinity-js",
-  "version": "3.10.0",
+  "version": "3.11.0",
   "description": "MongoDB/Mongoose adapter and compatibility facade for Simfinity GraphQL APIs",
   "main": "src/index.js",
   "types": "./types/index.d.ts",
@@ -29,8 +29,8 @@
     "owner": "simtlix"
   },
   "dependencies": {
-    "@simtlix/simfinity-core": "3.10.0",
-    "@simtlix/simfinity-mcp": "3.10.0"
+    "@simtlix/simfinity-core": "3.11.0",
+    "@simtlix/simfinity-mcp": "3.11.0"
   },
   "files": [
     "src",

package/src/mongo/adapter.js

@@ -12,8 +12,9 @@ import {
 
 const withSession = (query, session) => (session ? query.session(session) : query);
 
-// Every filter operator the query compiler supports; the runtime rejects any other one.
-const QUERY_FEATURES = Object.freeze({ operators: MONGO_QUERY_OPERATORS });
+// Every filter operator the query compiler supports, and emptiness through references and collections (#108 part
+// 2b); the runtime rejects any other operator.
+const QUERY_FEATURES = Object.freeze({ operators: MONGO_QUERY_OPERATORS, referenceEmptiness: true });
 
 // A write that fails only on malformed ObjectId values reports NOT_VALID_ID, not a driver error.
 const mapIdCastErrors = async (write) => {

package/src/mongo/queries.js

@@ -10,8 +10,8 @@ import mongoose from 'mongoose';
 
 import { SimfinityError, QLOperator, paginationStages } from '@simtlix/simfinity-core';
 import {
-  TEXT_QUERY_OPERATORS, describeEmptiness, effectiveScalarName, isEmptinessOperator, textQueryOperator,
-  validateTextFilterValue,
+  TEXT_QUERY_OPERATORS, describeEmptiness, effectiveScalarName, emptinessScopePaths, isEmptinessOperator,
+  textQueryOperator, validateTextFilterValue,
 } from '@simtlix/simfinity-core/internal/query-operators';
 import {
   getFieldStorageName, getListShape, hasStoredIdentity, isVirtualField, normalizeConnectionField,
@@ -26,6 +26,9 @@ const isGraphQLisoDate = (typeName) => (
 
 // GraphQL reserves `__` names, so these aliases never replace a stored field.
 const LOOKUP_ALIAS_PREFIX = '__sf_l';
+// The scoped joins and frames of reference and collection emptiness (#108 part 2b). Each kind has its own counter, so
+// the lookup aliases keep their numbers.
+const RELATION_ALIAS_PREFIXES = { join: '__sf_j', frame: '__sf_q' };
 
 /**
  * The filter operators this compiler supports, which the adapter declares as its queryFeatures. They
@@ -51,10 +54,32 @@ const allocateLookupAlias = (aggregationsIncluded, relationPath) => {
   return aliases.get(relationPath);
 };
 
+// The scoped joins and frames of one build, keyed by the build's `included` accumulator like the lookup aliases:
+// their aliases by kind and key, a counter per kind, and each target scope compiled once.
+const relationBuilds = new WeakMap();
+
+const relationBuild = (aggregationsIncluded) => {
+  let build = relationBuilds.get(aggregationsIncluded);
+  if (!build) {
+    build = {
+      aliases: new Map(), counts: { join: 0, frame: 0 }, scopes: new WeakMap(), scopeCount: 0,
+    };
+    relationBuilds.set(aggregationsIncluded, build);
+  }
+  return build;
+};
+
+const allocateRelationAlias = (build, kind, key) => {
+  const entry = `${kind} ${key}`;
+  if (!build.aliases.has(entry)) build.aliases.set(entry, `${RELATION_ALIAS_PREFIXES[kind]}${build.counts[kind]++}`);
+  return build.aliases.get(entry);
+};
+
+// Lookup entries are `{ stages }` by alias: a join's `$lookup` and `$unwind`, or a frame's single `$lookup`.
 const appendLookups = (aggregateClauses, aggregationsIncluded, lookups) => {
-  for (const [alias, { lookup, unwind }] of Object.entries(lookups)) {
+  for (const [alias, { stages }] of Object.entries(lookups)) {
     if (!Object.hasOwn(aggregationsIncluded, alias)) {
-      aggregateClauses.push(lookup, unwind);
+      aggregateClauses.push(...stages);
       aggregationsIncluded[alias] = true;
     }
   }
@@ -63,12 +88,14 @@ const appendLookups = (aggregateClauses, aggregationsIncluded, lookups) => {
 const buildRelationLookup = ({
   collectionName, localField, foreignField, alias,
 }) => ({
-  lookup: {
-    $lookup: {
-      from: collectionName, foreignField, localField, as: alias,
+  stages: [
+    {
+      $lookup: {
+        from: collectionName, foreignField, localField, as: alias,
+      },
     },
-  },
-  unwind: { $unwind: { path: `$${alias}`, preserveNullAndEmptyArrays: true } },
+    { $unwind: { path: `$${alias}`, preserveNullAndEmptyArrays: true } },
+  ],
 });
 
 const escapeRegex = (s) => String(s).replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
@@ -230,72 +257,238 @@ const resolveQueryPath = (gqltype, path, aggregationsIncluded) => {
   throw filterError(`Invalid query path: ${path}`, 'INVALID_FILTER_PATH');
 };
 
-// The emptiness operators IS_EMPTY and IS_NOT_EMPTY (#108, part 2a). Core's describeEmptiness applies
-// their checks in the order every caller uses and describes what "empty" means for the path. It
-// rejects references, collections, interface and union fields and anything inside a list, so the
-// path and every member it reads are stored in the queried document, through single embedded objects.
-// The forms are plain query operators that need no lookup, valid wherever the $match runs.
+// The emptiness operators IS_EMPTY and IS_NOT_EMPTY (#108, parts 2a and 2b). Core's describeEmptiness applies their
+// checks in the order every caller uses and describes what "empty" means for the path. It rejects interface and
+// union fields and anything inside a list. On the queried document's own fields, through single embedded objects,
+// the forms are plain query operators that need no lookup, valid wherever the $match runs. A path may also cross
+// single references and end at a reference or a collection, and an embedded object may have reference and
+// collection members (part 2b). A crossing reads the target through a join, and a reference or collection end or
+// member through a frame: one `$lookup` that keeps at most one record id, so it never repeats a document, and that
+// is empty when no record exists. A construct the client sent carries the client scope marker that the runtime
+// fills, `__scope: { id, scopes }`, on its term or group condition; `scopes` holds each target's find scope
+// arguments by path, or null when the scope added nothing. Its crossings are then scoped joins and its frames scoped
+// frames, so a record the scope hides reads as missing. A construct without a marker, from middleware, a scope or a
+// direct buildQuery() call, reads stored data, and its leaves share the plain lookup aliases with the other
+// operators. The joins and frames sit with the other lookups before the $match, so they also filter counts,
+// aggregates and batched collection reads.
 const isPathless = (path) => path == null || path === '';
+const joinPath = (base, name) => (base ? `${base}.${name}` : name);
 
-// The stored key of the member `name` of `type`, an object stored at `schemaPath` ('' at the root):
-// a non-object `id` member follows identifierLeaf, as resolveQueryPath does; others keep their name.
-const emptinessKey = (model, type, schemaPath, name) => {
+const hasScopeMarker = (construct) => construct != null && typeof construct === 'object' && Object.hasOwn(construct, '__scope');
+// Only the runtime writes and fills client scope markers, after middleware, so a marker that cannot be honored here
+// is an internal error, never a construct compiled with stored semantics.
+const detachedFilter = (message) => new SimfinityError(
+  `${message}; only the runtime writes and fills client scope markers (docs/guide/middleware.md)`, 'DETACHED_CLIENT_FILTER', 500,
+);
+
+// Where a value is: the Mongo key that reads it, its storage path relative to the documents of `model` (which
+// identifierLeaf and the identity rule read), and its dotted GraphQL path from the queried record, which names
+// its scope. `{ key: '', schemaPath: '', path: '' }` is the queried document; a joined document has its alias as
+// key, an empty storage path and its crossing as path.
+const memberPlace = (model, parent, type, name) => {
   const field = type.getFields()[name];
+  // A non-object `id` member follows identifierLeaf, as resolveQueryPath does; others keep their name.
   const leaf = name === 'id' && !(getNamedType(field.type) instanceof GraphQLObjectType)
-    ? identifierLeaf(model, schemaPath, type)
+    ? identifierLeaf(model, parent.schemaPath, type)
     : name;
-  return schemaPath ? `${schemaPath}.${leaf}` : leaf;
+  return { key: joinPath(parent.key, leaf), schemaPath: joinPath(parent.schemaPath, leaf), path: joinPath(parent.path, name) };
 };
 
 // MongoDB stores the built-in scalars, dates and enums as single values, but keeps an application scalar
 // without a supported base as written, arrays included, and `{ p: null }` also matches an array that
 // holds a null. The null rule of such a scalar leaves arrays out, so `[null]` is a value, as GraphQL
-// reads it. The record identity `_id` is never an array, so it keeps the plain form.
+// reads it. The record identity `_id`, of the queried or a joined document, is never an array, so it keeps the
+// plain form.
 const SINGLE_VALUE_SCALARS = new Set(['String', 'ID', 'Int', 'Float', 'Boolean', 'DateTime', 'Date', 'Time']);
-const holdsArrays = (key, type) => (
-  key !== '_id' && isScalarType(type) && !SINGLE_VALUE_SCALARS.has(effectiveScalarName(type))
+const holdsArrays = (place, type) => (
+  place.schemaPath !== '_id' && isScalarType(type) && !SINGLE_VALUE_SCALARS.has(effectiveScalarName(type))
 );
-const nullForm = (key, type) => (holdsArrays(key, type)
-  ? { $and: [{ [key]: null }, { [key]: { $not: { $type: 'array' } } }] }
-  : { [key]: null });
-
-// The IS_EMPTY form of `shape` for the value stored at `key`; `type` is the named type of its field.
-// An object is missing, null, or has every stored member empty. Without stored members it is always
-// empty: MongoDB rejects an empty $and, so the empty query, which matches every document, stands in.
-const emptyForm = (model, shape, key, type) => {
-  if (shape.rule === 'string') return { [key]: { $in: [null, ''] } };
-  if (shape.rule === 'null') return nullForm(key, type);
-  if (shape.rule === 'list') return { [`${key}.0`]: { $exists: false } };
-  const fields = type.getFields();
-  const members = shape.members.map((member) => emptyForm(
-    model, member.shape, emptinessKey(model, type, key, member.name), getNamedType(fields[member.name].type),
-  ));
-  return { $or: [{ [key]: null }, members.length ? { $and: members } : {}] };
+const nullForm = (place, type) => (holdsArrays(place, type)
+  ? { $and: [{ [place.key]: null }, { [place.key]: { $not: { $type: 'array' } } }] }
+  : { [place.key]: null });
+
+// A frame: NONE over the reference or collection `name` of the object at `parent`, whose type is `type`. Its
+// `$lookup` keeps at most one id of a target record that the reference points to (`foreignField: '_id'`) or of a
+// child whose connection field, `shape.connectionField`, holds the object's `_id`, among those the target scope
+// admits. A frame is keyed by its path and by the scopes of every crossing before it and its own, so a frame after
+// a scoped crossing never shares a trusted one, even when its own scope is null. A collection frame also tests its
+// local key for null: `$lookup` matches a missing local key with every child whose connection field is null.
+const frameForm = (ctx, shape, parent, type, name) => {
+  const field = type.getFields()[name];
+  const path = joinPath(parent.path, name);
+  const scope = ctx.scopes.get(path) ?? null;
+  const alias = allocateRelationAlias(ctx.build, 'frame', `${path} ${[...ctx.chain, scope ? scope.token : '-'].join(',')}`);
+  const collection = shape.rule === 'collection';
+  const localField = joinPath(parent.key, collection ? '_id' : getFieldStorageName(name, field));
+  ctx.lookups[alias] = {
+    stages: [{
+      $lookup: {
+        from: getModel(getNamedType(field.type)).collection.collectionName,
+        localField,
+        foreignField: collection ? shape.connectionField : '_id',
+        pipeline: [...(scope?.stages ?? []), { $limit: 1 }, { $project: { _id: 1 } }],
+        as: alias,
+      },
+    }],
+  };
+  const none = { [`${alias}.0`]: { $exists: false } };
+  return collection ? { $or: [{ [localField]: null }, none] } : none;
+};
+
+// The IS_EMPTY form of `shape` for the member `name` of the object at `parent`, whose type is `type`. An object is
+// missing, null, or has every stored member empty. Without stored members it is always empty: MongoDB rejects an
+// empty $and, so the empty query, which matches every document, stands in.
+const emptyForm = (ctx, shape, parent, type, name) => {
+  if (shape.rule === 'reference' || shape.rule === 'collection') return frameForm(ctx, shape, parent, type, name);
+  const place = memberPlace(ctx.model, parent, type, name);
+  const valueType = getNamedType(type.getFields()[name].type);
+  if (shape.rule === 'string') return { [place.key]: { $in: [null, ''] } };
+  if (shape.rule === 'null') return nullForm(place, valueType);
+  if (shape.rule === 'list') return { [`${place.key}.0`]: { $exists: false } };
+  const members = shape.members.map((member) => emptyForm(ctx, member.shape, place, valueType, member.name));
+  return { $or: [{ [place.key]: null }, members.length ? { $and: members } : {}] };
+};
+
+// IS_NOT_EMPTY is the exact complement: the positive form of a leaf, which an index can serve, or the $nor of the
+// IS_EMPTY form of an object, a reference, a collection or a scalar that may hold arrays.
+const notEmptyForm = (ctx, shape, parent, type, name) => {
+  if (shape.rule === 'reference' || shape.rule === 'collection' || shape.rule === 'object') {
+    return { $nor: [emptyForm(ctx, shape, parent, type, name)] };
+  }
+  const place = memberPlace(ctx.model, parent, type, name);
+  const valueType = getNamedType(type.getFields()[name].type);
+  if (shape.rule === 'string') return { [place.key]: { $nin: [null, ''] } };
+  if (shape.rule === 'null') return holdsArrays(place, valueType) ? { $nor: [nullForm(place, valueType)] } : { [place.key]: { $ne: null } };
+  return { [`${place.key}.0`]: { $exists: true } };
+};
+
+// The GraphQL target type of each reference and collection path an emptiness construct reads (its crossings, its end
+// and its members), by dotted path from the queried type.
+const relationTargets = (gqltype, segments, shape) => {
+  const targets = new Map();
+  let type = gqltype;
+  for (const [index, name] of segments.entries()) {
+    type = getNamedType(type.getFields()[name].type);
+    targets.set(segments.slice(0, index + 1).join('.'), type);
+  }
+  const visit = (node, objectType, path) => {
+    if (node.rule !== 'object') return;
+    for (const member of node.members) {
+      const memberType = getNamedType(objectType.getFields()[member.name].type);
+      targets.set(`${path}.${member.name}`, memberType);
+      visit(member.shape, memberType, `${path}.${member.name}`);
+    }
+  };
+  visit(shape, type, segments.join('.'));
+  return targets;
+};
+
+// A target's find scope from a filled client scope marker, compiled once per build and target type with this
+// compiler, in a fresh alias space: its own lookups live on the joined documents, and ID casting and state names are
+// the target's. Scope arguments are trusted. Gives `{ token, stages }`, or null for a null scope or one that adds no
+// stage, which reads like no scope.
+const compileTargetScope = async (build, scope, targetType) => {
+  if (scope === null) return null;
+  const compiled = build.scopes.get(scope);
+  if (compiled?.has(targetType)) return compiled.get(targetType);
+  const stages = [];
+  const match = await collectFiltersAndLookups(scope, targetType, stages, {}, 'find');
+  if (match) stages.push(match);
+  const result = stages.length ? { token: `s${build.scopeCount++}`, stages } : null;
+  if (!build.scopes.has(scope)) build.scopes.set(scope, new Map());
+  build.scopes.get(scope).set(targetType, result);
+  return result;
 };
 
-// IS_NOT_EMPTY is the exact complement: the positive form of a leaf, which an index can serve, or
-// the $nor of the IS_EMPTY form of an object or of a scalar that may hold arrays.
-const notEmptyForm = (model, shape, key, type) => {
-  if (shape.rule === 'string') return { [key]: { $nin: [null, ''] } };
-  if (shape.rule === 'null') return holdsArrays(key, type) ? { $nor: [nullForm(key, type)] } : { [key]: { $ne: null } };
-  if (shape.rule === 'list') return { [`${key}.0`]: { $exists: true } };
-  return { $nor: [emptyForm(model, shape, key, type)] };
+// The scopes of a client construct's filled marker, or null for a construct without one. Fails closed: a marker
+// on a construct that reads no reference or collection, or whose `scopes` lack a path the construct reads, is an
+// internal error.
+const clientScopes = (construct, label, paths) => {
+  if (!hasScopeMarker(construct)) return null;
+  if (!paths.length) throw detachedFilter(`${label} carries a client scope marker, but reads no reference or collection`);
+  const scopes = construct.__scope?.scopes;
+  if (!scopes || typeof scopes !== 'object' || Array.isArray(scopes)) {
+    throw detachedFilter(`${label} carries a client scope marker whose scopes are not filled`);
+  }
+  for (const path of paths) {
+    if (!Object.hasOwn(scopes, path) || scopes[path] === undefined) {
+      throw detachedFilter(`${label} carries a client scope marker without the scope of ${path}`);
+    }
+  }
+  return scopes;
 };
 
-// The match clause of one emptiness condition on `path`, the field itself or a dotted path from it.
-const buildEmptinessClause = (gqltype, path, operator, value) => {
+// The match clause of one emptiness condition on `path`, the field itself or a dotted path from it. `construct` is
+// the term or group condition that holds it, the only objects whose client scope marker is read, or null. Its joins
+// and frames are added to `lookups` as entries by alias, which the caller appends before the $match. A crossing, end
+// or member whose target has no model fails with INVALID_FILTER_PATH before any entry is added.
+const buildEmptinessClause = async (gqltype, path, operator, value, construct, lookups, aggregationsIncluded) => {
   const segments = path.split('.');
   const shape = describeEmptiness(gqltype, segments, { operator, value, path });
-  const model = getModel(gqltype);
+  const { joins, frames } = emptinessScopePaths(shape, segments);
+  const paths = [...joins, ...frames];
+  const targets = relationTargets(gqltype, segments, shape);
+  if (paths.some((relationPath) => !getModel(targets.get(relationPath)))) {
+    throw filterError(`Related model is not available for ${path}`, 'INVALID_FILTER_PATH');
+  }
+  const scopes = clientScopes(construct, `${operator} on ${path}`, paths);
+  const build = relationBuild(aggregationsIncluded);
+  const compiled = new Map();
+  if (scopes) {
+    for (const relationPath of paths) compiled.set(relationPath, await compileTargetScope(build, scopes[relationPath], targets.get(relationPath)));
+  }
+  const ctx = {
+    build, lookups, scopes: compiled, chain: [], model: getModel(gqltype),
+  };
+  // Crossings: a plain lookup, shared with the other operators, while every crossing so far is trusted or has a null
+  // scope; from the first scoped crossing on, a scoped join keyed by the scopes of every crossing up to it, also when
+  // its own scope is null, correlated through the joins before it.
+  const crossings = new Set(joins);
+  let parent = { key: '', schemaPath: '', path: '' };
   let type = gqltype;
-  let schemaPath = '';
+  let scoped = false;
   for (const name of segments.slice(0, -1)) {
-    schemaPath = schemaPath ? `${schemaPath}.${name}` : name;
-    type = getNamedType(type.getFields()[name].type);
+    const field = type.getFields()[name];
+    const fieldType = getNamedType(field.type);
+    const crossing = joinPath(parent.path, name);
+    if (!crossings.has(crossing)) {
+      parent = { key: joinPath(parent.key, name), schemaPath: joinPath(parent.schemaPath, name), path: crossing };
+    } else {
+      const scope = compiled.get(crossing) ?? null;
+      ctx.chain.push(scope ? scope.token : '-');
+      scoped ||= !!scope;
+      const target = getModel(fieldType);
+      const localField = joinPath(parent.key, getFieldStorageName(name, field));
+      let alias;
+      if (!scoped) {
+        alias = allocateLookupAlias(aggregationsIncluded, crossing);
+        lookups[alias] = buildRelationLookup({
+          collectionName: target.collection.collectionName, localField, foreignField: '_id', alias,
+        });
+      } else {
+        alias = allocateRelationAlias(build, 'join', `${crossing} ${ctx.chain.join(',')}`);
+        lookups[alias] = {
+          stages: [
+            {
+              $lookup: {
+                from: target.collection.collectionName,
+                localField,
+                foreignField: '_id',
+                pipeline: [...(scope?.stages ?? []), { $limit: 1 }],
+                as: alias,
+              },
+            },
+            { $unwind: { path: `$${alias}`, preserveNullAndEmptyArrays: true } },
+          ],
+        };
+      }
+      parent = { key: alias, schemaPath: '', path: crossing };
+      ctx.model = target;
+    }
+    type = fieldType;
   }
-  const name = segments.at(-1);
   const form = operator === 'IS_EMPTY' ? emptyForm : notEmptyForm;
-  return form(model, shape, emptinessKey(model, type, schemaPath, name), getNamedType(type.getFields()[name].type));
+  return form(ctx, shape, parent, type, segments.at(-1));
 };
 
 const castFilterValue = (value, resolved) => {
@@ -401,10 +594,14 @@ const buildQueryTerms = async (filterField, qlField, fieldName, gqltype, aggrega
   for (const term of terms) {
     if (!term || typeof term !== 'object' || Array.isArray(term)) throw filterError(`Invalid term for ${fieldName}`);
     // An emptiness term whose path is omitted, null or "" tests the field itself.
+    // Only the terms of an object filter are constructs; a scalar filter is its own term.
     if (isEmptinessOperator(term.operator)) {
       const pathed = isObject && !isPathless(term.path);
       if (pathed) assertValidFilterPath(term.path);
-      matchesClauses.push(buildEmptinessClause(gqltype, pathed ? `${fieldName}.${term.path}` : fieldName, term.operator, term.value));
+      matchesClauses.push(await buildEmptinessClause(
+        gqltype, pathed ? `${fieldName}.${term.path}` : fieldName, term.operator, term.value, isObject ? term : null,
+        aggregateClauses, aggregationsIncluded,
+      ));
       continue;
     }
     if (isObject) assertValidFilterPath(term.path);
@@ -412,6 +609,9 @@ const buildQueryTerms = async (filterField, qlField, fieldName, gqltype, aggrega
     const resolved = resolveQueryPath(gqltype, path, aggregationsIncluded);
     Object.assign(aggregateClauses, resolved.aggregateClauses);
     matchesClauses.push(buildMatchesClause(resolved, term.operator, term.value, path));
+    if (isObject && hasScopeMarker(term)) {
+      throw detachedFilter(`${term.operator ?? 'EQ'} on ${path} carries a client scope marker, but is not an emptiness condition`);
+    }
   }
   return { aggregateClauses, matchesClauses };
 };
@@ -440,9 +640,12 @@ const buildFilterGroupMatch = async (filterGroup, gqltype, aggregateClauses, agg
       const condition = normalizeFilterCondition(rawCondition);
       // An emptiness condition without a path tests its field, also an object field; a dotted field
       // is resolved as a whole, as the query plan resolves it.
+      const path = rawCondition.path ? `${rawCondition.field}.${rawCondition.path}` : rawCondition.field;
       if (isEmptinessOperator(condition.operator)) {
-        const path = rawCondition.path ? `${rawCondition.field}.${rawCondition.path}` : rawCondition.field;
-        parts.push(buildEmptinessClause(gqltype, path, condition.operator, condition.value));
+        const lookups = {};
+        const match = await buildEmptinessClause(gqltype, path, condition.operator, condition.value, rawCondition, lookups, aggregationsIncluded);
+        appendLookups(aggregateClauses, aggregationsIncluded, lookups);
+        parts.push(match);
         continue;
       }
       const qlField = fields[condition.field];
@@ -466,6 +669,9 @@ const buildFilterGroupMatch = async (filterGroup, gqltype, aggregateClauses, agg
       }
 
       const result = await buildQueryTerms(filterInput, qlField, condition.field, gqltype, aggregationsIncluded);
+      if (hasScopeMarker(rawCondition)) {
+        throw detachedFilter(`${condition.operator ?? 'EQ'} on ${path} carries a client scope marker, but is not an emptiness condition`);
+      }
       appendLookups(aggregateClauses, aggregationsIncluded, result.aggregateClauses);
       for (const matchClause of Object.values(result.matchesClauses)) {
         for (const [matchKey, match] of Object.entries(matchClause)) {

package/types/index.d.ts

@@ -92,13 +92,14 @@ export interface MongoAdapterOptions {
 export interface MongoAdapter extends DatabaseAdapter {
   readonly referentialIntegrity: 'off' | 'transactional';
   /**
-   * Frozen; lists every `QLOperator` value. The literal text operators compile to escaped regular
-   * expressions: `ILIKE` unanchored, `STARTS_WITH` and `ISTARTS_WITH` anchored with `^`, `ENDS_WITH` and
-   * `IENDS_WITH` with `\z`, and `IEQ` with both, with the `i` option for the case-insensitive ones.
-   * `IS_EMPTY` and `IS_NOT_EMPTY` compile to plain query operators on the record's own fields (see
-   * {@link buildQuery}); declaring them does not lift the `UNSUPPORTED_QUERY` that emptiness through
-   * references and collections gets in this release. A copy of the adapter without this member
-   * supports only the operators of 3.7.0.
+   * Frozen; lists every `QLOperator` value and declares `referenceEmptiness: true`. The literal text
+   * operators compile to escaped regular expressions: `ILIKE` unanchored, `STARTS_WITH` and
+   * `ISTARTS_WITH` anchored with `^`, `ENDS_WITH` and `IENDS_WITH` with `\z`, and `IEQ` with both, with
+   * the `i` option for the case-insensitive ones. `IS_EMPTY` and `IS_NOT_EMPTY` compile to plain query
+   * operators on the record's own fields, and through single references and at references and
+   * collections to joins and frames (see {@link buildQuery}). A copy of the adapter without this member
+   * supports only the operators of 3.7.0, and one whose declaration leaves out `referenceEmptiness`
+   * gets `UNSUPPORTED_QUERY` for emptiness through references and collections.
    */
   readonly queryFeatures: QueryFeatures;
   /**
@@ -259,9 +260,18 @@ export function preventCreatingCollection(prevent: boolean): void;
 export interface QueryLimitsOptions {
   /** Positive safe integer; defaults to 1000. Unpaged lists use min(100, maxPageSize). */
   maxPageSize?: number;
+  /**
+   * Positive safe integer; defaults to 10. The most distinct scoped joins and frames that the
+   * client's reference and collection emptiness may need in one list, aggregate or collection-field
+   * call; more fail with `FILTER_DEPTH_EXCEEDED` (400).
+   */
+  maxReferenceFrames?: number;
 }
 
-/** Configure at startup. Invalid configuration throws INVALID_QUERY_LIMITS (400). */
+/**
+ * Configure at startup. Invalid configuration throws INVALID_QUERY_LIMITS (400) and keeps both
+ * limits; an omitted option takes its default again.
+ */
 export function configureQueryLimits(options?: QueryLimitsOptions): void;
 
 /**
@@ -347,13 +357,34 @@ export interface FilterGroupInput {
  * without stored members is always empty (`{}` stands for the members). Embedded `id` members follow
  * the `.id` rule above, so an object of a supplied subdocument schema with an automatic `_id` is never
  * empty. A term without a path (omitted, null or `""`) and a group condition without a path test the
- * field itself. The checks, and their codes, are those of `createQueryPlan` in the same order:
- * `MISSING_FILTER_PATH` for an emptiness operator on an object filter without terms or beside terms,
- * `INVALID_FILTER_PATH` for a value inside an embedded list or a collection and for an interface or
- * union field or member, `INVALID_FILTER_OPERATOR` for an embedded type that reaches itself through
- * single embedded members, `INVALID_FILTER_VALUE` for a value that is not omitted or null, and
- * `UNSUPPORTED_QUERY` for a reference or collection on the path or among the members. Other operators
- * keep their pipelines.
+ * field itself.
+ *
+ * Their path may also cross single references and end at a reference or a collection, and an
+ * embedded object may have reference and collection members. A crossing reads its target through a
+ * `$lookup` and `$unwind` before the `$match`, and the target's `id` right after it is the joined
+ * `_id`, which takes the null rule. A reference or collection end or member is a frame, one `$lookup`
+ * with `pipeline: [{ $limit: 1 }, { $project: { _id: 1 } }]` under an alias `__sf_qN`, matched with
+ * `{ "__sf_qN.0": { $exists: false } }` (for a collection, `$or` a null local key, since `$lookup`
+ * matches a missing local key with every child whose connection field is null) and its `$nor`. A
+ * collection frame reads the child's connection field, and its local key is `<path>._id`. Arguments
+ * from this function, middleware and scopes read stored data: a reference is empty when no record it
+ * points to exists, so a dangling reference is empty, and leaves share the lookup aliases of the other
+ * operators. A term or group condition that the runtime marked for a client, `__scope: { id, scopes }`,
+ * reads through the target find scopes in `scopes`, compiled with this function into the `$lookup`
+ * pipelines: crossings become scoped joins (`__sf_jN`), frames take their target's scope, and a record
+ * that a scope hides reads as missing. A crossing or frame after a scoped crossing stays in that chain,
+ * also when its own scope is null. The aliases are unset at the end of find pipelines. A target type
+ * without a model fails with `INVALID_FILTER_PATH` (400), and a marker that is not filled for every
+ * path the construct reads, or that sits on a construct that reads none or on another operator, with
+ * `DETACHED_CLIENT_FILTER` (500), after the 400 checks. Supplied models on another database or
+ * connection and discriminator targets are not supported here, as for the other relation paths.
+ *
+ * The checks, and their codes, are those of `createQueryPlan` in the same order: `MISSING_FILTER_PATH`
+ * for an emptiness operator on an object filter without terms or beside terms, `INVALID_FILTER_PATH`
+ * for a value inside an embedded list or a collection and for an interface or union field or member,
+ * `INVALID_FILTER_OPERATOR` for an embedded type that reaches itself through single embedded members,
+ * and `INVALID_FILTER_VALUE` for a value that is not omitted or null. Other operators keep their
+ * pipelines.
  */
 export function buildQuery(
   input: Record<string, any>,
@@ -364,7 +395,9 @@ export function buildQuery(
 /**
  * Translate a QLFilterGroup into a MongoDB `$match` expression, collecting required lookups. A
  * condition path through a virtual field rejects with `INVALID_FILTER_FIELD` (400). `IS_EMPTY` and
- * `IS_NOT_EMPTY` conditions compile and reject as in {@link buildQuery}, and add no lookup.
+ * `IS_NOT_EMPTY` conditions compile and reject as in {@link buildQuery}: on the record's own fields
+ * they add no lookup, and through references and collections they append their joins and frames to
+ * `aggregateClauses` and record their aliases in `aggregationsIncluded`.
  */
 export function buildFilterGroupMatch(
   filterGroup: FilterGroupInput,