coll-fns 1.4.3 → 1.6.0

package/README.md

@@ -13,6 +13,7 @@ Stop repeating business logic all over your code base. Define hooks on collectio
 - [Rationale](#rationale)
 - [Installation and configuration](#installation-and-configuration)
   - [`setProtocol(protocol)`](#setprotocolprotocol)
+    - [Execution context (`bindEnvironment`)](#execution-context-bindenvironment)
   - [Bypassing `coll-fns`](#bypassing-coll-fns)
 - [Joins and fetch](#joins-and-fetch)
   - [Quick start examples](#quick-start-examples)
@@ -53,6 +54,18 @@ Stop repeating business logic all over your code base. Define hooks on collectio
     - [Default behavior](#default-behavior)
     - [Options](#options)
   - [Hook best practices](#hook-best-practices)
+- [Nested reactive publications](#nested-reactive-publications)
+  - [When to use `publish`](#when-to-use-publish)
+  - [`publish(publication, Coll, selector, options)`](#publishpublication-coll-selector-options)
+  - [Publication readiness](#publication-readiness)
+  - [Publication context (`this` in Meteor)](#publication-context-this-in-meteor)
+  - [How child declarations work](#how-child-declarations-work)
+  - [Ancestors chain](#ancestors-chain)
+  - [How `deps` works](#how-deps-works)
+  - [Debugging](#debugging)
+  - [Built-in optimizations](#built-in-optimizations)
+  - [Using `publish` outside Meteor](#using-publish-outside-meteor)
+  - [Current limitations](#current-limitations)
 - [License](#license)
 
 # Rationale
@@ -122,7 +135,7 @@ And since it uses a protocol, it can be used with the **native MongoDB driver**
 
 </details>
 
-<details style="margin-bottom: 1rem">
+<details>
 <summary><strong>Functional API</strong></summary>
 
 A lot of libraries that add functionalities to the database layer mutate the collection instances themselves or, when more respectful, offer ways to extend the collection constructor somehow.
@@ -139,6 +152,26 @@ For Meteor developers, it also means being able to enhance the `Meteor.users` co
 
 </details>
 
+<details style="margin-bottom: 1rem">
+<summary><strong>Nested reactive publications</strong></summary>
+
+In Meteor in particular, publication of data over DDP that gets synced on the client in Minimongo is very convenient. It makes for really reactive applications where data doesn't get stale. Optimistic UI optimizations is built-in and doesn't require additional complex logic.
+
+However, it can get **difficult to publish exactly the right documents**, especially when using a very useful library that allows joining collections together and keeping data much more normalized!
+
+Although returning Meteor cursors from a publish function is still the most optimized path, it sometimes makes sense to **publish children documents based on their relationship with their parent**. But doing so is usually very difficult to implement.
+
+Some existing community libraries that promise to do so either:
+
+- don't actually work (might understand `added` and `removed` callbacks, but won't react to updates)
+- are too simplistic (cause a lot of duplicated observers)
+- come with a much more complex data layer
+- add somewhat unpredictable reactivity on the server.
+
+Since `coll-fns` is already very good at understanding relationships, a `publish` helper was introduced to allow such fine-tuned reactivity out of the box. 💪
+
+</details>
+
 # Installation and configuration
 
 **IMPORTANT**: For concision, the **examples will use the synchronous Meteor protocol** to avoid `async/await` boilerplate. Of course, your code will have to be adapted when used with an asynchronous protocol.
@@ -220,10 +253,19 @@ const customProtocol = {
    * and return the inserted _id. */
   insert(/* Coll, doc, options */) {},
 
+  /* Observe document changes.
+   * Must return a handle with a `stop()` method. */
+  observe(/* Coll, selector = {}, callbacks = {}, options = {} */) {},
+
   /* Remove documents in a collection
    * and return the number of removed documents. */
   remove(/* Coll, selector, options */) {},
 
+  /* Stable stringify function used for internal query keys.
+   * EJSON canonical stringify is used as a good default,
+   * but can be overridden. */
+  stringify(/* value */) {},
+
   /* Update documents in a collection
    * and return the number of modified documents. */
   update(/* Coll, selector, modifier, options */) {},
@@ -232,6 +274,33 @@ const customProtocol = {
 setProtocol(customProtocol);
 ```
 
+`observe` expected shape:
+
+```js
+observe(Coll, selector = {}, callbacks = {}, options = {}) {
+  const { added, changed, removed } = callbacks;
+
+  // Your implementation subscribes reactively to selector/options changes.
+  // Callbacks should be invoked with:
+  // - added(id, fields)
+  // - changed(id, fields)
+  // - removed(id)
+  //
+  // Then return an object exposing a stop() method.
+  return {
+    stop() {
+      // Tear down underlying observer/resources.
+    },
+  };
+}
+```
+
+Notes:
+
+- `observe` can be sync or async (returning a Promise of the stop-handle).
+- `fields` should contain changed/added fields payload expected by your transport.
+- `publish()` relies on this contract to keep nested observers in sync.
+
 If your runtime has special callback context requirements, implement `bindEnvironment`.
 
 ```js
@@ -248,6 +317,11 @@ const customProtocol = {
 
 If your runtime has no such requirement, simply omit `bindEnvironment`.
 
+If you want to use the [`publish()`](#publishpublication-args-options) composite publication helper:
+
+- `observe` must be defined on the protocol.
+- `stringify` can be defined if the default EJSON implementation is not sufficient.
+
 </details>
 
 ## Bypassing `coll-fns`
@@ -449,7 +523,7 @@ join(Workers, {
 
 When joins are too complex to be defined with an array or object (although rare), a function can be used as the `on` property. Each parent document will be passed to this function, which should return a selector to use on the child collection.
 
-When using function-based joins, **a `fields` property should be added** to the join definition to declare which fields the parent document needs for the join to work:
+When using function-based joins, **a `deps` property should be added** to the join definition to declare which parent fields are required for the join to work:
 
 ```js
 import { join } from "coll-fns";
@@ -472,7 +546,7 @@ join(Posts, {
       };
     },
     /* Parent fields needed in the join function */
-    fields: {
+    deps: {
       _id: 1, // Optional. _id is implicit in any fetch.
       postedAt: 1,
     },
@@ -480,6 +554,8 @@ join(Posts, {
 });
 ```
 
+`fields` remains accepted as a backward-compatible alias for `deps`.
+
 ### Recursive joins
 
 A collection can define joins on itself.
@@ -499,7 +575,7 @@ join(Users, {
 
 ### Join additional options
 
-Any additional properties defined on the join (other than `Coll`, `on`, `single`, `postFetch`) will be treated as options to pass to the nested documents `fetchList`. It usually includes:
+Any additional properties defined on the join (other than `Coll`, `on`, `single`, `postFetch`, `deps` and legacy `fields`) will be treated as options to pass to the nested documents `fetchList`. It usually includes:
 
 - `limit`: Maximum joined documents count
 - `skip`: Documents to skip in the fetch
@@ -509,7 +585,7 @@ Any additional properties defined on the join (other than `Coll`, `on`, `single`
 
 Children documents might need to be modified (transformed, ordered, filtered...) after being fetched. The `postFetch: (childrenDocs, parentDoc) => childrenDocs` join definition property can be used to do so.
 
-The second argument of the function is the parent document. If some of its properties are needed, they should be declared in the `fields` property to ensure they are not missing from the requested fetched fields.
+The second argument of the function is the parent document. If some of its properties are needed, they should be declared in the join `deps` property so they are guaranteed to be fetched on the parent.
 
 ```js
 import { join } from "coll-fns";
@@ -521,8 +597,8 @@ join(Resources, {
     Coll: Tasks,
     on: ["_id", "resourceId"],
 
-    /* Ensure `tasksOrder` will be fetched */
-    fields: { tasksOrder: 1 },
+    /* Ensure `tasksOrder` will be fetched on parent docs */
+    deps: { tasksOrder: 1 },
 
     /* Transform the joined tasks documents based on parent resource. */
     postFetch(tasks, resource) {
@@ -1521,6 +1597,370 @@ hook(Users, {
 
 </details>
 
+# Nested reactive publications
+
+If `coll-fns` is used in a Meteor project, using publications is a way to create fully reactive applications. `publish` function helps to publish complex hierarchical data.
+
+## When to use `publish`
+
+Use `publish()` when the publication tree is dynamic and cannot be represented as a simple static list of cursors.
+
+**Important caveat (Meteor):** if a publication can return plain cursor(s) directly from `Meteor.publish`, prefer that approach. Native cursor-return publications are simpler and usually more optimized by Meteor internals than any userland helper.
+
+`publish()` is intended for cases where you need one or more of:
+
+- nested reactive children depending on parent documents
+- selector recomputation based on changed parent fields
+- observer reuse and invalidation logic you do not want to hand-roll repeatedly
+
+## `publish(publication, Coll, selector, options)`
+
+Create a reactive publication tree (Meteor-style) with support for:
+
+- explicit child observers (`{ Coll, on, ... }`)
+- join shorthand children (`{ join: "joinKey", ... }`)
+- implicit join children derived from parent requested join fields
+
+`publish` internally uses protocol methods:
+
+- `observe` to track cursor changes
+- `getName` to emit DDP collection names
+- `stringify` to build stable query reuse keys
+
+As with normal joins, child `on` values can be:
+
+- static selector objects
+- functions `(parent, ...ancestors) => selector`
+- join-array selectors: `[from, to, toSelector?]`
+
+Refer to the [`join()`](#joincoll-joindefinitions) section for more details.
+
+```js
+import { Meteor } from "meteor/meteor";
+import { join, publish, setJoinPrefix } from "coll-fns";
+import {
+  Posts,
+  Users,
+  Comments,
+  Tags,
+  FeatureFlags,
+  PostStats,
+} from "/imports/api/collections";
+
+setJoinPrefix("+");
+
+join(Posts, {
+  author: { Coll: Users, on: ["authorId", "_id"], single: true },
+  comments: { Coll: Comments, on: ["_id", "postId"], sort: { createdAt: -1 } },
+  stats: { Coll: PostStats, on: ["_id", "postId"], single: true },
+});
+
+join(Comments, {
+  author: { Coll: Users, on: ["authorId", "_id"], single: true },
+});
+
+Meteor.publish("posts.tree", function postsTree() {
+  return publish(
+    this,
+    Posts,
+    { status: "published" },
+    {
+      fields: {
+        title: 1,
+        authorId: 1,
+        tagIds: 1,
+        editorId: 1,
+        "+": {
+          stats: 1,
+          comments: {
+            body: 1,
+            createdAt: 1,
+            "+": {
+              author: { displayName: 1 },
+            },
+          },
+        },
+      },
+      children: [
+        /* Predefined join on Posts collection */
+        {
+          join: "author",
+          fields: { displayName: 1, avatarUrl: 1 },
+        },
+
+        /* Array selector */
+        {
+          Coll: Tags,
+          on: [["tagIds"], "_id"],
+          fields: { label: 1, color: 1 },
+          deps: undefined, // Array selector children implicitely derive deps
+        },
+
+        /* Function selector */
+        {
+          Coll: Users,
+          on: (post) => ({ _id: post.editorId }),
+          fields: { displayName: 1 },
+          deps: ["editorId"],
+        },
+
+        /* Object selector. Always returns the same data irrespective of parent. */
+        {
+          Coll: FeatureFlags,
+          on: { scope: "posts_publication" },
+          fields: { key: 1, enabled: 1 },
+          deps: false,
+        },
+      ],
+    }
+  );
+});
+```
+
+## Publication readiness and concurrency
+
+Readiness is controlled per child with `awaited` (default: `true`):
+
+- `awaited: true`: wait for this child subtree before calling `ready()`
+- `awaited: false`: do not wait for this child subtree; it loads/reacts in background
+
+Use `awaited: true` for data the screen must have immediately, and `awaited: false` for optional or heavy branches.
+If not specified, a child inherits `awaited` from its parent.
+
+`options.maxConcurrent` controls how many child observer creations can run at
+the same time (`10` by default). This can only be defined at the publication root, not on children.
+
+Why it matters:
+
+- each parent `added`/`changed` can trigger many child observer creations
+- unbounded concurrency can create CPU spikes and DB pressure
+- too little concurrency can slow initial publication warm-up
+
+What it controls exactly:
+
+- only child observer creation tasks are throttled
+- observer reuse still applies (duplicate query keys are collapsed)
+- invalidation logic still runs; the option just smooths creation bursts
+
+Practical guidance:
+
+- decrease it if your publication causes heavy DB load or event-loop stalls
+- increase it if your DB/runtime handles parallelism well and warm-up is slow
+- keep in mind this is per `publish()` call, not a single global cap
+
+Example:
+
+```js
+Meteor.publish("posts.tree", function postsTree() {
+  return publish(
+    this,
+    Posts,
+    { status: "published" },
+    {
+      maxConcurrent: 5,
+      children: [
+        { Coll: Users, on: ["authorId", "_id"], awaited: true },
+        {
+          Coll: FeatureFlags,
+          on: { scope: "posts_publication" },
+          awaited: false,
+        },
+      ],
+    }
+  );
+});
+```
+
+## Publication context (`this` in Meteor)
+
+`publish()` is designed first for Meteor publications. In Meteor usage, the first
+argument should be the publication session/context (`this`) received in
+`Meteor.publish(name, function () { ... })`.
+
+Minimal expected shape of `publication`:
+
+- `ready()` (required)
+- `added(collectionName, id, fields)` (optional but normally provided by Meteor)
+- `changed(collectionName, id, fields)` (optional but normally provided by Meteor)
+- `removed(collectionName, id)` (optional but normally provided by Meteor)
+- `onStop(fn)` (optional, used for cleanup registration)
+- `error(err)` (optional, used as error sink)
+
+Example:
+
+```js
+Meteor.publish("posts.tree", function () {
+  // `this` is the publication context/session.
+  return publish(this, Posts, { status: "published" });
+});
+```
+
+If `ready` is missing, `publish()` throws.
+
+## How child declarations work
+
+`children` entries can be objects or falsy values (`false`, `null`, `undefined`).
+Falsy entries are ignored, which allows short-circuit declarations like
+`isEnabled && { ...childArgs }`.
+
+Object entries can be defined with either:
+
+- explicit child args:
+  - `{ Coll, on, fields?, deps?, awaited?, children?, ...cursorOptions }`
+- join shorthand:
+  - `{ join: "joinKey", ...overrides }`
+
+Additionally, join children can be derived implicitly from requested parent join fields.
+
+Conflict rule:
+
+- If the same join key is declared both as explicit child (`{ join: "..." }`) and in parent join fields (`fields["+"][joinKey]` or root join key without prefix), `publish()` throws and asks you to choose one style.
+
+## Ancestors chain
+
+For function selectors and function deps, the helper passes:
+
+- first argument: direct parent document
+- remaining arguments: full ancestors chain (grandparent, great-grandparent, ...)
+
+This is useful when deep children must depend on context from higher levels.
+
+```js
+Meteor.publish("resources.tasks.actions", function () {
+  return publish(
+    this,
+    Resources,
+    { archived: false },
+    {
+      children: [
+        {
+          Coll: Tasks,
+          on: (resource) => ({ resourceId: resource._id }),
+          deps: ["_id"],
+          children: [
+            {
+              Coll: Actions,
+              on: (task, resource) => ({
+                taskId: task._id,
+                tenantId: resource.tenantId,
+              }),
+              deps(changedFields, task, resource) {
+                // Re-run when task link changes or resource tenancy changes.
+                if ("tenantId" in changedFields) return true;
+                return ["_id", "tenantId"];
+              },
+            },
+          ],
+        },
+      ],
+    }
+  );
+});
+```
+
+## How `deps` works
+
+`deps` controls when child observers are invalidated and recomputed after parent `changed` events.
+
+Supported values:
+
+- `true`: always invalidate
+- `false`: never invalidate
+- `"field"` or `["fieldA", "fieldB"]`: invalidate only when those keys are present in changed fields
+- `{ fieldA: 1, fieldB: true }`: object shorthand converted to watched keys (truthy top-level keys only)
+- function `(changedFields, parent, ...ancestors) => depsLike`: dynamic rule
+- `undefined`: special behavior
+
+Special behavior when `deps` is `undefined`:
+
+- static selector object child: treated as no invalidation (`[]`)
+- array/function selector child: treated as potentially dependent and will always invalidate conservatively
+
+For array selectors, implicit deps are auto-derived from the `from` key.
+
+`deps` matching is flat:
+
+- matching is done against exact keys present in `changedFields`
+- no deep path traversal is performed by `publish()`
+- nested object deps only contribute top-level keys
+
+## Debugging
+
+`publish()` supports lightweight lifecycle debugging with:
+
+- `debug: true` to log all internal debug events
+- `debug: ["EVENT_NAME", ...]` to log selected events only
+- `debug: { EVENT_NAME: true, ... }` to log selected events only
+
+Observer events:
+
+- `CREATED`: a new observer was successfully created and registered.
+- `BYPASSED`: observer creation was skipped because selector resolved to a void selector.
+- `REUSED`: an existing observer for the same query key was reused.
+- `INVALIDATED`: child observer graph for a parent document was recomputed after relevant parent changes.
+- `UNFOLLOWED`: one follower link to a sub-observer was removed.
+- `CANCELLED`: an observer was cancelled and its local cleanup started.
+
+Observer documents events:
+
+- `DOC_ADDED`: a document was published through `added`.
+- `DOC_CHANGED`: a published document emitted a `changed` update.
+- `DOC_REMOVED`: a document was removed from publication (observer count for that document dropped to zero).
+
+Publication events (available on root only):
+
+- `READY`: emitted once when `publish()` calls `publication.ready()`.
+- `STOPPED`: emitted once when the publication stop handler runs.
+
+Debug scope:
+
+- Root `debug` applies to the root observer and root publication locations (`READY`, `STOPPED`).
+- Each child defines its own `debug` argument.
+- Implicit join-derived children (from parent `fields`) inherit parent `debug`.
+
+## Built-in optimizations
+
+`publish()` is designed to stay controlled even with nested reactive trees.
+
+In practical terms, it aims to protect you from:
+
+- creating the same observer repeatedly for equivalent child queries
+- runaway bursts of child observer creation
+- stale async creations being attached after data already changed
+- leaked child observers when parent links disappear
+- duplicate add/remove churn for documents shared by multiple branches
+
+Why this matters:
+
+- lower risk of memory growth from forgotten/stale observers
+- fewer unnecessary observers and DB watches
+- more predictable behavior during frequent parent changes
+- safer use of nested publications in real apps, not only toy examples
+
+`maxConcurrent` is part of this safety model: it prevents uncontrolled parallel
+creation bursts and lets you tune throughput vs load.
+
+## Using `publish` outside Meteor
+
+The helper can be used outside Meteor only if both layers are provided:
+
+- protocol layer (`setProtocol`) supporting at least:
+  - `observe`
+  - `getName` (optional, default implementation provided)
+  - `stringify` (optional, default implementation provided)
+- publication transport/context object implementing the callbacks [listed above](#publication-context-this-in-meteor)
+  (`added/changed/removed/ready/onStop/error`)
+
+Protocol methods handle database reactivity.
+`publication` handles how data changes are emitted to clients.
+
+## Current limitations
+
+- `publish()` is a helper, not a replacement for simple cursor-return publications.
+- Deep/dynamic trees can still be expensive if selectors are broad and highly volatile.
+- If `deps` are too broad (or omitted for dynamic selectors), invalidations may be frequent.
+- For best results, keep parent selectors selective and declare precise `deps`.
+
 # License
 
 MIT