@procedures/mongo 0.9.1 → 0.10.0

package/CHANGELOG.md

@@ -1,5 +1,36 @@
 # @procedures/mongo
 
+## 0.10.0
+
+### Patch Changes
+
+- Updated dependencies [254ee7b]
+  - @procedures/client@0.10.0
+  - @procedures/collections@0.10.0
+
+## 0.9.2
+
+### Patch Changes
+
+- e192622: New `ViewMaterializer`: materialized views driven by change streams. A view is a document of a views collection, with a pipeline over source collections and a target collection, for one partition. The materializer computes each view again when its sources change, in the order of the dependencies. It keeps the status of each view (`ok`, `errored`, `cycle`).
+  
+  - The rows of a view are a generation, and the status names the current one. A reader filters with the new `viewFilter(view)`, so it reads one whole generation.
+  - A run writes only the rows that changed, and removes the rows that the result does not have, in one transaction. A larger view (`maxTransactionRows`), or a failed transaction, writes a whole new generation in groups. The previous generation stays for `generationGrace`. A pass in groups holds a lease on its view (`passLease`), so two processes can keep the same views. The time options are durations of `@procedures/client`.
+  - The stored `_id` of a row carries its partition and its generation: `{ partition, generation, id }` by default (`partitionTargetId`), or the form of the option `targetId`. Two partitions with the same keys share a target.
+  - Targets from an older materializer, with the plain key as `_id`, need no migration step. The first pass of each view replaces the rows of its partition.
+  - Each pipeline goes through `scopePipeline` by default. It reads only its partition, in its sources, and the current generation of a target.
+  
+  The code comes from the server of Graph (its split plan, part P4). The jobs of CI with a replica set start MongoDB with the test commands, for the failpoint tests.
+- 054ba7d: `ViewMaterializer`: a change reaches only the views of its partition when the partition is known (roadmap CLI-37). The filter only removes work: a change whose partition is unknown reaches the views of each partition, as before.
+  
+  - A delete has no document in its change event. Its partition comes from the pre-image, when its collection has `changeStreamPreAndPostImages`.
+  - The new option `storageSources` gives, for a collection of `MongoDocumentStorage`, the function from the storage partition (at the start of the id) to the partition of the views. `identityPartition` is for two partitions that are the same.
+  - With pre-images, an update that moves a document to another partition also reaches the views of its old partition.
+  - Without pre-images, a replace and an update of the partition field reach the views of each partition, as a delete does. The document can leave any partition.
+  - With `sourceFields`, the keys of an update are paths with dots: a `$push` (`tags.3`), a `$set` of `a.b`, a removal or a truncated array. Each key counts at the field, under it or above it, and the positions in arrays do not count (`lines.0.price` is `lines.price`). An update of the partition field starts a pass too.
+- @procedures/client@0.9.2
+  - @procedures/collections@0.9.2
+
 ## 0.9.1
 
 ### Patch Changes

package/README.md

@@ -699,6 +699,49 @@ const { pipeline: bounded, limited } = boundPipeline(scoped, 5000);
 - The function does not change the pipeline of the caller.
 - `MongoDocumentStorage.aggregate(pipeline, { maxTime, maxResults })` is read-only, reads only its own collection, and keeps the tombstones out. It gives `{ docs, truncated }`. The defaults are `seconds(10)` and 5000 documents. `maxTime` is a duration of `@procedures/client`.
 
+### Materialized views: `ViewMaterializer`
+
+A view is a document of a views collection: a pipeline over source collections, whose result goes into a target collection, for one partition. `ViewMaterializer` keeps each target equal to the result of its pipeline, with change streams. Change streams need a replica set.
+
+```ts
+import { ViewMaterializer, viewFilter } from "@procedures/mongo";
+
+await db.collection("_aggregations").insertOne({
+  _id: "totals",
+  _part: "tenant-1",
+  collections: ["orders"],
+  pipeline: [{ $group: { _id: "$status", count: { $sum: 1 } } }],
+  targetCollection: "orderTotals",
+});
+const views = new ViewMaterializer({ db: () => db });
+await views.start();
+
+// A reader of the target reads the current generation of the view
+const view = await db.collection("_aggregations").findOne({ _id: "totals" });
+const totals = await db.collection("orderTotals").find(viewFilter(view)).toArray();
+```
+
+- A change of a source starts a pass of its partition after the burst (`debounce`). A pass computes only the views whose sources changed, in the order of their dependencies. A view whose result changed makes its target dirty, so the views that read it are in the same pass.
+- A run reads the result in groups of 1000. Each stored row keeps the hash of its content (`_viewHash`), so a row that did not change writes nothing.
+- The rows of a view are a generation (`_viewGen`), and the status of the view names the current one. A reader of a target filters with `viewFilter(view)`, so it reads one whole generation. A plain query can see two generations.
+- A run writes in one transaction: the changed rows of the current generation, and the removal of the rows that the result does not have. A reader does not see a part of a run.
+- A view with more rows than `maxTransactionRows` (default 10000), or whose transaction fails, writes a whole new generation in groups. Then its status names the new generation. The previous one stays for `generationGrace` (default 60 s, on the time of the server), so a reader of it can finish. Then it goes.
+- A pass in groups holds a lease on its view (`passLease`, default 60 s, on the time of the server), and renews it after each group. Another process leaves the view and the rows of the lease alone. The change of generation needs the lease and the generation of the start of the pass. A pass that loses either one gives way, and the view runs again later.
+- A view in a cycle of views stays idle. Its status is `cycle` until the cycle ends.
+- The status of each view (`ok`, `errored` with `lastError`, `cycle`) goes into the field `_materializerStatus` of its document.
+- Each pipeline goes through `scopePipeline` by default. It reads only the documents of the partition, in its sources, and only the current generation of the target of another view. It cannot have `$merge` or `$out`. The option `scope` gives another guard, with the same filter.
+- A stream that ends opens again after `rewatch`, and a pass catches up. The times of the options are durations (`ms(100)`, `seconds(60)`) of `@procedures/client`.
+- A delete has no document in its change event. It reaches only the views of its partition when its pre-image tells the partition (a collection with `changeStreamPreAndPostImages`). Without a pre-image, a delete, a replace and an update of the partition field reach the views of each partition.
+- In a source of `storageSources`, the id of a `MongoDocumentStorage` document tells the storage partition of such a change. The function of that source gives the partition of the views (`identityPartition` when the two are the same). Else the delete reaches the views of each partition.
+- With pre-images, an update that moves a document to another partition also reaches the views of its old partition.
+- `sourceFields` gives the fields of each source, as paths with dots. An update starts a pass when it changes one of them (at the field, under it or above it), or the partition field. The positions in arrays do not count: `lines.0.price` is `lines.price`. The partition field is a top-level field.
+- The stop has the parts of a transport of a service: `stopIntake()`, `drain(deadline, signal)`, `inFlight()` and `close()`. `stop()` does the four. A host composes them with no adapter.
+- The option `clock` gives the clock of the timers, for example a fake clock in a test. The timers do not keep a process alive.
+- The target is a plain collection that only the materializer writes. Two views of one partition do not share a target. The stored `_id` of a row carries its partition and its generation: by default `{ partition, generation, id }` (`partitionTargetId`), where `id` is the `_id` of the row. Thus two partitions with the same keys share a target.
+- The option `targetId` gives another form of the stored `_id`. Two partitions, or two generations, must not give the same `_id`. The write of a row checks the partition, so the view of the second partition fails, and the row of the first one stays. A form without the generation works only in one transaction.
+- A pass keeps in memory the key of each row of the result (at most `maxRows`) and one group of rows. It keeps nothing between passes.
+- An app with targets from an older materializer (the plain key as `_id`, no generation) needs no migration step. The first pass of each view writes its rows in a generation, and removes the rows of its partition with no generation. A row without the partition field stays.
+
 ## Integration with Ecosystem
 
 ### Bundle Packages
@@ -737,42 +780,12 @@ const call = createTypedCaller<AllProcedures>(client.call);
 
 ## Package Structure
 
-```
-client-mongo/
-├── src/
-│   ├── connection.ts           # Connection management
-│   ├── types.ts                # Core type definitions
-│   ├── procedures.types.ts     # Typed procedures interface
-│   ├── register.ts             # Auto-registration
-│   ├── index.ts                # Main entry point
-│   ├── procedures/
-│   │   ├── schema.ts           # Minimal schema helpers
-│   │   ├── index.ts            # Procedure barrel export
-│   │   ├── database.ping.ts
-│   │   ├── database.info.ts
-│   │   ├── collections.list.ts
-│   │   ├── collections.create.ts
-│   │   ├── collections.drop.ts
-│   │   ├── collections.stats.ts
-│   │   ├── documents.find.ts
-│   │   ├── documents.get.ts
-│   │   ├── documents.insert.ts
-│   │   ├── documents.update.ts
-│   │   ├── documents.delete.ts
-│   │   ├── documents.count.ts
-│   │   ├── documents.aggregate.ts
-│   │   ├── documents.distinct.ts
-│   │   ├── documents.bulk.ts
-│   │   ├── documents.explain.ts
-│   │   ├── indexes.list.ts
-│   │   ├── indexes.create.ts
-│   │   ├── indexes.drop.ts
-│   │   └── watch.ts            # mongo.watch
-├── dist/                       # Compiled output
-├── package.json
-├── tsconfig.json
-└── README.md
-```
+- `src/index.ts`: the entry. It registers nothing.
+- `src/module.ts`: `mongoModule`, the procedures as data. `src/register.ts` registers them in the global registry.
+- `src/connection.ts`: the connection and its management.
+- `src/scope.ts`: the database and the collection of a call.
+- `src/procedures/`: one file for each procedure, with the schemas of the inputs.
+- `src/storage/`: `MongoStorage`, `MongoDocumentStorage`, `scopePipeline` and `ViewMaterializer`.
 
 ## Development