@procedures/mongo 0.35.1 → 0.37.0

package/CHANGELOG.md

@@ -1,5 +1,55 @@
 # @procedures/mongo
 
+## 0.37.0
+
+### Minor Changes
+
+- 1cb4ebf: `ViewMaterializer` writes the rows of a view as documents of a `MongoDocumentStorage` when the target is in `storageTargets` (roadmap CLI-81). Then `changes(since)` and `watchStorageChanges()` have each write of a run. For another target, the option `publish` gets one hint for each run.
+  
+  MINOR (pre-1.0): a run with a write conflict (code 112) tries again in place before it gives way. This is a changed default for each view. Before 1.0.0, a changed default raises the MINOR. The other parts are new optional options.
+  
+  - The new option `storageTargets` names the targets in a stored form, by the name of the target collection. Give a storage, a stored form, or `{ target, storagePartitionOf }` (the new types `ViewStorageTarget` and `ViewStorageTargetEntry`). A function `(collection) => entry | undefined` also works: the materializer uses it at each run, and keeps the storage that it made for the same form. The materializer writes with its own database. It takes the form, the counters, `writeLease` and `maxTombstones` of a storage.
+  - The materializer encodes each row as a write of the storage does. A row gets `_key`, `_seq`, `_ord` and the partition fields of the form. It also gets `_kid`, `_kpf` or `_json`. The fields of a storage in a row of the result do not go into the target. Thus `_json` comes from the content of the row, and each row has its own `_key`, `_kpf` and `_ord`.
+  - The id of a row is `storageRowIdOf(_id)` (new option). The default, the new export `defaultStorageRowId`, keeps a text as it is. It gives another value its relaxed Extended JSON text, for example `"null"` and `"5"`. Two rows of a result with the same id are an error of the view. A row also gets the partition field of the materializer, so a view that reads the target finds it.
+  - A run compares each row with its stored document, without the numbers. A row that did not change writes nothing, and a tombstone is not written again.
+  - A run takes pending numbers of the storage after its reads, writes with them in its transaction, and ends them after the commit. A late run moves to new numbers, so a reader of `changes(since)` misses no row. The transaction does not write the counters.
+  - The numbers are not noted as writes of this process. Thus a watcher of the target with `ownWrites: "skip"` publishes a hint for each write of a run, and has no raw write to adopt.
+  - A row that the result does not have becomes a tombstone, and the prune of the storage raises the floor of the change log. A document that tells no id becomes a tombstone without `_key`.
+  - A run that writes more than `maxTransactionRows` rows writes in place in groups, under its lease. A run also writes in groups when the target has more than `maxTransactionRows` live rows at its start. Its rows have no generation, so `viewFilter()` and `MongoViewStorage` do not read them.
+  - The new option `publish` gets the hints of the other targets. Each run that changes rows gives one hint, with the ids of its changed and removed rows. Some runs give a reset hint (`reset: true`): a new generation in groups, an id that is not a text, or more than 10000 ids. The new options `hintTopic` and `rowIdOf` give the topic (default `collectionChangeTopic(view._id)`) and the id of a stored `_id`.
+  - **A storage target partition must hold only the rows of this one view.** The run makes a tombstone of each other live document in that partition. A second view of the same target collection and partition of the storage gets the state `errored`.
+  - **A target collection gets no images before the change from the materializer.** Turn on `changeStreamPreAndPostImages` on it if raw hard deletes can occur.
+  - A transaction with a write conflict (code 112) tries again two times after a short wait on the clock of the materializer. Then the run gives way, and the view runs again after `rewatch`.
+  - Fix: the prune of the tombstones of `MongoDocumentStorage` raises the floor of the change log before it deletes the tombstones. Before, a reader far behind could read between the two steps and get a delta without the deletes.
+  
+  Migration:
+  
+  - No step for a target that is not in `storageTargets`.
+  - The site, MiniMongo, Graph and jarvis need no step, unless one of them uses `ViewMaterializer` with a target in a stored form. Graph sets `storageTargets` at its step 8.
+  - To move a target to a stored form, add it to `storageTargets` in each materializer of the views. The first pass writes each row as a document of the storage. A row of the older form with the partition of the storage becomes a tombstone. A reader then reads the target through a `MongoDocumentStorage` with the same form.
+
+### Patch Changes
+
+- 9659afd: The `ViewMaterializer` does not lose a change of a view document or of a source that comes at once after `start()`, or during the open of a stream (roadmap O63).
+  
+  PATCH: a fix, with no change to the API. Before 1.0.0, a fix raises the PATCH.
+  
+  - The cause: the driver sends the `aggregate` of a change stream only after `watch()` returns. `start()` read the views, and then started the stream of the views without a wait. A change of a view before the stream was open on the server did not reach the materializer.
+  - `start()` now waits until the stream of the views is open on the server, and then reads the views. A change between the two reaches both.
+  - A run of a view now starts after the streams of its sources are open on the server. A pass after a rewatch also waits for the new stream.
+  - The changes of the set of views run one at a time, in the order of the stream. A change of a view during a resync applies after the read of the resync.
+  - When a change of a view document replaces a view while its streams open, the run of the old view does not start.
+- @procedures/client@0.37.0
+  - @procedures/collections@0.37.0
+
+## 0.36.0
+
+### Patch Changes
+
+- Updated dependencies [f1fae86]
+  - @procedures/client@0.36.0
+  - @procedures/collections@0.36.0
+
 ## 0.35.1
 
 ### Patch Changes

package/README.md

@@ -797,13 +797,41 @@ const totals = await db.collection("orderTotals").find(viewFilter(view)).toArray
 - The option `targetId` gives another form of the stored `_id`. Two partitions, or two generations, must not give the same `_id`. The write of a row checks the partition, so the view of the second partition fails, and the row of the first one stays. A form without the generation works only in one transaction.
 - A pass keeps in memory the key of each row of the result (at most `maxRows`) and one group of rows. It keeps nothing between passes.
 - An app with targets from an older materializer (the plain key as `_id`, no generation) needs no migration step. The first pass of each view writes its rows in a generation, and removes the rows of its partition with no generation. A row without the partition field stays.
+- A transaction that has a write conflict with another process (code 112) tries again two times after a short wait. Then the run gives way, and the view runs again after `rewatch`. Thus a steady stream of writes does not stop a view.
+- The option `publish` (a function or a bus) gets the hints of the targets that are not in a stored form (roadmap CLI-81). Each run that changes rows publishes one hint with the ids of the changed rows and of the removed rows. `rowIdOf` gives the id of a stored `_id`. The topic is `collectionChangeTopic(view._id)`, or the result of `hintTopic(view, partition)`. Some runs publish a reset hint (`reset: true`): a new generation in groups, an id that is not a text, or more than 10000 ids. Then the subscribers read the view in full.
+
+#### A target in a stored form: `storageTargets`
+
+The option `storageTargets` names the targets whose rows are documents of a `MongoDocumentStorage` (roadmap CLI-81). Give the storage of each target, its stored form, or `{ target, storagePartitionOf }`, by the name of the target collection. A reader then reads the target through a `MongoDocumentStorage` with the same form, and `changes(since)` and `watchStorageChanges()` have each write of a run.
+
+```typescript
+const totals = new MongoDocumentStorage({ collection: "orderTotals", db: () => db });
+const views = new ViewMaterializer({ db: () => db, storageTargets: { orderTotals: totals } });
+const changes = await totals.forPartition("t1").getChangesSince(version);
+```
+
+> **Warning:** A storage target partition must hold only the rows of this one view. The run makes a tombstone of each other live document in that partition. The materializer refuses a second view of the same target collection and partition of the storage: that view gets the state `errored`.
+>
+> **Warning:** A target collection gets no images before the change from the materializer. Turn on `changeStreamPreAndPostImages` on the target if raw hard deletes can occur.
+
+`storageTargets` can also be a function, `(collection) => entry | undefined`, when the app names its targets at run time. The materializer uses it at each run. It keeps the storage that it made for a collection while the entry gives the same form, counters, `writeLease` and `maxTombstones`.
+
+- The materializer encodes each row as a write of the storage does. A row gets `_key`, `_seq`, `_ord` and the partition fields of the form. It also gets `_kid`, `_kpf` or `_json` when it needs them. The fields of a storage in a row (for example of a source document that the pipeline passes through) do not go into the target. Thus `_json` comes from the content of the row, and each row has its own `_key`, also after `$unwind`, `$lookup` or `$unionWith`.
+- The id of a row is `storageRowIdOf(_id)`. The default `defaultStorageRowId` keeps a text `_id` as it is. Another value gets its relaxed Extended JSON text: `null` gives `"null"`, `5` gives `"5"`, and an ObjectId gives `{"$oid":"…"}`. Thus a `$group` with `_id: null` or a number key works, and a row keeps its id at each run. The item has that id as its `_id`. Two rows of a result must not have the same id.
+- Each row also gets the partition field of the materializer with the partition of the view, so a view that reads the target finds it. When that field is a partition field of the form, the form must give it the partition of the view. `storagePartitionOf` gives the partition of the storage from the partition of the view (default the same partition).
+- A run compares each row with its stored document, without the numbers. A row that did not change writes nothing, and a tombstone is not written again. A row keeps the place (`_ord`) of its stored document.
+- A run takes the numbers of its writes from the sequence of the storage, as pending numbers. It writes with them in its transaction, and ends them after the commit. A late run (its numbers left their `writeLease`) moves to new numbers, so a reader of `changes(since)` misses no row. The transaction does not write the counters, so the writes of the storage do not conflict with it.
+- The materializer does not note its numbers as writes of this process. Thus a watcher of the target with `ownWrites: "skip"` publishes a hint for each write of a run, with its version. It has no raw write to adopt.
+- A row that the result does not have becomes a tombstone. The prune of the tombstones (`maxTombstones`) raises the floor of the change log. Thus a reader that is far behind gets a full result, and it misses no delete. A document of the partition that tells no id becomes a tombstone without `_key`.
+- A run that writes more than `maxTransactionRows` rows writes in place in groups, under its lease. A run also writes in groups when the target has more than `maxTransactionRows` live rows at its start, or when its transaction fails. A reader can then read a part of the run.
+- The rows have no generation, so `viewFilter()` and `MongoViewStorage` do not read them. Give the same `storageTargets` to each materializer of the views.
 
 To serve a view through the collection procedures, give its storage `new MongoViewStorage({ view: "totals", db })` (roadmap CLI-38):
 
 - Each read finds the current generation of the view one time, and reads only its rows. A reader of `get`, `getAll`, `find` or `changes` thus gets one generation only.
 - A read that starts before a change of generation ends on the previous generation. The materializer keeps its rows for `generationGrace`, so a read must end in that time.
 - The key of a row is its `_id` in the result, a text. The value has that `_id` and the fields of the row.
-- The storage is read-only: a write rejects with `FORBIDDEN`. It has no change log.
+- The storage is read-only: a write rejects with `FORBIDDEN`. It has no change log. The option `publish` of the materializer gives its subscribers a hint for each run.
 - Give it the same `views`, `partitionField`, `statusField` and `targetId` as the materializer.
 
 ## Integration with Ecosystem