@procedures/mongo 0.34.0 → 0.35.1

package/CHANGELOG.md

@@ -1,5 +1,74 @@
 # @procedures/mongo
 
+## 0.35.1
+
+### Patch Changes
+
+- 9f0405d: `watchStorageChanges()` publishes the change stream of a `MongoDocumentStorage` collection as collection hints, and `versionOfStored()` gives the version of a raw stored document (roadmap CLI-77).
+  
+  PATCH: the change only adds exports and new optional options, and keeps each behavior of 0.35.0 (the watcher is not released yet). Before 1.0.0, a new feature raises the PATCH. The fixed group shares one version, so this rides the next release (0.35.0 when it is not out yet).
+  
+  - `versionOfStored(doc, form?, partition?)` gives the version that `storage.versionOf(id)` gives for a stored document: `_seq` as text, `0` for a document without `_seq`, and `undefined` for a tombstone or a document without an id. It gives the value that `versionOf()` reads.
+  - `watchStorageChanges(storage, publish, options?)` watches the change stream of the collection. It publishes each change on the topic of the collection procedures, `collectionChangeTopic(name, partition)`, with the partition from `storage.form.partitionOf()`. The event is a `CollectionChangeEvent` without `since` (a hint), with the id in `changed` or `deleted` and the version of the item in `versions`. A tombstone and a delete are deletes. It returns `{ ready, stop() }`.
+  - With `ownWrites: "skip"` (the default), the watcher skips the events of the writes of this process through the storage, because the collection procedures publish them already. Each write of the storage notes its sequence numbers while a watcher is open. `ownWrites: "publish"` publishes each change.
+  - With `rawWrites: "adopt"` (the default), the watcher gives a raw write a new `_seq` through the write path of the storage, so `changes(since)` has it. It reads from the event alone whether a write is raw. It gives back the storage fields and the partition fields that a raw replace took, and never changes a field of the item. The write matches only the state that the event saw, so one of two racing watchers wins and the other does nothing. The event of that write publishes the hint with the new version; a raw write does not carry its old version. The adoptions of a watcher wait in one bounded queue, one at a time, in batches that take one block of numbers. The watcher keeps the ids of the raw writes that do not fit in a full queue or whose last try failed (up to 10,000), and adopts them again when the queue is empty, with the documents then. A failure that can go away tries again after the backoff. A failed last try, a second failure, or an id over that limit gives the partition a reset hint. The adoption is exact only with `changeStreamPreAndPostImages`. With the images before the change (`changeStreamPreAndPostImages`), a raw delete becomes a tombstone; without them, it is a hint without a version. A prune of tombstones publishes nothing. `rawWrites: "ignore"` writes nothing and publishes a hint without a version. A read-only user gets the same hint, with one log.
+  - With `sweep` (on by default with `rawWrites: "adopt"`), the watcher adopts each document without `_seq` after each gap, in pages of the index of `_id`. `sweepAtOpen: true` (off by default) also sweeps when the stream opens: the first watcher on a legacy collection adopts every legacy document, and the next delta of every subscriber is the whole collection. A document that the sweep cannot place goes to the log once, and the next sweeps leave it.
+  - `ready` rejects when the first open fails with an error that does not go away (a single server, a refused access, a wrong option). The watcher asks the server with `hello` first, so on a single server `ready` rejects at once. The watcher tries again after other errors.
+  - After an error of the stream, the watcher opens it again after a backoff (`backoff`, on the injected `clock`), from the resume token of the last event. When the server does not know the token any more, or the collection was dropped or renamed, it publishes a reset hint (no ids, `reset: true`) on the plain topic, on each partition of the counters, on each partition that it saw, and on each channel of the bus of the collection. The option `gapPartitions` adds more.
+  - `@procedures/client`: `CollectionChangeEvent` has the new optional field `versions`, `[{ id, version }]`: the version of each changed item, as `versionOf(id)` of the storage gives it. A publisher that knows the versions sets it. The new optional field `reset`: `true` tells the subscriber to read the collection in full, with `changes` and no `since`. PATCH: new optional fields of an output type, and before 1.0.0 a new feature raises the PATCH.
+  - `@procedures/store`: `EventsChangeFeed` reads `versions` and `reset` of an event, and `HybridStorage` does a full pull for a hint with `reset: true`. PATCH: a fix, so that a store does not miss the changes of a reset.
+  - `storageChangeOf(change, form?)` reads the change of an item from a change event, for an app that watches the collection itself.
+  - Fix: `ViewMaterializer` leaves `_seq` and `_tomb` of a source out of the content of a row, before the hash. A row is no write of a storage, so the watcher of the target does not read it as a raw write. The other fields of the source stay in the row.
+- cc80f74: PATCH: internal, no change to the public API. The procedure boilerplate of tracker item O65, after O58.
+  
+  - The `src/module.ts` of 37 packages gives its module and its register function with `definePackage()`. Each export keeps its name, its type and its behavior. The root and `./module` register nothing. `./register` registers in the global registry, as before.
+  - The output schemas of `ics.parse`, `ics.fetch`, `json.query`, `json.patch`, `json.merge` and `yaml.parse` have the type of the declared output (`z.ZodType<T>`). The validation and the JSON Schema stay the same.
+  - The `described()` helper of `@procedures/mongo` uses `typed<T>()` instead of the deprecated `zodAdapter()`. Its schemas validate and describe the input as before.
+- c9239a5: `MongoDocumentStorage` writes the floor document of its partition at its first write (roadmap O66). A gap of `watchStorageChanges()` then sends a reset hint to each partition with a write through a storage, also without a prune.
+  
+  PATCH: a fix, with no change to the API. Before 1.0.0, a fix raises the PATCH.
+  
+  - The floor document has the floor `0`: the change log reads a missing floor as `0` too, so no result of `getChangesSince` changes. The upsert writes only a new document, so the floor of a prune stays.
+  - The upsert runs once for each storage, before its first write and outside each transaction. A failure does not fail the write: the next write tries again.
+  - `gapPartitions` is necessary only for the partitions that only raw writes wrote.
+- Updated dependencies [9f0405d]
+- Updated dependencies [f5c2407]
+  - @procedures/client@0.35.1
+  - @procedures/collections@0.35.1
+
+## 0.35.0
+
+### Minor Changes
+
+- ab54137: `MongoDocumentStorage` can keep the stored form of an app (roadmap CLI-76), and `ViewMaterializer` leaves out the tombstones of its sources (roadmap CLI-79).
+  
+  MINOR: some default behaviors change. A source document with `_tomb: true` is no longer a row of a view. An item with a key with `$` or a dot below the top level is stored flat, not as JSON text. An item with a partition field of another value is refused. Before 1.0.0, a change of a default behavior raises the MINOR, and the fixed group shares one version.
+  
+  - The new option `form` takes a `StoredForm`: one object with `storedId(partition, id)`, `idOf(storedId, partition)`, `partitionFields(partition)` and `partitionOf(doc)`. `storage.form` gives it back. The stored `_id` can be a value that is not a text.
+  - `partitionFields` gives the fields that name a partition. Each document of the partition has them, and they are the filter of the reads, the change log and the aggregations. A field with the value `undefined` is absent: the filter is `{ $exists: false }`.
+  - `partitionOf(doc)` gives the partition of a stored document. A delete event has only `documentKey`, so with another `storedId`, the `_id` must tell the partition, or the collection needs `changeStreamPreAndPostImages`. The storage writes tombstones (replace events with the whole document), so the gap is only for a raw hard delete.
+  - `DEFAULT_STORED_FORM` writes the documents of 0.34.0, byte for byte. A test pins this. Its `partitionOf` reads `_part`, else the part of `_id` before U+001F.
+  - A source of the `ViewMaterializer` option `storageSources` is a function (the default form) or the new `ViewStorageSource` `{ form, viewPartitionOf }`. The materializer reads the partition of a change without a pre-image with `partitionOf`, from `fullDocument`, else `fullDocumentBeforeChange`, else `documentKey`.
+  - The storage keeps its fields (`_key`, `_seq`, `_ord`, `_tomb`, `_kid`, `_kpf`, `_json`) in each form. `DOCUMENT_STORAGE_FIELDS` names them, and `TOMBSTONE_FIELD` names the field of a tombstone.
+  - The new `decodeStoredDocument(doc, options)` reads a raw document as the storage reads it: `undefined` for a tombstone, `_json` parsed, the fields of the storage left out, and the id as `_id`.
+  - Keys with `$` or dots below the top level, and a top-level key with a dot, stay as they are in the document (MongoDB 5.0 or later). `_json` stays for an item that is not a plain object, a top-level key of the storage or with `$`, a key `__proto__`, and a class instance other than `Date`. An object with a text `$ref` and an `$id` is refused with a `TypeError`: MongoDB gives it back as a `DBRef`.
+  - An item can have a partition field with the value of its partition. The write leaves it out and notes it in `_kpf`, and the read gives it back. Another value is refused with a `TypeError`.
+  - A document without the fields of the storage is an item. Its id comes from `idOf`, else `_key`. Its item gets the id as `_id`. Its partition fields follow the rule of a written document without `_kpf`: a field with the value of the partition stays out, and a field with another value stays in. So the item has the same form before and after its first write. Its version is `0`, and it comes before the other items, then by `_id`. Its first write through the storage gives it a version and keeps its place. A document whose id neither tells is not an item: the reads and `size()` leave it out.
+  - The order of the entries is `_ord`, then `_id`, so the documents from outside the storage (all at the place 0) have a fixed order.
+  - `selectorPrefilter(selector, options)` takes the new optional `storedId` and `storageFields`. With `storedId`, a condition on `_id` tests the stored `_id`, so `query()` uses the index of `_id`. A path with a part that starts with `$` does not go in the filter.
+  - The docs tell three limits. Write through the storage: a raw write takes no number of the sequence, so a delta does not give it. A raw writer must not hard-delete an item while the storage writes it. On a single server, a precondition of a document without `_seq` does not see a writer outside the storage between the test and the write; a replica set is safe. With `db` and no `client`, a write uses the watermark, also on a replica set.
+  - `ViewMaterializer` adds `_tomb: { $ne: true }` to the filter of the sources, so a deleted item of a `MongoDocumentStorage` is not in a view.
+  
+  Migration:
+  
+  - MiniMongo, Graph and jarvis: an item with a dotted key or a `$` key below the top level needs MongoDB 5.0 or later. On a 4.x server, the write of such an item fails, because 0.34.0 stored it in `_json` and 0.35.0 stores it flat. An item that is already in `_json` reads as before.
+  - A view pipeline reads `"a.b"` as a nested path. A literal dotted key of an item is not reachable there without `$getField`.
+
+### Patch Changes
+
+- @procedures/client@0.35.0
+  - @procedures/collections@0.35.0
+
 ## 0.34.0
 
 ### Patch Changes

package/README.md

@@ -667,18 +667,74 @@ server.usePartitions("notes", { storageOf: (partition) => notes.forPartition(par
 | `queryPolicy` | The query policy of `query()` without a policy of the call |
 | `counters` | The collection of the sequence counters (default `__storage_counters`) |
 | `writeLease` | The time after which a pending number of a write that stopped no longer holds the readers (default `seconds(60)`). A plain number is a `TypeError`. |
-| `client` | The client of the database, for the transactions. The default is the client of the default connection when `db` is absent. |
+| `client` | The client of the database, for the transactions. The default is the client of the default connection when `db` is absent. With `db` and no `client`, a write is not a transaction, also on a replica set: it uses the watermark. |
 | `transactions` | `auto` (the default): a transaction for each write on a replica set or a sharded cluster. `never`: the watermark only. |
+| `versionOf` | The version of an item, for the expected versions of `set`, `delete` and `setBatch` (roadmap CLI-16) |
+| `form` | The stored form (roadmap CLI-76): a `StoredForm` with `storedId`, `idOf`, `partitionFields` and `partitionOf`. The default `DEFAULT_STORED_FORM` writes the documents of 0.34.0. `storage.form` gives it back. |
 
-- **The documents.** A plain object with safe keys is stored flat: its fields beside the fields of the storage (`_key`, `_part`, `_seq`, `_ord`, `_tomb`). Another item is stored as JSON text in `_json`. Both ways give the item back as it was. An item whose `_id` is its id keeps it.
+- **The documents.** A plain object is stored flat: its fields beside the fields of the storage (`_key`, `_part`, `_seq`, `_ord`, `_tomb`). Keys with `$` or dots below the top level, and a top-level key with a dot, stay as they are (MongoDB 5.0 or later). Another item is stored as JSON text in `_json`: null, a primitive, an array, an object with a top-level key of the storage or a top-level key with `$`, a key `__proto__`, or a class instance other than `Date`. Both ways give the item back as it was. An object with a text `$ref` and an `$id` is refused with a `TypeError`, because MongoDB gives it back as a `DBRef`. An item whose `_id` is its id keeps it.
+- **Write through the storage.** A raw write (for example `findOneAndReplace` or a cleanup script) takes no number of the sequence, so a delta of the change log does not give it. A raw replace also drops `_seq`. A raw writer must not hard-delete an item while the storage writes it: the write of a document without `_seq` is an upsert with `{ _seq: { $exists: false } }`, so it inserts the document again.
 - **The change log.** Each write gets the next number of the sequence of the collection, and a delete leaves a tombstone. `getChangesSince()` thus gives the exact changes and deletes. A version of another epoch, or a version before the oldest dropped tombstone, gives a full result.
 - **The versions.** `versionOf(id)` gives the number of the last write of an item.
 - **The queries.** `query()` sends the equality of a string, a number or a boolean, and `$in` of them, to MongoDB as a filter. `queryEntries()` of the core then gives the exact result, so a query gives the result of the reference.
 - **Several writers.** On a replica set, a write takes its numbers and writes in one transaction. On a single server, a watermark keeps the log exact. Each write keeps its numbers pending until it ends, and a version stops below the lowest pending number. A write that ends after its lease moves its documents to new numbers.
-- **The case that stays on a single server.** A process that stops between a late write and the move of its documents leaves that change below the watermark. A reader that passed those numbers does not get the change. On a replica set, this case does not occur.
+- **The cases that stay on a single server.** A process that stops between a late write and the move of its documents leaves that change below the watermark. A reader that passed those numbers does not get the change. And a precondition (`expectedVersion`) of a document without `_seq` has the guard `{ _seq: { $exists: false } }`: a writer outside the storage between the test and the write is not seen. On a replica set, the test and the write are one transaction, and these cases do not occur.
 - **The partitions.** An id cannot have the character U+001F, and each write filter has the partition. A write cannot change a document of another partition.
+- **The stored form** (roadmap CLI-76). An app that has its own form of documents keeps it, with no migration: the option `form`, one `StoredForm` object. `DEFAULT_STORED_FORM` writes the documents of 0.34.0, byte for byte. Give the same object to `ViewMaterializer` (`storageSources`).
+  - `storedId(partition, id)` gives the stored `_id`, for example `tenant:key`. It can be a value that is not a text. `idOf(storedId, partition)` gives the id back. A read uses `idOf` first, then `_key`.
+  - `partitionFields(partition)` gives the fields that name the partition, for example `{ tenant: "acme", scope: undefined }`. Each document of the partition has them, and they are the filter of the reads, the change log and the aggregations. A field with the value `undefined` is absent: the document does not have it, and the filter is `{ $exists: false }`.
+  - `partitionOf(doc)` gives the partition of a stored document. `ViewMaterializer` gives it the document of a change event: `fullDocument`, else `fullDocumentBeforeChange`, else `documentKey`. A delete event has only `documentKey`, so with another `storedId`, the `_id` must tell the partition, or the collection needs `changeStreamPreAndPostImages`. The storage writes a delete as a tombstone (a replace event with the whole document), so the gap is only for a raw hard delete. The default reads `_part`, else the part of `_id` before U+001F.
+  - The storage adds its fields in each form: `_key`, `_seq`, `_ord`, `_tomb`, and `_kid`, `_kpf` or `_json` when an item needs them. `DOCUMENT_STORAGE_FIELDS` names them, so a reader of the raw documents can leave them out. A delete leaves a tombstone (`TOMBSTONE_FIELD` is `true`): a reader of the raw documents must leave it out. `ViewMaterializer` does.
+  - `decodeStoredDocument(doc, { form })` reads a raw document as the storage reads it: `undefined` for a tombstone or a document without an id, `_json` parsed, the fields of the storage left out, and the id as `_id`. It gives `{ id, partition, item }`.
+  - `versionOfStored(doc, form)` gives the version of a raw document: the version that `storage.versionOf(id)` gives (roadmap CLI-77). It is `_seq` as text, `0` for a document without `_seq`, and `undefined` for a tombstone or a document without an id. The storage uses it too.
+  - An item can have a partition field with the value of its partition (for example `scope: "board:1"` in the partition `acme/board:1`). The write leaves it out and notes it in `_kpf`, and the read gives it back. The same field with another value is refused with a `TypeError`.
+  - A document without the fields of the storage (for example a document from before the app used the storage) is an item. Its id comes from `idOf`, else `_key`. Its item gets the id as `_id`. Its partition fields follow the rule of a written document without `_kpf`: a field with the value of the partition stays out, and a field with another value stays in. So the item has the same form before and after its first write. Its version is `0`, and it comes before the other items (then by `_id`). A full result of the change log has it. Its first write through the storage gives it a version, keeps its place, and is in the next delta.
+  - A document without `_key` whose id `idOf` does not tell (for example an old `ObjectId`) is not an item: the reads and `size()` leave it out.
+
+```ts
+const notes = new MongoDocumentStorage<Note>({
+  collection: "notes",
+  form: {
+    // The stored _id is `tenant:key`, and the field `tenant` names the partition
+    storedId: (partition, id) => `${partition ?? "main"}:${id}`,
+    idOf: (stored, partition) => {
+      const prefix = `${partition ?? "main"}:`;
+      return typeof stored === "string" && stored.startsWith(prefix) ? stored.slice(prefix.length) : undefined;
+    },
+    partitionFields: (partition) => ({ tenant: partition ?? "main" }),
+    // The field, else the part of the stored _id before ':' (a delete event has only the _id)
+    partitionOf: (doc) => (typeof doc.tenant === "string" ? doc.tenant : typeof doc._id === "string" ? doc._id.split(":")[0] : undefined),
+  },
+});
+```
 - **The tests.** The storage passes the `CollectionStorage` contract on MongoDB 7 in CI. Set `MONGO_TEST_URL` to start them locally.
 
+### Change hints from the change stream: `watchStorageChanges`
+
+The collection procedures publish an event for each of their writes. A write that does not go through them (a raw write, a script, a view, another process) has no event. `watchStorageChanges(storage, publish, options)` watches the change stream of the collection of a storage and publishes a hint of each change (roadmap CLI-77). Change streams need a replica set.
+
+```ts
+const notes = new MongoDocumentStorage<Note>({ collection: "notes" });
+const watch = watchStorageChanges(notes, bus); // the bus of the procedures, or (topic, event) => ...
+await watch.ready; // the stream is open
+// A raw write of a note in the partition t1 reaches the subscribers of collections.notes.t1
+await watch.stop();
+```
+
+- **The same topic and the same event.** The topic is `collectionChangeTopic(name, partition)`, as the procedures publish it. The partition comes from `storage.form.partitionOf()` of `fullDocumentBeforeChange`, else `fullDocument`, else `documentKey`: the first one whose partition lets `form.idOf()` read the stored `_id`. Thus a raw write that changes the partition field stays in the partition of its stored `_id`. The event is a `CollectionChangeEvent` without `since`: a hint, so the subscriber reads the changes with `changes`. It has the id in `changed` or `deleted`, and `versions: [{ id, version }]` when the watcher knows the new version. A tombstone and a delete are deletes. The options `collection` and `topic` give the name and the topic rule of the procedures when they are not the defaults.
+- **A raw write gets a sequence number** (`rawWrites: "adopt"`, the default). The watcher reads from the event alone whether a write is raw: a write of the storage sets `_seq`, a raw write does not (a raw replace that copies `_seq` is raw only when the image before the change shows the same `_seq`). The raw event publishes nothing. The watcher gives the document a new `_seq` through the write path of the storage (the numbers, the late check and the move), and gives back the storage fields and the partition fields that a raw replace took (`_key`, `_ord`, `_kpf`), from the image before the change, else from `form.idOf()` and `form.partitionOf()` of the stored `_id`. It never changes a field of the item. The write matches only the state that the event saw: when two watchers race, or a write of the storage comes first, one write wins and the others do nothing. The event of that write publishes the hint, with the new version, so `changes(since)` has the raw write. A raw write does not carry its old version.
+- **One queue of adoptions.** The adoptions of a watcher wait in one queue, and one adoption runs at a time. The raw writes that wait together (up to 100) take one block of numbers for each partition. An adoption that fails with an error that can go away tries again after the backoff, up to 5 times. When more than 10,000 raw writes wait, the next hints go out without a version, with one log. The watcher keeps the ids of the raw writes that did not fit or whose last try failed (up to 10,000), and adopts them again when the queue is empty, after the sweep, with the documents then. A failed last try, a second failure, or an id over that limit gives the partition a reset hint (`reset: true`): its subscribers read the collection in full.
+- **The sweep** (`sweep`, on by default with `rawWrites: "adopt"`). After each gap, the watcher sends each document without `_seq` to the same queue, 500 at a time. Thus a raw write during a gap gets its number too, but only on a document without `_seq`: a raw update of a document with `_seq` keeps that `_seq`, and only the reset hint of the gap tells it. `sweep: false` turns it off, except for the sweep after more ids than the watcher can keep. The scan walks the index of `_id` and reads each document of the collection: a partial index cannot express `$exists: false`. A document that the sweep cannot place (no id, or another partition) goes to the log once, and the next sweeps leave it.
+- **The sweep at the open** (`sweepAtOpen: true`, off by default) also sweeps when the stream opens, for the raw writes before the watcher started. The first watcher on a legacy collection adopts every legacy document, and the next delta of every subscriber is the whole collection.
+- **The adoption is exact only with the images before the change** (`changeStreamPreAndPostImages: { enabled: true }`). An insert and a replace show their document, but an update shows only its fields. Without the image, the `_seq` that a raw update kept comes from the lookup of the document. When a write of the storage comes after the raw update, the adoption can give that write one more version. A client that holds the version of that write then gets a conflict of its precondition, and reads the item again.
+- **The watcher needs write access, or set `rawWrites: "ignore"`.** With `"ignore"`, the watcher writes nothing and publishes each raw write as a hint without a version. A read-only user gets the same hint: the watcher logs the first refusal once.
+- **Raw deletes need the images before the change.** A raw delete of a document becomes a tombstone only when the collection keeps the image before each change (`changeStreamPreAndPostImages: { enabled: true }`): the tombstone takes its id and its partition from the image, and its insert fails when the `_id` exists again. Without the image, the watcher publishes a hint without a version and writes nothing. The delete of a tombstone (a prune of the storage) publishes nothing.
+- **One event for each write.** With `ownWrites: "skip"` (the default), the watcher skips the events of the writes of this process through the storage: the procedures of this process publish them already, with `since`. Each write of the storage notes its sequence numbers while a watcher of the collection is open, and the event of each number is skipped once. A raw write, or a write of another process, is published. Use `ownWrites: "publish"` when the writes of this process do not go through the procedures, or when the procedures publish no events (`changeEvents: false`).
+- **No lost change.** After an error of the stream, the watcher opens the stream again after a backoff (option `backoff`, on the injected `clock`), from the resume token of the last event. When the server does not know the token any more, or the collection was dropped or renamed, the watcher publishes a reset hint (no ids, `reset: true`) on the plain topic, on each partition of the counters, on each partition that it saw, and on each channel of the bus of the collection. The option `gapPartitions: () => partitions` adds more. The first write of each storage writes the floor document of its partition in the counters, with the floor `0` (no prune). So the partitions of the counters include each partition with a write through a storage, in any process. Give `gapPartitions` only for the partitions that only raw writes wrote. The subscribers read the collection in full.
+- **`ready`** resolves when the stream is open. It rejects when the first open fails with an error that does not go away: a single server, a refused access, or a wrong option. Change streams need a replica set or a sharded cluster: the watcher asks the server with `hello` before it opens the stream, and on a single server `ready` rejects with code 40573. The watcher logs it once and stops. It tries again after other errors.
+- **The storage of a partition** (`forPartition()`) publishes and adopts only the documents of its partition. Its stream reads the whole collection. The documents of the counters are in another collection, and they never make a hint.
+- **Your own stream.** `storageChangeOf(change, form)` reads the change of an item from a change event, for an app that watches the collection itself: `{ id, partition, deleted, raw, version, seq }`, or `undefined`.
+
 ### The guard of a pipeline: `scopePipeline`
 
 `scopePipeline(pipeline, scope)` gives a pipeline that reads only the documents of a filter, on each collection that it reads. `MongoDocumentStorage.aggregate()` uses it with the filter of its partition. An app can use it for its own aggregations over shared collections:
@@ -729,10 +785,10 @@ const totals = await db.collection("orderTotals").find(viewFilter(view)).toArray
 - A pass in groups holds a lease on its view (`passLease`, default 60 s, on the time of the server), and renews it after each group. Another process leaves the view and the rows of the lease alone. The change of generation needs the lease and the generation of the start of the pass. A pass that loses either one gives way, and the view runs again later.
 - A view in a cycle of views stays idle. Its status is `cycle` until the cycle ends.
 - The status of each view (`ok`, `errored` with `lastError`, `cycle`) goes into the field `_materializerStatus` of its document.
-- Each pipeline goes through `scopePipeline` by default. It reads only the documents of the partition, in its sources, and only the current generation of the target of another view. It cannot have `$merge` or `$out`. The option `scope` gives another guard, with the same filter.
+- Each pipeline goes through `scopePipeline` by default. It reads only the documents of the partition, in its sources, and only the current generation of the target of another view. The tombstones of a `MongoDocumentStorage` (`_tomb` is `true`) stay out (roadmap CLI-79). It cannot have `$merge` or `$out`. The option `scope` gives another guard, with the same filter.
 - A stream that ends opens again after `rewatch`, and a pass catches up. The times of the options are durations (`ms(100)`, `seconds(60)`) of `@procedures/client`.
 - A delete has no document in its change event. It reaches only the views of its partition when its pre-image tells the partition (a collection with `changeStreamPreAndPostImages`). Without a pre-image, a delete, a replace and an update of the partition field reach the views of each partition.
-- In a source of `storageSources`, the id of a `MongoDocumentStorage` document tells the storage partition of such a change. The function of that source gives the partition of the views (`identityPartition` when the two are the same). Else the delete reaches the views of each partition.
+- In a source of `storageSources` (a collection of a `MongoDocumentStorage`), `partitionOf` of its stored form reads the storage partition of such a change from its document (for a delete, only `documentKey`). The function of that source gives the partition of the views (`identityPartition` when the two are the same). A source is that function (the default form), or `{ form: storage.form, viewPartitionOf }` for another form. Else the change reaches the views of each partition.
 - With pre-images, an update that moves a document to another partition also reaches the views of its old partition.
 - `sourceFields` gives the fields of each source, as paths with dots. An update starts a pass when it changes one of them (at the field, under it or above it), or the partition field. The positions in arrays do not count: `lines.0.price` is `lines.price`. The partition field is a top-level field.
 - The stop has the parts of a transport of a service: `stopIntake()`, `drain(deadline, signal)`, `inFlight()` and `close()`. `stop()` does the four. A host composes them with no adapter.