@procedures/mongo 0.35.0 → 0.35.1

package/CHANGELOG.md

@@ -1,5 +1,41 @@
 # @procedures/mongo
 
+## 0.35.1
+
+### Patch Changes
+
+- 9f0405d: `watchStorageChanges()` publishes the change stream of a `MongoDocumentStorage` collection as collection hints, and `versionOfStored()` gives the version of a raw stored document (roadmap CLI-77).
+  
+  PATCH: the change only adds exports and new optional options, and keeps each behavior of 0.35.0 (the watcher is not released yet). Before 1.0.0, a new feature raises the PATCH. The fixed group shares one version, so this rides the next release (0.35.0 when it is not out yet).
+  
+  - `versionOfStored(doc, form?, partition?)` gives the version that `storage.versionOf(id)` gives for a stored document: `_seq` as text, `0` for a document without `_seq`, and `undefined` for a tombstone or a document without an id. It gives the value that `versionOf()` reads.
+  - `watchStorageChanges(storage, publish, options?)` watches the change stream of the collection. It publishes each change on the topic of the collection procedures, `collectionChangeTopic(name, partition)`, with the partition from `storage.form.partitionOf()`. The event is a `CollectionChangeEvent` without `since` (a hint), with the id in `changed` or `deleted` and the version of the item in `versions`. A tombstone and a delete are deletes. It returns `{ ready, stop() }`.
+  - With `ownWrites: "skip"` (the default), the watcher skips the events of the writes of this process through the storage, because the collection procedures publish them already. Each write of the storage notes its sequence numbers while a watcher is open. `ownWrites: "publish"` publishes each change.
+  - With `rawWrites: "adopt"` (the default), the watcher gives a raw write a new `_seq` through the write path of the storage, so `changes(since)` has it. It reads from the event alone whether a write is raw. It gives back the storage fields and the partition fields that a raw replace took, and never changes a field of the item. The write matches only the state that the event saw, so one of two racing watchers wins and the other does nothing. The event of that write publishes the hint with the new version; a raw write does not carry its old version. The adoptions of a watcher wait in one bounded queue, one at a time, in batches that take one block of numbers. The watcher keeps the ids of the raw writes that do not fit in a full queue or whose last try failed (up to 10,000), and adopts them again when the queue is empty, with the documents then. A failure that can go away tries again after the backoff. A failed last try, a second failure, or an id over that limit gives the partition a reset hint. The adoption is exact only with `changeStreamPreAndPostImages`. With the images before the change (`changeStreamPreAndPostImages`), a raw delete becomes a tombstone; without them, it is a hint without a version. A prune of tombstones publishes nothing. `rawWrites: "ignore"` writes nothing and publishes a hint without a version. A read-only user gets the same hint, with one log.
+  - With `sweep` (on by default with `rawWrites: "adopt"`), the watcher adopts each document without `_seq` after each gap, in pages of the index of `_id`. `sweepAtOpen: true` (off by default) also sweeps when the stream opens: the first watcher on a legacy collection adopts every legacy document, and the next delta of every subscriber is the whole collection. A document that the sweep cannot place goes to the log once, and the next sweeps leave it.
+  - `ready` rejects when the first open fails with an error that does not go away (a single server, a refused access, a wrong option). The watcher asks the server with `hello` first, so on a single server `ready` rejects at once. The watcher tries again after other errors.
+  - After an error of the stream, the watcher opens it again after a backoff (`backoff`, on the injected `clock`), from the resume token of the last event. When the server does not know the token any more, or the collection was dropped or renamed, it publishes a reset hint (no ids, `reset: true`) on the plain topic, on each partition of the counters, on each partition that it saw, and on each channel of the bus of the collection. The option `gapPartitions` adds more.
+  - `@procedures/client`: `CollectionChangeEvent` has the new optional field `versions`, `[{ id, version }]`: the version of each changed item, as `versionOf(id)` of the storage gives it. A publisher that knows the versions sets it. The new optional field `reset`: `true` tells the subscriber to read the collection in full, with `changes` and no `since`. PATCH: new optional fields of an output type, and before 1.0.0 a new feature raises the PATCH.
+  - `@procedures/store`: `EventsChangeFeed` reads `versions` and `reset` of an event, and `HybridStorage` does a full pull for a hint with `reset: true`. PATCH: a fix, so that a store does not miss the changes of a reset.
+  - `storageChangeOf(change, form?)` reads the change of an item from a change event, for an app that watches the collection itself.
+  - Fix: `ViewMaterializer` leaves `_seq` and `_tomb` of a source out of the content of a row, before the hash. A row is no write of a storage, so the watcher of the target does not read it as a raw write. The other fields of the source stay in the row.
+- cc80f74: PATCH: internal, no change to the public API. The procedure boilerplate of tracker item O65, after O58.
+  
+  - The `src/module.ts` of 37 packages gives its module and its register function with `definePackage()`. Each export keeps its name, its type and its behavior. The root and `./module` register nothing. `./register` registers in the global registry, as before.
+  - The output schemas of `ics.parse`, `ics.fetch`, `json.query`, `json.patch`, `json.merge` and `yaml.parse` have the type of the declared output (`z.ZodType<T>`). The validation and the JSON Schema stay the same.
+  - The `described()` helper of `@procedures/mongo` uses `typed<T>()` instead of the deprecated `zodAdapter()`. Its schemas validate and describe the input as before.
+- c9239a5: `MongoDocumentStorage` writes the floor document of its partition at its first write (roadmap O66). A gap of `watchStorageChanges()` then sends a reset hint to each partition with a write through a storage, also without a prune.
+  
+  PATCH: a fix, with no change to the API. Before 1.0.0, a fix raises the PATCH.
+  
+  - The floor document has the floor `0`: the change log reads a missing floor as `0` too, so no result of `getChangesSince` changes. The upsert writes only a new document, so the floor of a prune stays.
+  - The upsert runs once for each storage, before its first write and outside each transaction. A failure does not fail the write: the next write tries again.
+  - `gapPartitions` is necessary only for the partitions that only raw writes wrote.
+- Updated dependencies [9f0405d]
+- Updated dependencies [f5c2407]
+  - @procedures/client@0.35.1
+  - @procedures/collections@0.35.1
+
 ## 0.35.0
 
 ### Minor Changes

package/README.md

@@ -686,6 +686,7 @@ server.usePartitions("notes", { storageOf: (partition) => notes.forPartition(par
   - `partitionOf(doc)` gives the partition of a stored document. `ViewMaterializer` gives it the document of a change event: `fullDocument`, else `fullDocumentBeforeChange`, else `documentKey`. A delete event has only `documentKey`, so with another `storedId`, the `_id` must tell the partition, or the collection needs `changeStreamPreAndPostImages`. The storage writes a delete as a tombstone (a replace event with the whole document), so the gap is only for a raw hard delete. The default reads `_part`, else the part of `_id` before U+001F.
   - The storage adds its fields in each form: `_key`, `_seq`, `_ord`, `_tomb`, and `_kid`, `_kpf` or `_json` when an item needs them. `DOCUMENT_STORAGE_FIELDS` names them, so a reader of the raw documents can leave them out. A delete leaves a tombstone (`TOMBSTONE_FIELD` is `true`): a reader of the raw documents must leave it out. `ViewMaterializer` does.
   - `decodeStoredDocument(doc, { form })` reads a raw document as the storage reads it: `undefined` for a tombstone or a document without an id, `_json` parsed, the fields of the storage left out, and the id as `_id`. It gives `{ id, partition, item }`.
+  - `versionOfStored(doc, form)` gives the version of a raw document: the version that `storage.versionOf(id)` gives (roadmap CLI-77). It is `_seq` as text, `0` for a document without `_seq`, and `undefined` for a tombstone or a document without an id. The storage uses it too.
   - An item can have a partition field with the value of its partition (for example `scope: "board:1"` in the partition `acme/board:1`). The write leaves it out and notes it in `_kpf`, and the read gives it back. The same field with another value is refused with a `TypeError`.
   - A document without the fields of the storage (for example a document from before the app used the storage) is an item. Its id comes from `idOf`, else `_key`. Its item gets the id as `_id`. Its partition fields follow the rule of a written document without `_kpf`: a field with the value of the partition stays out, and a field with another value stays in. So the item has the same form before and after its first write. Its version is `0`, and it comes before the other items (then by `_id`). A full result of the change log has it. Its first write through the storage gives it a version, keeps its place, and is in the next delta.
   - A document without `_key` whose id `idOf` does not tell (for example an old `ObjectId`) is not an item: the reads and `size()` leave it out.
@@ -708,6 +709,32 @@ const notes = new MongoDocumentStorage<Note>({
 ```
 - **The tests.** The storage passes the `CollectionStorage` contract on MongoDB 7 in CI. Set `MONGO_TEST_URL` to start them locally.
 
+### Change hints from the change stream: `watchStorageChanges`
+
+The collection procedures publish an event for each of their writes. A write that does not go through them (a raw write, a script, a view, another process) has no event. `watchStorageChanges(storage, publish, options)` watches the change stream of the collection of a storage and publishes a hint of each change (roadmap CLI-77). Change streams need a replica set.
+
+```ts
+const notes = new MongoDocumentStorage<Note>({ collection: "notes" });
+const watch = watchStorageChanges(notes, bus); // the bus of the procedures, or (topic, event) => ...
+await watch.ready; // the stream is open
+// A raw write of a note in the partition t1 reaches the subscribers of collections.notes.t1
+await watch.stop();
+```
+
+- **The same topic and the same event.** The topic is `collectionChangeTopic(name, partition)`, as the procedures publish it. The partition comes from `storage.form.partitionOf()` of `fullDocumentBeforeChange`, else `fullDocument`, else `documentKey`: the first one whose partition lets `form.idOf()` read the stored `_id`. Thus a raw write that changes the partition field stays in the partition of its stored `_id`. The event is a `CollectionChangeEvent` without `since`: a hint, so the subscriber reads the changes with `changes`. It has the id in `changed` or `deleted`, and `versions: [{ id, version }]` when the watcher knows the new version. A tombstone and a delete are deletes. The options `collection` and `topic` give the name and the topic rule of the procedures when they are not the defaults.
+- **A raw write gets a sequence number** (`rawWrites: "adopt"`, the default). The watcher reads from the event alone whether a write is raw: a write of the storage sets `_seq`, a raw write does not (a raw replace that copies `_seq` is raw only when the image before the change shows the same `_seq`). The raw event publishes nothing. The watcher gives the document a new `_seq` through the write path of the storage (the numbers, the late check and the move), and gives back the storage fields and the partition fields that a raw replace took (`_key`, `_ord`, `_kpf`), from the image before the change, else from `form.idOf()` and `form.partitionOf()` of the stored `_id`. It never changes a field of the item. The write matches only the state that the event saw: when two watchers race, or a write of the storage comes first, one write wins and the others do nothing. The event of that write publishes the hint, with the new version, so `changes(since)` has the raw write. A raw write does not carry its old version.
+- **One queue of adoptions.** The adoptions of a watcher wait in one queue, and one adoption runs at a time. The raw writes that wait together (up to 100) take one block of numbers for each partition. An adoption that fails with an error that can go away tries again after the backoff, up to 5 times. When more than 10,000 raw writes wait, the next hints go out without a version, with one log. The watcher keeps the ids of the raw writes that did not fit or whose last try failed (up to 10,000), and adopts them again when the queue is empty, after the sweep, with the documents then. A failed last try, a second failure, or an id over that limit gives the partition a reset hint (`reset: true`): its subscribers read the collection in full.
+- **The sweep** (`sweep`, on by default with `rawWrites: "adopt"`). After each gap, the watcher sends each document without `_seq` to the same queue, 500 at a time. Thus a raw write during a gap gets its number too, but only on a document without `_seq`: a raw update of a document with `_seq` keeps that `_seq`, and only the reset hint of the gap tells it. `sweep: false` turns it off, except for the sweep after more ids than the watcher can keep. The scan walks the index of `_id` and reads each document of the collection: a partial index cannot express `$exists: false`. A document that the sweep cannot place (no id, or another partition) goes to the log once, and the next sweeps leave it.
+- **The sweep at the open** (`sweepAtOpen: true`, off by default) also sweeps when the stream opens, for the raw writes before the watcher started. The first watcher on a legacy collection adopts every legacy document, and the next delta of every subscriber is the whole collection.
+- **The adoption is exact only with the images before the change** (`changeStreamPreAndPostImages: { enabled: true }`). An insert and a replace show their document, but an update shows only its fields. Without the image, the `_seq` that a raw update kept comes from the lookup of the document. When a write of the storage comes after the raw update, the adoption can give that write one more version. A client that holds the version of that write then gets a conflict of its precondition, and reads the item again.
+- **The watcher needs write access, or set `rawWrites: "ignore"`.** With `"ignore"`, the watcher writes nothing and publishes each raw write as a hint without a version. A read-only user gets the same hint: the watcher logs the first refusal once.
+- **Raw deletes need the images before the change.** A raw delete of a document becomes a tombstone only when the collection keeps the image before each change (`changeStreamPreAndPostImages: { enabled: true }`): the tombstone takes its id and its partition from the image, and its insert fails when the `_id` exists again. Without the image, the watcher publishes a hint without a version and writes nothing. The delete of a tombstone (a prune of the storage) publishes nothing.
+- **One event for each write.** With `ownWrites: "skip"` (the default), the watcher skips the events of the writes of this process through the storage: the procedures of this process publish them already, with `since`. Each write of the storage notes its sequence numbers while a watcher of the collection is open, and the event of each number is skipped once. A raw write, or a write of another process, is published. Use `ownWrites: "publish"` when the writes of this process do not go through the procedures, or when the procedures publish no events (`changeEvents: false`).
+- **No lost change.** After an error of the stream, the watcher opens the stream again after a backoff (option `backoff`, on the injected `clock`), from the resume token of the last event. When the server does not know the token any more, or the collection was dropped or renamed, the watcher publishes a reset hint (no ids, `reset: true`) on the plain topic, on each partition of the counters, on each partition that it saw, and on each channel of the bus of the collection. The option `gapPartitions: () => partitions` adds more. The first write of each storage writes the floor document of its partition in the counters, with the floor `0` (no prune). So the partitions of the counters include each partition with a write through a storage, in any process. Give `gapPartitions` only for the partitions that only raw writes wrote. The subscribers read the collection in full.
+- **`ready`** resolves when the stream is open. It rejects when the first open fails with an error that does not go away: a single server, a refused access, or a wrong option. Change streams need a replica set or a sharded cluster: the watcher asks the server with `hello` before it opens the stream, and on a single server `ready` rejects with code 40573. The watcher logs it once and stops. It tries again after other errors.
+- **The storage of a partition** (`forPartition()`) publishes and adopts only the documents of its partition. Its stream reads the whole collection. The documents of the counters are in another collection, and they never make a hint.
+- **Your own stream.** `storageChangeOf(change, form)` reads the change of an item from a change event, for an app that watches the collection itself: `{ id, partition, deleted, raw, version, seq }`, or `undefined`.
+
 ### The guard of a pipeline: `scopePipeline`
 
 `scopePipeline(pipeline, scope)` gives a pipeline that reads only the documents of a filter, on each collection that it reads. `MongoDocumentStorage.aggregate()` uses it with the filter of its partition. An app can use it for its own aggregations over shared collections: