@procedures/mongo 0.33.0 → 0.35.0

package/CHANGELOG.md

@@ -1,5 +1,64 @@
 # @procedures/mongo
 
+## 0.35.0
+
+### Minor Changes
+
+- ab54137: `MongoDocumentStorage` can keep the stored form of an app (roadmap CLI-76), and `ViewMaterializer` leaves out the tombstones of its sources (roadmap CLI-79).
+  
+  MINOR: some default behaviors change. A source document with `_tomb: true` is no longer a row of a view. An item with a key with `$` or a dot below the top level is stored flat, not as JSON text. An item with a partition field of another value is refused. Before 1.0.0, a change of a default behavior raises the MINOR, and the fixed group shares one version.
+  
+  - The new option `form` takes a `StoredForm`: one object with `storedId(partition, id)`, `idOf(storedId, partition)`, `partitionFields(partition)` and `partitionOf(doc)`. `storage.form` gives it back. The stored `_id` can be a value that is not a text.
+  - `partitionFields` gives the fields that name a partition. Each document of the partition has them, and they are the filter of the reads, the change log and the aggregations. A field with the value `undefined` is absent: the filter is `{ $exists: false }`.
+  - `partitionOf(doc)` gives the partition of a stored document. A delete event has only `documentKey`, so with another `storedId`, the `_id` must tell the partition, or the collection needs `changeStreamPreAndPostImages`. The storage writes tombstones (replace events with the whole document), so the gap is only for a raw hard delete.
+  - `DEFAULT_STORED_FORM` writes the documents of 0.34.0, byte for byte. A test pins this. Its `partitionOf` reads `_part`, else the part of `_id` before U+001F.
+  - A source of the `ViewMaterializer` option `storageSources` is a function (the default form) or the new `ViewStorageSource` `{ form, viewPartitionOf }`. The materializer reads the partition of a change without a pre-image with `partitionOf`, from `fullDocument`, else `fullDocumentBeforeChange`, else `documentKey`.
+  - The storage keeps its fields (`_key`, `_seq`, `_ord`, `_tomb`, `_kid`, `_kpf`, `_json`) in each form. `DOCUMENT_STORAGE_FIELDS` names them, and `TOMBSTONE_FIELD` names the field of a tombstone.
+  - The new `decodeStoredDocument(doc, options)` reads a raw document as the storage reads it: `undefined` for a tombstone, `_json` parsed, the fields of the storage left out, and the id as `_id`.
+  - Keys with `$` or dots below the top level, and a top-level key with a dot, stay as they are in the document (MongoDB 5.0 or later). `_json` stays for an item that is not a plain object, a top-level key of the storage or with `$`, a key `__proto__`, and a class instance other than `Date`. An object with a text `$ref` and an `$id` is refused with a `TypeError`: MongoDB gives it back as a `DBRef`.
+  - An item can have a partition field with the value of its partition. The write leaves it out and notes it in `_kpf`, and the read gives it back. Another value is refused with a `TypeError`.
+  - A document without the fields of the storage is an item. Its id comes from `idOf`, else `_key`. Its item gets the id as `_id`. Its partition fields follow the rule of a written document without `_kpf`: a field with the value of the partition stays out, and a field with another value stays in. So the item has the same form before and after its first write. Its version is `0`, and it comes before the other items, then by `_id`. Its first write through the storage gives it a version and keeps its place. A document whose id neither tells is not an item: the reads and `size()` leave it out.
+  - The order of the entries is `_ord`, then `_id`, so the documents from outside the storage (all at the place 0) have a fixed order.
+  - `selectorPrefilter(selector, options)` takes the new optional `storedId` and `storageFields`. With `storedId`, a condition on `_id` tests the stored `_id`, so `query()` uses the index of `_id`. A path with a part that starts with `$` does not go in the filter.
+  - The docs tell three limits. Write through the storage: a raw write takes no number of the sequence, so a delta does not give it. A raw writer must not hard-delete an item while the storage writes it. On a single server, a precondition of a document without `_seq` does not see a writer outside the storage between the test and the write; a replica set is safe. With `db` and no `client`, a write uses the watermark, also on a replica set.
+  - `ViewMaterializer` adds `_tomb: { $ne: true }` to the filter of the sources, so a deleted item of a `MongoDocumentStorage` is not in a view.
+  
+  Migration:
+  
+  - MiniMongo, Graph and jarvis: an item with a dotted key or a `$` key below the top level needs MongoDB 5.0 or later. On a 4.x server, the write of such an item fails, because 0.34.0 stored it in `_json` and 0.35.0 stores it flat. An item that is already in `_json` reads as before.
+  - A view pipeline reads `"a.b"` as a nested path. A literal dotted key of an item is not reachable there without `$getField`.
+
+### Patch Changes
+
+- @procedures/client@0.35.0
+  - @procedures/collections@0.35.0
+
+## 0.34.0
+
+### Patch Changes
+
+- fa1a61a: Less boilerplate in each procedure definition (dedupe review F1, tracker item O58).
+  
+  PATCH (pre-1.0): this change adds exports and deprecates names. It removes no export and changes no behavior.
+  
+  - `@procedures/client` has the new export `typed<T>()`. It is the one pass-through schema that gives only the type `T`.
+  - `@procedures/client` has the new export `definePackage(name, procedures, requires?)`. It gives `{ module, register }` and registers nothing at the call. The new type `PackageDefinition` is its result.
+  - `zodAdapter` is deprecated. `input()` and `output()` take each Standard Schema and infer its type, so a zod schema goes in directly.
+  - `typedAnySchema` and `outputSchema` are deprecated. Use `typed<T>()`. They give the same pass-through schema.
+  - The procedures of the packages give their zod schemas to `input()` and `output()` directly. The `.build() as AnyProcedure` casts are gone. The schemas, the types of the exports and the validation do not change.
+  - `mark procedure new` writes `.input(<Name>InputSchema)` and `.output(<Name>OutputSchema)`, without `zodAdapter`.
+  
+  Migration:
+  
+  - The site, MiniMongo, Graph and jarvis: no change at this time. The deprecated names work until the next MINOR.
+  - Before the next MINOR, replace `zodAdapter<T>(schema)` with `schema`. To keep the check of a schema against a type, write `const schema: z.ZodType<T> = ...`.
+  - Before the next MINOR, replace `typedAnySchema<T>()` and `outputSchema<T>()` with `typed<T>()`.
+  - A parse-only object (`{ parse }`) in `zodAdapter` becomes a Standard Schema with a `~standard` property.
+- Updated dependencies [d4376c2]
+- Updated dependencies [fa1a61a]
+  - @procedures/client@0.34.0
+  - @procedures/collections@0.34.0
+
 ## 0.33.0
 
 ### Patch Changes

package/README.md

@@ -667,16 +667,45 @@ server.usePartitions("notes", { storageOf: (partition) => notes.forPartition(par
 | `queryPolicy` | The query policy of `query()` without a policy of the call |
 | `counters` | The collection of the sequence counters (default `__storage_counters`) |
 | `writeLease` | The time after which a pending number of a write that stopped no longer holds the readers (default `seconds(60)`). A plain number is a `TypeError`. |
-| `client` | The client of the database, for the transactions. The default is the client of the default connection when `db` is absent. |
+| `client` | The client of the database, for the transactions. The default is the client of the default connection when `db` is absent. With `db` and no `client`, a write is not a transaction, also on a replica set: it uses the watermark. |
 | `transactions` | `auto` (the default): a transaction for each write on a replica set or a sharded cluster. `never`: the watermark only. |
+| `versionOf` | The version of an item, for the expected versions of `set`, `delete` and `setBatch` (roadmap CLI-16) |
+| `form` | The stored form (roadmap CLI-76): a `StoredForm` with `storedId`, `idOf`, `partitionFields` and `partitionOf`. The default `DEFAULT_STORED_FORM` writes the documents of 0.34.0. `storage.form` gives it back. |
 
-- **The documents.** A plain object with safe keys is stored flat: its fields beside the fields of the storage (`_key`, `_part`, `_seq`, `_ord`, `_tomb`). Another item is stored as JSON text in `_json`. Both ways give the item back as it was. An item whose `_id` is its id keeps it.
+- **The documents.** A plain object is stored flat: its fields beside the fields of the storage (`_key`, `_part`, `_seq`, `_ord`, `_tomb`). Keys with `$` or dots below the top level, and a top-level key with a dot, stay as they are (MongoDB 5.0 or later). Another item is stored as JSON text in `_json`: null, a primitive, an array, an object with a top-level key of the storage or a top-level key with `$`, a key `__proto__`, or a class instance other than `Date`. Both ways give the item back as it was. An object with a text `$ref` and an `$id` is refused with a `TypeError`, because MongoDB gives it back as a `DBRef`. An item whose `_id` is its id keeps it.
+- **Write through the storage.** A raw write (for example `findOneAndReplace` or a cleanup script) takes no number of the sequence, so a delta of the change log does not give it. A raw replace also drops `_seq`. A raw writer must not hard-delete an item while the storage writes it: the write of a document without `_seq` is an upsert with `{ _seq: { $exists: false } }`, so it inserts the document again.
 - **The change log.** Each write gets the next number of the sequence of the collection, and a delete leaves a tombstone. `getChangesSince()` thus gives the exact changes and deletes. A version of another epoch, or a version before the oldest dropped tombstone, gives a full result.
 - **The versions.** `versionOf(id)` gives the number of the last write of an item.
 - **The queries.** `query()` sends the equality of a string, a number or a boolean, and `$in` of them, to MongoDB as a filter. `queryEntries()` of the core then gives the exact result, so a query gives the result of the reference.
 - **Several writers.** On a replica set, a write takes its numbers and writes in one transaction. On a single server, a watermark keeps the log exact. Each write keeps its numbers pending until it ends, and a version stops below the lowest pending number. A write that ends after its lease moves its documents to new numbers.
-- **The case that stays on a single server.** A process that stops between a late write and the move of its documents leaves that change below the watermark. A reader that passed those numbers does not get the change. On a replica set, this case does not occur.
+- **The cases that stay on a single server.** A process that stops between a late write and the move of its documents leaves that change below the watermark. A reader that passed those numbers does not get the change. And a precondition (`expectedVersion`) of a document without `_seq` has the guard `{ _seq: { $exists: false } }`: a writer outside the storage between the test and the write is not seen. On a replica set, the test and the write are one transaction, and these cases do not occur.
 - **The partitions.** An id cannot have the character U+001F, and each write filter has the partition. A write cannot change a document of another partition.
+- **The stored form** (roadmap CLI-76). An app that has its own form of documents keeps it, with no migration: the option `form`, one `StoredForm` object. `DEFAULT_STORED_FORM` writes the documents of 0.34.0, byte for byte. Give the same object to `ViewMaterializer` (`storageSources`).
+  - `storedId(partition, id)` gives the stored `_id`, for example `tenant:key`. It can be a value that is not a text. `idOf(storedId, partition)` gives the id back. A read uses `idOf` first, then `_key`.
+  - `partitionFields(partition)` gives the fields that name the partition, for example `{ tenant: "acme", scope: undefined }`. Each document of the partition has them, and they are the filter of the reads, the change log and the aggregations. A field with the value `undefined` is absent: the document does not have it, and the filter is `{ $exists: false }`.
+  - `partitionOf(doc)` gives the partition of a stored document. `ViewMaterializer` gives it the document of a change event: `fullDocument`, else `fullDocumentBeforeChange`, else `documentKey`. A delete event has only `documentKey`, so with another `storedId`, the `_id` must tell the partition, or the collection needs `changeStreamPreAndPostImages`. The storage writes a delete as a tombstone (a replace event with the whole document), so the gap is only for a raw hard delete. The default reads `_part`, else the part of `_id` before U+001F.
+  - The storage adds its fields in each form: `_key`, `_seq`, `_ord`, `_tomb`, and `_kid`, `_kpf` or `_json` when an item needs them. `DOCUMENT_STORAGE_FIELDS` names them, so a reader of the raw documents can leave them out. A delete leaves a tombstone (`TOMBSTONE_FIELD` is `true`): a reader of the raw documents must leave it out. `ViewMaterializer` does.
+  - `decodeStoredDocument(doc, { form })` reads a raw document as the storage reads it: `undefined` for a tombstone or a document without an id, `_json` parsed, the fields of the storage left out, and the id as `_id`. It gives `{ id, partition, item }`.
+  - An item can have a partition field with the value of its partition (for example `scope: "board:1"` in the partition `acme/board:1`). The write leaves it out and notes it in `_kpf`, and the read gives it back. The same field with another value is refused with a `TypeError`.
+  - A document without the fields of the storage (for example a document from before the app used the storage) is an item. Its id comes from `idOf`, else `_key`. Its item gets the id as `_id`. Its partition fields follow the rule of a written document without `_kpf`: a field with the value of the partition stays out, and a field with another value stays in. So the item has the same form before and after its first write. Its version is `0`, and it comes before the other items (then by `_id`). A full result of the change log has it. Its first write through the storage gives it a version, keeps its place, and is in the next delta.
+  - A document without `_key` whose id `idOf` does not tell (for example an old `ObjectId`) is not an item: the reads and `size()` leave it out.
+
+```ts
+const notes = new MongoDocumentStorage<Note>({
+  collection: "notes",
+  form: {
+    // The stored _id is `tenant:key`, and the field `tenant` names the partition
+    storedId: (partition, id) => `${partition ?? "main"}:${id}`,
+    idOf: (stored, partition) => {
+      const prefix = `${partition ?? "main"}:`;
+      return typeof stored === "string" && stored.startsWith(prefix) ? stored.slice(prefix.length) : undefined;
+    },
+    partitionFields: (partition) => ({ tenant: partition ?? "main" }),
+    // The field, else the part of the stored _id before ':' (a delete event has only the _id)
+    partitionOf: (doc) => (typeof doc.tenant === "string" ? doc.tenant : typeof doc._id === "string" ? doc._id.split(":")[0] : undefined),
+  },
+});
+```
 - **The tests.** The storage passes the `CollectionStorage` contract on MongoDB 7 in CI. Set `MONGO_TEST_URL` to start them locally.
 
 ### The guard of a pipeline: `scopePipeline`
@@ -729,10 +758,10 @@ const totals = await db.collection("orderTotals").find(viewFilter(view)).toArray
 - A pass in groups holds a lease on its view (`passLease`, default 60 s, on the time of the server), and renews it after each group. Another process leaves the view and the rows of the lease alone. The change of generation needs the lease and the generation of the start of the pass. A pass that loses either one gives way, and the view runs again later.
 - A view in a cycle of views stays idle. Its status is `cycle` until the cycle ends.
 - The status of each view (`ok`, `errored` with `lastError`, `cycle`) goes into the field `_materializerStatus` of its document.
-- Each pipeline goes through `scopePipeline` by default. It reads only the documents of the partition, in its sources, and only the current generation of the target of another view. It cannot have `$merge` or `$out`. The option `scope` gives another guard, with the same filter.
+- Each pipeline goes through `scopePipeline` by default. It reads only the documents of the partition, in its sources, and only the current generation of the target of another view. The tombstones of a `MongoDocumentStorage` (`_tomb` is `true`) stay out (roadmap CLI-79). It cannot have `$merge` or `$out`. The option `scope` gives another guard, with the same filter.
 - A stream that ends opens again after `rewatch`, and a pass catches up. The times of the options are durations (`ms(100)`, `seconds(60)`) of `@procedures/client`.
 - A delete has no document in its change event. It reaches only the views of its partition when its pre-image tells the partition (a collection with `changeStreamPreAndPostImages`). Without a pre-image, a delete, a replace and an update of the partition field reach the views of each partition.
-- In a source of `storageSources`, the id of a `MongoDocumentStorage` document tells the storage partition of such a change. The function of that source gives the partition of the views (`identityPartition` when the two are the same). Else the delete reaches the views of each partition.
+- In a source of `storageSources` (a collection of a `MongoDocumentStorage`), `partitionOf` of its stored form reads the storage partition of such a change from its document (for a delete, only `documentKey`). The function of that source gives the partition of the views (`identityPartition` when the two are the same). A source is that function (the default form), or `{ form: storage.form, viewPartitionOf }` for another form. Else the change reaches the views of each partition.
 - With pre-images, an update that moves a document to another partition also reaches the views of its old partition.
 - `sourceFields` gives the fields of each source, as paths with dots. An update starts a pass when it changes one of them (at the field, under it or above it), or the partition field. The positions in arrays do not count: `lines.0.price` is `lines.price`. The partition field is a top-level field.
 - The stop has the parts of a transport of a service: `stopIntake()`, `drain(deadline, signal)`, `inFlight()` and `close()`. `stop()` does the four. A host composes them with no adapter.