@procedures/collections 0.8.0 → 0.8.1

package/CHANGELOG.md

@@ -1,5 +1,34 @@
 # @procedures/collections
 
+## 0.8.1
+
+### Patch Changes
+
+- 1aa9ef7: The collection storages get a JSON query language with a query policy, change batches and patches in one step, and the sync of `@procedures/store` runs on the service clock and backoff (server stack CLI-20, CLI-14, CLI-13, CLI-34).
+  
+  - The query language is in the core, in the new entry `@procedures/client/query`, which imports nothing else. It is the one copy (CLI-34). `@procedures/collections` now depends on `@procedures/client` for it, and exports the same names again.
+  - The selector is JSON: a subset of the query selectors of MongoDB. `queryEntries()` is the reference, and `selectorProblems()`, `queryOptionsProblems()`, `compileSelector()`, `matchesSelector()` and `compareStorageValues()` are exported.
+  - A query policy (`StorageQueryPolicy`) limits the operators, the length and the flags of a `$regex`, the clauses and the depth of a selector, the lists of `$in`, `$nin` and `$all`, and the limit. The default policy does not permit `$regex`. A pattern that repeats a repetition, for example `(a+)+`, is always an error.
+  - `CollectionStorage` of `@procedures/collections` has the optional `query(selector, { sort, skip, limit }, policy)`, `observe(observer)` and `patch(id, fields)`. A caller tests for each one. A storage without them still follows the contract.
+  - `observe()` gives one batch of `set`, `delete` and `clear` changes for each write that changes the storage, before the write resolves. The write methods take `{ origin }`, and each change has it.
+  - `patch()` takes a JSON Merge Patch (RFC 7386), or the operations `{ $set, $unset }`, which can set `null` and nested field paths. A patch that mixes the two forms is an error. `applyPatch()`, `mergePatch()` and `patchProblems()` are exported.
+  - `InMemoryStorage` has the three methods, and the options `queryPolicy` and `onObserverError`. `StorageListeners` is the set of listeners that the storages share.
+  - The contract suite of `@procedures/collections/testing` tests the selectors against `queryEntries()`, with fixed and random selectors, the refusal of `$regex` by default, and, with `createWithQueryPolicy`, the opt-in of `$regex` and each limit of a policy. It also tests the two forms of a patch, the patches at the same time, and the batches with their origins.
+  - The core collection procedures have `collections.<name>.find` (it reads) and `collections.<name>.patch` (it reads and writes, idempotent). `find` refuses a selector or a limit that the policy does not permit with `VALIDATION_ERROR` and the reason, and a `find` without a limit gets at most `DEFAULT_FIND_LIMIT` (1000) items. It gives the selector and the policy to `query()` of the storage, or runs it on the entries. `patch` uses `patch()` of the storage, or a read and a write, one patch of an id at a time.
+  - `createCollectionProcedures(name, { queryPolicy })` and `ProcedureServer.registerCollectionProcedures(name, { queryPolicy })` take the policy of the host.
+  - The `CollectionStorage` of the core has the same optional methods and the write options. Its new types are exported from the root, `./browser` and `./procedures`.
+  - `ApiStorage` has `query()` (the procedure `find`, or the entries of an older server) and `patch()` (the procedure `patch`). Its option `queryPolicy` adds limits of the client.
+  - `ApiStorage` waits `backoffDelay` of `@mark1russell7/service/backoff` between its tries, with full jitter, on the option `clock`. The timeout uses the clock too. The new options are `retryFactor`, `maxRetryDelay`, `retryJitter`, `clock` and `random`. An error with `retryAfterMs` gets at least that wait.
+  - `HybridStorage` has `query()` and the option `queryPolicy`. Its timers run on the option `clock`, and the delay after a failure grows with the option `backoff`. `getStatus()` and `onStatus()` give the state of the sync and its failures. The storage writes nothing to the console.
+  - `SyncedProcedureRegistry` takes the option `clock` for its auto-sync. Its status has the failures of the auto-sync and of the write-through writes, and `onStatus()` sends them.
+  - `SqliteStorage` has `query()`, `patch()` and the option `queryPolicy`. A patch runs in one transaction.
+  - `@procedures/store` depends on `@mark1russell7/service`, and `@procedures/collections` depends on `@procedures/client`.
+- Updated dependencies [2d3f235]
+- Updated dependencies [1aa9ef7]
+- Updated dependencies [2523b1b]
+- Updated dependencies [c559b00]
+  - @procedures/client@0.8.1
+
 ## 0.8.0
 
 ### Patch Changes

package/README.md

@@ -89,17 +89,20 @@ interface CollectionStorage<T> {
   getIds?(): Promise<string[]>;
   getChangesSince?(since?: string): Promise<StorageChanges<T>>;
   find(predicate: (item: T) => boolean): Promise<T[]>;
+  query?(selector: StorageSelector, options?: StorageQueryOptions): Promise<Array<[string, T]>>;
+  observe?(observer: StorageObserver<T>): () => void;
   has(id: string): Promise<boolean>;
   size(): Promise<number>;
 
-  // Write Operations
-  set(id: string, value: T): Promise<void>;
-  delete(id: string): Promise<boolean>;
-  clear(): Promise<void>;
+  // Write Operations (the options give the origin of the write to the observers)
+  set(id: string, value: T, options?: StorageWriteOptions): Promise<void>;
+  delete(id: string, options?: StorageWriteOptions): Promise<boolean>;
+  clear(options?: StorageWriteOptions): Promise<void>;
+  patch?(id: string, fields: StoragePatch, options?: StorageWriteOptions): Promise<T | undefined>;
 
   // Bulk Operations
-  setBatch(items: Array<[string, T]>): Promise<void>;
-  deleteBatch(ids: string[]): Promise<number>;
+  setBatch(items: Array<[string, T]>, options?: StorageWriteOptions): Promise<void>;
+  deleteBatch(ids: string[], options?: StorageWriteOptions): Promise<number>;
   getBatch(ids: string[]): Promise<Map<string, T>>;
 
   // Lifecycle
@@ -116,6 +119,72 @@ Each storage agrees on these points:
 - A delta of `getChangesSince` follows the order of the last change of each id. A full result follows the order of the entries.
 - `getIds()` gives the ids in the order of the entries, and it does not read or copy the values. A storage that cannot give the ids cheaply does not have it. `HybridStorage` uses it for a full sync.
 - After `close()`, each operation rejects with an error whose message says "closed". A second `close()` does nothing.
+- `query()` gives the result of `queryEntries()` for the entries of the storage. Refer to "The query language".
+- `observe()` gives one batch of changes for each write that changes the storage, before the write resolves. Each change has the `origin` of its write.
+- `patch()` merges fields into one item in one step, as a JSON Merge Patch (RFC 7386). Two patches of one item at the same time keep each field.
+- The optional methods are features. A caller tests for each one before it uses it.
+
+### The query language
+
+A selector is JSON: a subset of the query selectors of MongoDB. The same selector goes to `query()` of a storage and to the procedure `collections.<name>.find` of `@procedures/client`.
+
+The language is a part of the wire contract of the core, so its one copy is in `@procedures/client/query`, an entry that imports nothing else. This package depends on `@procedures/client` for it, and it exports the same names again.
+
+```typescript
+const storage = new InMemoryStorage<Task>();
+const open = await storage.query(
+  { done: false, "owner.team": "core", tags: { $in: ["bug", "urgent"] }, $or: [{ due: { $lt: "2026-11-01" } }, { due: null }] },
+  { sort: { due: 1, title: 1 }, skip: 0, limit: 20 },
+);
+// open: [id, task] pairs, with copies of the tasks
+```
+
+- A key is a field path. A number in a path is an index of an array. A path through an array of objects looks into each object.
+- A literal is a test for equality. An array field also matches an equal element. `null` also matches a missing field.
+- The operators are `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$nin`, `$exists`, `$not`, `$regex` with `$options`, `$size`, `$all`, `$elemMatch`, `$and`, `$or` and `$nor`.
+- An operator that is not in the list is an error. `selectorProblems()` and `queryOptionsProblems()` give the problems, and `query()` rejects with a `TypeError`.
+- The comparisons compare only values of one type. The order of a sort is `null`, numbers, strings, objects, arrays, then booleans.
+- `queryEntries(entries, selector, options, policy)` is the reference. A storage that runs a selector itself, for example a database, gives the same result.
+- `compileSelector(selector, policy)` gives a test for many items.
+
+#### The query policy
+
+A query policy limits the queries of a caller. The default policy does not permit `$regex`: a pattern runs in the process, and some patterns take an exponential time.
+
+| Limit | Default | What a larger query gets |
+|---|---|---|
+| `operators` | `DEFAULT_QUERY_OPERATORS`: each operator except `$regex` | A `TypeError`, or `VALIDATION_ERROR` from `find` |
+| `maxRegexLength` | 128 characters | The same |
+| `regexFlags` | `"imsu"` | The same |
+| `maxClauses` | 64: each condition of a field, each operator, each `$and`, `$or` and `$nor` | The same |
+| `maxDepth` | 6: each `$and`, `$or`, `$nor`, `$not` and `$elemMatch` adds a level | The same |
+| `maxListLength` | 256 values of `$in`, `$nin` and `$all` | The same |
+| `maxLimit` | No limit for a storage. 1000 (`DEFAULT_FIND_LIMIT`) for the procedure `find`. | A larger `limit` is an error. A query without a limit gets `maxLimit`. |
+
+```typescript
+import { QUERY_OPERATORS } from "@procedures/collections";
+
+// A storage of trusted data turns $regex on
+const notes = new InMemoryStorage<Note>({ queryPolicy: { operators: QUERY_OPERATORS, maxRegexLength: 64 } });
+await notes.query({ title: { $regex: "^draft", $options: "i" } });
+```
+
+- The option `queryPolicy` of a storage applies to each call without a policy. The third argument of `query()` gives the policy of one call.
+- The procedure `find` gives the policy of its host to `query()` of the storage, so the host sets the policy in one place.
+- A pattern that repeats a group that repeats without a bound, for example `(a+)+`, is always an error.
+
+#### The two forms of a patch
+
+`patch(id, fields)` takes one of two forms. A patch whose keys each start with `$` is the operations form. A patch that mixes the two forms is an error.
+
+| Form | Example | What it does |
+|---|---|---|
+| JSON Merge Patch (RFC 7386) | `{ title: "New", meta: { score: 2 }, draft: null }` | Each field replaces the field of the item. An object merges into an object field. `null` removes the field. |
+| Operations | `{ $set: { "meta.score": 2, draft: null }, $unset: ["tags.0"] }` | `$set` sets field paths, also to `null`. `$unset` removes field paths. An element of an array becomes `null`. |
+
+- `applyPatch(value, fields)` applies either form, and `mergePatch(value, fields)` applies only a JSON Merge Patch. `patchProblems(fields)` gives the problems of a patch.
+- `$set` makes each missing or `null` part of a path an object. A path through another value, or past the end of an array, is a `TypeError`, and the item does not change.
+- `$set` comes before `$unset`. Two paths that overlap, for example `a` and `a.b`, are an error.
 
 ### The contract suite: `@procedures/collections/testing`
 
@@ -138,6 +207,7 @@ describeCollectionStorageContract(
 - The suite tests the reads, the writes and the batches, the order of the entries, `getIds`, and the results as new arrays.
 - It tests the copies, the JSON values with `null`, and the writes at the same time.
 - It tests `getChangesSince` with its versions and its order, `flush`, `getMetadata`, `close` and the errors.
+- It tests `query()` with fixed and random selectors against `queryEntries()`, `patch()` with patches at the same time, and the batches and origins of `observe()`.
 - A case for an optional method skips a storage without the method.
 - A random sequence of writes runs against a `Map`. Each saved version must give the right changes, in the order of the last change of each id.
 - The suite imports no test framework. The caller gives `describe` and `it`, and a failed check throws `ContractViolation`. The runtime entries do not import this entry.
@@ -145,10 +215,10 @@ describeCollectionStorageContract(
 
 | Storage | Package | Notes |
 |---|---|---|
-| `InMemoryStorage` | `@procedures/collections` | It copies with `structuredClone`. The run with `copies: false` skips only the case of the copies. |
-| `SqliteStorage` (each engine, in memory and in a file) | `@procedures/sqlite` | It keeps JSON text, so it gives copies. |
-| `ApiStorage` | `@procedures/store` | It reads `{ found, value }` from the procedure `get`, so a stored `null` stays `null`. Its `getIds` uses the procedure `ids`. |
-| `HybridStorage` (each mode, and a queue in IndexedDB) | `@procedures/store` | It has no `getChangesSince`. |
+| `InMemoryStorage` | `@procedures/collections` | It copies with `structuredClone`. The run with `copies: false` skips only the case of the copies. It has `query`, `observe` and `patch`. |
+| `SqliteStorage` (each engine, in memory and in a file) | `@procedures/sqlite` | It keeps JSON text, so it gives copies. Its `patch` runs in one transaction. |
+| `ApiStorage` | `@procedures/store` | It reads `{ found, value }` from the procedure `get`, so a stored `null` stays `null`. Its `getIds` uses the procedure `ids`, its `query` the procedure `find`, and its `patch` the procedure `patch`. |
+| `HybridStorage` (each mode, and a queue in IndexedDB) | `@procedures/store` | It has no `getChangesSince`. Its `query` reads the local copy, or the remote in `remote-only` mode. |
 
 ## Storage Backends
 
@@ -167,6 +237,7 @@ const user = await storage.get("user-1");
 - The storage keeps a copy of each written value, and each read gives a new copy (`structuredClone`). A `Date` stays a `Date`. A function in a value makes the write reject.
 - The option `copies: false` turns the copies off: the storage keeps the values themselves, and the reads give them. It is faster. Give it only when no caller changes a value after a write or a read.
 - `getIds()` gives the ids without a copy of the values.
+- `observe()` gives each batch before the write resolves. The error of an observer goes to the option `onObserverError`. The default throws it again in a microtask, as an `EventTarget` does.
 - `close()` drops the items. After it, each operation rejects, and the methods of the `Map` interface (`keys()`, `values()`, `entries()`, `forEach()`) throw.
 
 ### ApiStorage and HybridStorage