@procedures/store 0.8.0 → 0.9.0

package/CHANGELOG.md

@@ -1,5 +1,67 @@
 # @procedures/store
 
+## 0.9.0
+
+### Patch Changes
+
+- dca28c1: The packages accept `@mark1russell7/service` 0.5.1 and later 0.5 versions, and `@procedures/client` accepts `@mark1russell7/algebra` 0.7.1. Before, the range `^0.4.1` stopped below 0.5. An application on service 0.5 then got a second copy of service inside the client.
+  
+  - The client uses only the `/backoff`, `/heartbeat` and `/testing` entries of service. The code of `/backoff`, `/heartbeat` and the clock is the same in 0.4.1 and 0.5.1. `/testing` only gets new exports.
+  - The public API of the client does not change.
+- Updated dependencies [8bb469c]
+- Updated dependencies [0dd1c86]
+- Updated dependencies [dc03903]
+- Updated dependencies [dca28c1]
+  - @procedures/client@0.9.0
+  - @procedures/collections@0.9.0
+
+## 0.8.1
+
+### Patch Changes
+
+- 1aa9ef7: The collection storages get a JSON query language with a query policy, change batches and patches in one step, and the sync of `@procedures/store` runs on the service clock and backoff (server stack CLI-20, CLI-14, CLI-13, CLI-34).
+  
+  - The query language is in the core, in the new entry `@procedures/client/query`, which imports nothing else. It is the one copy (CLI-34). `@procedures/collections` now depends on `@procedures/client` for it, and exports the same names again.
+  - The selector is JSON: a subset of the query selectors of MongoDB. `queryEntries()` is the reference, and `selectorProblems()`, `queryOptionsProblems()`, `compileSelector()`, `matchesSelector()` and `compareStorageValues()` are exported.
+  - A query policy (`StorageQueryPolicy`) limits the operators, the length and the flags of a `$regex`, the clauses and the depth of a selector, the lists of `$in`, `$nin` and `$all`, and the limit. The default policy does not permit `$regex`. A pattern that repeats a repetition, for example `(a+)+`, is always an error.
+  - `CollectionStorage` of `@procedures/collections` has the optional `query(selector, { sort, skip, limit }, policy)`, `observe(observer)` and `patch(id, fields)`. A caller tests for each one. A storage without them still follows the contract.
+  - `observe()` gives one batch of `set`, `delete` and `clear` changes for each write that changes the storage, before the write resolves. The write methods take `{ origin }`, and each change has it.
+  - `patch()` takes a JSON Merge Patch (RFC 7386), or the operations `{ $set, $unset }`, which can set `null` and nested field paths. A patch that mixes the two forms is an error. `applyPatch()`, `mergePatch()` and `patchProblems()` are exported.
+  - `InMemoryStorage` has the three methods, and the options `queryPolicy` and `onObserverError`. `StorageListeners` is the set of listeners that the storages share.
+  - The contract suite of `@procedures/collections/testing` tests the selectors against `queryEntries()`, with fixed and random selectors, the refusal of `$regex` by default, and, with `createWithQueryPolicy`, the opt-in of `$regex` and each limit of a policy. It also tests the two forms of a patch, the patches at the same time, and the batches with their origins.
+  - The core collection procedures have `collections.<name>.find` (it reads) and `collections.<name>.patch` (it reads and writes, idempotent). `find` refuses a selector or a limit that the policy does not permit with `VALIDATION_ERROR` and the reason, and a `find` without a limit gets at most `DEFAULT_FIND_LIMIT` (1000) items. It gives the selector and the policy to `query()` of the storage, or runs it on the entries. `patch` uses `patch()` of the storage, or a read and a write, one patch of an id at a time.
+  - `createCollectionProcedures(name, { queryPolicy })` and `ProcedureServer.registerCollectionProcedures(name, { queryPolicy })` take the policy of the host.
+  - The `CollectionStorage` of the core has the same optional methods and the write options. Its new types are exported from the root, `./browser` and `./procedures`.
+  - `ApiStorage` has `query()` (the procedure `find`, or the entries of an older server) and `patch()` (the procedure `patch`). Its option `queryPolicy` adds limits of the client.
+  - `ApiStorage` waits `backoffDelay` of `@mark1russell7/service/backoff` between its tries, with full jitter, on the option `clock`. The timeout uses the clock too. The new options are `retryFactor`, `maxRetryDelay`, `retryJitter`, `clock` and `random`. An error with `retryAfterMs` gets at least that wait.
+  - `HybridStorage` has `query()` and the option `queryPolicy`. Its timers run on the option `clock`, and the delay after a failure grows with the option `backoff`. `getStatus()` and `onStatus()` give the state of the sync and its failures. The storage writes nothing to the console.
+  - `SyncedProcedureRegistry` takes the option `clock` for its auto-sync. Its status has the failures of the auto-sync and of the write-through writes, and `onStatus()` sends them.
+  - `SqliteStorage` has `query()`, `patch()` and the option `queryPolicy`. A patch runs in one transaction.
+  - `@procedures/store` depends on `@mark1russell7/service`, and `@procedures/collections` depends on `@procedures/client`.
+- c559b00: `HybridStorage` captures the writes of the application to its local copy, keeps its sync state after a reload and takes a push feed of the remote changes. Each write of the core collection procedures publishes a change event (server stack CLI-21, CLI-15, decision D9).
+  
+  - `HybridStorage.observe(observer)` gives one batch of changes for each change of the storage. A write of the storage gives its batch with the origin of its options. A pull and a delta of the feed give their changes with the origin of the option `syncOrigin` (`"sync"` by default). In `remote-only` mode, the batches tell the writes of the storage to the remote.
+  - The option `origin` of `HybridStorage` is the tag of the storage (a random `hybrid-…` tag by default). Each write of the storage to the local copy and to the remote has it. The write methods of `HybridStorage` and `ApiStorage` take `{ origin }`.
+  - `capture: "observe"`: the storage observes the local copy and queues each change with another origin. Thus an application can write to the local copy directly, for example with the synchronous writes of a SignalDB collection. In `write-through` and `remote-only` mode, the storage sends a captured change at once.
+  - The option `syncState` takes a `SyncStateStore`: `MemorySyncStateStore`, or `IndexedDbSyncStateStore` of `@procedures/store/indexeddb`. The store keeps the cursor and the base versions. A new storage with the same store and a durable local copy asks only for the changes since the cursor, and it finds the conflicts of the base versions. A failure of the store goes to the status (`operation: "store"`, `SyncStateError`), and the storage goes on.
+  - The option `feed` takes a `RemoteChangeFeed`. A delta whose `since` is the cursor applies without a read of the remote. A gap, a lost event, a new subscription or another delta makes one pull. A delta that comes during a pull applies after the pull, so the pull does not erase it (the LiveLog rule of Graph).
+  - In `write-through` mode, a pull or a delta does not write over an own write that waits for the remote, a set or a delete. The storage keeps the remote value of the id, and after the write it reads the id again: then the rules of a pull decide. A read of a missing id does not bring back an item whose own delete waits.
+  - With `versionOf`, the storage keeps the versions of its own writes until their echoes come (option `echoWindow`, 30 seconds by default). The echo of an older own write does not write over a newer local write (the EchoLedger rule of Graph).
+  - `EventsChangeFeed` of `@procedures/store` is the feed of the change events. Its source is a `Client`, a `WebSocketTransport` or another transport. It treats the events as hints: a gap of the numbers, a lost event, a hint and each confirmation of the subscription call `onGap()`.
+  - The core collection procedures publish `CollectionChangeEvent` `{ since, version, changed, deleted, origin }` on `ctx.bus`, under the topic `collections.<name>` (`collectionChangeTopic()`), while a subscriber listens. The changes come from `getChangesSince()` of the storage, and the events of one storage make a chain of versions. The option `changeEvents` of `createCollectionProcedures()` turns them off, or gives the topic of each call: the partition of CLI-17 goes there.
+  - A value reaches a subscriber only through the checks of a read. By default, an event has the ids and the versions only, and `EventsChangeFeed` gives it to `onHint()`: `HybridStorage` reads the values with `changes`, and a hint with only deletes applies without a read. With `changeEvents: { values: true }`, an event also has `values`: each row `{ id, value }` passes the output filter of the host (`admitItem`) for each subscriber, and a refused id goes in `withheld`, so the storage pulls it.
+  - `EventPublisher` refuses a topic whose reads the capability of the call does not permit (`FORBIDDEN`, `effects-exceeded`). Its new option `reads` gives the reads of a topic. The default, `collectionTopicReads()`, gives `collection:<name>` for `collections.<name>` and `collections.<name>.<partition>`. `ctx.admit` gives a procedure the output filter of its host for the data that it sends later.
+  - The syncs of one `HybridStorage` run one at a time: the deltas and the hints of the feed, the pulls, the pushes, the reads after an own write and the mode switches. A remote delta and a full sync never run at the same time (Graph R18).
+  - The write inputs of the collection procedures take an optional `origin` (1 to 256 characters). The procedure gives it to the storage and to the event. A server before this change ignores the field.
+  - `Client.subscribe()` and `Client.events()` subscribe to the topics of the bus of the server through the middleware and the context of the client. A subscription has no deadline: the timeouts of the client do not apply to it, and only `metadata: { timeout }` in its options gives one.
+  - New exports: `collectionChangeTopic`, `CollectionChangeEvent` and `CollectionChangeEventsOptions` from `@procedures/client`, `./browser` and `./procedures`. `EventsChangeFeed`, `MemorySyncStateStore`, `SyncStateError` and the types `RemoteChangeFeed`, `RemoteChangeListener`, `RemoteChanges`, `EventSubscriber`, `ChangeEventSource`, `EventsChangeFeedOptions`, `SyncState`, `SyncStateChange` and `SyncStateStore` from `@procedures/store`. `IndexedDbSyncStateStore` and `MemorySyncStateStore` from `@procedures/store/indexeddb`.
+- Updated dependencies [2d3f235]
+- Updated dependencies [1aa9ef7]
+- Updated dependencies [2523b1b]
+- Updated dependencies [c559b00]
+  - @procedures/client@0.8.1
+  - @procedures/collections@0.8.1
+
 ## 0.8.0
 
 ### Patch Changes

package/README.md

@@ -91,9 +91,18 @@ The contract suite of `@procedures/collections/testing` tests `ApiStorage`, and
 - **The ids.** `getIds()` gives the ids in the order of the entries. `ApiStorage.getIds()` uses the procedure `collections.<name>.ids`, so the values stay on the server.
 - **After `close()`.** Each operation rejects with an error whose message says "closed". A second `close()` does nothing.
 
+- **The queries.** `query(selector, { sort, skip, limit })` gives the result of `queryEntries()` of `@procedures/collections`. `ApiStorage.query()` sends the JSON selector to the procedure `collections.<name>.find`, so the server applies it.
+- **The patches.** `ApiStorage.patch(id, fields)` sends a patch to the procedure `collections.<name>.patch`. The server changes the item in one step. A patch is a JSON Merge Patch, or the operations `{ $set, $unset }`.
+- **The query policy.** The server applies the policy of its host to each `find`. The default policy does not permit `$regex`, and a `find` gets at most 1000 items.
+
 Notes:
 
 - A custom `service` or a custom `get` operation gets the input `{ id }`, as before. Its `null` reads back as `undefined`.
+- A server before the procedure `find` answers `NOT_FOUND`. Then `ApiStorage.query()` reads the entries and applies the query in the client, with the same result.
+- `ApiStorage.find(predicate)` reads each item, because a function cannot go over the network. Use `query()` with a selector to filter on the server.
+- `HybridStorage.query()` reads the local copy, or the remote in `remote-only` mode. Its option `queryPolicy` goes to the storage under it.
+- `ApiStorage.query()` examines the form of a selector before it sends it. Its option `queryPolicy` adds the limits of the client.
+- A host turns `$regex` on with `server.registerCollectionProcedures(name, { queryPolicy })` or `createCollectionProcedures(name, { queryPolicy })`.
 - A server before `found` ignores it and gives the item. Then a stored `null` reads back as `undefined`, as before.
 - A server before the procedure `ids` answers `NOT_FOUND`. Then `ApiStorage.getIds()` reads the ids from the entries, and it does not ask for `ids` again.
 - `HybridStorage.close()` waits for the operations, the switch and the syncs that run. Then it sends the queue, and it closes the remote and the local copy.
@@ -183,6 +192,85 @@ const users = new HybridStorage(new ApiStorage(client, { collection: "users" }),
 - Without versions, a pull reads each local item without a queued write. It compares the JSON of the local item and the remote item.
 - The option `pullInterval` starts a periodic pull in the modes `write-through` and `write-back`. `close()` stops it.
 
+### The clock, the backoff and the status
+
+The timers and the retries run on a clock: `Clock` of `@mark1russell7/service/clock`. A test gives `fakeClock` of `@mark1russell7/service/testing`, and the times are then exact.
+
+- The option `clock` of `HybridStorage` and of `ApiStorage` gives the clock. The default is `realClock` of `@procedures/client`.
+- The timers of `HybridStorage` do not keep a Node process alive. These timers are the push of `write-back`, the periodic pull, and the send of the queue after a failure.
+- After a failed run of a timer, the next delay grows with `backoffDelay` of `@mark1russell7/service/backoff`. The option `backoff` gives the `factor`, the `maxDelay` and the `jitter`.
+- A run that succeeds sets the delay back to its interval.
+- `ApiStorage` waits `backoffDelay` between its tries, with full jitter. The options are `retryDelay`, `retryFactor`, `maxRetryDelay` and `retryJitter`.
+- An error with `retryAfterMs` gets at least that wait.
+- The timeout of each call of `ApiStorage` uses the clock too.
+
+`HybridStorage.getStatus()` gives the state of the sync. `onStatus(listener)` sends the status at each change. The storages of this package do not write to the console.
+
+| Field | Value |
+|---|---|
+| `connected` | False after a remote call failed, until a remote call succeeds |
+| `pendingChanges` | The number of operations in the queue |
+| `syncing` | True while a push or a pull runs |
+| `lastSync` | The time of the last pull that succeeded |
+| `mode` | The mode of the storage |
+| `error` | The newest failure of a load, a push, a pull, the queue store or the sync state store, with its `operation` and its `error`. A later success of the same work clears it. |
+
+```ts
+const stop = users.onStatus((status) => {
+  if (status.error) showBanner(`The sync failed: ${String(status.error.error)}`);
+  else if (status.pendingChanges > 0) showBanner(`${status.pendingChanges} changes wait for the server`);
+});
+```
+
+`SyncedProcedureRegistry` takes the option `clock` for its auto-sync. Its `getStatus()` and `onStatus()` give the failures of the auto-sync and of the write-through writes.
+
+### The changes, the capture, the sync state and the push feed
+
+`HybridStorage` is a sync engine for each local copy: `InMemoryStorage`, `SqliteStorage`, IndexedDB or a SignalDB collection (server stack D9, CLI-21, CLI-15).
+
+- `observe(observer)` gives one batch of `set`, `delete` and `clear` changes for each change of the storage.
+- A write of the storage gives its batch with the `origin` of its options. A pull and a delta of the feed give their changes with the origin `syncOrigin`, `"sync"` by default.
+- The option `origin` is the tag of the storage. Each write of the storage to the local copy and to the remote has it.
+- With `capture: "observe"`, the storage observes the local copy. It puts each change with another origin in the queue. Thus the application can write to the local copy directly.
+- A captured change always goes in the queue, because the application made it already. In `write-through` mode, the storage sends it at once.
+- The option `syncState` keeps the cursor and the base versions. A new storage with the same store asks only for the changes since the cursor.
+- `MemorySyncStateStore` keeps the state in memory. `IndexedDbSyncStateStore` of `@procedures/store/indexeddb` keeps it in a browser.
+- The storage uses a loaded cursor only with the option `local` and a local copy that has items. Give the state store with a durable local copy.
+- The option `feed` takes a push feed of the remote changes, for example `EventsChangeFeed`. The storage subscribes in the modes `write-through` and `write-back`.
+- A delta whose `since` is the cursor of the storage applies without a read of the remote.
+- A gap, a lost event, a new subscription or another delta makes the storage pull the changes since its cursor.
+- A delta that comes during a pull applies after the pull. Thus a pull does not erase a change that came during it.
+- In `write-through` mode, a pull or a delta does not write over an own write that waits for the remote. After the write, the storage reads the id again, and the rules of a pull decide.
+- With `versionOf`, the storage keeps the versions of its own writes until their echoes come. The echo of an older own write does not write over a newer local write.
+- The option `echoWindow` gives the time that the storage waits for an echo. The default is 30 seconds.
+
+`EventsChangeFeed` subscribes to the change events of the core collection procedures. Each write of `collections.<name>.*` publishes `{ since, version, changed, deleted, origin }` on the topic `collections.<name>`.
+
+- An event has the ids and the versions, and no value. The storage reads the values with `changes`, so the checks of a read apply. A hint with only deletes applies without a read.
+- A subscriber must have a capability that reads `collection:<name>`, as for a read.
+- With `changeEvents: { values: true }` on the server, the events also have the values. Each row passes the output filter of the host for each subscriber, and a refused id goes in `withheld`.
+- An event with a value for each changed id applies without a read. An event with a withheld id makes the storage pull.
+- The syncs of one storage run one at a time: the deltas, the hints, the pulls, the pushes and the mode switches. Thus a remote delta and a full sync do not run at the same time.
+
+```ts
+import { ApiStorage, EventsChangeFeed, HybridStorage } from "@procedures/store";
+import { IndexedDbQueueStore, IndexedDbSyncStateStore } from "@procedures/store/indexeddb";
+
+const users = new HybridStorage(new ApiStorage(client, { collection: "users" }), {
+  local: signalDbStorage,
+  capture: "observe",
+  versionOf: (user) => user.rev,
+  queueStore: new IndexedDbQueueStore("users-queue"),
+  syncState: new IndexedDbSyncStateStore("users-sync"),
+  feed: new EventsChangeFeed(client, { collection: "users" }),
+});
+```
+
+- The source of an `EventsChangeFeed` is a `Client`, a `WebSocketTransport`, or another transport of `@procedures/client`. `Client.subscribe()` sends the subscription through the middleware of the client.
+- The delivery of the events is at most once. Thus the feed gives each loss to the storage, and the storage pulls.
+- The server serves the events with the option `events` of `ProcedureServer`, for example `{ topics: ["collections.*"] }`. Without a subscriber, the procedures publish nothing.
+- `ApiStorage` sends the `origin` of a write to the procedure. Thus the storage of the server and the event have it.
+
 ### The change of a storage in a server
 
 `ProcedureServer.useStorage(name, storage, { drain: true, close: true })` first starts `flush()` of the old storage. Then it uses the new storage and closes the old one. When the flush fails, the old storage stays.