@smthrs/sync 0.0.0-stage → 1.0.0-rc.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +208 -0
- package/LICENSE +21 -0
- package/README.md +208 -2
- package/dist/cjs/BranchCommands.d.ts +228 -0
- package/dist/cjs/BranchCommands.d.ts.map +1 -0
- package/dist/cjs/BranchCommands.js +316 -0
- package/dist/cjs/BranchCommands.js.map +7 -0
- package/dist/cjs/BranchIds.d.ts +65 -0
- package/dist/cjs/BranchIds.d.ts.map +1 -0
- package/dist/cjs/BranchIds.js +53 -0
- package/dist/cjs/BranchIds.js.map +7 -0
- package/dist/cjs/BranchPresence.d.ts +195 -0
- package/dist/cjs/BranchPresence.d.ts.map +1 -0
- package/dist/cjs/BranchPresence.js +276 -0
- package/dist/cjs/BranchPresence.js.map +7 -0
- package/dist/cjs/BranchProjection.d.ts +179 -0
- package/dist/cjs/BranchProjection.d.ts.map +1 -0
- package/dist/cjs/BranchProjection.js +197 -0
- package/dist/cjs/BranchProjection.js.map +7 -0
- package/dist/cjs/BranchProtocol.d.ts +415 -0
- package/dist/cjs/BranchProtocol.d.ts.map +1 -0
- package/dist/cjs/BranchProtocol.js +172 -0
- package/dist/cjs/BranchProtocol.js.map +7 -0
- package/dist/cjs/BranchRpcs.d.ts +177 -0
- package/dist/cjs/BranchRpcs.d.ts.map +1 -0
- package/dist/cjs/BranchRpcs.js +85 -0
- package/dist/cjs/BranchRpcs.js.map +7 -0
- package/dist/cjs/BranchServer.d.ts +55 -0
- package/dist/cjs/BranchServer.d.ts.map +1 -0
- package/dist/cjs/BranchServer.js +147 -0
- package/dist/cjs/BranchServer.js.map +7 -0
- package/dist/cjs/BranchShare.d.ts +159 -0
- package/dist/cjs/BranchShare.d.ts.map +1 -0
- package/dist/cjs/BranchShare.js +147 -0
- package/dist/cjs/BranchShare.js.map +7 -0
- package/dist/cjs/RunCatalog.d.ts +207 -0
- package/dist/cjs/RunCatalog.d.ts.map +1 -0
- package/dist/cjs/RunCatalog.js +122 -0
- package/dist/cjs/RunCatalog.js.map +7 -0
- package/dist/cjs/SyncAuth.d.ts +70 -0
- package/dist/cjs/SyncAuth.d.ts.map +1 -0
- package/dist/cjs/SyncAuth.js +109 -0
- package/dist/cjs/SyncAuth.js.map +7 -0
- package/dist/cjs/SyncClient.d.ts +274 -0
- package/dist/cjs/SyncClient.d.ts.map +1 -0
- package/dist/cjs/SyncClient.js +474 -0
- package/dist/cjs/SyncClient.js.map +7 -0
- package/dist/cjs/SyncError.d.ts +107 -0
- package/dist/cjs/SyncError.d.ts.map +1 -0
- package/dist/cjs/SyncError.js +110 -0
- package/dist/cjs/SyncError.js.map +7 -0
- package/dist/cjs/SyncPrincipal.d.ts +100 -0
- package/dist/cjs/SyncPrincipal.d.ts.map +1 -0
- package/dist/cjs/SyncPrincipal.js +48 -0
- package/dist/cjs/SyncPrincipal.js.map +7 -0
- package/dist/cjs/SyncProtocol.d.ts +592 -0
- package/dist/cjs/SyncProtocol.d.ts.map +1 -0
- package/dist/cjs/SyncProtocol.js +165 -0
- package/dist/cjs/SyncProtocol.js.map +7 -0
- package/dist/cjs/SyncRpcs.d.ts +92 -0
- package/dist/cjs/SyncRpcs.d.ts.map +1 -0
- package/dist/cjs/SyncRpcs.js +35 -0
- package/dist/cjs/SyncRpcs.js.map +7 -0
- package/dist/cjs/SyncServer.d.ts +184 -0
- package/dist/cjs/SyncServer.d.ts.map +1 -0
- package/dist/cjs/SyncServer.js +513 -0
- package/dist/cjs/SyncServer.js.map +7 -0
- package/dist/cjs/WorkspaceShare.d.ts +174 -0
- package/dist/cjs/WorkspaceShare.d.ts.map +1 -0
- package/dist/cjs/WorkspaceShare.js +149 -0
- package/dist/cjs/WorkspaceShare.js.map +7 -0
- package/dist/cjs/index.d.ts +91 -0
- package/dist/cjs/index.d.ts.map +1 -0
- package/dist/cjs/index.js +67 -0
- package/dist/cjs/index.js.map +7 -0
- package/dist/cjs/internal/Admission.d.ts +43 -0
- package/dist/cjs/internal/Admission.d.ts.map +1 -0
- package/dist/cjs/internal/Admission.js +90 -0
- package/dist/cjs/internal/Admission.js.map +7 -0
- package/dist/cjs/internal/CauseText.d.ts +55 -0
- package/dist/cjs/internal/CauseText.d.ts.map +1 -0
- package/dist/cjs/internal/CauseText.js +38 -0
- package/dist/cjs/internal/CauseText.js.map +7 -0
- package/dist/cjs/internal/JournalErrorCode.d.ts +23 -0
- package/dist/cjs/internal/JournalErrorCode.d.ts.map +1 -0
- package/dist/cjs/internal/JournalErrorCode.js +34 -0
- package/dist/cjs/internal/JournalErrorCode.js.map +7 -0
- package/dist/cjs/internal/PolicyOptions.d.ts +57 -0
- package/dist/cjs/internal/PolicyOptions.d.ts.map +1 -0
- package/dist/cjs/internal/PolicyOptions.js +62 -0
- package/dist/cjs/internal/PolicyOptions.js.map +7 -0
- package/dist/cjs/internal/ProtocolVersion.d.ts +17 -0
- package/dist/cjs/internal/ProtocolVersion.d.ts.map +1 -0
- package/dist/cjs/internal/ProtocolVersion.js +44 -0
- package/dist/cjs/internal/ProtocolVersion.js.map +7 -0
- package/dist/cjs/internal/ShareSigner.d.ts +173 -0
- package/dist/cjs/internal/ShareSigner.d.ts.map +1 -0
- package/dist/cjs/internal/ShareSigner.js +150 -0
- package/dist/cjs/internal/ShareSigner.js.map +7 -0
- package/dist/cjs/internal/SnapshotBoundary.d.ts +17 -0
- package/dist/cjs/internal/SnapshotBoundary.d.ts.map +1 -0
- package/dist/cjs/internal/SnapshotBoundary.js +97 -0
- package/dist/cjs/internal/SnapshotBoundary.js.map +7 -0
- package/dist/cjs/package.json +1 -0
- package/dist/cjs/test/TestSocket.d.ts +55 -0
- package/dist/cjs/test/TestSocket.d.ts.map +1 -0
- package/dist/cjs/test/TestSocket.js +145 -0
- package/dist/cjs/test/TestSocket.js.map +7 -0
- package/dist/cjs/test/TestSync.d.ts +54 -0
- package/dist/cjs/test/TestSync.d.ts.map +1 -0
- package/dist/cjs/test/TestSync.js +96 -0
- package/dist/cjs/test/TestSync.js.map +7 -0
- package/dist/esm/BranchCommands.d.ts +228 -0
- package/dist/esm/BranchCommands.d.ts.map +1 -0
- package/dist/esm/BranchCommands.js +496 -0
- package/dist/esm/BranchCommands.js.map +1 -0
- package/dist/esm/BranchIds.d.ts +65 -0
- package/dist/esm/BranchIds.d.ts.map +1 -0
- package/dist/esm/BranchIds.js +61 -0
- package/dist/esm/BranchIds.js.map +1 -0
- package/dist/esm/BranchPresence.d.ts +195 -0
- package/dist/esm/BranchPresence.d.ts.map +1 -0
- package/dist/esm/BranchPresence.js +361 -0
- package/dist/esm/BranchPresence.js.map +1 -0
- package/dist/esm/BranchProjection.d.ts +179 -0
- package/dist/esm/BranchProjection.d.ts.map +1 -0
- package/dist/esm/BranchProjection.js +269 -0
- package/dist/esm/BranchProjection.js.map +1 -0
- package/dist/esm/BranchProtocol.d.ts +415 -0
- package/dist/esm/BranchProtocol.d.ts.map +1 -0
- package/dist/esm/BranchProtocol.js +323 -0
- package/dist/esm/BranchProtocol.js.map +1 -0
- package/dist/esm/BranchRpcs.d.ts +177 -0
- package/dist/esm/BranchRpcs.d.ts.map +1 -0
- package/dist/esm/BranchRpcs.js +103 -0
- package/dist/esm/BranchRpcs.js.map +1 -0
- package/dist/esm/BranchServer.d.ts +55 -0
- package/dist/esm/BranchServer.d.ts.map +1 -0
- package/dist/esm/BranchServer.js +168 -0
- package/dist/esm/BranchServer.js.map +1 -0
- package/dist/esm/BranchShare.d.ts +159 -0
- package/dist/esm/BranchShare.d.ts.map +1 -0
- package/dist/esm/BranchShare.js +213 -0
- package/dist/esm/BranchShare.js.map +1 -0
- package/dist/esm/RunCatalog.d.ts +207 -0
- package/dist/esm/RunCatalog.d.ts.map +1 -0
- package/dist/esm/RunCatalog.js +196 -0
- package/dist/esm/RunCatalog.js.map +1 -0
- package/dist/esm/SyncAuth.d.ts +70 -0
- package/dist/esm/SyncAuth.d.ts.map +1 -0
- package/dist/esm/SyncAuth.js +119 -0
- package/dist/esm/SyncAuth.js.map +1 -0
- package/dist/esm/SyncClient.d.ts +274 -0
- package/dist/esm/SyncClient.d.ts.map +1 -0
- package/dist/esm/SyncClient.js +586 -0
- package/dist/esm/SyncClient.js.map +1 -0
- package/dist/esm/SyncError.d.ts +107 -0
- package/dist/esm/SyncError.d.ts.map +1 -0
- package/dist/esm/SyncError.js +133 -0
- package/dist/esm/SyncError.js.map +1 -0
- package/dist/esm/SyncPrincipal.d.ts +100 -0
- package/dist/esm/SyncPrincipal.d.ts.map +1 -0
- package/dist/esm/SyncPrincipal.js +58 -0
- package/dist/esm/SyncPrincipal.js.map +1 -0
- package/dist/esm/SyncProtocol.d.ts +592 -0
- package/dist/esm/SyncProtocol.d.ts.map +1 -0
- package/dist/esm/SyncProtocol.js +434 -0
- package/dist/esm/SyncProtocol.js.map +1 -0
- package/dist/esm/SyncRpcs.d.ts +92 -0
- package/dist/esm/SyncRpcs.d.ts.map +1 -0
- package/dist/esm/SyncRpcs.js +35 -0
- package/dist/esm/SyncRpcs.js.map +1 -0
- package/dist/esm/SyncServer.d.ts +184 -0
- package/dist/esm/SyncServer.d.ts.map +1 -0
- package/dist/esm/SyncServer.js +897 -0
- package/dist/esm/SyncServer.js.map +1 -0
- package/dist/esm/WorkspaceShare.d.ts +174 -0
- package/dist/esm/WorkspaceShare.d.ts.map +1 -0
- package/dist/esm/WorkspaceShare.js +208 -0
- package/dist/esm/WorkspaceShare.js.map +1 -0
- package/dist/esm/index.d.ts +91 -0
- package/dist/esm/index.d.ts.map +1 -0
- package/dist/esm/index.js +91 -0
- package/dist/esm/index.js.map +1 -0
- package/dist/esm/internal/Admission.d.ts +43 -0
- package/dist/esm/internal/Admission.d.ts.map +1 -0
- package/dist/esm/internal/Admission.js +80 -0
- package/dist/esm/internal/Admission.js.map +1 -0
- package/dist/esm/internal/CauseText.d.ts +55 -0
- package/dist/esm/internal/CauseText.d.ts.map +1 -0
- package/dist/esm/internal/CauseText.js +67 -0
- package/dist/esm/internal/CauseText.js.map +1 -0
- package/dist/esm/internal/JournalErrorCode.d.ts +23 -0
- package/dist/esm/internal/JournalErrorCode.d.ts.map +1 -0
- package/dist/esm/internal/JournalErrorCode.js +44 -0
- package/dist/esm/internal/JournalErrorCode.js.map +1 -0
- package/dist/esm/internal/PolicyOptions.d.ts +57 -0
- package/dist/esm/internal/PolicyOptions.d.ts.map +1 -0
- package/dist/esm/internal/PolicyOptions.js +70 -0
- package/dist/esm/internal/PolicyOptions.js.map +1 -0
- package/dist/esm/internal/ProtocolVersion.d.ts +17 -0
- package/dist/esm/internal/ProtocolVersion.d.ts.map +1 -0
- package/dist/esm/internal/ProtocolVersion.js +24 -0
- package/dist/esm/internal/ProtocolVersion.js.map +1 -0
- package/dist/esm/internal/ShareSigner.d.ts +173 -0
- package/dist/esm/internal/ShareSigner.d.ts.map +1 -0
- package/dist/esm/internal/ShareSigner.js +219 -0
- package/dist/esm/internal/ShareSigner.js.map +1 -0
- package/dist/esm/internal/SnapshotBoundary.d.ts +17 -0
- package/dist/esm/internal/SnapshotBoundary.d.ts.map +1 -0
- package/dist/esm/internal/SnapshotBoundary.js +68 -0
- package/dist/esm/internal/SnapshotBoundary.js.map +1 -0
- package/dist/esm/test/TestSocket.d.ts +55 -0
- package/dist/esm/test/TestSocket.d.ts.map +1 -0
- package/dist/esm/test/TestSocket.js +128 -0
- package/dist/esm/test/TestSocket.js.map +1 -0
- package/dist/esm/test/TestSync.d.ts +54 -0
- package/dist/esm/test/TestSync.d.ts.map +1 -0
- package/dist/esm/test/TestSync.js +76 -0
- package/dist/esm/test/TestSync.js.map +1 -0
- package/docs/README.md +156 -0
- package/docs/api.md +328 -0
- package/docs/concepts/authorization.md +117 -0
- package/docs/concepts/branches.md +195 -0
- package/docs/concepts/compaction.md +144 -0
- package/docs/concepts/replay-then-follow.md +129 -0
- package/docs/concepts/scopes-and-cursors.md +135 -0
- package/docs/guides/authorize-a-connection.md +174 -0
- package/docs/guides/follow-a-run.md +174 -0
- package/docs/guides/handle-a-compacted-run.md +158 -0
- package/docs/guides/list-workspace-runs.md +120 -0
- package/docs/guides/run-a-long-soak.md +80 -0
- package/docs/guides/serve-the-read-path.md +146 -0
- package/docs/guides/test-a-follower.md +273 -0
- package/docs/installation.md +92 -0
- package/docs/protocol.md +183 -0
- package/docs/quickstart.md +159 -0
- package/docs/troubleshooting.md +234 -0
- package/package.json +304 -3
- package/src/BranchCommands.ts +684 -0
- package/src/BranchIds.ts +76 -0
- package/src/BranchPresence.ts +512 -0
- package/src/BranchProjection.ts +308 -0
- package/src/BranchProtocol.ts +411 -0
- package/src/BranchRpcs.ts +163 -0
- package/src/BranchServer.ts +202 -0
- package/src/BranchShare.ts +297 -0
- package/src/RunCatalog.ts +308 -0
- package/src/SyncAuth.ts +146 -0
- package/src/SyncClient.ts +902 -0
- package/src/SyncError.ts +147 -0
- package/src/SyncPrincipal.ts +115 -0
- package/src/SyncProtocol.ts +594 -0
- package/src/SyncRpcs.ts +40 -0
- package/src/SyncServer.ts +1184 -0
- package/src/WorkspaceShare.ts +292 -0
- package/src/index.ts +107 -0
- package/src/internal/Admission.ts +119 -0
- package/src/internal/CauseText.ts +74 -0
- package/src/internal/JournalErrorCode.ts +46 -0
- package/src/internal/PolicyOptions.ts +97 -0
- package/src/internal/ProtocolVersion.ts +28 -0
- package/src/internal/ShareSigner.ts +300 -0
- package/src/internal/SnapshotBoundary.ts +91 -0
- package/src/test/TestSocket.ts +192 -0
- package/src/test/TestSync.ts +113 -0
|
@@ -0,0 +1,174 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Follow a run"
|
|
3
|
+
description: "Compose a sync client over a transport, subscribe to one run or a whole workspace, acknowledge entries only after applying them, and persist a cursor a restart can resume from."
|
|
4
|
+
sidebar:
|
|
5
|
+
order: 1
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
This guide builds a follower: the side that reads. It assumes a host is already
|
|
9
|
+
serving `SyncRpcs`, whether that is [`@smthrs/gateway`](/api/gateway) on
|
|
10
|
+
`POST /sync` and `/sync/ws` or a host of your own.
|
|
11
|
+
|
|
12
|
+
## Provide the client
|
|
13
|
+
|
|
14
|
+
`SyncClient.layer` derives an RPC client from the ambient
|
|
15
|
+
`RpcClient.Protocol`, so the composition supplies the protocol and the
|
|
16
|
+
serialization:
|
|
17
|
+
|
|
18
|
+
```ts
|
|
19
|
+
import * as SyncClient from "@smthrs/sync/SyncClient"
|
|
20
|
+
import * as Layer from "effect/Layer"
|
|
21
|
+
import * as RpcClient from "effect/unstable/rpc/RpcClient"
|
|
22
|
+
import * as RpcSerialization from "effect/unstable/rpc/RpcSerialization"
|
|
23
|
+
import type * as Socket from "effect/unstable/socket/Socket"
|
|
24
|
+
|
|
25
|
+
export const clientLayer: Layer.Layer<SyncClient.SyncClient, never, Socket.Socket> = SyncClient.layer.pipe(
|
|
26
|
+
Layer.provide(RpcClient.layerProtocolSocket()),
|
|
27
|
+
Layer.provide(RpcSerialization.layerJson)
|
|
28
|
+
)
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
Supply the socket that protocol runs over from your platform. A browser
|
|
32
|
+
follower provides a WebSocket; a Node follower provides
|
|
33
|
+
`@effect/platform-node`'s socket layer; a test provides the in-memory pair from
|
|
34
|
+
[Test a follower](./test-a-follower.md).
|
|
35
|
+
|
|
36
|
+
To tune the catch-up page size or the frame ceiling the client enforces on
|
|
37
|
+
responses, build the service directly with `SyncClient.makeWith`, which
|
|
38
|
+
validates both options and fails with `invalid_request` when one is not a
|
|
39
|
+
positive safe integer:
|
|
40
|
+
|
|
41
|
+
```ts
|
|
42
|
+
import { SyncRpcs } from "@smthrs/sync/SyncRpcs"
|
|
43
|
+
import * as Effect from "effect/Effect"
|
|
44
|
+
import * as RpcClient from "effect/unstable/rpc/RpcClient"
|
|
45
|
+
|
|
46
|
+
const makeClient = Effect.flatMap(
|
|
47
|
+
RpcClient.make(SyncRpcs),
|
|
48
|
+
(client) => SyncClient.makeWith({ client, bootstrapLimit: 512 })
|
|
49
|
+
)
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
## Subscribe
|
|
53
|
+
|
|
54
|
+
One call runs both phases. The stream starts with durable history and continues
|
|
55
|
+
with live entries:
|
|
56
|
+
|
|
57
|
+
```ts
|
|
58
|
+
import type { JournalEvent } from "@smthrs/journal"
|
|
59
|
+
import * as Stream from "effect/Stream"
|
|
60
|
+
|
|
61
|
+
const runId = "build-42" as JournalEvent.RunId
|
|
62
|
+
|
|
63
|
+
const entries = Effect.gen(function*() {
|
|
64
|
+
const sync = yield* SyncClient.SyncClient
|
|
65
|
+
return sync.subscribe({ scope: { _tag: "Run", runId }, cursors: [] })
|
|
66
|
+
})
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Pass `{ _tag: "Workspace" }` to follow every run the catalog lists and your
|
|
70
|
+
credential covers. A workspace subscription reconciles its covered run set on
|
|
71
|
+
every round, so a run created after it opened joins it.
|
|
72
|
+
|
|
73
|
+
## Acknowledge only what you applied
|
|
74
|
+
|
|
75
|
+
By default only a delivered bookmark advances as an entry is handed to the
|
|
76
|
+
consumer. Supply `apply` to record separately tagged applied progress after
|
|
77
|
+
the callback succeeds:
|
|
78
|
+
|
|
79
|
+
```ts
|
|
80
|
+
const followed = Effect.gen(function*() {
|
|
81
|
+
const sync = yield* SyncClient.SyncClient
|
|
82
|
+
return sync.subscribe({
|
|
83
|
+
scope: { _tag: "Run", runId },
|
|
84
|
+
cursors: [],
|
|
85
|
+
apply: (entry) => Effect.logInfo(`applied ${entry.eventType}@${entry.seq}`)
|
|
86
|
+
})
|
|
87
|
+
})
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
A failure in `apply` fails the subscription with that entry unacknowledged, so
|
|
91
|
+
the next subscription delivers it again. Make the callback idempotent:
|
|
92
|
+
redelivery is what a retry looks like here.
|
|
93
|
+
|
|
94
|
+
## Resume after a restart
|
|
95
|
+
|
|
96
|
+
`(yield* client.progress).delivered` reports delivery bookmarks.
|
|
97
|
+
`(yield* client.progress).applied` reports successfully applied positions, one
|
|
98
|
+
entry per run, sorted by run id.
|
|
99
|
+
Both maps are in memory. A durable consumer commits its projection and cursor
|
|
100
|
+
together in its application transaction, then seeds a fresh client with that
|
|
101
|
+
durable cursor after restart:
|
|
102
|
+
|
|
103
|
+
```ts
|
|
104
|
+
const resume = (
|
|
105
|
+
saved: ReadonlyArray<{ readonly runId: JournalEvent.RunId; readonly afterSeq: JournalEvent.Seq }>,
|
|
106
|
+
apply: NonNullable<SyncClient.SubscribeOptions["apply"]>
|
|
107
|
+
) =>
|
|
108
|
+
Effect.gen(function*() {
|
|
109
|
+
const sync = yield* SyncClient.SyncClient
|
|
110
|
+
return sync.subscribe({ scope: { _tag: "Workspace" }, cursors: saved, apply })
|
|
111
|
+
})
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
The effective start position is the later of your cursor and the matching
|
|
115
|
+
progress map: applied when `apply` is supplied, delivered otherwise. An earlier
|
|
116
|
+
delivery-only subscription cannot skip unapplied history. Independent
|
|
117
|
+
materializations and rebuilds use fresh clients with empty progress maps.
|
|
118
|
+
|
|
119
|
+
## Present a credential
|
|
120
|
+
|
|
121
|
+
A connection with no credential reads no non-branch run. Stamp every outgoing
|
|
122
|
+
request with a workspace capability:
|
|
123
|
+
|
|
124
|
+
```ts
|
|
125
|
+
import * as SyncAuth from "@smthrs/sync/SyncAuth"
|
|
126
|
+
import type * as WorkspaceShare from "@smthrs/sync/WorkspaceShare"
|
|
127
|
+
|
|
128
|
+
const authorized = (capability: WorkspaceShare.WorkspaceCapability) =>
|
|
129
|
+
SyncClient.layer.pipe(Layer.provide(SyncAuth.layerClient(capability)))
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
To read a shared branch instead, pass that branch's capability on the
|
|
133
|
+
subscription itself as `capability`. See
|
|
134
|
+
[Authorize a connection](./authorize-a-connection.md).
|
|
135
|
+
|
|
136
|
+
## Handle the failures that reach you
|
|
137
|
+
|
|
138
|
+
The stream fails with `SyncError` or `SyncGapError`. Two cases have recovery paths:
|
|
139
|
+
|
|
140
|
+
- `transport_failed` is retried under exponential backoff capped at five
|
|
141
|
+
seconds and only surfaces if the stream is otherwise ended.
|
|
142
|
+
- `compacted` requires an explicit `onResync` handler to restore a checkpoint
|
|
143
|
+
and return its exact cursor before restarting. Without one it fails closed. See
|
|
144
|
+
[Handle a compacted run](./handle-a-compacted-run.md).
|
|
145
|
+
|
|
146
|
+
Everything else is yours:
|
|
147
|
+
|
|
148
|
+
| Failure | What it means |
|
|
149
|
+
| -------------------- | -------------------------------------------------------------------------------------------------------- |
|
|
150
|
+
| `SyncGapError` | The server's frame started beyond the cursor you covered. History was missed; rebuild rather than retry. |
|
|
151
|
+
| `unauthorized` | No credential, a refused one, or a credential that expired while the subscription was open. |
|
|
152
|
+
| `protocol_violation` | The server's page or frame contradicted itself. No cursor moved. |
|
|
153
|
+
| `frame_too_large` | A page or frame exceeded the client's frame ceiling. |
|
|
154
|
+
| `closed` | The server ended the subscription with a `Closed` frame. |
|
|
155
|
+
|
|
156
|
+
[Troubleshooting](../troubleshooting.md) has the full code table with causes and
|
|
157
|
+
fixes.
|
|
158
|
+
|
|
159
|
+
## Tune the window
|
|
160
|
+
|
|
161
|
+
`credit` bounds the frames one subscription round carries before the follow
|
|
162
|
+
replenishes it by resubscribing. The default is 256, and the ceiling the wire
|
|
163
|
+
allows is `SyncProtocol.maxSubscribeCredit`:
|
|
164
|
+
|
|
165
|
+
```ts
|
|
166
|
+
const narrow = Effect.gen(function*() {
|
|
167
|
+
const sync = yield* SyncClient.SyncClient
|
|
168
|
+
return sync.subscribe({ scope: { _tag: "Run", runId }, cursors: [], credit: 32 })
|
|
169
|
+
})
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
Lower it only for a deliberate reason. A small window makes the round-trip cost
|
|
173
|
+
proportional to traffic rather than to the window, which is what
|
|
174
|
+
[Replay then follow](../concepts/replay-then-follow.md) explains.
|
|
@@ -0,0 +1,158 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Handle a compacted run"
|
|
3
|
+
description: "Restore a checkpoint and return its exact cursor before replaying the remaining history."
|
|
4
|
+
sidebar:
|
|
5
|
+
order: 5
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
A cursor below a run's compaction floor names deleted history. The server
|
|
9
|
+
refuses it with `compacted` and `resync: { runId, checkpointSeq, generation }`.
|
|
10
|
+
The client fails without moving its cursor unless you supply `onResync`.
|
|
11
|
+
|
|
12
|
+
The handler must restore the missing prefix and return
|
|
13
|
+
`{ runId, afterSeq }` for the snapshot actually applied. A newer snapshot can
|
|
14
|
+
have a sequence above the reported floor. Returning the older floor after
|
|
15
|
+
applying that snapshot would replay part of its state twice. The client binds
|
|
16
|
+
the returned cursor to `resync.generation`, the run's history after any rewinds,
|
|
17
|
+
so a fresh follower needs no earlier cursor to resume it.
|
|
18
|
+
|
|
19
|
+
## Restore a remote public projection
|
|
20
|
+
|
|
21
|
+
The host must configure `SyncServer.SnapshotSource` with a provider that selects
|
|
22
|
+
only public state for the requested run, lineage, projection and schema version.
|
|
23
|
+
All authorized readers of that run can fetch it. Narrower or private data needs
|
|
24
|
+
its own authorization boundary. A raw `Journal.latestCheckpoint` passthrough is
|
|
25
|
+
unsafe because execution checkpoints are intentionally unredacted.
|
|
26
|
+
|
|
27
|
+
The client can fetch that projection in its recovery handler:
|
|
28
|
+
|
|
29
|
+
```ts
|
|
30
|
+
import type * as SyncClient from "@smthrs/sync/SyncClient"
|
|
31
|
+
import type { SyncError } from "@smthrs/sync/SyncError"
|
|
32
|
+
import type * as SyncProtocol from "@smthrs/sync/SyncProtocol"
|
|
33
|
+
import * as Effect from "effect/Effect"
|
|
34
|
+
|
|
35
|
+
const recoverPublic = <E>(
|
|
36
|
+
sync: SyncClient.Service,
|
|
37
|
+
identity: Pick<SyncProtocol.SnapshotRequest, "lineageId" | "projection" | "projectionVersion" | "capability">,
|
|
38
|
+
applySnapshot: (snapshot: SyncProtocol.Snapshot) => Effect.Effect<void, E>
|
|
39
|
+
) =>
|
|
40
|
+
(resync: SyncProtocol.Resync): Effect.Effect<SyncProtocol.RunCursor, SyncError | E> =>
|
|
41
|
+
Effect.gen(function*() {
|
|
42
|
+
const snapshot = yield* sync.snapshot({
|
|
43
|
+
...identity,
|
|
44
|
+
protocolVersion: 1,
|
|
45
|
+
runId: resync.runId,
|
|
46
|
+
atLeastSeq: resync.checkpointSeq
|
|
47
|
+
})
|
|
48
|
+
const cursor = { runId: snapshot.runId, afterSeq: snapshot.seq }
|
|
49
|
+
yield* applySnapshot(snapshot)
|
|
50
|
+
return cursor
|
|
51
|
+
})
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
`snapshot` validates identity, version, sequence and byte limits; your callback
|
|
55
|
+
must still decode the projection's application schema and transactionally apply
|
|
56
|
+
its state and durable cursor. The handler fails with `snapshot`'s `SyncError`
|
|
57
|
+
or with your callback's own error `E`, and the subscription carries either
|
|
58
|
+
unchanged. A fetch alone never acknowledges missing history.
|
|
59
|
+
The host must retain a snapshot covering the compaction floor. If the floor
|
|
60
|
+
moves again before replay, recovery runs again for the newer floor.
|
|
61
|
+
|
|
62
|
+
## Restore through an authorized local source
|
|
63
|
+
|
|
64
|
+
A trusted local follower may alternatively read its execution checkpoint from
|
|
65
|
+
the journal and supply its own application callback:
|
|
66
|
+
|
|
67
|
+
```ts
|
|
68
|
+
import { Journal } from "@smthrs/journal"
|
|
69
|
+
import type * as SyncProtocol from "@smthrs/sync/SyncProtocol"
|
|
70
|
+
import * as Data from "effect/Data"
|
|
71
|
+
import * as Effect from "effect/Effect"
|
|
72
|
+
import * as Option from "effect/Option"
|
|
73
|
+
|
|
74
|
+
class CheckpointUnavailable extends Data.TaggedError("CheckpointUnavailable")<{
|
|
75
|
+
readonly message: string
|
|
76
|
+
readonly resync: SyncProtocol.Resync
|
|
77
|
+
}> {}
|
|
78
|
+
|
|
79
|
+
const recoverFromJournal = <E>(
|
|
80
|
+
journal: Journal.Service,
|
|
81
|
+
applySnapshot: (checkpoint: Journal.Checkpoint) => Effect.Effect<void, E>
|
|
82
|
+
) =>
|
|
83
|
+
(resync: SyncProtocol.Resync): Effect.Effect<SyncProtocol.RunCursor, CheckpointUnavailable | E> =>
|
|
84
|
+
Effect.gen(function*() {
|
|
85
|
+
const saved = yield* journal.latestCheckpoint(resync.runId).pipe(
|
|
86
|
+
Effect.mapError(() => new CheckpointUnavailable({ message: "Could not read the checkpoint", resync }))
|
|
87
|
+
)
|
|
88
|
+
if (
|
|
89
|
+
Option.isNone(saved) ||
|
|
90
|
+
saved.value.runId !== resync.runId ||
|
|
91
|
+
saved.value.seq < resync.checkpointSeq
|
|
92
|
+
) {
|
|
93
|
+
return yield* Effect.fail(
|
|
94
|
+
new CheckpointUnavailable({ message: "No checkpoint covers the requested history", resync })
|
|
95
|
+
)
|
|
96
|
+
}
|
|
97
|
+
const checkpoint = saved.value
|
|
98
|
+
const cursor = { runId: checkpoint.runId, afterSeq: checkpoint.seq }
|
|
99
|
+
yield* applySnapshot(checkpoint)
|
|
100
|
+
return cursor
|
|
101
|
+
})
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
Pass the returned function as `onResync`. `CheckpointUnavailable` is the
|
|
105
|
+
follower's own error, not a wire code, so the subscription reports it as a
|
|
106
|
+
local restoration failure rather than as the server's `compacted` refusal. The
|
|
107
|
+
required `applySnapshot` callback must actually decode and restore the
|
|
108
|
+
projection; logging a checkpoint is not restoration. Validate its run/lineage
|
|
109
|
+
and projection schema/version before using the state. Journal checkpoints are generic and intentionally
|
|
110
|
+
unredacted: never expose raw execution checkpoints to a remote follower merely
|
|
111
|
+
because that follower can read redacted history.
|
|
112
|
+
|
|
113
|
+
For a durable projection, commit the restored state **and its durable cursor**
|
|
114
|
+
in one application-owned transaction. On restart, seed a fresh sync client with
|
|
115
|
+
that committed cursor. The sync client's cursor is only in memory; its
|
|
116
|
+
interruption mask cannot make writes to separate stores atomic. Make recovery
|
|
117
|
+
idempotent so a crash or a moving compaction floor can safely repeat it.
|
|
118
|
+
|
|
119
|
+
Apply the same rule to every suffix entry: the `apply` callback must commit its
|
|
120
|
+
projection changes and durable cursor together before returning. Keep the
|
|
121
|
+
cursor with its run/lineage and projection version, so a cursor for one
|
|
122
|
+
projection cannot skip the history needed by another. On restart, read that
|
|
123
|
+
durable cursor, not a cursor cached separately before the crash.
|
|
124
|
+
|
|
125
|
+
| Crash point | Durable recovery |
|
|
126
|
+
| ---------------------------------------------------------- | --------------------------------------------------------------------------------- |
|
|
127
|
+
| Between state and cursor writes, before transaction commit | Both writes roll back; restore the snapshot again. |
|
|
128
|
+
| After snapshot commit, before `onResync` returns | Seed the fresh client with the committed snapshot cursor; replay only its suffix. |
|
|
129
|
+
| After an event commit, before `apply` returns | Seed from that event's committed cursor; do not apply the event twice. |
|
|
130
|
+
|
|
131
|
+
These are application-store transaction guarantees. The sync client does not
|
|
132
|
+
make arbitrary callbacks or external side effects exactly-once.
|
|
133
|
+
|
|
134
|
+
The client validates the receipt's run and sequence before advancing. Missing,
|
|
135
|
+
invalid, foreign-run or behind-floor receipts fail with the cursor unchanged.
|
|
136
|
+
If compaction moves again before the next read, recovery runs again for the
|
|
137
|
+
new floor.
|
|
138
|
+
|
|
139
|
+
## Refuse when restoration is unavailable
|
|
140
|
+
|
|
141
|
+
Omit `onResync` to preserve the original `compacted` refusal and cursor.
|
|
142
|
+
A handler may also fail with its own error if its snapshot is missing,
|
|
143
|
+
unauthorized, incompatible, or cannot be applied; the subscription fails with
|
|
144
|
+
that error unchanged. Failed or interrupted application never acknowledges the
|
|
145
|
+
deleted prefix.
|
|
146
|
+
|
|
147
|
+
A browser or remote follower without a configured public snapshot source must
|
|
148
|
+
not pretend to have rebuilt complete state. The RPC fails closed when the
|
|
149
|
+
host has not provided the requested public projection.
|
|
150
|
+
|
|
151
|
+
Returning the reported floor without restoring state would claim application
|
|
152
|
+
of deleted history. Keep the typed `compacted` result when reconstruction is
|
|
153
|
+
unavailable; clearing a projection does not recreate its missing prefix.
|
|
154
|
+
|
|
155
|
+
## Related pages
|
|
156
|
+
|
|
157
|
+
- [Compaction and resync](../concepts/compaction.md): scope and failure semantics.
|
|
158
|
+
- [Checkpoints and compaction](/pkg/journal/concepts/compaction): journal retention.
|
|
@@ -0,0 +1,120 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "List a workspace's runs"
|
|
3
|
+
description: "Choose the run catalog a workspace subscription reconciles against: a fixed set, an in-process registry, or a poll of the durable run set another engine writes."
|
|
4
|
+
sidebar:
|
|
5
|
+
order: 4
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
A workspace subscription serves the runs a `RunCatalog` lists. The catalog is
|
|
9
|
+
supplied by the host rather than derived from the journal, because
|
|
10
|
+
[`@smthrs/journal`](/api/journal) has no workspace-wide list or watch contract.
|
|
11
|
+
This guide picks the right implementation.
|
|
12
|
+
|
|
13
|
+
## The contract
|
|
14
|
+
|
|
15
|
+
```ts
|
|
16
|
+
import type * as RunCatalog from "@smthrs/sync/RunCatalog"
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
`list` is the authoritative run set. `changes` is a low-latency wake and
|
|
20
|
+
nothing more: a subscriber that misses an announcement re-lists rather than
|
|
21
|
+
losing state. A workspace subscription reconciles its covered set against
|
|
22
|
+
`list` on every round, so a dropped notification costs latency and never state.
|
|
23
|
+
|
|
24
|
+
`list` must name each run at most once. Every implementation here deduplicates,
|
|
25
|
+
and `SyncServer` deduplicates again at the seam rather than trusting a host's
|
|
26
|
+
catalog, because both fan-out paths key a run's served position by run id: a
|
|
27
|
+
run named twice is read twice from the same position and its entries are served
|
|
28
|
+
twice.
|
|
29
|
+
|
|
30
|
+
Every implementation returns a fresh array from `list`, so a caller may retain
|
|
31
|
+
and sort what it receives without disturbing another reader.
|
|
32
|
+
|
|
33
|
+
## A fixed set
|
|
34
|
+
|
|
35
|
+
```ts
|
|
36
|
+
import type { JournalEvent } from "@smthrs/journal"
|
|
37
|
+
import * as RunCatalogNs from "@smthrs/sync/RunCatalog"
|
|
38
|
+
|
|
39
|
+
const fixed = RunCatalogNs.layerStatic(["build-42" as JournalEvent.RunId])
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
`changes` is empty, so nothing is ever announced. Use it when the covered set
|
|
43
|
+
is known at composition time, and for tests. `RunCatalog.layerNoop` is
|
|
44
|
+
`layerStatic([])`.
|
|
45
|
+
|
|
46
|
+
## Runs this process creates
|
|
47
|
+
|
|
48
|
+
`makeMemory` returns a catalog and the `register` function that grows it. A run
|
|
49
|
+
is published exactly once, when it is first registered:
|
|
50
|
+
|
|
51
|
+
```ts
|
|
52
|
+
import * as Effect from "effect/Effect"
|
|
53
|
+
|
|
54
|
+
const local = Effect.gen(function*() {
|
|
55
|
+
const { catalog, register } = yield* RunCatalogNs.makeMemory()
|
|
56
|
+
yield* register("build-43" as JournalEvent.RunId)
|
|
57
|
+
return catalog
|
|
58
|
+
})
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
`RunCatalog.layerMemory` provides the same catalog when a composition needs the
|
|
62
|
+
service and not the `register` function.
|
|
63
|
+
|
|
64
|
+
The announcement feed slides at `RunCatalog.defaultChangesCapacity`
|
|
65
|
+
announcements: registering never waits on a stalled subscriber and never grows
|
|
66
|
+
the process on its behalf. A subscriber that falls further behind loses the
|
|
67
|
+
oldest announcements and converges by re-listing.
|
|
68
|
+
|
|
69
|
+
This catalog only ever hears about runs the same process registered. A follower
|
|
70
|
+
composed against it never learns of a run another engine created.
|
|
71
|
+
|
|
72
|
+
## Runs any engine creates
|
|
73
|
+
|
|
74
|
+
`makePolling` re-reads a durable source on an interval.
|
|
75
|
+
[`@smthrs/engine-store`](/api/engine-store) supplies that read as
|
|
76
|
+
`RunCatalogRead`, over its own `flows_runs` table:
|
|
77
|
+
|
|
78
|
+
```ts
|
|
79
|
+
const durable = (read: Effect.Effect<ReadonlyArray<JournalEvent.RunId>>) =>
|
|
80
|
+
RunCatalogNs.layerPolling({ read, intervalMs: 1000 })
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
Anything that answers the same question works; the catalog holds no assumption
|
|
84
|
+
about which. The interval defaults to `RunCatalog.defaultPollIntervalMs`, one
|
|
85
|
+
second, and the read costs one bounded query per interval per composition, not
|
|
86
|
+
per subscriber.
|
|
87
|
+
|
|
88
|
+
Four behaviors are worth knowing before you rely on it:
|
|
89
|
+
|
|
90
|
+
- **It primes itself before returning.** The first read a follower makes is
|
|
91
|
+
already current, and a workspace that cannot be read at all fails the
|
|
92
|
+
composition instead of serving an empty run set.
|
|
93
|
+
- **A failed read is a warning and nothing else.** The previous view stands,
|
|
94
|
+
the interval holds, and no subscription attached to the catalog is torn down.
|
|
95
|
+
The warning is logged once per outage with the first failure's cause, and one
|
|
96
|
+
info line marks the read that ends it; the failed polls in between are silent.
|
|
97
|
+
- **A run is announced once**, when it first appears. A run the read stops
|
|
98
|
+
naming, which is what retention collecting it looks like, leaves the view.
|
|
99
|
+
`list` is the workspace's run set, not a log of every run it ever had.
|
|
100
|
+
- **The poll fiber belongs to the caller's scope.** Closing the scope stops it.
|
|
101
|
+
|
|
102
|
+
Polling rather than waking is deliberate at 1.0.0-rc.0: there is no
|
|
103
|
+
cross-process event delivery, so a follower learns of another engine's run by
|
|
104
|
+
asking the workspace again, and the interval is the whole of the policy.
|
|
105
|
+
|
|
106
|
+
## Choosing an interval
|
|
107
|
+
|
|
108
|
+
The interval is how long a run can exist before a follower sees it. One second
|
|
109
|
+
is short enough that a run appears in a dashboard while the operator is still
|
|
110
|
+
looking at the command that started it, and long enough that an idle workspace
|
|
111
|
+
is close to free. Lower it only when a real user is waiting on the difference,
|
|
112
|
+
and remember that the workspace subscription's own `tailIntervalMs` is a second
|
|
113
|
+
bound on the same latency.
|
|
114
|
+
|
|
115
|
+
## Related pages
|
|
116
|
+
|
|
117
|
+
- [Serve the read path](./serve-the-read-path.md): where the catalog is
|
|
118
|
+
provided.
|
|
119
|
+
- [Coordinate two processes over one store](/pkg/engine-store/guides/coordinate-two-processes):
|
|
120
|
+
the durable read this catalog polls.
|
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Run the optional long soak"
|
|
3
|
+
description: "Execute and verify repeated resource-growth samples across TCP reconnects, compaction, retention and a stalled sync consumer."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Ordinary package gates keep the bounded `ServerSoak.test.ts` tests. The timed
|
|
7
|
+
tier is off until `SMITHERS_SOAK_MINUTES` is set. Run from the sync package:
|
|
8
|
+
|
|
9
|
+
```sh
|
|
10
|
+
SMITHERS_SOAK_MINUTES=60 \
|
|
11
|
+
SMITHERS_SOAK_ARTIFACT=/tmp/sync-soak/node26.json \
|
|
12
|
+
pnpm exec vitest run test/ServerLongSoak.test.ts --maxWorkers=1 --coverage.enabled=false
|
|
13
|
+
node test/fixtures/verify-soak.ts /tmp/sync-soak/node26.json 60
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
The focused timed tier does not replace the full package coverage gate. It
|
|
17
|
+
spawns the same Node executable with `--expose-gc`. Supported pinned lane
|
|
18
|
+
runtimes are Node 26.4.0 and 26.10.0. Minutes must be finite and within 1..720.
|
|
19
|
+
The artifact path is required. For a bounded local rehearsal use 3 minutes,
|
|
20
|
+
and label the receipt as local bounded evidence.
|
|
21
|
+
|
|
22
|
+
The workload uses an on-disk SQLite journal and schema-aware NDJSON RPC over
|
|
23
|
+
loopback TCP. Each cycle appends eight entries and opens four concurrent
|
|
24
|
+
connections. Each consumer reconstructs the complete count and checks its
|
|
25
|
+
applied cursor. A separate workspace consumer stalls inside `apply` for the
|
|
26
|
+
whole measurement. Journal changes are also held unread to measure their
|
|
27
|
+
bounded queue. Compaction advances a checkpoint while retaining 32 history
|
|
28
|
+
rows, collects superseded checkpoints, and repeatedly triggers snapshot
|
|
29
|
+
recovery in reconnecting clients. This exercises retention of history and
|
|
30
|
+
checkpoints, not whole-run deletion or retention of externally stored blobs.
|
|
31
|
+
|
|
32
|
+
## Artifact verifier contract
|
|
33
|
+
|
|
34
|
+
`test/soakArtifact.ts` defines schema version 1. CI must run the independent
|
|
35
|
+
`verify-soak.ts` command and fail on missing, malformed, incomplete, or failed
|
|
36
|
+
artifacts. The exit code of the workload is also required; a JSON file alone
|
|
37
|
+
is not a passing job. Preserve stdout/stderr and the JSON even on failure.
|
|
38
|
+
|
|
39
|
+
Required top-level fields:
|
|
40
|
+
|
|
41
|
+
| Field | Contract |
|
|
42
|
+
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
43
|
+
| `schemaVersion`, `status` | `1` and `complete` for a passing receipt; failure is explicit. |
|
|
44
|
+
| `runtime` | Exact Node version, platform, architecture. |
|
|
45
|
+
| `candidate` | Git head, dirty flag, SHA-256 over sorted sync source paths and bytes. A dirty local receipt is not immutable release certification. |
|
|
46
|
+
| `workload` | Requested minutes, seed `20260904`, warmup `20000` ms, sampling interval `10000` ms. |
|
|
47
|
+
| `samples` | At least five strictly increasing post-warmup checkpoints, continued useful work, no sample gap over 30 seconds, final sample at or beyond requested duration. |
|
|
48
|
+
| `slopes` | Ordinary least-squares regression of every metric against elapsed minutes. The verifier recomputes these values. |
|
|
49
|
+
| `cleanup` | Zero active journal reads, pending writes, stalled subscribers and connected TCP sockets after scope release. |
|
|
50
|
+
| `failure` | Diagnostic reason when the run fails. |
|
|
51
|
+
|
|
52
|
+
Every sample must carry elapsed time, cycle count, emitted events, connections,
|
|
53
|
+
compactions, retained event/checkpoint counts, stalled-subscriber count, and
|
|
54
|
+
all metrics below. Memory is measured after explicit GC. Handles come from
|
|
55
|
+
Node's `_getActiveHandles`; connected sockets are actual `net.Socket` handles.
|
|
56
|
+
Queues measure outstanding journal reads/writes, socket readable/writable
|
|
57
|
+
bytes, and the stalled journal notification subscription. These are named
|
|
58
|
+
observed queues, not a claim to inspect every runtime-internal RPC queue.
|
|
59
|
+
|
|
60
|
+
| Metric | Maximum positive slope per minute |
|
|
61
|
+
| ------------------------------------------------------------------ | --------------------------------- |
|
|
62
|
+
| Retained heap | 1 MiB |
|
|
63
|
+
| RSS | 4 MiB |
|
|
64
|
+
| Open handles, connected sockets | 0.5 each |
|
|
65
|
+
| Active journal reads, pending writes, queued journal notifications | 1 each |
|
|
66
|
+
| Queued socket bytes | 64 KiB |
|
|
67
|
+
| Database bytes, WAL bytes | 1 MiB each |
|
|
68
|
+
|
|
69
|
+
The verifier also requires at most 40 retained events, exactly one retained
|
|
70
|
+
checkpoint, exactly one stalled subscriber during sampling, zero pending
|
|
71
|
+
writes at checkpoints, and bounded journal notifications. SQLite's normal WAL
|
|
72
|
+
policy remains active; samples do not truncate the WAL to manufacture a flat
|
|
73
|
+
curve. The thresholds are provisional regression budgets, not a production
|
|
74
|
+
capacity or latency guarantee. Run the scheduled tier on a controlled host
|
|
75
|
+
and retain at least 30 days of artifacts for baseline comparison.
|
|
76
|
+
|
|
77
|
+
The scheduled owner should run a fresh 60-minute job on each pinned Node
|
|
78
|
+
version, serially or on separate workers, without a cached test result. A
|
|
79
|
+
three-minute local success verifies the harness and its bounded observation;
|
|
80
|
+
it does not close the hours-long leak campaign or certify remote CI execution.
|
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Serve the read path"
|
|
3
|
+
description: "Compose SyncServer over a journal and a run catalog, mount the RPC group on a transport with its authentication middleware, and set the policy that bounds one follower's cost."
|
|
4
|
+
sidebar:
|
|
5
|
+
order: 2
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
This guide builds the serving side. A host that already runs
|
|
9
|
+
[`@smthrs/gateway`](/api/gateway) has this done: the gateway mounts `SyncRpcs`
|
|
10
|
+
on `POST /sync` and `/sync/ws`. Build it yourself when you are embedding the
|
|
11
|
+
read path in a host of your own.
|
|
12
|
+
|
|
13
|
+
## Compose the server
|
|
14
|
+
|
|
15
|
+
`SyncServer.layer` requires a `Journal` and a `RunCatalog`, and nothing else:
|
|
16
|
+
|
|
17
|
+
```ts
|
|
18
|
+
import * as RunCatalog from "@smthrs/sync/RunCatalog"
|
|
19
|
+
import * as SyncServer from "@smthrs/sync/SyncServer"
|
|
20
|
+
import * as Layer from "effect/Layer"
|
|
21
|
+
|
|
22
|
+
const serverLayer = SyncServer.layer.pipe(
|
|
23
|
+
Layer.provide(RunCatalog.layerStatic([]))
|
|
24
|
+
)
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
The journal is still required, and the composition supplies it: in production
|
|
28
|
+
that is `@smthrs/journal`'s `SqlJournal` over
|
|
29
|
+
[`@smthrs/database`](/api/database). The catalog is the workspace's run set;
|
|
30
|
+
`layerStatic([])` serves a workspace with no runs, which is what a run-scoped
|
|
31
|
+
deployment wants. For a catalog that learns of runs another engine created, see
|
|
32
|
+
[List a workspace's runs](./list-workspace-runs.md).
|
|
33
|
+
|
|
34
|
+
Provide `BranchShare` as well if any run in the workspace is a shared branch.
|
|
35
|
+
Without it in scope, every branch run is closed.
|
|
36
|
+
|
|
37
|
+
## Mount the RPC group
|
|
38
|
+
|
|
39
|
+
`SyncServer.layerHandlers` projects the service onto the wire. It is a thin
|
|
40
|
+
projection with no policy of its own, so the handlers and an in-process caller
|
|
41
|
+
face the same rules:
|
|
42
|
+
|
|
43
|
+
```ts
|
|
44
|
+
import { SyncRpcs } from "@smthrs/sync/SyncRpcs"
|
|
45
|
+
import * as RpcServer from "effect/unstable/rpc/RpcServer"
|
|
46
|
+
|
|
47
|
+
const mount = RpcServer.layer(SyncRpcs, { disableFatalDefects: true }).pipe(
|
|
48
|
+
Layer.provide(SyncServer.layerHandlers),
|
|
49
|
+
Layer.provideMerge(RpcServer.layerProtocolHttp({ path: "/sync" }))
|
|
50
|
+
)
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Mount both protocols against the same handlers when clients read a page over
|
|
54
|
+
the request path and follow over a socket. Pointing them at two servers would
|
|
55
|
+
let the two halves disagree.
|
|
56
|
+
|
|
57
|
+
## Supply the authentication middleware
|
|
58
|
+
|
|
59
|
+
`SyncRpcs` is bound to the `SyncAuth` middleware tag, so serving it requires an
|
|
60
|
+
implementation. The production one verifies the capability header:
|
|
61
|
+
|
|
62
|
+
```ts
|
|
63
|
+
import * as SyncAuth from "@smthrs/sync/SyncAuth"
|
|
64
|
+
import * as WorkspaceShare from "@smthrs/sync/WorkspaceShare"
|
|
65
|
+
|
|
66
|
+
const authorized = SyncAuth.layer.pipe(Layer.provide(WorkspaceShare.layerConfig))
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Without a middleware-installed principal every request runs as anonymous and
|
|
70
|
+
every non-branch read is refused. That is the fail-closed default, not a
|
|
71
|
+
misconfiguration to work around. See
|
|
72
|
+
[Authorize a connection](./authorize-a-connection.md) for minting and rotation,
|
|
73
|
+
and [Authorization](../concepts/authorization.md) for why the two boundaries
|
|
74
|
+
are shaped this way.
|
|
75
|
+
|
|
76
|
+
## Read in process, without a transport
|
|
77
|
+
|
|
78
|
+
A caller that already owns the journal skips the wire and provides the
|
|
79
|
+
principal directly:
|
|
80
|
+
|
|
81
|
+
```ts
|
|
82
|
+
import * as SyncPrincipal from "@smthrs/sync/SyncPrincipal"
|
|
83
|
+
import * as Effect from "effect/Effect"
|
|
84
|
+
|
|
85
|
+
const page = Effect.gen(function*() {
|
|
86
|
+
const server = yield* SyncServer.SyncServer
|
|
87
|
+
return yield* server.read({ protocolVersion: 1, scope: { _tag: "Workspace" }, cursors: [], limit: 100 })
|
|
88
|
+
}).pipe(Effect.provide(SyncPrincipal.layerWorkspace("host")))
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
That is the only sanctioned bypass of the header check, and it is never a
|
|
92
|
+
transport's to take. An in-process owner presented no credential, so nothing
|
|
93
|
+
can expire under it and its subscriptions have no deadline.
|
|
94
|
+
|
|
95
|
+
## Set the policy
|
|
96
|
+
|
|
97
|
+
`SyncServer.layerWith` takes the read-path policy. Every field defaults, and
|
|
98
|
+
every field is validated as a positive safe integer at construction, so a bad
|
|
99
|
+
value fails loudly instead of quietly disabling the bound it configures:
|
|
100
|
+
|
|
101
|
+
```ts
|
|
102
|
+
const tuned = SyncServer.layerWith({
|
|
103
|
+
maxFrameBytes: 512 * 1024,
|
|
104
|
+
concurrency: 32,
|
|
105
|
+
tailIntervalMs: 250
|
|
106
|
+
})
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
| Option | Default | What it bounds |
|
|
110
|
+
| ---------------- | ------- | ------------------------------------------------------------------------------------------------------ |
|
|
111
|
+
| `maxFrameBytes` | 2 MiB | Summed encoded entries of one read page or subscription frame. |
|
|
112
|
+
| `concurrency` | 64 | Journal reads one workspace subscription holds open at once. |
|
|
113
|
+
| `tailIntervalMs` | 1000 | Milliseconds a workspace subscription waits before revisiting every covered run when nothing wakes it. |
|
|
114
|
+
|
|
115
|
+
`concurrency` bounds what one follower costs without bounding what it sees: the
|
|
116
|
+
workspace tail visits every covered run each round, so a run past the bound
|
|
117
|
+
waits one round for a slot rather than being starved.
|
|
118
|
+
|
|
119
|
+
`tailIntervalMs` is the freshness policy for runs another process owns, not for
|
|
120
|
+
runs written beside the follower. An entry this process commits is published on
|
|
121
|
+
`Journal.changes` and wakes the round at once. A run-scoped subscription follows
|
|
122
|
+
one journal stream directly and keeps that stream's wake.
|
|
123
|
+
|
|
124
|
+
## What the server refuses
|
|
125
|
+
|
|
126
|
+
- A request whose cursor set names one run twice, with `invalid_request`.
|
|
127
|
+
- A single entry whose own encoded size exceeds `maxFrameBytes`, with
|
|
128
|
+
`frame_too_large`. A page that merely sums past the ceiling is truncated and
|
|
129
|
+
reports `done: false` instead.
|
|
130
|
+
- Every non-branch read for an anonymous caller, with `unauthorized`.
|
|
131
|
+
- A subscription whose credential has expired, which ends the stream with
|
|
132
|
+
`unauthorized`.
|
|
133
|
+
|
|
134
|
+
A journal failure crosses as its stable journal code with a constant public
|
|
135
|
+
message. The driver's own sentence stays on the server, because a follower may
|
|
136
|
+
hold nothing but a share link and a SQLite message routinely carries SQL text,
|
|
137
|
+
table names, and constraint identifiers.
|
|
138
|
+
|
|
139
|
+
## Related pages
|
|
140
|
+
|
|
141
|
+
- [List a workspace's runs](./list-workspace-runs.md): the catalog this server
|
|
142
|
+
reconciles against.
|
|
143
|
+
- [Authorize a connection](./authorize-a-connection.md): the credential the
|
|
144
|
+
middleware verifies.
|
|
145
|
+
- [Test a follower](./test-a-follower.md): a real server and client over an
|
|
146
|
+
in-memory socket.
|