@smthrs/sync 0.0.0-stage → 1.0.0-rc.3

package/docs/guides/test-a-follower.md

@@ -0,0 +1,273 @@
+---
+title: "Test a follower"
+description: "Bind a real sync server to a real sync client over an in-memory socket, then drop, stall, and disconnect frames to prove your follower survives a faulty transport."
+sidebar:
+  order: 6
+---
+
+This guide tests the code you wrote against
+[Follow a run](./follow-a-run.md), using the production server and the
+production client. Only the transport is a double: an in-memory socket pair
+stands in for a WebSocket, and it can drop, stall, and break frames on demand.
+
+Two subpaths ship for this:
+
+| Import                         | What it gives you                                                                                                        |
+| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------ |
+| `@smthrs/sync/test/TestSocket` | `makePair`, the `Pair` it returns, its `TestFaults` controls, and the `FrameFilter` type. Runs anywhere.                 |
+| `@smthrs/sync/test/TestSync`   | `layerTest`, `layerTrustAllAsOwner`, `layerNoop`, and `connect`. Binds the Node SQLite test journal, so it is Node only. |
+
+## Bind a server to a client
+
+`TestSync.layerTest` provides the in-memory SQLite journal, an empty mutable
+run catalog, a middleware that trusts the connection as the workspace owner,
+and the production `SyncServer`. `TestSocket.makePair` allocates the socket,
+and `TestSync.connect` starts an RPC server over its server end and returns a
+`SyncClient.Service` over its client end:
+
+```ts
+import { Journal, JournalEvent } from "@smthrs/journal"
+import * as TestSocket from "@smthrs/sync/test/TestSocket"
+import * as TestSync from "@smthrs/sync/test/TestSync"
+import { Effect, Fiber, Stream } from "effect"
+
+const runId = "build-42" as JournalEvent.RunId
+const sourceId = "test" as JournalEvent.SourceId
+
+const collectTwo = Effect.gen(function*() {
+  const journal = yield* Journal.Journal
+  const pair = yield* TestSocket.makePair()
+  const client = yield* TestSync.connect(pair)
+
+  yield* journal.emitDurableUnfenced({ runId, sourceId, eventType: "first", payload: { value: 0 } })
+  yield* journal.flush
+
+  return yield* Stream.runCollect(
+    client.subscribe({ scope: { _tag: "Run", runId }, cursors: [] }).pipe(Stream.take(1))
+  )
+}).pipe(Effect.provide(TestSync.layerTest), Effect.scoped)
+```
+
+Both `makePair` and `connect` are scoped, so `Effect.scoped` closes the socket
+and the RPC server together. Everything below builds on this shape.
+
+## Use real elapsed time
+
+Run these cases with `it.live`, not `it.effect`. A follower's reconnect
+backoff, the server's tail interval, and the socket's own delivery loop all
+advance on the wall clock, and `it.effect` freezes the `TestClock` so a live
+follow never wakes:
+
+```ts
+import { describe, expect, it } from "@effect/vitest"
+
+describe("my follower", () => {
+  it.live("receives the durable history", () =>
+    Effect.gen(function*() {
+      const entries = yield* collectTwo
+      expect(Array.from(entries).map((entry) => entry.eventType)).toEqual(["first"])
+    }))
+})
+```
+
+Fork a subscription with `Effect.forkChild({ startImmediately: true })` when
+the test commits an entry after subscribing. Without it the entry can land
+before the stream attaches, and the follower reads it out of the durable
+history instead of following it, which proves less than the test intends.
+
+## Drop the frames that carry one range
+
+`faults.dropRange(runId, from, to)` drops every `Entries` frame whose covered
+interval intersects `from..to`. Install it before you connect, then assert
+that the follower never delivers entries past the hole:
+
+```ts
+const withHole = Effect.gen(function*() {
+  const pair = yield* TestSocket.makePair()
+  pair.faults.dropRange(runId, 1, 1)
+  return yield* TestSync.connect(pair)
+})
+```
+
+The follower stalls at sequence 0 forever. That is the correct outcome:
+sequence 1 is not a legitimate journal hole, it is a frame the transport ate,
+and skipping it would lose an entry. The package's own transport-fault suite
+asserts exactly this, by requiring the subscription to time out rather than
+complete.
+
+`dropRange` recognizes the JSON form of an `Entries` frame without importing
+the protocol, so it works only when the connection uses JSON serialization,
+which `TestSync.connect` does.
+
+## Stall and resume delivery
+
+`faults.stall()` holds every frame in the socket, and `faults.resume()`
+releases them. Nothing is lost, so this reproduces a slow link rather than a
+broken one:
+
+```ts
+const stallThenResume = Effect.gen(function*() {
+  const journal = yield* Journal.Journal
+  const pair = yield* TestSocket.makePair()
+  const client = yield* TestSync.connect(pair)
+
+  yield* journal.emitDurableUnfenced({ runId, sourceId, eventType: "before-stall", payload: { value: 0 } })
+  yield* journal.flush
+  pair.faults.stall()
+
+  const fiber = yield* Stream.runCollect(
+    client.subscribe({ scope: { _tag: "Run", runId }, cursors: [] }).pipe(Stream.take(1))
+  ).pipe(Effect.forkChild({ startImmediately: true }))
+
+  const stalled = yield* Fiber.join(fiber).pipe(Effect.timeoutOption("200 millis"))
+  pair.faults.resume()
+  return { stalled, entries: yield* Fiber.join(fiber) }
+})
+```
+
+`stalled` is `None`, and `entries` carries the frame the stall held. Assert
+both: the first proves the stall took effect, and the second proves the resume
+delivered rather than dropped.
+
+## Break the connection
+
+`faults.disconnect()` fails both queues with a socket error, which is what a
+dropped WebSocket looks like to the client. The subscription does not fail:
+`SyncClient` classifies the failure as `transport_failed` and retries under
+exponential backoff capped at five seconds, resuming from the subscription's
+progress. Assert that the stream stays open, not that it errors:
+
+```ts
+const survivesDisconnect = Effect.gen(function*() {
+  const journal = yield* Journal.Journal
+  const pair = yield* TestSocket.makePair()
+  const client = yield* TestSync.connect(pair)
+
+  yield* journal.emitDurableUnfenced({ runId, sourceId, eventType: "first", payload: { value: 0 } })
+  yield* journal.flush
+
+  const fiber = yield* Stream.runCollect(
+    client.subscribe({ scope: { _tag: "Run", runId }, cursors: [] }).pipe(Stream.take(2))
+  ).pipe(Effect.forkChild({ startImmediately: true }))
+
+  yield* pair.faults.disconnect()
+  const settled = yield* Fiber.await(fiber).pipe(Effect.timeoutOption("500 millis"))
+  yield* Fiber.interrupt(fiber)
+  return settled._tag
+})
+```
+
+`survivesDisconnect` returns `"None"`: the fiber was still running when the
+timeout elapsed. A test that expects a failure here is asserting the opposite
+of the contract.
+
+## Corrupt a frame
+
+`faults.installFilter` takes a `FrameFilter`, which is called with each frame's
+bytes. Return `false` to drop the frame, or return replacement bytes to rewrite
+it. Use it to feed your follower a frame the server would never send, and prove
+the client refuses it:
+
+```ts
+import type * as TestSocket from "@smthrs/sync/test/TestSocket"
+
+const misattribute: TestSocket.FrameFilter = (bytes) => {
+  const text = new TextDecoder().decode(bytes)
+  if (!text.includes("\"Entries\"")) return true
+  return new TextEncoder().encode(text.replace("\"runId\":\"build-42\"", "\"runId\":\"other-run\""))
+}
+
+pair.faults.installFilter(misattribute)
+```
+
+The rewrite changes exactly one of the two run ids an `Entries` frame carries,
+its own or its first entry's, so the frame ends up claiming one run and
+carrying another. The subscription fails with `protocol_violation` before any
+cursor moves. The client admits server frames rather than trusting them, and
+this is how you prove your own error handling sees that.
+
+Filters run in installation order on every frame in both directions. A filter
+that returns bytes hands them to the next filter.
+
+## Choose the authorization the test needs
+
+`layerTest` uses `TestSync.layerTrustAllAsOwner`, which trusts every connection
+as the workspace owner. That is what a replication-mechanics test wants: it
+exercises paging and streaming without provisioning capabilities.
+
+To test authorization itself, compose the production middleware instead.
+`SyncAuth.layer` verifies a `WorkspaceShare` capability from the
+`flows-sync-workspace` header, so the test provides a real keyring and mints a
+real capability:
+
+```ts
+import * as TestJournal from "@smthrs/journal/test/TestJournal"
+import * as RunCatalog from "@smthrs/sync/RunCatalog"
+import * as SyncAuth from "@smthrs/sync/SyncAuth"
+import * as SyncServer from "@smthrs/sync/SyncServer"
+import * as WorkspaceShare from "@smthrs/sync/WorkspaceShare"
+import { Layer, Redacted } from "effect"
+
+const keyring: WorkspaceShare.Keyring = {
+  activeKid: "k1",
+  keys: [{ kid: "k1", secret: Redacted.make("test-secret-fixture-only-not-for-prod") }]
+}
+
+const servingStack = Layer.mergeAll(SyncServer.layerHandlers, SyncAuth.layer).pipe(
+  Layer.provideMerge(Layer.mergeAll(SyncServer.layer, WorkspaceShare.layerHmac(keyring))),
+  Layer.provideMerge(Layer.mergeAll(TestJournal.layer(), RunCatalog.layerStatic([runId])))
+)
+```
+
+A connection that presents no header reads no non-branch run, so the
+unauthenticated case is the one to assert first. See
+[Authorize a connection](./authorize-a-connection.md) for minting and
+presenting the capability.
+
+For an in-process caller with no wire at all, provide the principal directly
+with `SyncPrincipal.layerWorkspace("my-suite")`. That path presents no
+credential and therefore has no expiry.
+
+## Register runs a workspace subscription can see
+
+`layerTest`'s run catalog starts empty, and a workspace subscription covers
+only what the catalog lists. Build your own catalog when the test needs runs in
+it. `RunCatalog.makeMemory` returns both the service and a `register` function:
+
+```ts
+const catalog = Effect.gen(function*() {
+  const memory = yield* RunCatalog.makeMemory()
+  yield* memory.register(runId)
+  return memory.catalog
+})
+```
+
+Provide that catalog as `RunCatalog.RunCatalog` in place of the one `layerTest`
+supplies. See
+[List a workspace's runs](./list-workspace-runs.md) for the production forms.
+
+## Stub the ports entirely
+
+A consumer that only needs the sync services to exist, and never exercises
+them, takes `TestSync.layerNoop`. It provides a closed client, a closed server,
+an empty catalog, and a pass-through middleware. Every subscription it hands
+out fails with `closed`, so it is a compile-time convenience and not a fixture
+to assert against.
+
+## What these fixtures do not cover
+
+`TestSync` binds the read path only. It mounts `SyncRpcs`, never `BranchRpcs`,
+so a branch test drives `BranchServer` and the branch services directly. See
+[Branch collaboration](../concepts/branches.md) for why the branch surface
+ships unserved.
+
+The socket pair's queues hold 16 frames each. A test that writes more than
+that without a reader blocks, which is a bounded transport behaving correctly
+rather than a fixture defect.
+
+## Next steps
+
+- [Troubleshooting](../troubleshooting.md): every failure code these faults
+  produce, and what each one means.
+- [Replay then follow](../concepts/replay-then-follow.md): why the client
+  retries a transport failure and propagates everything else.

package/docs/installation.md

@@ -0,0 +1,92 @@
+---
+title: "Installation"
+description: "Install @smthrs/sync, choose an import form, and add the journal, run catalog, and transport that a serving composition needs."
+sidebar:
+  order: 1
+---
+
+## Install the package
+
+Not on npm yet; see [Installation](https://github.com/smithersai/smithers/blob/main/packages/smithers/flows/flow/docs/installation.md#use-the-libraries).
+
+## Requirements
+
+- Node.js 26.4.0 or later, or a browser with Web Crypto. The signing paths
+  call `crypto.subtle` directly so one module serves both.
+- `effect` exactly at the version the package declares. It is a peer dependency
+  so the follower and Smithers use one runtime.
+- [`@smthrs/journal`](/api/journal), also a direct dependency. The entry
+  envelope every frame carries is the journal's `JournalEvent.Entry`.
+
+## Import forms
+
+The root is browser safe and re-exports every namespace:
+
+```ts
+import { RunCatalog, SyncClient, SyncServer } from "@smthrs/sync"
+```
+
+Every module is also importable on its own path, which is what a bundler
+targeting a browser wants because it leaves the server modules out:
+
+```ts
+import * as SyncClient from "@smthrs/sync/SyncClient"
+import * as SyncProtocol from "@smthrs/sync/SyncProtocol"
+```
+
+The two test doubles live under `test/` subpaths and are documented in
+[Test a follower](./guides/test-a-follower.md):
+
+```ts
+import * as TestSocket from "@smthrs/sync/test/TestSocket"
+import * as TestSync from "@smthrs/sync/test/TestSync"
+```
+
+`@smthrs/sync/test/TestSync` binds the Node SQLite test journal, so it is a
+Node-only import. `@smthrs/sync/test/TestSocket` is an in-memory socket pair
+and runs anywhere. Install the optional SQLite driver when using `TestSync`:
+
+```bash
+pnpm add @effect/sql-sqlite-node@4.0.0-rc.115
+```
+
+`@smthrs/sync/internal/*` is not exported. A path under it fails to resolve.
+
+## What a follower adds
+
+A follower needs a transport. `SyncClient.layer` derives its RPC client from
+`RpcClient.Protocol`, so the composition supplies the protocol and the
+serialization:
+
+```bash
+pnpm add @effect/platform-node@4.0.0-rc.115 @effect/platform-node-shared@4.0.0-rc.115
+```
+
+A browser follower needs no extra package: the WebSocket protocol and the JSON
+serialization both come from `effect/unstable/rpc`.
+
+## What a server adds
+
+Serving the read path requires a real journal and a run catalog:
+
+- [`@smthrs/journal`](/api/journal) supplies `Journal`. Its production layer,
+  `SqlJournal`, sits above [`@smthrs/database`](/api/database), so a serving
+  host installs a database driver and the journal migrations too.
+- `RunCatalog` is supplied by the host. `RunCatalog.layerStatic` covers a fixed
+  set. [`@smthrs/engine-store`](/api/engine-store) supplies `RunCatalogRead`,
+  the durable read a polling catalog uses so a follower learns of runs another
+  engine created. See [List a workspace's runs](./guides/list-workspace-runs.md).
+- A transport mounts the RPC group. [`@smthrs/gateway`](/api/gateway) already
+  does it, on `POST /sync` and `/sync/ws`.
+
+Authorization is a separate decision with its own dependency: `SyncAuth.layer`
+requires a `WorkspaceShare` authority, and `WorkspaceShare.layerConfig` reads
+its secret from the environment. See
+[Authorize a connection](./guides/authorize-a-connection.md).
+
+## Next steps
+
+- [Quickstart](./quickstart.md) runs a follower end to end with no transport
+  and no database of your own.
+- [Serve the read path](./guides/serve-the-read-path.md) composes the server
+  side against a real journal.

package/docs/protocol.md

@@ -0,0 +1,183 @@
+---
+title: "Wire protocol"
+description: "The normative message shapes of the sync read path: scopes, cursors, read requests and responses, subscription frames, and the limits every side enforces."
+---
+
+## Scopes and cursors
+
+A client reads either one run or every run in a workspace:
+
+```ts
+import { SyncProtocol } from "@smthrs/sync"
+import { Schema } from "effect"
+
+const scope = Schema.decodeUnknownSync(SyncProtocol.RunScope)({
+  _tag: "Run",
+  runId: "build-42"
+})
+const cursor = Schema.decodeUnknownSync(SyncProtocol.RunCursor)({
+  runId: "build-42",
+  afterSeq: 17
+})
+```
+
+A run cursor stores the last sequence the client was delivered. A workspace
+cursor is an array of per-run cursors with at most one entry per run: a request
+that names one run twice is refused with `invalid_request`, because the read
+position and the echoed response state would otherwise disagree. Journal
+sequences may have holes, so a cursor means "read entries after this number",
+not "expect the next number to be exactly one greater".
+
+A read and a subscribe request carry the same envelope, `SyncProtocol.RequestEnvelope`:
+one scope and one cursor set. Server read, server subscribe and client
+subscribe admit a caller-built request against that single schema.
+
+`generation` is asymmetric on purpose. A request may omit it, which is how a
+cursor persisted before rewind support still decodes; a response may not. The
+server-side shapes state that requirement in the type system:
+`SyncProtocol.ServerCursor`, `ServerReadResponse`, `ServerEntriesFrame` and
+`ServerFrame` all require `generation`, and `SyncServer.Service` answers with
+them, so a server that omits one does not compile. The wire schemas stay
+lenient so a non-conforming server is refused with `protocol_violation` naming
+the missing generation rather than a bare decode failure.
+
+## Protocol version
+
+`SyncProtocol.protocolVersion` is the one accepted wire revision, and the only
+place the number is written. Read, subscribe and snapshot requests all decode a
+version this revision does not speak, so the refusal is a typed
+`protocol_violation` with cause `protocol_version_mismatch` on every path,
+in process and over the wire, request and response alike. There is no second
+code for a version mismatch.
+
+## Read
+
+`Sync.Read` accepts `protocolVersion: 1`, a scope, a cursor set, and a limit, and returns journal
+entries plus the cursor set the page served through. `RunCatalog` supplies the
+run list for workspace reads; `SyncServer.layer` combines it with `Journal`.
+
+`limit` is bounded at `SyncProtocol.maxReadLimit`. A page is also bounded by
+the encoded-entry ceiling, `maxFrameBytes`, which defaults to 2 MiB. The
+ceiling is a budget on the page, not a verdict on the read: the server serves
+entries until the next one would cross it, then returns `done: false` so the
+client asks for the rest. Only a single entry whose own encoded size exceeds
+the ceiling fails, with `frame_too_large`, because no page can carry it.
+
+Persist the returned cursors only after applying the returned batch.
+
+There is no application acknowledgement RPC. Response cursors are delivery
+bookmarks. `SyncClient.progress` reports `Progress`, two cursor sets:
+`delivered` holds delivery bookmarks and `applied` holds application
+positions. Only successful `apply` and snapshot-restore callbacks advance the
+applied set. An applying subscription never inherits delivery-only progress.
+
+Both server and client validate complete admitted batches before serving or
+applying them. A malformed envelope or non-JSON payload fails `decode_failed`;
+foreign runs, foreign branch identities, and non-monotonic sequences fail
+`protocol_violation`. Causes retain a bounded classification without publishing
+private record contents. Unknown event namespaces remain open JSON values.
+Known branch commands must decode; existing defaults for absent `args` and
+`target` remain unchanged. No stored row is rewritten or silently skipped.
+
+## Subscribe
+
+`Sync.Subscribe` requires the same `protocolVersion: 1` and streams `Entries` frames, each carrying one run's entries and
+the interval the server covered (`runId`, `fromSeq`, `toSeq`), and a terminal
+`Closed` frame when the server ends the subscription. `fromSeq` and `toSeq`
+describe the covered interval, not the sequences actually carried: dropped
+admissions leave legitimate holes. No frame carries a cursor; a client tracks
+its own, which is what `SyncClient` does.
+
+The `Heartbeat` variant is RESERVED. No server emits one at 1.0.0-rc.0, so a
+client must not wait for one; it stays in the union so adding it later is not a
+wire break. Keepalive is a transport concern, because a heartbeat here would
+spend the subscription's credit.
+
+The request includes a credit count between 1 and
+`SyncProtocol.maxSubscribeCredit`. Credit is a hard limit on frames emitted by
+that subscription, not a sliding acknowledgement window, and there is no Ack
+RPC. A client that needs more resubscribes from its current cursor positions,
+which is what `SyncClient` does once per window. These are applied positions
+when an `apply` callback is supplied and delivery bookmarks otherwise.
+
+A subscription also ends when the credential that opened it expires, with
+`SyncError` code `unauthorized`. Authorization happens once, at open, so
+without that deadline a signed expiry could not revoke a stream that simply
+stayed connected.
+
+`SyncClient.subscribe` wraps the RPC stream. It detects invalid cursor movement
+as `SyncGapError`, refuses a frame or a catch-up page that contradicts itself
+as a `protocol_violation`, reconnects transport failures under exponential
+backoff capped at five seconds, and resumes from the subscription's matching
+progress map. Authentication is `SyncAuth`, and both are part of the package
+rather than left to the application.
+
+## Compacted runs
+
+A cursor below a run's compaction floor names entries the journal has deleted.
+The read or the subscription fails with `SyncError` code `compacted`, and the
+error carries a `Resync` of `{ runId, checkpointSeq, generation }`: the floor to
+resume from and the run history it belongs to. `SyncClient.subscribe` fails
+closed unless `onResync` restores the missing prefix and returns
+`{ runId, afterSeq }`. The cursor advances to that actual restored sequence, not
+necessarily the older reported floor, before replaying the suffix. A receipt
+without a generation takes the refusal's, so a fresh follower of a rewound run
+resumes the right history. Foreign-run and behind-floor receipts are refused.
+
+:::warning
+The resync moves a cursor, not state. The entries below `checkpointSeq` are
+never delivered. Fetch an explicitly public projection with `Sync.Snapshot`
+or use an authorized local source. `SubscribeOptions.onResync` runs before the cursor moves and must
+succeed with a valid restored cursor, so a consumer that cannot restore the prefix fails instead of silently
+skipping it. A Node follower reads that prefix out of band with
+`journal.latestCheckpoint(runId)`, applies `checkpoint.state`, and continues
+from the sync stream. See [Checkpoints and compaction](/pkg/journal/concepts/compaction).
+:::
+
+## Public snapshots
+
+`Sync.Snapshot` accepts `SnapshotRequest`: `protocolVersion: 1`, `runId`,
+`lineageId`, `projection`, `projectionVersion`, `atLeastSeq`, and an optional
+branch `capability`. The response repeats that identity with `seq` and JSON
+`state`, representing the complete public projection through `seq` inclusive.
+Lineage IDs are nonempty and at most 512 characters; projection names are
+nonempty and at most 128. Projection versions are positive safe integers.
+
+Run/branch read authorization is the same as for history. An explicit host
+`SyncServer.SnapshotSource` must select a public projection safe for every reader
+of that run; raw journal checkpoints are never served automatically. Missing
+providers or unavailable projections fail closed. Credential expiry is checked
+before and after provider work, and provider failures do not expose private text.
+
+Both ends enforce the requested identity, its protocol version, and the
+minimum sequence; each mismatch is a `protocol_violation`. Both also bound
+the entire encoded UTF-8 snapshot response by `maxFrameBytes` (default 2 MiB,
+excluding the outer RPC envelope), and reject non-JSON provider state. The
+response is detached from mutable provider objects.
+
+`SyncClient.snapshot` fetches only: it does not apply state or move cursors.
+After transactional application, `onResync` returns the actual restored cursor.
+If compaction advances before the next suffix read, recovery runs again at the
+newer floor. Snapshot bytes are inline, so no temporary download reference can
+expire during application. Providers must retain a usable latest snapshot
+covering every compacted prefix and version they continue to serve.
+
+## Directionality
+
+`SyncRpcs` is read only: `Sync.Read`, `Sync.Subscribe`, and `Sync.Snapshot`. Client-to-server
+submission is not planned work, it is `BranchRpcs`: `Branch.Submit` admits one
+command onto a branch's journal run under a client-minted idempotency key, and
+`Branch.Announce`, `Branch.Leave`, `Branch.Roster`, and `Branch.WatchRoster`
+carry ephemeral presence. `BranchRpcs` has no production mount at
+1.0.0-rc.0; the gateway serves `SyncRpcs` alone.
+
+Bidirectional reconciliation of the read path, acknowledgement windows, and
+resumable transport sessions remain unplanned.
+
+Command admission separately permits up to 1 MiB of UTF-8 JSON submission
+bytes. Before writing, it also checks the complete journal envelope against
+`BranchCommands.Options.maxFrameBytes` (2 MiB by default), reserving the largest
+generated sequence/timestamp representation. Custom admission, server, and
+client budgets must agree. `maxFrameBytes` sums encoded entries; an RPC transport
+must additionally allow its frame/envelope overhead. Neither journal cursors nor
+entries change when admission refuses a command.