reflectdb 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +1260 -0
- package/dist/cjs/client/index.cjs +1252 -0
- package/dist/cjs/client/index.d.cts +629 -0
- package/dist/cjs/client/storage/indexeddb.cjs +253 -0
- package/dist/cjs/client/storage/indexeddb.d.cts +85 -0
- package/dist/cjs/core/index.cjs +202 -0
- package/dist/cjs/core/index.d.cts +473 -0
- package/dist/cjs/react/index.cjs +1512 -0
- package/dist/cjs/react/index.d.cts +748 -0
- package/dist/cjs/server/drizzle.cjs +413 -0
- package/dist/cjs/server/drizzle.d.cts +233 -0
- package/dist/cjs/server/index.cjs +4486 -0
- package/dist/cjs/server/index.d.cts +1988 -0
- package/dist/cjs/svelte/index.cjs +1414 -0
- package/dist/cjs/svelte/index.d.cts +645 -0
- package/dist/cjs/transport/bun-ws.cjs +244 -0
- package/dist/cjs/transport/bun-ws.d.cts +215 -0
- package/dist/cjs/transport/polling.cjs +285 -0
- package/dist/cjs/transport/polling.d.cts +210 -0
- package/dist/cjs/transport/sse.cjs +312 -0
- package/dist/cjs/transport/sse.d.cts +205 -0
- package/dist/cjs/transport/ws.cjs +330 -0
- package/dist/cjs/transport/ws.d.cts +236 -0
- package/dist/cjs/vanilla/index.cjs +1430 -0
- package/dist/cjs/vanilla/index.d.cts +657 -0
- package/dist/client/index.d.ts +629 -0
- package/dist/client/index.js +106 -0
- package/dist/client/storage/indexeddb.d.ts +85 -0
- package/dist/client/storage/indexeddb.js +213 -0
- package/dist/core/index.d.ts +473 -0
- package/dist/core/index.js +56 -0
- package/dist/react/index.d.ts +748 -0
- package/dist/react/index.js +366 -0
- package/dist/server/drizzle.d.ts +233 -0
- package/dist/server/drizzle.js +9 -0
- package/dist/server/index.d.ts +1988 -0
- package/dist/server/index.js +3959 -0
- package/dist/shared/esm-3tkwvysa.js +54 -0
- package/dist/shared/esm-b7xs9cde.js +4 -0
- package/dist/shared/esm-rw7jjtrv.js +58 -0
- package/dist/shared/esm-wkwx6bd9.js +25 -0
- package/dist/shared/esm-ytrd3hbq.js +1007 -0
- package/dist/shared/esm-z1xse19c.js +369 -0
- package/dist/svelte/index.d.ts +645 -0
- package/dist/svelte/index.js +262 -0
- package/dist/transport/bun-ws.d.ts +215 -0
- package/dist/transport/bun-ws.js +140 -0
- package/dist/transport/polling.d.ts +210 -0
- package/dist/transport/polling.js +181 -0
- package/dist/transport/sse.d.ts +205 -0
- package/dist/transport/sse.js +208 -0
- package/dist/transport/ws.d.ts +236 -0
- package/dist/transport/ws.js +226 -0
- package/dist/vanilla/index.d.ts +657 -0
- package/dist/vanilla/index.js +278 -0
- package/package.json +253 -0
package/README.md
ADDED
|
@@ -0,0 +1,1260 @@
|
|
|
1
|
+
# reflectdb
|
|
2
|
+
|
|
3
|
+
[](https://www.npmjs.com/package/reflectdb)
|
|
4
|
+
[](https://www.npmjs.com/package/reflectdb)
|
|
5
|
+
[](https://github.com/TimMikeladze/reflectdb/actions/workflows/ci.yml)
|
|
6
|
+
[](https://www.typescriptlang.org)
|
|
7
|
+
[](./LICENSE)
|
|
8
|
+
|
|
9
|
+
A real-time sync engine for TypeScript. Keeps a server-side database in sync with any number of browser clients — offline-first, with optimistic local writes, automatic conflict resolution, and end-to-end type inference.
|
|
10
|
+
|
|
11
|
+
You bring your own types and your own database. reflectdb handles the protocol, the op log, conflicts, reconnection, and subscriptions.
|
|
12
|
+
|
|
13
|
+
```
|
|
14
|
+
┌──────────────┐ writes ┌──────────────┐ writes ┌──────────────┐
|
|
15
|
+
│ Browser A │ ──────────▶ │ Server │ ◀────────── │ Browser B │
|
|
16
|
+
│ (optimistic) │ deltas │ (authoritive)│ deltas │ (optimistic) │
|
|
17
|
+
│ │ ◀────────── │ │ ──────────▶ │ │
|
|
18
|
+
└──────────────┘ └──────────────┘ └──────────────┘
|
|
19
|
+
▲ │
|
|
20
|
+
│ offline ▼
|
|
21
|
+
└────── IndexedDB ───── op log (in-memory / SQLite / Postgres)
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
## Table of Contents
|
|
25
|
+
|
|
26
|
+
- [Why reflectdb](#why-reflectdb)
|
|
27
|
+
- [Features](#features)
|
|
28
|
+
- [Use Cases](#use-cases)
|
|
29
|
+
- [Installation](#installation)
|
|
30
|
+
- [Quick Start](#quick-start)
|
|
31
|
+
- [Recipes](#recipes)
|
|
32
|
+
- [WebSocket sync with SQLite + Drizzle](#websocket-sync-with-sqlite--drizzle)
|
|
33
|
+
- [Typed params for multi-tenant queries](#typed-params-for-multi-tenant-queries)
|
|
34
|
+
- [Authentication and room-based access control](#authentication-and-room-based-access-control)
|
|
35
|
+
- [Per-column merge for collaborative editing](#per-column-merge-for-collaborative-editing)
|
|
36
|
+
- [Custom conflict resolvers](#custom-conflict-resolvers)
|
|
37
|
+
- [Validating client payloads](#validating-client-payloads)
|
|
38
|
+
- [Ephemeral messages (cursors, presence, typing)](#ephemeral-messages-cursors-presence-typing)
|
|
39
|
+
- [Per-user query results](#per-user-query-results)
|
|
40
|
+
- [Server-driven game loops](#server-driven-game-loops)
|
|
41
|
+
- [Windowed sync and pagination](#windowed-sync-and-pagination)
|
|
42
|
+
- [Auto-generated REST API](#auto-generated-rest-api)
|
|
43
|
+
- [High availability with Postgres](#high-availability-with-postgres)
|
|
44
|
+
- [Whiteboard + Pictionary example](#whiteboard--pictionary-example)
|
|
45
|
+
- [Architecture](#architecture)
|
|
46
|
+
- [Core Concepts](#core-concepts)
|
|
47
|
+
- [Hybrid Logical Clocks](#hybrid-logical-clocks)
|
|
48
|
+
- [Conflict Resolution](#conflict-resolution)
|
|
49
|
+
- [The Sync Protocol](#the-sync-protocol)
|
|
50
|
+
- [Two stores, one sync](#two-stores-one-sync)
|
|
51
|
+
- [The Op Log and Resume](#the-op-log-and-resume)
|
|
52
|
+
- [API Reference](#api-reference)
|
|
53
|
+
- [`reflectdb/core`](#reflectdbcore)
|
|
54
|
+
- [`reflectdb/server`](#reflectdbserver)
|
|
55
|
+
- [`reflectdb/client`](#reflectdbclient)
|
|
56
|
+
- [`reflectdb/react`](#reflectdbreact)
|
|
57
|
+
- [`reflectdb/svelte`](#reflectdbsvelte)
|
|
58
|
+
- [`reflectdb/vanilla`](#reflectdbvanilla)
|
|
59
|
+
- [`reflectdb/transport/*`](#reflectdbtransport)
|
|
60
|
+
- [Configuration Reference](#configuration-reference)
|
|
61
|
+
- [Query definition](#query-definition)
|
|
62
|
+
- [Server configuration](#server-configuration)
|
|
63
|
+
- [`implement()` options](#implement-options)
|
|
64
|
+
- [Rate limiting](#rate-limiting)
|
|
65
|
+
- [Compaction](#compaction)
|
|
66
|
+
- [Client configuration](#client-configuration)
|
|
67
|
+
- [Storage adapters](#storage-adapters)
|
|
68
|
+
- [Transport configuration](#transport-configuration)
|
|
69
|
+
- [Development](#development)
|
|
70
|
+
- [License](#license)
|
|
71
|
+
|
|
72
|
+
## Why reflectdb
|
|
73
|
+
|
|
74
|
+
Most real-time sync libraries force you to choose: CRDTs (powerful but opaque), or simple pub/sub (fast but brittle). reflectdb sits in the middle — **per-row operations** with **hybrid logical clocks** for causal ordering, validated through a server-side pipeline so your database stays authoritative.
|
|
75
|
+
|
|
76
|
+
You define your schema once, and the same types flow to both sides:
|
|
77
|
+
|
|
78
|
+
```ts
|
|
79
|
+
const { rows, insert } = useSync("todos");
|
|
80
|
+
// ^? Todo[] ^? (id, { title, done, createdAt? }) => void
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
No code generation. No glue layer. No second source of truth.
|
|
84
|
+
|
|
85
|
+
**Bring your own stack.** reflectdb is agnostic about:
|
|
86
|
+
|
|
87
|
+
- **Your database** — any TypeScript ORM, raw SQL driver, Map, or REST API works. The `query`/`mutate` callbacks hand you `db` untouched.
|
|
88
|
+
- **Your row types** — plain TypeScript types, Drizzle `$inferSelect`, Kysely, Prisma, anything. Declare them with `t<MyRow>()`.
|
|
89
|
+
- **Your HTTP server** — Bun, Node, Deno, Cloudflare Workers, anything fetch-compatible. Transports expose handler functions you wire to routes.
|
|
90
|
+
|
|
91
|
+
Optional bits (use what you want):
|
|
92
|
+
|
|
93
|
+
- **Drizzle ORM** — if you point `table` at a Drizzle table, row types are auto-inferred.
|
|
94
|
+
- **Server op log storage** — SQLite (for single-node) or Postgres (for HA). Omit it and the op log is in-memory.
|
|
95
|
+
- **React / Svelte bindings** — use the core client directly if you prefer.
|
|
96
|
+
|
|
97
|
+
## Features
|
|
98
|
+
|
|
99
|
+
- **Real-time sync** over WebSocket, Server-Sent Events, or HTTP long-polling
|
|
100
|
+
- **Offline-first** — optimistic local writes, queued and replayed on reconnect
|
|
101
|
+
- **End-to-end type safety** — schema defines row types, query params, writable fields, and which columns the server owns
|
|
102
|
+
- **Per-row and per-column conflict resolution** — `lww`, `merge`, `server`, or a custom resolver
|
|
103
|
+
- **Causal ordering** via hybrid logical clocks (HLC) — no dependence on synchronized wall clocks
|
|
104
|
+
- **Pluggable storage** — in-memory, SQLite, or Postgres for the server op log; memory or IndexedDB for the browser
|
|
105
|
+
- **Auto-generated REST** — `server.rest()` turns your schema into CRUD endpoints that broadcast deltas
|
|
106
|
+
- **Room-based access control** — scope clients to `org/:orgId` or arbitrary patterns
|
|
107
|
+
- **Rate limiting** — global and per-table, fail-open
|
|
108
|
+
- **Op log compaction** — configurable retention for old accepted ops
|
|
109
|
+
- **High availability** — shared Postgres + optional cross-instance polling
|
|
110
|
+
- **Framework bindings** — React hooks, Svelte stores, and a vanilla-JS helper; the core client works anywhere
|
|
111
|
+
- **Ephemeral channels** — presence, cursors, typing indicators that never touch the op log
|
|
112
|
+
- **Windowed sync** — paginate large tables with `loadMore` + `useTotalCount`
|
|
113
|
+
|
|
114
|
+
## Use Cases
|
|
115
|
+
|
|
116
|
+
- Collaborative editing (docs, whiteboards, spreadsheets)
|
|
117
|
+
- Multi-device note apps, todo apps, inbox-like UIs
|
|
118
|
+
- Live dashboards where multiple clients view and edit the same state
|
|
119
|
+
- Local-first apps that need to work offline and merge on reconnect
|
|
120
|
+
- Admin tools that should "just update" when someone else changes a row
|
|
121
|
+
- Field-service or retail apps on spotty networks
|
|
122
|
+
- Games or canvases with presence indicators and live cursors
|
|
123
|
+
|
|
124
|
+
## Installation
|
|
125
|
+
|
|
126
|
+
```bash
|
|
127
|
+
bun add reflectdb
|
|
128
|
+
# or
|
|
129
|
+
npm install reflectdb
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
Ships both ESM and CommonJS, so `import` and `require` both work:
|
|
133
|
+
|
|
134
|
+
```ts
|
|
135
|
+
import { defineSyncQueries, t } from "reflectdb"; // or "reflectdb/core"
|
|
136
|
+
```
|
|
137
|
+
```js
|
|
138
|
+
const { defineSyncQueries, t } = require("reflectdb");
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
Everything else is a subpath — `reflectdb/server`, `reflectdb/client`, `reflectdb/react`, and so on. The bare `reflectdb` specifier is an alias for `reflectdb/core`, the surface both sides share.
|
|
142
|
+
|
|
143
|
+
Peer dependencies are all optional:
|
|
144
|
+
|
|
145
|
+
```bash
|
|
146
|
+
bun add react # for reflectdb/react
|
|
147
|
+
bun add drizzle-orm # if you want auto-inferred row types from Drizzle tables
|
|
148
|
+
# Svelte + vanilla have no peer deps
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
## Quick Start
|
|
152
|
+
|
|
153
|
+
A complete sync server in ~30 lines. No ORM, no database — just plain types and an in-memory Map.
|
|
154
|
+
|
|
155
|
+
### 1. Define your schema
|
|
156
|
+
|
|
157
|
+
```ts
|
|
158
|
+
// schema.ts
|
|
159
|
+
import { defineSyncQueries, t } from "reflectdb/core";
|
|
160
|
+
|
|
161
|
+
export type Todo = {
|
|
162
|
+
id: string;
|
|
163
|
+
title: string;
|
|
164
|
+
done: boolean;
|
|
165
|
+
createdAt: Date;
|
|
166
|
+
};
|
|
167
|
+
|
|
168
|
+
export const queries = defineSyncQueries({
|
|
169
|
+
todos: {
|
|
170
|
+
row: t<Todo>(),
|
|
171
|
+
conflict: "lww",
|
|
172
|
+
serverSet: ["createdAt"], // server always sets this, clients cannot
|
|
173
|
+
},
|
|
174
|
+
});
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
### 2. Create the server
|
|
178
|
+
|
|
179
|
+
```ts
|
|
180
|
+
// server.ts
|
|
181
|
+
import { serve } from "bun";
|
|
182
|
+
import { createSyncServer } from "reflectdb/server";
|
|
183
|
+
import { createWsServerTransport } from "reflectdb/transport/ws";
|
|
184
|
+
import { queries, type Todo } from "./schema";
|
|
185
|
+
|
|
186
|
+
const todos = new Map<string, Todo>();
|
|
187
|
+
const transport = createWsServerTransport();
|
|
188
|
+
|
|
189
|
+
const server = createSyncServer({ queries, transport, serverId: "s1" });
|
|
190
|
+
|
|
191
|
+
server.auth(async (req) => {
|
|
192
|
+
// validate req.headers.get("authorization")
|
|
193
|
+
return { userId: "user-1" };
|
|
194
|
+
});
|
|
195
|
+
|
|
196
|
+
server.implement("todos", {
|
|
197
|
+
query: () => [...todos.values()],
|
|
198
|
+
mutate: async (op) => {
|
|
199
|
+
if (op.type === "delete") todos.delete(op.rowId);
|
|
200
|
+
else todos.set(op.rowId, { id: op.rowId, ...(op.payload as Partial<Todo>) } as Todo);
|
|
201
|
+
},
|
|
202
|
+
serverSet: { createdAt: () => new Date() },
|
|
203
|
+
});
|
|
204
|
+
|
|
205
|
+
// Wire WebSocket handlers to your HTTP server
|
|
206
|
+
serve({
|
|
207
|
+
port: 3001,
|
|
208
|
+
fetch(req, srv) {
|
|
209
|
+
const url = new URL(req.url);
|
|
210
|
+
if (url.pathname === "/sync") {
|
|
211
|
+
const clientId = crypto.randomUUID();
|
|
212
|
+
if (srv.upgrade(req, { data: { clientId } })) return;
|
|
213
|
+
}
|
|
214
|
+
return new Response("ok");
|
|
215
|
+
},
|
|
216
|
+
websocket: {
|
|
217
|
+
open(ws) { transport.handleOpen(ws.data.clientId, ws); },
|
|
218
|
+
message(ws, data) { transport.handleMessage(ws.data.clientId, String(data)); },
|
|
219
|
+
close(ws) { transport.handleClose(ws.data.clientId); },
|
|
220
|
+
pong(ws) { transport.handlePong(ws.data.clientId); },
|
|
221
|
+
},
|
|
222
|
+
});
|
|
223
|
+
```
|
|
224
|
+
|
|
225
|
+
### 3. Connect from the browser
|
|
226
|
+
|
|
227
|
+
```tsx
|
|
228
|
+
// app.tsx
|
|
229
|
+
import { SyncProvider, useSync, useSyncStatus } from "reflectdb/react";
|
|
230
|
+
import { createIndexedDBStorage } from "reflectdb/client/storage/indexeddb";
|
|
231
|
+
|
|
232
|
+
export function App() {
|
|
233
|
+
return (
|
|
234
|
+
<SyncProvider
|
|
235
|
+
url="ws://localhost:3001/sync"
|
|
236
|
+
token="..."
|
|
237
|
+
tables={["todos"]}
|
|
238
|
+
storage={createIndexedDBStorage({ dbName: "myapp" })}
|
|
239
|
+
>
|
|
240
|
+
<TodoList />
|
|
241
|
+
</SyncProvider>
|
|
242
|
+
);
|
|
243
|
+
}
|
|
244
|
+
|
|
245
|
+
function TodoList() {
|
|
246
|
+
const { rows, insert, update, remove } = useSync("todos");
|
|
247
|
+
const status = useSyncStatus();
|
|
248
|
+
|
|
249
|
+
return (
|
|
250
|
+
<div>
|
|
251
|
+
<p>Status: {status}</p>
|
|
252
|
+
{rows.map((t) => (
|
|
253
|
+
<label key={t.id}>
|
|
254
|
+
<input type="checkbox" checked={t.done} onChange={() => update(t.id, { done: !t.done })} />
|
|
255
|
+
{t.title}
|
|
256
|
+
<button onClick={() => remove(t.id)}>x</button>
|
|
257
|
+
</label>
|
|
258
|
+
))}
|
|
259
|
+
<button onClick={() => insert(crypto.randomUUID(), { title: "New", done: false })}>
|
|
260
|
+
Add
|
|
261
|
+
</button>
|
|
262
|
+
</div>
|
|
263
|
+
);
|
|
264
|
+
}
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
Open two tabs — edits in one appear in the other within a round-trip. Close the laptop, edit offline, reopen — pending ops replay automatically.
|
|
268
|
+
|
|
269
|
+
## Recipes
|
|
270
|
+
|
|
271
|
+
The repo ships one end-to-end example — [`examples/whiteboard/`](./examples/whiteboard/) — a collaborative drawing app with two modes (freeform and **Pictionary**). It exercises the patterns below in one place. The snippets here are minimal, copy-paste-friendly references; see the example for how they fit together.
|
|
272
|
+
|
|
273
|
+
### WebSocket sync with SQLite + Drizzle
|
|
274
|
+
|
|
275
|
+
If you use Drizzle, point `table` at it and row types flow automatically. Swap the Map for `bun:sqlite` + Drizzle and add a persistent op log:
|
|
276
|
+
|
|
277
|
+
```ts
|
|
278
|
+
import { Database } from "bun:sqlite";
|
|
279
|
+
import { drizzle } from "drizzle-orm/bun-sqlite";
|
|
280
|
+
import { sqliteTable, text, integer } from "drizzle-orm/sqlite-core";
|
|
281
|
+
import { eq } from "drizzle-orm";
|
|
282
|
+
import { defineSyncQueries } from "reflectdb/core";
|
|
283
|
+
import { createSyncServer, createSqliteStorage } from "reflectdb/server";
|
|
284
|
+
|
|
285
|
+
const todos = sqliteTable("todos", {
|
|
286
|
+
id: text("id").primaryKey(),
|
|
287
|
+
title: text("title").notNull(),
|
|
288
|
+
done: integer("done", { mode: "boolean" }).notNull().default(false),
|
|
289
|
+
createdAt: integer("created_at", { mode: "timestamp" }).notNull(),
|
|
290
|
+
});
|
|
291
|
+
|
|
292
|
+
const queries = defineSyncQueries({
|
|
293
|
+
todos: { table: todos, conflict: "lww", serverSet: ["createdAt"] },
|
|
294
|
+
});
|
|
295
|
+
|
|
296
|
+
const db = drizzle(new Database("app.db"));
|
|
297
|
+
const storage = createSqliteStorage({ path: "sync.db" });
|
|
298
|
+
|
|
299
|
+
const server = createSyncServer({ queries, db, transport, storage, serverId: "s1" });
|
|
300
|
+
|
|
301
|
+
server.implement("todos", {
|
|
302
|
+
query: (_ctx, db) => db.select().from(todos),
|
|
303
|
+
mutate: async (op, _ctx, db) => {
|
|
304
|
+
if (op.type === "delete") {
|
|
305
|
+
await db.delete(todos).where(eq(todos.id, op.rowId));
|
|
306
|
+
} else {
|
|
307
|
+
await db.insert(todos)
|
|
308
|
+
.values({ id: op.rowId, ...op.payload })
|
|
309
|
+
.onConflictDoUpdate({ target: todos.id, set: op.payload });
|
|
310
|
+
}
|
|
311
|
+
},
|
|
312
|
+
serverSet: { createdAt: () => new Date() },
|
|
313
|
+
});
|
|
314
|
+
```
|
|
315
|
+
|
|
316
|
+
### Typed params for multi-tenant queries
|
|
317
|
+
|
|
318
|
+
Declare query params with `t<T>()` so the client must pass them and the server can use them to scope queries:
|
|
319
|
+
|
|
320
|
+
```ts
|
|
321
|
+
import { defineSyncQueries, t } from "reflectdb/core";
|
|
322
|
+
|
|
323
|
+
type Post = { id: string; title: string; orgId: string };
|
|
324
|
+
|
|
325
|
+
const queries = defineSyncQueries({
|
|
326
|
+
posts: {
|
|
327
|
+
row: t<Post>(),
|
|
328
|
+
params: t<{ orgId: string }>(),
|
|
329
|
+
tables: ["posts"], // change-detection hint for delta computation
|
|
330
|
+
pk: "id",
|
|
331
|
+
conflict: "lww",
|
|
332
|
+
readonly: ["orgId"], // clients cannot write this
|
|
333
|
+
},
|
|
334
|
+
});
|
|
335
|
+
|
|
336
|
+
// server
|
|
337
|
+
server.implement("posts", {
|
|
338
|
+
query: (ctx, kyselyDb) =>
|
|
339
|
+
kyselyDb.selectFrom("posts").where("orgId", "=", ctx.params.orgId).selectAll().execute(),
|
|
340
|
+
mutate: async (op, ctx, kyselyDb) => { /* ... */ },
|
|
341
|
+
});
|
|
342
|
+
|
|
343
|
+
// client
|
|
344
|
+
client.sync("posts", { orgId: "org-42" });
|
|
345
|
+
```
|
|
346
|
+
|
|
347
|
+
Works with any ORM or raw driver.
|
|
348
|
+
|
|
349
|
+
### Authentication and room-based access control
|
|
350
|
+
|
|
351
|
+
`auth()` runs on every connection. Return an `AuthContext` — anything with a `userId`. It's passed to every `query`, `mutate`, and `authorize` call.
|
|
352
|
+
|
|
353
|
+
```ts
|
|
354
|
+
server.auth(async (req) => {
|
|
355
|
+
const token = req.headers.get("authorization")?.replace("Bearer ", "");
|
|
356
|
+
const session = await validateToken(token);
|
|
357
|
+
if (!session) throw new Error("unauthorized");
|
|
358
|
+
return { userId: session.userId, orgId: session.orgId };
|
|
359
|
+
});
|
|
360
|
+
```
|
|
361
|
+
|
|
362
|
+
For multi-tenant apps, use `room()` to pin a client to a subset of data:
|
|
363
|
+
|
|
364
|
+
```ts
|
|
365
|
+
server.room("org/:orgId", async ({ params, auth }) => {
|
|
366
|
+
if (!auth.memberships.includes(params.orgId)) {
|
|
367
|
+
throw new Error("not a member of this org");
|
|
368
|
+
}
|
|
369
|
+
return { scope: { orgId: params.orgId } };
|
|
370
|
+
});
|
|
371
|
+
```
|
|
372
|
+
|
|
373
|
+
The whiteboard example wires this up with [better-auth](https://better-auth.com) — see [`examples/whiteboard/auth.ts`](./examples/whiteboard/auth.ts).
|
|
374
|
+
|
|
375
|
+
### Per-column merge for collaborative editing
|
|
376
|
+
|
|
377
|
+
When two users edit different fields of the same row, `lww` would throw one write away. `merge` keeps both:
|
|
378
|
+
|
|
379
|
+
```ts
|
|
380
|
+
const queries = defineSyncQueries({
|
|
381
|
+
docs: { row: t<Doc>(), conflict: "merge" }, // per-column HLCs
|
|
382
|
+
});
|
|
383
|
+
```
|
|
384
|
+
|
|
385
|
+
```
|
|
386
|
+
User A writes { title: "Hello" } at HLC 100
|
|
387
|
+
User B writes { body: "world" } at HLC 200
|
|
388
|
+
→ Result: { title: "Hello", body: "world" } (both accepted)
|
|
389
|
+
```
|
|
390
|
+
|
|
391
|
+
### Custom conflict resolvers
|
|
392
|
+
|
|
393
|
+
For domain logic — counters, highest-bid-wins, append-only lists — supply a resolver:
|
|
394
|
+
|
|
395
|
+
```ts
|
|
396
|
+
const queries = defineSyncQueries({
|
|
397
|
+
auctions: {
|
|
398
|
+
row: t<Auction>(),
|
|
399
|
+
conflict: {
|
|
400
|
+
policy: "custom",
|
|
401
|
+
resolve: (incoming, existing) => {
|
|
402
|
+
const bid = (incoming.payload.bid as number) ?? 0;
|
|
403
|
+
if (bid <= (existing.row?.bid as number ?? 0)) {
|
|
404
|
+
throw new Error("bid too low"); // rejects the op
|
|
405
|
+
}
|
|
406
|
+
return { row: { ...existing.row, ...incoming.payload } };
|
|
407
|
+
},
|
|
408
|
+
},
|
|
409
|
+
},
|
|
410
|
+
});
|
|
411
|
+
```
|
|
412
|
+
|
|
413
|
+
### Validating client payloads
|
|
414
|
+
|
|
415
|
+
`t<MyRow>()` is a **compile-time** phantom — it erases at runtime. reflectdb validates
|
|
416
|
+
protocol structure (an op's `payload` must be a non-array object or null) but never
|
|
417
|
+
its contents, so a client can send `{ title: 12345 }` or extra keys and they reach
|
|
418
|
+
your `mutate` untouched. `readonly` and `serverSet` strip named fields; they don't
|
|
419
|
+
type-check what's left.
|
|
420
|
+
|
|
421
|
+
Validate in `mutate`, with whatever library you already use — reflectdb has no opinion
|
|
422
|
+
and no dependency here:
|
|
423
|
+
|
|
424
|
+
```ts
|
|
425
|
+
import { z } from "zod";
|
|
426
|
+
import { MutationError } from "reflectdb/core";
|
|
427
|
+
|
|
428
|
+
const Todo = z.object({
|
|
429
|
+
id: z.string(),
|
|
430
|
+
title: z.string().min(1).max(200),
|
|
431
|
+
done: z.boolean(),
|
|
432
|
+
}).strict(); // reject unknown keys instead of passing them through
|
|
433
|
+
|
|
434
|
+
server.implement("todos", {
|
|
435
|
+
query: (ctx, db) => db.select().from(todos),
|
|
436
|
+
mutate: async (op, ctx, db) => {
|
|
437
|
+
if (op.type === "delete") {
|
|
438
|
+
await db.delete(todos).where(eq(todos.id, op.rowId));
|
|
439
|
+
return;
|
|
440
|
+
}
|
|
441
|
+
// Updates carry a partial delta, not a whole row.
|
|
442
|
+
const schema = op.type === "insert" ? Todo : Todo.partial();
|
|
443
|
+
const parsed = schema.safeParse(op.payload);
|
|
444
|
+
if (!parsed.success) {
|
|
445
|
+
throw new MutationError("outside_shape", parsed.error.message);
|
|
446
|
+
}
|
|
447
|
+
await db.insert(todos).values({ id: op.rowId, ...parsed.data })
|
|
448
|
+
.onConflictDoUpdate({ target: todos.id, set: parsed.data });
|
|
449
|
+
},
|
|
450
|
+
});
|
|
451
|
+
```
|
|
452
|
+
|
|
453
|
+
Two details that matter:
|
|
454
|
+
|
|
455
|
+
- **Throw `MutationError`, not a plain `Error`.** Only `MutationError` carries an
|
|
456
|
+
`ErrorReason` through to the client's `onError`; anything else is reported as
|
|
457
|
+
`server_error`.
|
|
458
|
+
- **Write the parsed value, not `op.payload`.** Writing the raw payload after
|
|
459
|
+
validating it defeats `.strict()` and any coercion the schema applied — and it
|
|
460
|
+
also feeds unvalidated data into reflectdb's mirror, which is what conflict
|
|
461
|
+
resolution compares against.
|
|
462
|
+
|
|
463
|
+
The same applies to `authorize`, and to writes arriving through `server.rest()` —
|
|
464
|
+
both run the identical pipeline.
|
|
465
|
+
|
|
466
|
+
### Ephemeral messages (cursors, presence, typing)
|
|
467
|
+
|
|
468
|
+
Ephemeral events are room-scoped broadcasts that bypass the op log — ideal for high-frequency signals:
|
|
469
|
+
|
|
470
|
+
```tsx
|
|
471
|
+
import { useEphemeral } from "reflectdb/react";
|
|
472
|
+
|
|
473
|
+
const { events, broadcast } = useEphemeral({
|
|
474
|
+
key: "cursor",
|
|
475
|
+
userId: currentUserId,
|
|
476
|
+
ttlMs: 10_000,
|
|
477
|
+
});
|
|
478
|
+
|
|
479
|
+
// on mouse move
|
|
480
|
+
broadcast({ x: e.clientX, y: e.clientY });
|
|
481
|
+
|
|
482
|
+
// render peers
|
|
483
|
+
Object.values(events).map((c) => <Cursor x={c.x} y={c.y} />);
|
|
484
|
+
```
|
|
485
|
+
|
|
486
|
+
The whiteboard renders peer cursors this way — see [`examples/whiteboard/app.tsx`](./examples/whiteboard/app.tsx).
|
|
487
|
+
|
|
488
|
+
### Per-user query results
|
|
489
|
+
|
|
490
|
+
A `query` callback is just a function — it can return different rows depending on the caller's `auth`. reflectdb re-runs it whenever the listed `tables` change, so each subscriber gets a personalized view that stays live.
|
|
491
|
+
|
|
492
|
+
The whiteboard uses this to keep the round's secret word out of the wire for everyone except the active drawer:
|
|
493
|
+
|
|
494
|
+
```ts
|
|
495
|
+
const queries = defineSyncQueries({
|
|
496
|
+
roundWord: {
|
|
497
|
+
row: t<{ id: string; gameId: string; word: string }>(),
|
|
498
|
+
params: t<{ gameId: string }>(),
|
|
499
|
+
tables: ["games", "game_secrets"], // re-run on these
|
|
500
|
+
},
|
|
501
|
+
});
|
|
502
|
+
|
|
503
|
+
server.implement("roundWord", {
|
|
504
|
+
query: async (ctx, db) => {
|
|
505
|
+
const game = await db.select().from(games).where(eq(games.id, ctx.params.gameId)).get();
|
|
506
|
+
if (!game || game.state !== "drawing") return [];
|
|
507
|
+
if (game.currentDrawerId !== ctx.auth.userId) return []; // guessers see []
|
|
508
|
+
const secret = await db.select().from(gameSecrets)
|
|
509
|
+
.where(eq(gameSecrets.gameId, ctx.params.gameId)).get();
|
|
510
|
+
return secret?.word ? [{ id: ctx.params.gameId, gameId: ctx.params.gameId, word: secret.word }] : [];
|
|
511
|
+
},
|
|
512
|
+
mutate: async () => { throw new Error("read-only"); },
|
|
513
|
+
tables: ["games", "game_secrets"],
|
|
514
|
+
});
|
|
515
|
+
```
|
|
516
|
+
|
|
517
|
+
The `game_secrets` table isn't registered in `defineSyncQueries`, so it's never broadcast directly. Calling `server.notifyChange("game_secrets")` from the engine fans out the recomputed `roundWord` result to whichever client is now the drawer.
|
|
518
|
+
|
|
519
|
+
### Server-driven game loops
|
|
520
|
+
|
|
521
|
+
Some apps need state that advances on a clock, not on user input — round timers, expiring claims, scheduled rotations. Pair a `setInterval` with `notifyChange` and the server stays the single source of truth.
|
|
522
|
+
|
|
523
|
+
```ts
|
|
524
|
+
setInterval(async () => {
|
|
525
|
+
const now = Date.now();
|
|
526
|
+
const active = await db.select().from(games).where(eq(games.mode, "pictionary"));
|
|
527
|
+
for (const g of active) {
|
|
528
|
+
if (g.state === "drawing" && now >= g.roundEndsAt) {
|
|
529
|
+
await endRound(g.id); // raw SQL writes
|
|
530
|
+
await server.notifyChange("games"); // fan-out to subscribers
|
|
531
|
+
}
|
|
532
|
+
}
|
|
533
|
+
}, 500);
|
|
534
|
+
```
|
|
535
|
+
|
|
536
|
+
Pair with a per-row mutex if a tick can outrun its interval. The whiteboard example does both — see [`examples/whiteboard/server.tsx`](./examples/whiteboard/server.tsx).
|
|
537
|
+
|
|
538
|
+
### Windowed sync and pagination
|
|
539
|
+
|
|
540
|
+
For large tables, sync a sliding window instead of the whole set:
|
|
541
|
+
|
|
542
|
+
```ts
|
|
543
|
+
const queries = defineSyncQueries({
|
|
544
|
+
messages: {
|
|
545
|
+
row: t<Message>(),
|
|
546
|
+
conflict: "lww",
|
|
547
|
+
countHints: true, // emit count_changed deltas
|
|
548
|
+
},
|
|
549
|
+
});
|
|
550
|
+
|
|
551
|
+
// React
|
|
552
|
+
const { rows } = useSync("messages", { window: 50 });
|
|
553
|
+
const total = useTotalCount("messages");
|
|
554
|
+
const loadMore = useLoadMore("messages");
|
|
555
|
+
|
|
556
|
+
// show "Load 50 more" when rows.length < total
|
|
557
|
+
```
|
|
558
|
+
|
|
559
|
+
Make the window a **real** limit by reading `ctx.limit` in the query and supplying a `count`:
|
|
560
|
+
|
|
561
|
+
```ts
|
|
562
|
+
server.implement("messages", {
|
|
563
|
+
query: ({ params, limit }, db) =>
|
|
564
|
+
db.select().from(messages)
|
|
565
|
+
.where(eq(messages.roomId, params.roomId))
|
|
566
|
+
.orderBy(desc(messages.createdAt))
|
|
567
|
+
.limit(limit ?? 1000),
|
|
568
|
+
count: async ({ params }, db) =>
|
|
569
|
+
(await db.select({ n: count() }).from(messages)
|
|
570
|
+
.where(eq(messages.roomId, params.roomId)))[0].n,
|
|
571
|
+
});
|
|
572
|
+
```
|
|
573
|
+
|
|
574
|
+
Without them, the server fetches every matching row on every broadcast and slices in JS — pagination reduces bytes on the wire and nothing else. `ctx.limit` is `undefined` when the caller genuinely needs the full set, so a plain `limit ?? <max>` is safe.
|
|
575
|
+
|
|
576
|
+
A window is an **entitlement**, not a row count: a subscriber with `window: 50` whose query matched 3 rows still receives the next 47 inserts, and `loadMore(20)` widens the entitlement by 20 regardless of how many rows actually arrived. Reconnecting restores the widened window, not the initial one.
|
|
577
|
+
|
|
578
|
+
### Auto-generated REST API
|
|
579
|
+
|
|
580
|
+
`server.rest()` returns a fetch-style handler that responds to CRUD URLs derived from your schema:
|
|
581
|
+
|
|
582
|
+
```ts
|
|
583
|
+
const rest = server.rest({ prefix: "/api" });
|
|
584
|
+
|
|
585
|
+
serve({
|
|
586
|
+
port: 3001,
|
|
587
|
+
async fetch(req, srv) {
|
|
588
|
+
const url = new URL(req.url);
|
|
589
|
+
if (url.pathname.startsWith("/api/")) return rest(req);
|
|
590
|
+
// ...WebSocket upgrade, etc.
|
|
591
|
+
return new Response("ok");
|
|
592
|
+
},
|
|
593
|
+
});
|
|
594
|
+
```
|
|
595
|
+
|
|
596
|
+
Endpoints generated for every `implement()`'d table:
|
|
597
|
+
|
|
598
|
+
```
|
|
599
|
+
GET /api/<table> → list (supports ?where=…&limit=&offset=)
|
|
600
|
+
GET /api/<table>/:id → single row
|
|
601
|
+
POST /api/<table> → insert (body = row, or array = batch)
|
|
602
|
+
PATCH /api/<table>/:id → update
|
|
603
|
+
DELETE /api/<table>/:id → delete
|
|
604
|
+
```
|
|
605
|
+
|
|
606
|
+
REST writes go through the same pipeline as sync writes and broadcast deltas to connected clients.
|
|
607
|
+
|
|
608
|
+
### High availability with Postgres
|
|
609
|
+
|
|
610
|
+
Share a Postgres op log between server instances. Clients reconnecting to a different instance resume seamlessly from their HLC watermark.
|
|
611
|
+
|
|
612
|
+
```ts
|
|
613
|
+
import pg from "pg";
|
|
614
|
+
import { createPostgresStorage } from "reflectdb/server";
|
|
615
|
+
|
|
616
|
+
const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });
|
|
617
|
+
|
|
618
|
+
const server = createSyncServer({
|
|
619
|
+
queries, db, transport,
|
|
620
|
+
storage: createPostgresStorage(pool),
|
|
621
|
+
serverId: process.env.FLY_ALLOC_ID,
|
|
622
|
+
poll: 500, // 500ms cross-instance poll for active-active
|
|
623
|
+
});
|
|
624
|
+
```
|
|
625
|
+
|
|
626
|
+
| Mode | Config | Use case |
|
|
627
|
+
|------|--------|----------|
|
|
628
|
+
| **Failover only** | Shared Postgres, no `poll` | Clients resume on reconnect |
|
|
629
|
+
| **Active-active** | Shared Postgres + `poll: 500` | Real-time cross-instance updates |
|
|
630
|
+
|
|
631
|
+
Each poll tick first probes the shared op log's head HLC; an idle tick costs one `MAX(hlc)` query and broadcasts nothing. Only tables that actually changed are re-broadcast. The tick also re-merges the shared clock watermark, so an instance whose wall clock lags its peers stops stamping writes below HLCs clients have already seen.
|
|
632
|
+
|
|
633
|
+
## Whiteboard + Pictionary example
|
|
634
|
+
|
|
635
|
+
A complete React + Bun + Drizzle app that exercises most of reflectdb in one place: [`examples/whiteboard/`](./examples/whiteboard/).
|
|
636
|
+
|
|
637
|
+
```bash
|
|
638
|
+
cd examples/whiteboard
|
|
639
|
+
bun install
|
|
640
|
+
bun dev
|
|
641
|
+
# open http://localhost:3003 in two tabs
|
|
642
|
+
```
|
|
643
|
+
|
|
644
|
+
Two modes:
|
|
645
|
+
|
|
646
|
+
- **Freeform draw** — every player can draw on a shared canvas. Strokes are LWW per row.
|
|
647
|
+
- **Pictionary** — players take turns drawing while the others guess in chat. The server picks a word, runs a per-round timer, awards points based on remaining time, advances the drawer, and ends the game after N full rotations.
|
|
648
|
+
|
|
649
|
+
What it demonstrates:
|
|
650
|
+
|
|
651
|
+
| Pattern | Where |
|
|
652
|
+
|---------|-------|
|
|
653
|
+
| Drizzle-typed schema, SQLite op log | [`schema.ts`](./examples/whiteboard/schema.ts) |
|
|
654
|
+
| WebSocket transport on Bun | [`server.tsx`](./examples/whiteboard/server.tsx) |
|
|
655
|
+
| Authentication via better-auth (email/password + anonymous) | [`auth.ts`](./examples/whiteboard/auth.ts) |
|
|
656
|
+
| `params`-scoped queries (`strokes`, `messages` per game) | [`server.tsx`](./examples/whiteboard/server.tsx) |
|
|
657
|
+
| Per-user query results — only the drawer receives the secret word | `roundWord` in [`server.tsx`](./examples/whiteboard/server.tsx) |
|
|
658
|
+
| Server-side game loop with a mutex + `notifyChange` | `tick`, `withLock` in [`server.tsx`](./examples/whiteboard/server.tsx) |
|
|
659
|
+
| Server-side guess detection (text replacement so the answer never broadcasts) | `mutateMessageWithGuesses` in [`server.tsx`](./examples/whiteboard/server.tsx) |
|
|
660
|
+
| `readonly` field enforcement to keep the engine state out of client hands | [`schema.ts`](./examples/whiteboard/schema.ts) |
|
|
661
|
+
| Ephemeral cursors per game, scoped via `key: \`cursor:${gameId}\`` | [`app.tsx`](./examples/whiteboard/app.tsx) |
|
|
662
|
+
| Per-table rate limiting (loose for strokes, tight for chat) | `server.rateLimit` in [`server.tsx`](./examples/whiteboard/server.tsx) |
|
|
663
|
+
|
|
664
|
+
## Architecture
|
|
665
|
+
|
|
666
|
+
```
|
|
667
|
+
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
668
|
+
│ SHARED CORE (core/) │
|
|
669
|
+
│ │
|
|
670
|
+
│ defineSyncQueries({ ... }) ── one schema, shared by every layer │
|
|
671
|
+
│ │
|
|
672
|
+
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────────────────┐ │
|
|
673
|
+
│ │ types.ts │ │ hlc.ts │ │ schema.ts │ │
|
|
674
|
+
│ │ • SyncOp │ │ • HLC │ │ • SyncQueryDef │ │
|
|
675
|
+
│ │ • Messages │ │ • send/recv │ │ • InferRow / InferParams │ │
|
|
676
|
+
│ │ • ErrorReason│ │ • pack/cmp │ │ • t<T>() phantom helper │ │
|
|
677
|
+
│ │ • Protocol │ │ │ │ • ConflictPolicy │ │
|
|
678
|
+
│ └──────────────┘ └──────────────┘ └──────────────────────────────────┘ │
|
|
679
|
+
└───────────────────────────────┬─────────────────────────────────────────────┘
|
|
680
|
+
┌───────────────┴────────────────┐
|
|
681
|
+
▼ ▼
|
|
682
|
+
┌───────────────────────────────────┐ ┌──────────────────────────────────────┐
|
|
683
|
+
│ SERVER (server/) │ │ CLIENT (client/) │
|
|
684
|
+
│ │ │ │
|
|
685
|
+
│ createSyncServer<TQueries>() │ │ createSyncClient<TQueries>() │
|
|
686
|
+
│ ├─ .implement(name, opts) │ │ ├─ .sync(name, params?) │
|
|
687
|
+
│ ├─ .auth(token → AuthContext) │ │ ├─ .insert/.update/.delete │
|
|
688
|
+
│ ├─ .room(pattern, cb) │ │ ├─ .subscribe / .subscribeTable │
|
|
689
|
+
│ ├─ .rateLimit({...}) │ │ ├─ .getRows / .getRow / .getState │
|
|
690
|
+
│ ├─ .compaction({...}) │ │ ├─ .loadMore / .getTotalCount │
|
|
691
|
+
│ ├─ .rest({ prefix }) │ │ └─ .sendEphemeral / .subscribeEph. │
|
|
692
|
+
│ ├─ .notifyChange(table) │ │ │
|
|
693
|
+
│ └─ .close() │ │ Internal: │
|
|
694
|
+
│ │ │ • SyncClient (state machine) │
|
|
695
|
+
│ Pipeline (per op): │ │ • ClientStore (row cache + queue) │
|
|
696
|
+
│ 1. clock drift check │ │ • OpCreator (HLC stamping) │
|
|
697
|
+
│ 2. rate limit (fail-open) │ │ │
|
|
698
|
+
│ 3. batch-size check │ │ State machine: │
|
|
699
|
+
│ 4. readonly enforcement │ │ hydrating → disconnected → … │
|
|
700
|
+
│ 5. serverSet injection │ │ connecting → bootstrapping → synced│
|
|
701
|
+
│ 6. conflict resolution* │ │ │
|
|
702
|
+
│ │ │ Storage adapters: │
|
|
703
|
+
│ (* skipped by eager modes) │ │ • memory (ephemeral) │
|
|
704
|
+
│ │ │ • indexeddb (persistent) │
|
|
705
|
+
│ BroadcastEngine (per write): │ │ │
|
|
706
|
+
│ group subscribers → run query │ │ │
|
|
707
|
+
│ once per group → diff per │ │ │
|
|
708
|
+
│ client → send → commit cache │ │ │
|
|
709
|
+
│ │ │ │
|
|
710
|
+
│ Op log storage (optional): │ │ │
|
|
711
|
+
│ • in-memory (default) │ │ │
|
|
712
|
+
│ • sqlite (bun:sqlite) │ │ │
|
|
713
|
+
│ • postgres (any pg-compatible) │ │ │
|
|
714
|
+
└───────────────────────────────────┘ └──────────────────────────────────────┘
|
|
715
|
+
|
|
716
|
+
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
717
|
+
│ TRANSPORT LAYER (transport/) │
|
|
718
|
+
│ │
|
|
719
|
+
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────────────┐ │
|
|
720
|
+
│ │ WebSocket │ │ SSE │ │ Polling │ │
|
|
721
|
+
│ │ real-time │ │ event-stream + │ │ 3 HTTP endpoints — │ │
|
|
722
|
+
│ │ bi-directional │ │ POST back-chan │ │ works anywhere HTTP does│ │
|
|
723
|
+
│ └──────────────────┘ └──────────────────┘ └──────────────────────────┘ │
|
|
724
|
+
└─────────────────────────────────────────────────────────────────────────────┘
|
|
725
|
+
|
|
726
|
+
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
727
|
+
│ FRAMEWORK BINDINGS (react/, svelte/, vanilla/) │
|
|
728
|
+
│ │
|
|
729
|
+
│ • createSyncReact(queries) → typed hooks + <SyncProvider> │
|
|
730
|
+
│ • createSyncSvelte(queries) → typed Svelte stores │
|
|
731
|
+
│ • createSyncVanilla(queries) → typed callback API │
|
|
732
|
+
└─────────────────────────────────────────────────────────────────────────────┘
|
|
733
|
+
```
|
|
734
|
+
|
|
735
|
+
## Core Concepts
|
|
736
|
+
|
|
737
|
+
### Hybrid Logical Clocks
|
|
738
|
+
|
|
739
|
+
reflectdb uses [hybrid logical clocks](https://muratbuffalo.blogspot.com/2014/07/hybrid-logical-clocks.html) (HLCs) to order events across machines without requiring synchronized clocks.
|
|
740
|
+
|
|
741
|
+
An HLC has three parts:
|
|
742
|
+
|
|
743
|
+
| Component | Purpose |
|
|
744
|
+
|-----------|---------|
|
|
745
|
+
| `ms` | Physical wall time |
|
|
746
|
+
| `counter` | Logical counter (breaks ties) |
|
|
747
|
+
| `nodeId` | Machine that generated it |
|
|
748
|
+
|
|
749
|
+
HLCs pack to zero-padded strings (`0000001711234567890.0003.client-abc`), so **string comparison gives correct causal ordering** — no parsing needed. Conflict resolution is essentially free.
|
|
750
|
+
|
|
751
|
+
Two operations define the clock:
|
|
752
|
+
|
|
753
|
+
- **send** (`sendHlc`): advance `max(wall, lastMs)`; increment counter on tie, else reset.
|
|
754
|
+
- **receive** (`receiveHlc`): advance past both local and remote state. Remote timestamps are clamped to `wall + MAX_CLOCK_DRIFT_MS` (default 5 min) so a runaway client can't push the clock into the future.
|
|
755
|
+
|
|
756
|
+
The clock **ratchets forward** through every exchange, so causal ordering is preserved across the network.
|
|
757
|
+
|
|
758
|
+
### Conflict Resolution
|
|
759
|
+
|
|
760
|
+
Four built-in policies, chosen per-query:
|
|
761
|
+
|
|
762
|
+
```ts
|
|
763
|
+
defineSyncQueries({
|
|
764
|
+
posts: { row: t<Post>(), conflict: "lww" },
|
|
765
|
+
docs: { row: t<Doc>(), conflict: "merge" },
|
|
766
|
+
config: { row: t<Config>(), conflict: "server" },
|
|
767
|
+
scores: { row: t<Score>(), conflict: { policy: "custom", resolve: fn } },
|
|
768
|
+
});
|
|
769
|
+
```
|
|
770
|
+
|
|
771
|
+
| Policy | Granularity | Concurrent edits to different fields | Use case |
|
|
772
|
+
|---------|-------------|--------------------------------------|----------|
|
|
773
|
+
| `lww` | Row | One wins, the other is lost | Simple data, rare conflicts |
|
|
774
|
+
| `merge` | Column | Both preserved | Collaborative editing |
|
|
775
|
+
| `server`| Row | Only first write; all others rejected | Config, reference data |
|
|
776
|
+
| custom | You choose | Your logic | Counters, "highest bid wins", business rules |
|
|
777
|
+
|
|
778
|
+
A custom resolver receives the incoming op, the existing row + per-column clocks, and metadata, and returns the resolved row. Throw to reject.
|
|
779
|
+
|
|
780
|
+
`merge` is a **server-side** guarantee: the server resolves per column using the client op HLCs held in its mirror, so two clients editing different fields both land. The per-column clocks a client sees on a broadcast are a different domain — a diff-driven broadcast can't attribute a column to the op that produced it, so every column changed in one broadcast carries that broadcast's HLC. Client-side merge orders *broadcasts* against each other and against local optimistic state; it does not reconstruct per-column causality between clients.
|
|
781
|
+
|
|
782
|
+
Both eager broadcast modes skip conflict resolution entirely — writes land last-writer-wins regardless of the declared policy.
|
|
783
|
+
|
|
784
|
+
### The Sync Protocol
|
|
785
|
+
|
|
786
|
+
```
|
|
787
|
+
1. client ──▶ hello server ──▶ hello_ack (protocol, serverId)
|
|
788
|
+
2. client ──▶ sync_declare server ──▶ snapshot / bootstrap_complete
|
|
789
|
+
3. client ──▶ ops (optimistic) server runs pipeline
|
|
790
|
+
4. server ──▶ ack / reject
|
|
791
|
+
5. server ──▶ delta (broadcast to subscribers)
|
|
792
|
+
6. client reconnects ──▶ resume (watermark HLC) server ──▶ missed deltas
|
|
793
|
+
```
|
|
794
|
+
|
|
795
|
+
All messages are JSON; the transport is just a pipe. WebSocket gives bi-directional real-time; SSE gives server-push with POST for upstream; polling is stateless HTTP for constrained environments.
|
|
796
|
+
|
|
797
|
+
### Two stores, one sync
|
|
798
|
+
|
|
799
|
+
reflectdb keeps its own store alongside yours, and it helps to know which one answers what:
|
|
800
|
+
|
|
801
|
+
| Read | Source |
|
|
802
|
+
|------|--------|
|
|
803
|
+
| Snapshots (`bootstrap`, `resume`) | **Your database**, via the `query` callback |
|
|
804
|
+
| Broadcast deltas | **Your database**, diffed against a per-client cached result set |
|
|
805
|
+
| Conflict resolution (`lww` / `merge` / `server` / custom) | **reflectdb's mirror** — a JSONB row store plus per-column HLCs |
|
|
806
|
+
| Which tables changed since an HLC | **reflectdb's op log** |
|
|
807
|
+
|
|
808
|
+
A write therefore lands in two places: your `mutate` callback commits to your database, and reflectdb commits the mirror row plus its op-log entry. Those are **separate commits** — "atomic" in this codebase means the mirror row and its op-log entry commit together, not that they commit with your write. Consequences worth designing around:
|
|
809
|
+
|
|
810
|
+
- A crash between the two leaves your database ahead of the mirror. Clients still converge (snapshots come from your database), but conflict resolution decides against slightly stale state until the next write.
|
|
811
|
+
- If `mutate` transforms the payload, or database defaults/triggers rewrite it, or something writes the table out of band, the mirror drifts from what clients actually see. Keep `mutate` a faithful application of `op.payload` when conflict policy is load-bearing, and route out-of-band writes through `server.applyServerOp` / `server.emit` / `server.tx`.
|
|
812
|
+
|
|
813
|
+
### The Op Log and Resume
|
|
814
|
+
|
|
815
|
+
Every accepted mutation is appended to the server's op log with its HLC. On reconnect, the client sends its last seen HLC as a watermark, and the server replays only what the client missed. This makes cross-server failover automatic when the log is shared (Postgres).
|
|
816
|
+
|
|
817
|
+
Old ops are compacted on a schedule based on client inactivity and minimum op age. Reconnecting clients whose watermark has been compacted receive a fresh bootstrap.
|
|
818
|
+
|
|
819
|
+
## API Reference
|
|
820
|
+
|
|
821
|
+
### `reflectdb/core`
|
|
822
|
+
|
|
823
|
+
```ts
|
|
824
|
+
import {
|
|
825
|
+
defineSyncQueries, t,
|
|
826
|
+
createHlc, sendHlc, receiveHlc, packHlc, unpackHlc, compareHlc,
|
|
827
|
+
MutationError, TransportSendError, isErrorReason, reasonFromError,
|
|
828
|
+
PROTOCOL_VERSION, MAX_CLOCK_DRIFT_MS, MAX_BATCH_SIZE,
|
|
829
|
+
TOMBSTONE_RETENTION_MS, SERVER_TOMBSTONE_RETENTION_MS,
|
|
830
|
+
} from "reflectdb/core";
|
|
831
|
+
```
|
|
832
|
+
|
|
833
|
+
| Export | Description |
|
|
834
|
+
|--------|-------------|
|
|
835
|
+
| `defineSyncQueries(map)` | Identity function that pins your schema's literal types. Feed its result to both server and client. |
|
|
836
|
+
| `t<T>()` | Phantom helper to declare a row or params type. Returns `undefined as T`. |
|
|
837
|
+
| `createHlc(nodeId)` / `sendHlc` / `receiveHlc` | HLC constructors and transitions. |
|
|
838
|
+
| `packHlc` / `unpackHlc` / `compareHlc` | Serialize, deserialize, compare HLC values. |
|
|
839
|
+
| `MutationError(reason, message?)` | Throw from `mutate`/`authorize` to reject a write with a specific `ErrorReason`. |
|
|
840
|
+
| `TransportSendError(clientId, message)` | Throw from a custom `ServerTransport.send` when a frame did not reach the peer. |
|
|
841
|
+
| `isErrorReason(v)` / `reasonFromError(e)` | Validate / extract an `ErrorReason`. |
|
|
842
|
+
|
|
843
|
+
Types: `HLC`, `SyncOp`, `OpType`, `OpStatus`, `ClientMessage`, `ServerMessage`, `ErrorReason`, `ConflictPolicy`, `ConflictResolver`, `SyncQueryDef`, `InferRow`, `InferParams`, `InferWritableRow`, `RequiresParams`, `RateLimitConfig`, `CompactionConfig`, `ShapeConfig`, `AuthContext`, `DrizzleTableLike`.
|
|
844
|
+
|
|
845
|
+
### `reflectdb/server`
|
|
846
|
+
|
|
847
|
+
```ts
|
|
848
|
+
import {
|
|
849
|
+
createServer, createSyncServer,
|
|
850
|
+
createSqliteStorage, createPostgresStorage,
|
|
851
|
+
resolveConflict, processOp,
|
|
852
|
+
enforceClockDrift, enforceReadonly, enforceServerSet, enforceBatchSize, createRateLimiter,
|
|
853
|
+
MutationError,
|
|
854
|
+
} from "reflectdb/server";
|
|
855
|
+
```
|
|
856
|
+
|
|
857
|
+
**`createSyncServer<TQueries, TDb, TAuth>(config)`** — the typed entry point. Returns a server with:
|
|
858
|
+
|
|
859
|
+
| Method | Purpose |
|
|
860
|
+
|--------|---------|
|
|
861
|
+
| `.implement(name, options)` | Register a query handler (required for every query in the schema). |
|
|
862
|
+
| `.auth(callback)` | Validate the connection request and return an `AuthContext`. |
|
|
863
|
+
| `.room(pattern, callback)` | Scope clients to a subset of data, matched against URL-style patterns (`org/:orgId`). |
|
|
864
|
+
| `.rateLimit(config)` | Set per-user/per-table limits. Fail-open on limiter errors. |
|
|
865
|
+
| `.compaction(config)` | Configure op-log compaction. |
|
|
866
|
+
| `.rest({ prefix })` | Generate a CRUD fetch handler. |
|
|
867
|
+
| `.minSchemaVersion(n)` | Reject clients on older schema versions. |
|
|
868
|
+
| `.notifyChange(table)` | Manually trigger a broadcast (for external writes). |
|
|
869
|
+
| `.runCompaction()` | Manually run one compaction pass. |
|
|
870
|
+
| `.close()` | Shut down, disconnect clients, close storage. |
|
|
871
|
+
|
|
872
|
+
`createServer()` is the lower-level untyped variant — use it only if you need to register queries dynamically or don't have a `defineSyncQueries` map.
|
|
873
|
+
|
|
874
|
+
### `reflectdb/client`
|
|
875
|
+
|
|
876
|
+
```ts
|
|
877
|
+
import {
|
|
878
|
+
createSyncClient,
|
|
879
|
+
SyncClient,
|
|
880
|
+
ClientStore,
|
|
881
|
+
createMemoryStorage,
|
|
882
|
+
createOpCreator,
|
|
883
|
+
} from "reflectdb/client";
|
|
884
|
+
|
|
885
|
+
import { createIndexedDBStorage } from "reflectdb/client/storage/indexeddb";
|
|
886
|
+
```
|
|
887
|
+
|
|
888
|
+
**`createSyncClient<TQueries>(config)`** — fully typed client. Methods:
|
|
889
|
+
|
|
890
|
+
| Category | Methods |
|
|
891
|
+
|----------|---------|
|
|
892
|
+
| Lifecycle | `init()`, `connect()`, `bootstrap()`, `resume()`, `push()`, `close()` |
|
|
893
|
+
| Subscriptions | `sync(name, params?)`, `unsync(name)` |
|
|
894
|
+
| Mutations | `insert(name, id, payload)`, `update(name, id, patch)`, `delete(name, id)`, `batch(ops)` |
|
|
895
|
+
| Reads | `getRows(name)`, `getRow(name, id)`, `getState()`, `getVersion()`, `getPendingCount()` |
|
|
896
|
+
| Observation | `subscribe(listener)`, `subscribeTable(name, listener)` (returns unsubscribe fn) |
|
|
897
|
+
| Windowing | `loadMore(name, count)`, `getTotalCount(name)` |
|
|
898
|
+
| Ephemeral | `sendEphemeral({ key, userId, data, ttlMs? })`, `subscribeEphemeral(key, listener)` |
|
|
899
|
+
|
|
900
|
+
State machine: `hydrating → disconnected → connecting → bootstrapping → synced`. Reconnects with exponential backoff (capped by `maxReconnectDelayMs`, default 30s).
|
|
901
|
+
|
|
902
|
+
### `reflectdb/react`
|
|
903
|
+
|
|
904
|
+
```ts
|
|
905
|
+
import {
|
|
906
|
+
SyncProvider, useSyncClient,
|
|
907
|
+
useSync, useSyncStatus, useRow,
|
|
908
|
+
usePendingCount, useEphemeral,
|
|
909
|
+
useTotalCount, useLoadMore,
|
|
910
|
+
createSyncReact,
|
|
911
|
+
} from "reflectdb/react";
|
|
912
|
+
```
|
|
913
|
+
|
|
914
|
+
**`<SyncProvider>` props:**
|
|
915
|
+
|
|
916
|
+
| Prop | Type | Description |
|
|
917
|
+
|------|------|-------------|
|
|
918
|
+
| `url` | `string` | WebSocket URL (required) |
|
|
919
|
+
| `token` | `string` | Auth token passed to `server.auth()` (required) |
|
|
920
|
+
| `tables` | `string[]` | Tables to auto-sync on mount |
|
|
921
|
+
| `clientId` | `string` | Stable ID for this client (generated if omitted) |
|
|
922
|
+
| `storage` | `ClientStorageAdapter` | Defaults to memory |
|
|
923
|
+
| `onReauth` | `() => Promise<string>` | Called when server revokes auth |
|
|
924
|
+
| `onError` | `(e) => void` | Connection / sync error callback |
|
|
925
|
+
|
|
926
|
+
**Hooks:**
|
|
927
|
+
|
|
928
|
+
| Hook | Returns |
|
|
929
|
+
|------|---------|
|
|
930
|
+
| `useSync(table, options?)` | `{ rows, insert, update, remove, loading }` — options: `{ params?, includeDeleted?, window? }` |
|
|
931
|
+
| `useSyncStatus()` | `"hydrating" \| "disconnected" \| "connecting" \| "bootstrapping" \| "synced"` |
|
|
932
|
+
| `useRow(table, id)` | Single row or `null` |
|
|
933
|
+
| `usePendingCount()` | Total unsynced op count |
|
|
934
|
+
| `useEphemeral({ key, userId, ttlMs? })` | `{ events, broadcast }` |
|
|
935
|
+
| `useTotalCount(table)` | Server-side count (requires `countHints: true`) |
|
|
936
|
+
| `useLoadMore(table)` | Function to expand the sync window |
|
|
937
|
+
|
|
938
|
+
**`createSyncReact<TQueries>()`** returns the same hook set with row and param types inferred from your schema.
|
|
939
|
+
|
|
940
|
+
### `reflectdb/svelte`
|
|
941
|
+
|
|
942
|
+
```ts
|
|
943
|
+
import { createSyncStore, createSyncSvelte, createBrowserWsTransport } from "reflectdb/svelte";
|
|
944
|
+
```
|
|
945
|
+
|
|
946
|
+
`createSyncStore(config)` returns a `SyncStore`:
|
|
947
|
+
|
|
948
|
+
```ts
|
|
949
|
+
const store = createSyncStore({ url, token, tables: ["notes"] });
|
|
950
|
+
|
|
951
|
+
const { rows, insert, update, remove } = store.sync<Note>("notes");
|
|
952
|
+
// rows is a Readable<Note[]> — subscribe with Svelte's $rows
|
|
953
|
+
|
|
954
|
+
store.status // Readable<SyncClientState>
|
|
955
|
+
store.pendingCount // Readable<number>
|
|
956
|
+
|
|
957
|
+
store.connect();
|
|
958
|
+
store.onStateChange((s) => …);
|
|
959
|
+
store.onError((e) => …);
|
|
960
|
+
```
|
|
961
|
+
|
|
962
|
+
`createSyncSvelte(queries)` returns fully-typed store factories.
|
|
963
|
+
|
|
964
|
+
### `reflectdb/vanilla`
|
|
965
|
+
|
|
966
|
+
```ts
|
|
967
|
+
import { createSync, createSyncVanilla, createBrowserWsTransport } from "reflectdb/vanilla";
|
|
968
|
+
|
|
969
|
+
const sync = createSync({ url, token, tables: ["notes"] });
|
|
970
|
+
const notes = sync.sync<Note>("notes");
|
|
971
|
+
|
|
972
|
+
notes.onChange((rows) => render(rows));
|
|
973
|
+
notes.insert(id, { title: "…" });
|
|
974
|
+
|
|
975
|
+
sync.onStateChange((s) => …);
|
|
976
|
+
sync.onPendingChange((n) => …);
|
|
977
|
+
sync.onError((e) => …);
|
|
978
|
+
sync.connect();
|
|
979
|
+
```
|
|
980
|
+
|
|
981
|
+
Also supports ephemeral: `sync.sendEphemeral({ key, userId, data })`, `sync.onEphemeral(key, listener)`.
|
|
982
|
+
|
|
983
|
+
### `reflectdb/transport/*`
|
|
984
|
+
|
|
985
|
+
```ts
|
|
986
|
+
import { createWsServerTransport, isOriginAllowed } from "reflectdb/transport/ws";
|
|
987
|
+
import { createBunWsServerTransport } from "reflectdb/transport/bun-ws";
|
|
988
|
+
import { createSseServerTransport } from "reflectdb/transport/sse";
|
|
989
|
+
import { createPollingServerTransport, pollingBodyTooLarge } from "reflectdb/transport/polling";
|
|
990
|
+
```
|
|
991
|
+
|
|
992
|
+
Each server transport returns a `ServerTransport` object plus framework-agnostic handlers (`handleOpen`, `handleMessage`, `handleClose`, `handlePong` for WS; `handleSubscribe`, `handleMessage`, `handleDisconnect`, `createEventStream` for SSE; `handleConnect`, `handlePoll`, `handleSend`, `handleDisconnect` for polling). Wire them to your HTTP server's routes — reflectdb does not ship a specific HTTP server.
|
|
993
|
+
|
|
994
|
+
`reflectdb/transport/bun-ws` is the same WebSocket transport shaped for `Bun.serve`: `createBunWsServerTransport()` returns `{ transport, websocket }`, where `websocket` is the handlers object you pass straight to `Bun.serve({ websocket })`. Use it only under Bun; `reflectdb/transport/ws` is the runtime-agnostic one.
|
|
995
|
+
|
|
996
|
+
Client-side, use the `createBrowserWsTransport(url)` helper exported from `reflectdb/svelte` or `reflectdb/vanilla`, or let `<SyncProvider>` create one internally.
|
|
997
|
+
|
|
998
|
+
## Configuration Reference
|
|
999
|
+
|
|
1000
|
+
### Query definition
|
|
1001
|
+
|
|
1002
|
+
Every entry in `defineSyncQueries({ ... })`:
|
|
1003
|
+
|
|
1004
|
+
```ts
|
|
1005
|
+
{
|
|
1006
|
+
// declare the row type — choose ONE:
|
|
1007
|
+
row: t<MyRow>(), // plain type (recommended default)
|
|
1008
|
+
// OR
|
|
1009
|
+
table: someDrizzleTable, // auto-infers row type + table list + pk
|
|
1010
|
+
|
|
1011
|
+
// optional:
|
|
1012
|
+
params: t<{ orgId: string }>(), // typed query params (required on sync() if declared)
|
|
1013
|
+
tables: ["posts", "post_tags"], // change-detection tables; defaults to the query key
|
|
1014
|
+
pk: "id", // primary-key column name (default "id")
|
|
1015
|
+
conflict: "lww", // "lww" | "merge" | "server" | { policy: "custom", resolve }
|
|
1016
|
+
readonly: ["createdBy"], // fields the client cannot write
|
|
1017
|
+
serverSet: ["createdAt", "updatedAt"], // fields the server always sets — required in `implement.serverSet`
|
|
1018
|
+
countHints: true, // emit count_changed deltas for windowed sync
|
|
1019
|
+
}
|
|
1020
|
+
```
|
|
1021
|
+
|
|
1022
|
+
`conflict` resolves the incoming op against **reflectdb's mirror** (its own JSONB row store and per-column clocks), not against your database. The two agree as long as every write goes through reflectdb and `mutate` persists the resolved payload verbatim — see [Two stores, one sync](#two-stores-one-sync).
|
|
1023
|
+
|
|
1024
|
+
### Server configuration
|
|
1025
|
+
|
|
1026
|
+
```ts
|
|
1027
|
+
createSyncServer({
|
|
1028
|
+
queries, // from defineSyncQueries()
|
|
1029
|
+
db, // optional — anything you want handed to query/mutate callbacks
|
|
1030
|
+
transport, // ServerTransport (required)
|
|
1031
|
+
storage, // StorageAdapter (optional; defaults to in-memory op log)
|
|
1032
|
+
serverId: "server-1", // unique within the deployment
|
|
1033
|
+
poll: 500, // ms — enable HA active-active polling
|
|
1034
|
+
maxConnectionsPerUser: 10, // backpressure guard
|
|
1035
|
+
queryTimeoutMs: 5_000, // abort a broadcast query that hangs (0 = disabled)
|
|
1036
|
+
maxBroadcastConcurrency: 8, // subscriber groups queried in parallel per broadcast
|
|
1037
|
+
allowAnonymous: false, // serve connections with no auth() callback
|
|
1038
|
+
onEvent: (event) => { … }, // lifecycle telemetry
|
|
1039
|
+
});
|
|
1040
|
+
```
|
|
1041
|
+
|
|
1042
|
+
A server with no `auth()` callback rejects the handshake — every message after `hello` would fail the authentication gate anyway. Pass `allowAnonymous: true` to serve unauthenticated clients; each session gets an `anon:<clientId>` identity.
|
|
1043
|
+
|
|
1044
|
+
### `implement()` options
|
|
1045
|
+
|
|
1046
|
+
```ts
|
|
1047
|
+
server.implement("todos", {
|
|
1048
|
+
query: (ctx, db) => /* fetch rows — return anything iterable */,
|
|
1049
|
+
mutate: async (op, ctx, db) => { /* apply op.type/op.rowId/op.payload */ },
|
|
1050
|
+
authorize: async (action, ctx, db) => {
|
|
1051
|
+
// action.type is "read" | "write"
|
|
1052
|
+
// throw to deny
|
|
1053
|
+
},
|
|
1054
|
+
serverSet: { createdAt: () => new Date() }, // required if schema declares serverSet fields
|
|
1055
|
+
broadcast: "consistent", // "consistent" | "eager" | "eager-durable"
|
|
1056
|
+
flushInterval: 50, // ms, for eager broadcasting
|
|
1057
|
+
maxBufferSize: 100, // ops per eager batch
|
|
1058
|
+
tables: ["todos"], // override change-detection set
|
|
1059
|
+
count: (ctx, db) => db.count(…), // total row count for windowed queries
|
|
1060
|
+
groupBy: ({ auth }) => String(auth.orgId), // collapse subscribers into one query execution
|
|
1061
|
+
room: "org/:orgId", // require this room pattern on every subscription
|
|
1062
|
+
});
|
|
1063
|
+
```
|
|
1064
|
+
|
|
1065
|
+
`broadcast` modes:
|
|
1066
|
+
|
|
1067
|
+
- **`consistent`** (default): run the conflict pipeline, persist, then broadcast by diffing each subscriber's re-executed query result.
|
|
1068
|
+
- **`eager-durable`**: skip conflict resolution; run `mutate`, persist to reflectdb's mirror atomically, then broadcast the delta directly. The recommended low-latency mode.
|
|
1069
|
+
- **`eager`**: same, but the mirror write is batched in the background — a crash can lose it. Only safe when `mutate` is durable to your own database.
|
|
1070
|
+
|
|
1071
|
+
Both eager modes still enforce `readonly`, `serverSet`, clock drift, the batch cap and rate limits. What they skip is **conflict resolution**: a declared `conflict` policy does not apply and writes land last-writer-wins.
|
|
1072
|
+
|
|
1073
|
+
#### Scaling broadcasts with `groupBy`
|
|
1074
|
+
|
|
1075
|
+
A write re-executes each dependent query once per subscriber *group*. Groups default to `(auth, params, roomKey)`, so with per-user auth they collapse to roughly one per connected client — N clients means N query executions per write.
|
|
1076
|
+
|
|
1077
|
+
`groupBy` returns the coarser key a query actually depends on:
|
|
1078
|
+
|
|
1079
|
+
```ts
|
|
1080
|
+
server.implement("posts", {
|
|
1081
|
+
query: ({ auth }, db) => db.select().from(posts).where(eq(posts.orgId, auth.orgId)),
|
|
1082
|
+
// Results depend only on orgId — every member of an org shares one execution.
|
|
1083
|
+
groupBy: ({ auth }) => String(auth.orgId),
|
|
1084
|
+
});
|
|
1085
|
+
```
|
|
1086
|
+
|
|
1087
|
+
Two clients sharing a key **must** be entitled to byte-identical rows. Collapsing clients that aren't leaks rows across the boundary.
|
|
1088
|
+
|
|
1089
|
+
### Rate limiting
|
|
1090
|
+
|
|
1091
|
+
```ts
|
|
1092
|
+
server.rateLimit({
|
|
1093
|
+
opsPerSecond: 20,
|
|
1094
|
+
opsPerMinute: 600,
|
|
1095
|
+
batchesPerMinute: 120,
|
|
1096
|
+
perTable: {
|
|
1097
|
+
todos: { opsPerSecond: 10 },
|
|
1098
|
+
},
|
|
1099
|
+
ephemeralPerSecond: 60, // presence/cursor ceiling per client (default 60; 0 disables)
|
|
1100
|
+
});
|
|
1101
|
+
```
|
|
1102
|
+
|
|
1103
|
+
The limiter is **fail-open**: if the limiter itself errors, ops still flow. Clients that exceed their limit receive `ErrorReason: "rate_limited"`.
|
|
1104
|
+
|
|
1105
|
+
Ephemeral messages are metered separately and **always** — even without a `rateLimit()` call — because each one fans out to every room subscriber, which makes an unmetered channel an amplification vector. Dropped messages surface as an `ephemeral_rate_limited` event on `onEvent`.
|
|
1106
|
+
|
|
1107
|
+
### Compaction
|
|
1108
|
+
|
|
1109
|
+
```ts
|
|
1110
|
+
server.compaction({
|
|
1111
|
+
clientInactivityTimeout: "24h", // clients idle longer are ignored
|
|
1112
|
+
interval: "1h", // compaction interval
|
|
1113
|
+
minOpAge: "5m", // don't compact ops younger than this
|
|
1114
|
+
});
|
|
1115
|
+
```
|
|
1116
|
+
|
|
1117
|
+
Durations accept `ms`, `s`, `m`, `h`, `d`.
|
|
1118
|
+
|
|
1119
|
+
### Client configuration
|
|
1120
|
+
|
|
1121
|
+
```ts
|
|
1122
|
+
createSyncClient({
|
|
1123
|
+
queries,
|
|
1124
|
+
clientId: "browser-xyz", // required; keep stable across reloads
|
|
1125
|
+
transport: createBrowserWsTransport("ws://…"),
|
|
1126
|
+
token: "auth-token", // required
|
|
1127
|
+
storage: createIndexedDBStorage({ dbName: "app" }),
|
|
1128
|
+
autoSync: true, // auto-sync param-less queries on connect
|
|
1129
|
+
maxReconnectDelayMs: 30_000,
|
|
1130
|
+
hydrateAllTables: false, // true = read every stored row at boot
|
|
1131
|
+
onSync: () => { … }, // fires once bootstrap completes
|
|
1132
|
+
onError: (e) => { … },
|
|
1133
|
+
onReauth: async () => newToken, // called after "auth_revoked"
|
|
1134
|
+
});
|
|
1135
|
+
```
|
|
1136
|
+
|
|
1137
|
+
At boot the client restores its persisted subscriptions first and hydrates only those tables — rows only ever reach local storage through a subscription, so nothing reachable is skipped. Set `hydrateAllTables: true` if you read rows for a table before calling `sync()` on it.
|
|
1138
|
+
|
|
1139
|
+
### Storage adapters
|
|
1140
|
+
|
|
1141
|
+
#### Server op log
|
|
1142
|
+
|
|
1143
|
+
| Adapter | Import | Best for |
|
|
1144
|
+
|---------|--------|----------|
|
|
1145
|
+
| _(none)_ | omit `storage` | In-memory op log; ephemeral, single node |
|
|
1146
|
+
| `createSqliteStorage({ path?, db? })` | `reflectdb/server` | Single server, development, embedded — **Bun only** |
|
|
1147
|
+
| `createPostgresStorage(poolOrConfig)` | `reflectdb/server` | Multi-server HA, production |
|
|
1148
|
+
|
|
1149
|
+
`createPostgresStorage` accepts any object with `query(text, values) => { rows }` — `pg.Pool`, `pg.Client`, `@neondatabase/serverless`, etc. Optional config: `{ client, tablePrefix: "_reflectdb" }`.
|
|
1150
|
+
|
|
1151
|
+
`createSqliteStorage` is backed by `bun:sqlite` and is resolved lazily, so importing `reflectdb/server` on Node is fine — only calling `createSqliteStorage` there throws. On Node, use `createPostgresStorage` (or omit `storage` for the in-memory op log).
|
|
1152
|
+
|
|
1153
|
+
#### Client
|
|
1154
|
+
|
|
1155
|
+
| Adapter | Import | Best for |
|
|
1156
|
+
|---------|--------|----------|
|
|
1157
|
+
| `createMemoryStorage()` | `reflectdb/client` | Testing, SSR, short sessions |
|
|
1158
|
+
| `createIndexedDBStorage({ dbName, version?, migrate? })` | `reflectdb/client/storage/indexeddb` | Production browser apps |
|
|
1159
|
+
|
|
1160
|
+
### Transport configuration
|
|
1161
|
+
|
|
1162
|
+
#### WebSocket
|
|
1163
|
+
|
|
1164
|
+
```ts
|
|
1165
|
+
createWsServerTransport({
|
|
1166
|
+
maxMessageBytes: 1_000_000, // reject larger frames
|
|
1167
|
+
pingIntervalMs: 30_000, // heartbeat; 0 disables
|
|
1168
|
+
pongTimeoutMs: 60_000, // close half-open connections
|
|
1169
|
+
maxBufferedBytes: 8_000_000, // outbound backpressure ceiling; 0 disables
|
|
1170
|
+
});
|
|
1171
|
+
```
|
|
1172
|
+
|
|
1173
|
+
Returns a `ServerTransport` plus `handleOpen`, `handleMessage`, `handleClose`, `handlePong`. Wire these to your HTTP server's WebSocket callbacks (see [Quick Start](#quick-start)). Use `isOriginAllowed(req, ["https://app.example"])` in your upgrade handler for CORS.
|
|
1174
|
+
|
|
1175
|
+
#### Server-Sent Events
|
|
1176
|
+
|
|
1177
|
+
```ts
|
|
1178
|
+
createSseServerTransport({
|
|
1179
|
+
replayBufferSize: 256, // Last-Event-ID replay window
|
|
1180
|
+
});
|
|
1181
|
+
```
|
|
1182
|
+
|
|
1183
|
+
Two endpoints to wire: `GET /sync/events/:clientId` (SSE stream) and `POST /sync/messages/:clientId` (client → server).
|
|
1184
|
+
|
|
1185
|
+
#### HTTP long-polling
|
|
1186
|
+
|
|
1187
|
+
```ts
|
|
1188
|
+
createPollingServerTransport({
|
|
1189
|
+
maxQueueLen: 1000,
|
|
1190
|
+
idleTimeoutMs: 60_000,
|
|
1191
|
+
reaperIntervalMs: 30_000,
|
|
1192
|
+
maxMessageBytes: 1_000_000,
|
|
1193
|
+
});
|
|
1194
|
+
```
|
|
1195
|
+
|
|
1196
|
+
Three endpoints: `POST /sync/connect/:id`, `GET /sync/poll/:id`, `POST /sync/send/:id`. Use the `pollingBodyTooLarge()` helper in your request handler.
|
|
1197
|
+
|
|
1198
|
+
#### Writing your own transport
|
|
1199
|
+
|
|
1200
|
+
`ServerTransport.send` **must reject when the frame did not reach the peer** — unknown or closed socket, full outbound queue, backpressure limit. The broadcast engine treats a resolved `send` as "this delta landed" and only then commits the client's cached result set; a transport that swallows failures makes the server believe a client holds rows it never received, and the divergence persists until reconnect. Throw `TransportSendError` from `reflectdb/core` so callers can distinguish delivery failures from bugs.
|
|
1201
|
+
|
|
1202
|
+
## Development
|
|
1203
|
+
|
|
1204
|
+
### Prerequisites
|
|
1205
|
+
|
|
1206
|
+
- [Bun](https://bun.sh) 1.0+
|
|
1207
|
+
- Node.js 22+ and npm — dev tooling, and `bun run verify:node`, which checks the published package works for Node consumers in both ESM and CommonJS
|
|
1208
|
+
|
|
1209
|
+
### Setup
|
|
1210
|
+
|
|
1211
|
+
```bash
|
|
1212
|
+
git clone https://github.com/TimMikeladze/reflectdb.git
|
|
1213
|
+
cd reflectdb
|
|
1214
|
+
bun install
|
|
1215
|
+
```
|
|
1216
|
+
|
|
1217
|
+
### Scripts
|
|
1218
|
+
|
|
1219
|
+
| Command | Description |
|
|
1220
|
+
|---------|-------------|
|
|
1221
|
+
| `bun test` | Run the full test suite |
|
|
1222
|
+
| `bun test --watch` | Watch mode |
|
|
1223
|
+
| `bun test --coverage` | Coverage report |
|
|
1224
|
+
| `bun run build` | Build with `bunup` (ESM + types) |
|
|
1225
|
+
| `bun run type-check` | TypeScript strict check |
|
|
1226
|
+
| `bun run lint` | Lint with `oxlint` |
|
|
1227
|
+
| `bun run format` | Format with `oxfmt` |
|
|
1228
|
+
| `bun run verify:exports` | Check the `exports` map against `dist/`, and type-check the emitted declarations without ambient Bun/React globals (run after `build`) |
|
|
1229
|
+
| `bun run verify:node` | Install the packed tarball into a throwaway Node project and check every subpath imports, requires, and type-checks there under both export conditions (run after `build`; needs `node` + `npm`) |
|
|
1230
|
+
|
|
1231
|
+
### Project Structure
|
|
1232
|
+
|
|
1233
|
+
```
|
|
1234
|
+
src/
|
|
1235
|
+
├── core/ HLC, types, schema
|
|
1236
|
+
├── server/ createSyncServer, pipeline, session, handler
|
|
1237
|
+
│ │ broadcast-engine, result-cache, eager-buffer,
|
|
1238
|
+
│ │ compaction-manager, replay-detector, ephemeral-manager
|
|
1239
|
+
│ └── storage/ SQLite + Postgres adapters
|
|
1240
|
+
├── client/ sync-client, store, ops, typed-client
|
|
1241
|
+
│ └── storage/ memory + IndexedDB adapters
|
|
1242
|
+
├── transport/ WebSocket (runtime-agnostic + Bun.serve), SSE, polling
|
|
1243
|
+
├── react/ <SyncProvider>, hooks, typed factory
|
|
1244
|
+
├── svelte/ createSyncStore, typed factory
|
|
1245
|
+
└── vanilla/ createSync, typed factory
|
|
1246
|
+
```
|
|
1247
|
+
|
|
1248
|
+
### Tech stack
|
|
1249
|
+
|
|
1250
|
+
- Runtime: Bun
|
|
1251
|
+
- Language: TypeScript (strict, ESM only)
|
|
1252
|
+
- Build: bunup
|
|
1253
|
+
- Test: `bun:test`
|
|
1254
|
+
- Lint: oxlint
|
|
1255
|
+
- Format: oxfmt
|
|
1256
|
+
- CI: GitHub Actions (Ubuntu + macOS + Windows)
|
|
1257
|
+
|
|
1258
|
+
## License
|
|
1259
|
+
|
|
1260
|
+
MIT
|