@trestleinc/replicate 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +201 -0
- package/LICENSE.package +201 -0
- package/README.md +871 -0
- package/dist/client/collection.d.ts +94 -0
- package/dist/client/index.d.ts +18 -0
- package/dist/client/logger.d.ts +3 -0
- package/dist/client/storage.d.ts +143 -0
- package/dist/component/_generated/api.js +5 -0
- package/dist/component/_generated/server.js +9 -0
- package/dist/component/convex.config.d.ts +2 -0
- package/dist/component/convex.config.js +3 -0
- package/dist/component/public.d.ts +99 -0
- package/dist/component/public.js +135 -0
- package/dist/component/schema.d.ts +22 -0
- package/dist/component/schema.js +22 -0
- package/dist/index.js +375 -0
- package/dist/server/index.d.ts +17 -0
- package/dist/server/replication.d.ts +122 -0
- package/dist/server/schema.d.ts +73 -0
- package/dist/server/ssr.d.ts +79 -0
- package/dist/server.js +96 -0
- package/dist/ssr.js +19 -0
- package/package.json +108 -0
- package/src/client/collection.ts +550 -0
- package/src/client/index.ts +31 -0
- package/src/client/logger.ts +31 -0
- package/src/client/storage.ts +206 -0
- package/src/component/_generated/api.d.ts +95 -0
- package/src/component/_generated/api.js +23 -0
- package/src/component/_generated/dataModel.d.ts +60 -0
- package/src/component/_generated/server.d.ts +149 -0
- package/src/component/_generated/server.js +90 -0
- package/src/component/convex.config.ts +3 -0
- package/src/component/public.ts +212 -0
- package/src/component/schema.ts +16 -0
- package/src/server/index.ts +26 -0
- package/src/server/replication.ts +244 -0
- package/src/server/schema.ts +97 -0
- package/src/server/ssr.ts +106 -0
package/README.md
ADDED
|
@@ -0,0 +1,871 @@
|
|
|
1
|
+
# Convex Replicate
|
|
2
|
+
|
|
3
|
+
**Offline-first sync library using Yjs CRDTs and Convex for real-time data synchronization.**
|
|
4
|
+
|
|
5
|
+
Convex Replicate provides a dual-storage architecture for building offline-capable applications with automatic conflict resolution. It combines Yjs CRDTs (96% smaller than Automerge, no WASM) with TanStack's offline transaction system and Convex's reactive backend for real-time synchronization and efficient querying.
|
|
6
|
+
|
|
7
|
+
## Features
|
|
8
|
+
|
|
9
|
+
- **Offline-first** - Works without internet, syncs when reconnected
|
|
10
|
+
- **Yjs CRDTs** - Automatic conflict-free replication with Yjs (96% smaller than Automerge, no WASM)
|
|
11
|
+
- **TanStack offline-transactions** - Proven outbox pattern for reliable offline sync
|
|
12
|
+
- **Real-time sync** - Convex WebSocket-based synchronization
|
|
13
|
+
- **TanStack DB integration** - Reactive state management for React and Svelte
|
|
14
|
+
- **Dual-storage pattern** - CRDT layer for conflict resolution + main tables for queries
|
|
15
|
+
- **Event sourcing** - Append-only event log preserves complete history
|
|
16
|
+
- **Type-safe** - Full TypeScript support
|
|
17
|
+
- **Multi-tab sync** - Changes sync instantly across browser tabs via TanStack coordination
|
|
18
|
+
- **SSR support** - Server-side rendering with data preloading
|
|
19
|
+
- **Network resilience** - Automatic retry with exponential backoff
|
|
20
|
+
- **Component-based** - Convex component for plug-and-play CRDT storage
|
|
21
|
+
- **React Native compatible** - No WASM dependency, works on mobile
|
|
22
|
+
|
|
23
|
+
## Architecture
|
|
24
|
+
|
|
25
|
+
### Data Flow: Real-Time Sync
|
|
26
|
+
|
|
27
|
+
```mermaid
|
|
28
|
+
sequenceDiagram
|
|
29
|
+
participant User
|
|
30
|
+
participant UI as React/Svelte Component
|
|
31
|
+
participant TDB as TanStack DB
|
|
32
|
+
participant Yjs as Yjs CRDT
|
|
33
|
+
participant Offline as Offline Executor
|
|
34
|
+
participant Convex as Convex Component
|
|
35
|
+
participant Table as Main Table
|
|
36
|
+
|
|
37
|
+
User->>UI: Create/Update Task
|
|
38
|
+
UI->>TDB: collection.insert/update
|
|
39
|
+
TDB->>Yjs: Update Yjs CRDT
|
|
40
|
+
Yjs-->>TDB: Notify change
|
|
41
|
+
TDB-->>UI: Re-render (optimistic)
|
|
42
|
+
|
|
43
|
+
Note over Offline: Automatic retry with backoff
|
|
44
|
+
Offline->>Yjs: Get CRDT delta
|
|
45
|
+
Offline->>Convex: insertDocument/updateDocument mutation
|
|
46
|
+
Convex->>Component: Append delta to event log
|
|
47
|
+
Convex->>Table: Insert/Update materialized doc
|
|
48
|
+
|
|
49
|
+
Note over Convex,Table: Change detected
|
|
50
|
+
Table-->>UI: Subscription update
|
|
51
|
+
UI-->>User: Re-render with synced data
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
### Dual-Storage Architecture
|
|
55
|
+
|
|
56
|
+
```mermaid
|
|
57
|
+
graph LR
|
|
58
|
+
Client[Client<br/>Yjs CRDT]
|
|
59
|
+
Component[Component Storage<br/>Event Log<br/>CRDT Deltas]
|
|
60
|
+
MainTable[Main Application Table<br/>Materialized Docs<br/>Efficient Queries]
|
|
61
|
+
|
|
62
|
+
Client -->|insertDocument/updateDocument| Component
|
|
63
|
+
Component -->|also writes to| MainTable
|
|
64
|
+
MainTable -->|subscription| Client
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
**Why both?**
|
|
68
|
+
- **Component Storage (Event Log)**: Append-only CRDT deltas, complete history, conflict resolution
|
|
69
|
+
- **Main Tables (Read Model)**: Current state, efficient server-side queries, indexes, and reactive subscriptions
|
|
70
|
+
- Similar to CQRS/Event Sourcing: component = event log, main table = materialized view
|
|
71
|
+
|
|
72
|
+
## Installation
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
# Using pnpm (recommended)
|
|
76
|
+
pnpm add @trestleinc/replicate
|
|
77
|
+
|
|
78
|
+
# Using npm (v7+)
|
|
79
|
+
npm install @trestleinc/replicate
|
|
80
|
+
|
|
81
|
+
# Using Bun
|
|
82
|
+
bun add @trestleinc/replicate
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
## Quick Start
|
|
86
|
+
|
|
87
|
+
### Step 1: Install the Convex Component
|
|
88
|
+
|
|
89
|
+
Add the replicate component to your Convex app configuration:
|
|
90
|
+
|
|
91
|
+
```typescript
|
|
92
|
+
// convex/convex.config.ts
|
|
93
|
+
import { defineApp } from 'convex/server';
|
|
94
|
+
import replicate from '@trestleinc/replicate/convex.config';
|
|
95
|
+
|
|
96
|
+
const app = defineApp();
|
|
97
|
+
app.use(replicate);
|
|
98
|
+
|
|
99
|
+
export default app;
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
### Step 2: Define Your Schema
|
|
103
|
+
|
|
104
|
+
Use the `replicatedTable` helper to automatically inject required fields:
|
|
105
|
+
|
|
106
|
+
```typescript
|
|
107
|
+
// convex/schema.ts
|
|
108
|
+
import { defineSchema } from 'convex/server';
|
|
109
|
+
import { v } from 'convex/values';
|
|
110
|
+
import { replicatedTable } from '@trestleinc/replicate/server';
|
|
111
|
+
|
|
112
|
+
export default defineSchema({
|
|
113
|
+
tasks: replicatedTable(
|
|
114
|
+
{
|
|
115
|
+
// Your application fields only!
|
|
116
|
+
// version and timestamp are automatically injected by replicatedTable
|
|
117
|
+
id: v.string(),
|
|
118
|
+
text: v.string(),
|
|
119
|
+
isCompleted: v.boolean(),
|
|
120
|
+
},
|
|
121
|
+
(table) => table
|
|
122
|
+
.index('by_user_id', ['id']) // Required for document lookups
|
|
123
|
+
.index('by_timestamp', ['timestamp']) // Required for incremental sync
|
|
124
|
+
),
|
|
125
|
+
});
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
**What `replicatedTable` does:**
|
|
129
|
+
- Automatically injects `version: v.number()` (for CRDT versioning)
|
|
130
|
+
- Automatically injects `timestamp: v.number()` (for incremental sync)
|
|
131
|
+
- You only define your business logic fields
|
|
132
|
+
|
|
133
|
+
**Required indexes:**
|
|
134
|
+
- `by_user_id` on `['id']` - Enables fast document lookups during updates
|
|
135
|
+
- `by_timestamp` on `['timestamp']` - Enables efficient incremental synchronization
|
|
136
|
+
|
|
137
|
+
### Step 3: Create Replication Functions
|
|
138
|
+
|
|
139
|
+
Create functions that use replication helpers for dual-storage pattern:
|
|
140
|
+
|
|
141
|
+
```typescript
|
|
142
|
+
// convex/tasks.ts
|
|
143
|
+
import { mutation, query } from './_generated/server';
|
|
144
|
+
import { components } from './_generated/api';
|
|
145
|
+
import { v } from 'convex/values';
|
|
146
|
+
import {
|
|
147
|
+
insertDocumentHelper,
|
|
148
|
+
updateDocumentHelper,
|
|
149
|
+
deleteDocumentHelper,
|
|
150
|
+
} from '@trestleinc/replicate/server'; // IMPORTANT: Use /server for Convex functions!
|
|
151
|
+
|
|
152
|
+
/**
|
|
153
|
+
* TanStack DB endpoints - called by convexCollectionOptions
|
|
154
|
+
* These receive Yjs CRDT deltas from client and write to both:
|
|
155
|
+
* 1. Component storage (Yjs CRDT deltas in event log)
|
|
156
|
+
* 2. Main table (materialized docs for efficient queries)
|
|
157
|
+
*/
|
|
158
|
+
|
|
159
|
+
export const insertDocument = mutation({
|
|
160
|
+
args: {
|
|
161
|
+
collectionName: v.string(),
|
|
162
|
+
documentId: v.string(),
|
|
163
|
+
crdtBytes: v.bytes(),
|
|
164
|
+
materializedDoc: v.any(),
|
|
165
|
+
version: v.number(),
|
|
166
|
+
},
|
|
167
|
+
handler: async (ctx, args) => {
|
|
168
|
+
return await insertDocumentHelper(ctx, components, 'tasks', {
|
|
169
|
+
id: args.documentId,
|
|
170
|
+
crdtBytes: args.crdtBytes,
|
|
171
|
+
materializedDoc: args.materializedDoc,
|
|
172
|
+
version: args.version,
|
|
173
|
+
});
|
|
174
|
+
},
|
|
175
|
+
});
|
|
176
|
+
|
|
177
|
+
export const updateDocument = mutation({
|
|
178
|
+
args: {
|
|
179
|
+
collectionName: v.string(),
|
|
180
|
+
documentId: v.string(),
|
|
181
|
+
crdtBytes: v.bytes(),
|
|
182
|
+
materializedDoc: v.any(),
|
|
183
|
+
version: v.number(),
|
|
184
|
+
},
|
|
185
|
+
handler: async (ctx, args) => {
|
|
186
|
+
return await updateDocumentHelper(ctx, components, 'tasks', {
|
|
187
|
+
id: args.documentId,
|
|
188
|
+
crdtBytes: args.crdtBytes,
|
|
189
|
+
materializedDoc: args.materializedDoc,
|
|
190
|
+
version: args.version,
|
|
191
|
+
});
|
|
192
|
+
},
|
|
193
|
+
});
|
|
194
|
+
|
|
195
|
+
export const deleteDocument = mutation({
|
|
196
|
+
args: {
|
|
197
|
+
collectionName: v.string(),
|
|
198
|
+
documentId: v.string(),
|
|
199
|
+
crdtBytes: v.bytes(),
|
|
200
|
+
version: v.number(),
|
|
201
|
+
},
|
|
202
|
+
handler: async (ctx, args) => {
|
|
203
|
+
return await deleteDocumentHelper(ctx, components, 'tasks', {
|
|
204
|
+
id: args.documentId,
|
|
205
|
+
crdtBytes: args.crdtBytes,
|
|
206
|
+
version: args.version,
|
|
207
|
+
});
|
|
208
|
+
},
|
|
209
|
+
});
|
|
210
|
+
|
|
211
|
+
/**
|
|
212
|
+
* Stream endpoint for real-time subscriptions
|
|
213
|
+
* Returns all active items (hard deletes are physically removed from table)
|
|
214
|
+
*/
|
|
215
|
+
export const stream = query({
|
|
216
|
+
handler: async (ctx) => {
|
|
217
|
+
return await ctx.db.query('tasks').collect();
|
|
218
|
+
},
|
|
219
|
+
});
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
### Step 4: Create a Custom Hook
|
|
223
|
+
|
|
224
|
+
Create a hook that wraps TanStack DB with Convex collection options:
|
|
225
|
+
|
|
226
|
+
```typescript
|
|
227
|
+
// src/useTasks.ts
|
|
228
|
+
import { createCollection } from '@tanstack/react-db';
|
|
229
|
+
import {
|
|
230
|
+
convexCollectionOptions,
|
|
231
|
+
createConvexCollection,
|
|
232
|
+
type ConvexCollection,
|
|
233
|
+
} from '@trestleinc/replicate/client';
|
|
234
|
+
import { api } from '../convex/_generated/api';
|
|
235
|
+
import { convexClient } from './router';
|
|
236
|
+
import { useMemo } from 'react';
|
|
237
|
+
|
|
238
|
+
export interface Task {
|
|
239
|
+
id: string;
|
|
240
|
+
text: string;
|
|
241
|
+
isCompleted: boolean;
|
|
242
|
+
}
|
|
243
|
+
|
|
244
|
+
// Module-level singleton to prevent multiple collection instances
|
|
245
|
+
// This ensures only one sync process runs, even across component remounts
|
|
246
|
+
let tasksCollection: ConvexCollection<Task>;
|
|
247
|
+
|
|
248
|
+
export function useTasks(initialData?: ReadonlyArray<Task>) {
|
|
249
|
+
return useMemo(() => {
|
|
250
|
+
if (!tasksCollection) {
|
|
251
|
+
// Step 1: Create raw TanStack DB collection with ALL config
|
|
252
|
+
const rawCollection = createCollection(
|
|
253
|
+
convexCollectionOptions<Task>({
|
|
254
|
+
convexClient,
|
|
255
|
+
api: {
|
|
256
|
+
stream: api.tasks.stream,
|
|
257
|
+
insertDocument: api.tasks.insertDocument,
|
|
258
|
+
updateDocument: api.tasks.updateDocument,
|
|
259
|
+
deleteDocument: api.tasks.deleteDocument,
|
|
260
|
+
},
|
|
261
|
+
collectionName: 'tasks',
|
|
262
|
+
getKey: (task) => task.id,
|
|
263
|
+
initialData,
|
|
264
|
+
})
|
|
265
|
+
);
|
|
266
|
+
|
|
267
|
+
// Step 2: Wrap with Convex offline support (Yjs + TanStack)
|
|
268
|
+
// Config is automatically extracted from rawCollection
|
|
269
|
+
tasksCollection = createConvexCollection(rawCollection);
|
|
270
|
+
}
|
|
271
|
+
return tasksCollection;
|
|
272
|
+
}, [initialData]);
|
|
273
|
+
}
|
|
274
|
+
```
|
|
275
|
+
|
|
276
|
+
### Step 5: Use in Components
|
|
277
|
+
|
|
278
|
+
```typescript
|
|
279
|
+
// src/routes/index.tsx
|
|
280
|
+
import { useLiveQuery } from '@tanstack/react-db';
|
|
281
|
+
import { useTasks } from '../useTasks';
|
|
282
|
+
|
|
283
|
+
export function TaskList() {
|
|
284
|
+
const collection = useTasks();
|
|
285
|
+
const { data: tasks, isLoading, isError } = useLiveQuery(collection);
|
|
286
|
+
|
|
287
|
+
const handleCreate = () => {
|
|
288
|
+
collection.insert({
|
|
289
|
+
id: crypto.randomUUID(),
|
|
290
|
+
text: 'New task',
|
|
291
|
+
isCompleted: false,
|
|
292
|
+
});
|
|
293
|
+
};
|
|
294
|
+
|
|
295
|
+
const handleUpdate = (id: string, isCompleted: boolean) => {
|
|
296
|
+
collection.update(id, (draft: Task) => {
|
|
297
|
+
draft.isCompleted = !isCompleted;
|
|
298
|
+
});
|
|
299
|
+
};
|
|
300
|
+
|
|
301
|
+
const handleDelete = (id: string) => {
|
|
302
|
+
// Hard delete - physically removes from main table
|
|
303
|
+
collection.delete(id);
|
|
304
|
+
};
|
|
305
|
+
|
|
306
|
+
if (isError) {
|
|
307
|
+
return <div>Error loading tasks. Please refresh.</div>;
|
|
308
|
+
}
|
|
309
|
+
|
|
310
|
+
if (isLoading) {
|
|
311
|
+
return <div>Loading tasks...</div>;
|
|
312
|
+
}
|
|
313
|
+
|
|
314
|
+
return (
|
|
315
|
+
<div>
|
|
316
|
+
<button onClick={handleCreate}>Add Task</button>
|
|
317
|
+
|
|
318
|
+
{tasks.map((task) => (
|
|
319
|
+
<div key={task.id}>
|
|
320
|
+
<input
|
|
321
|
+
type="checkbox"
|
|
322
|
+
checked={task.isCompleted}
|
|
323
|
+
onChange={() => handleUpdate(task.id, task.isCompleted)}
|
|
324
|
+
/>
|
|
325
|
+
<span>{task.text}</span>
|
|
326
|
+
<button onClick={() => handleDelete(task.id)}>Delete</button>
|
|
327
|
+
</div>
|
|
328
|
+
))}
|
|
329
|
+
</div>
|
|
330
|
+
);
|
|
331
|
+
}
|
|
332
|
+
```
|
|
333
|
+
|
|
334
|
+
## Delete Pattern: Hard Delete with Event History (v0.3.0+)
|
|
335
|
+
|
|
336
|
+
Convex Replicate uses **hard deletes** where items are physically removed from the main table, while the internal component preserves complete event history.
|
|
337
|
+
|
|
338
|
+
**Why hard delete?**
|
|
339
|
+
- Clean main table (no filtering required)
|
|
340
|
+
- Standard TanStack DB operations
|
|
341
|
+
- Complete audit trail preserved in component event log
|
|
342
|
+
- Proper CRDT conflict resolution maintained
|
|
343
|
+
- Foundation for future recovery features
|
|
344
|
+
|
|
345
|
+
**Implementation:**
|
|
346
|
+
|
|
347
|
+
```typescript
|
|
348
|
+
// Delete handler (uses collection.delete)
|
|
349
|
+
const handleDelete = (id: string) => {
|
|
350
|
+
collection.delete(id); // Hard delete - physically removes from main table
|
|
351
|
+
};
|
|
352
|
+
|
|
353
|
+
// UI usage - no filtering needed!
|
|
354
|
+
const { data: tasks } = useLiveQuery(collection);
|
|
355
|
+
|
|
356
|
+
// SSR loader - no filtering needed!
|
|
357
|
+
export const Route = createFileRoute('/')({
|
|
358
|
+
loader: async () => {
|
|
359
|
+
const tasks = await httpClient.query(api.tasks.stream);
|
|
360
|
+
return { tasks };
|
|
361
|
+
},
|
|
362
|
+
});
|
|
363
|
+
```
|
|
364
|
+
|
|
365
|
+
**How it works:**
|
|
366
|
+
1. Client calls `collection.delete(id)`
|
|
367
|
+
2. `onDelete` handler captures Yjs deletion delta
|
|
368
|
+
3. Delta appended to component event log (history preserved)
|
|
369
|
+
4. Main table: document physically removed
|
|
370
|
+
5. Other clients notified and item removed locally
|
|
371
|
+
|
|
372
|
+
**Server-side:** Returns only active items (deleted items are physically removed):
|
|
373
|
+
|
|
374
|
+
```typescript
|
|
375
|
+
// convex/tasks.ts
|
|
376
|
+
export const stream = query({
|
|
377
|
+
handler: async (ctx) => {
|
|
378
|
+
return await ctx.db.query('tasks').collect();
|
|
379
|
+
},
|
|
380
|
+
});
|
|
381
|
+
```
|
|
382
|
+
|
|
383
|
+
**Dual Storage Architecture:**
|
|
384
|
+
- **Component Storage**: Append-only event log with complete history (including deletions)
|
|
385
|
+
- **Main Table**: Current state only (deleted items removed)
|
|
386
|
+
|
|
387
|
+
## Advanced Usage
|
|
388
|
+
|
|
389
|
+
### Server-Side Rendering (SSR)
|
|
390
|
+
|
|
391
|
+
Preload data on the server for instant page loads:
|
|
392
|
+
|
|
393
|
+
**Step 1: Create an SSR-friendly query**
|
|
394
|
+
|
|
395
|
+
```typescript
|
|
396
|
+
// convex/tasks.ts
|
|
397
|
+
export const getTasks = query({
|
|
398
|
+
handler: async (ctx) => {
|
|
399
|
+
return await ctx.db.query('tasks').collect();
|
|
400
|
+
},
|
|
401
|
+
});
|
|
402
|
+
```
|
|
403
|
+
|
|
404
|
+
**Step 2: Load data in your route loader**
|
|
405
|
+
|
|
406
|
+
```typescript
|
|
407
|
+
// src/routes/index.tsx
|
|
408
|
+
import { createFileRoute } from '@tanstack/react-router';
|
|
409
|
+
import { ConvexHttpClient } from 'convex/browser';
|
|
410
|
+
import { api } from '../convex/_generated/api';
|
|
411
|
+
import type { Task } from '../useTasks';
|
|
412
|
+
|
|
413
|
+
const httpClient = new ConvexHttpClient(import.meta.env.VITE_CONVEX_URL);
|
|
414
|
+
|
|
415
|
+
export const Route = createFileRoute('/')({
|
|
416
|
+
loader: async () => {
|
|
417
|
+
const tasks = await httpClient.query(api.tasks.getTasks);
|
|
418
|
+
return { tasks };
|
|
419
|
+
},
|
|
420
|
+
});
|
|
421
|
+
|
|
422
|
+
function TasksPage() {
|
|
423
|
+
const { tasks: initialTasks } = Route.useLoaderData();
|
|
424
|
+
|
|
425
|
+
// Pass initialData to your hook
|
|
426
|
+
const collection = useTasks(initialTasks);
|
|
427
|
+
const { data: tasks } = useLiveQuery(collection);
|
|
428
|
+
|
|
429
|
+
// No loading state on first render!
|
|
430
|
+
return <TaskList tasks={tasks} />;
|
|
431
|
+
}
|
|
432
|
+
```
|
|
433
|
+
|
|
434
|
+
### Direct Component Usage (Advanced)
|
|
435
|
+
|
|
436
|
+
> **WARNING:** Using `ReplicateStorage` directly only writes to the component CRDT storage layer. It does NOT implement the dual-storage pattern (no writes to main table), which means:
|
|
437
|
+
> - You cannot query this data efficiently in Convex
|
|
438
|
+
> - You lose the benefits of reactive subscriptions on materialized docs
|
|
439
|
+
> - You'll need to manually handle materialization
|
|
440
|
+
>
|
|
441
|
+
> **Recommended:** Use the replication helpers (`insertDocumentHelper`, etc.) shown in Step 3 for the full dual-storage pattern.
|
|
442
|
+
|
|
443
|
+
For advanced use cases where you need direct component access:
|
|
444
|
+
|
|
445
|
+
```typescript
|
|
446
|
+
// convex/tasks.ts
|
|
447
|
+
import { ReplicateStorage } from '@trestleinc/replicate/client';
|
|
448
|
+
import { mutation, query } from './_generated/server';
|
|
449
|
+
import { components } from './_generated/api';
|
|
450
|
+
import { v } from 'convex/values';
|
|
451
|
+
|
|
452
|
+
interface Task {
|
|
453
|
+
id: string;
|
|
454
|
+
text: string;
|
|
455
|
+
isCompleted: boolean;
|
|
456
|
+
}
|
|
457
|
+
|
|
458
|
+
const tasksStorage = new ReplicateStorage<Task>(components.replicate, 'tasks');
|
|
459
|
+
|
|
460
|
+
export const insertTask = mutation({
|
|
461
|
+
args: {
|
|
462
|
+
id: v.string(),
|
|
463
|
+
crdtBytes: v.bytes(),
|
|
464
|
+
version: v.number(),
|
|
465
|
+
},
|
|
466
|
+
handler: async (ctx, args) => {
|
|
467
|
+
return await tasksStorage.insertDocument(
|
|
468
|
+
ctx,
|
|
469
|
+
args.id,
|
|
470
|
+
args.crdtBytes,
|
|
471
|
+
args.version
|
|
472
|
+
);
|
|
473
|
+
},
|
|
474
|
+
});
|
|
475
|
+
|
|
476
|
+
export const updateTask = mutation({
|
|
477
|
+
args: {
|
|
478
|
+
id: v.string(),
|
|
479
|
+
crdtBytes: v.bytes(),
|
|
480
|
+
version: v.number(),
|
|
481
|
+
},
|
|
482
|
+
handler: async (ctx, args) => {
|
|
483
|
+
return await tasksStorage.updateDocument(
|
|
484
|
+
ctx,
|
|
485
|
+
args.id,
|
|
486
|
+
args.crdtBytes,
|
|
487
|
+
args.version
|
|
488
|
+
);
|
|
489
|
+
},
|
|
490
|
+
});
|
|
491
|
+
|
|
492
|
+
export const streamChanges = query({
|
|
493
|
+
args: {
|
|
494
|
+
checkpoint: v.object({ lastModified: v.number() }),
|
|
495
|
+
limit: v.optional(v.number()),
|
|
496
|
+
},
|
|
497
|
+
handler: async (ctx, args) => {
|
|
498
|
+
return await tasksStorage.stream(ctx, args.checkpoint, args.limit);
|
|
499
|
+
},
|
|
500
|
+
});
|
|
501
|
+
```
|
|
502
|
+
|
|
503
|
+
### Logging Configuration
|
|
504
|
+
|
|
505
|
+
Configure logging for debugging and development using LogTape:
|
|
506
|
+
|
|
507
|
+
```typescript
|
|
508
|
+
// src/routes/__root.tsx or app entry point
|
|
509
|
+
import { configure, getConsoleSink } from '@logtape/logtape';
|
|
510
|
+
|
|
511
|
+
await configure({
|
|
512
|
+
sinks: { console: getConsoleSink() },
|
|
513
|
+
loggers: [
|
|
514
|
+
{
|
|
515
|
+
category: ['convex-replicate'],
|
|
516
|
+
lowestLevel: 'debug', // 'debug' | 'info' | 'warn' | 'error'
|
|
517
|
+
sinks: ['console']
|
|
518
|
+
}
|
|
519
|
+
],
|
|
520
|
+
});
|
|
521
|
+
```
|
|
522
|
+
|
|
523
|
+
Get a logger instance for custom logging:
|
|
524
|
+
|
|
525
|
+
```typescript
|
|
526
|
+
import { getLogger } from '@trestleinc/replicate/client';
|
|
527
|
+
|
|
528
|
+
const logger = getLogger(['my-module']); // Accepts string or string array
|
|
529
|
+
|
|
530
|
+
logger.info('Operation started', { userId: '123' });
|
|
531
|
+
logger.warn('Something unexpected', { reason: 'timeout' });
|
|
532
|
+
logger.error('Operation failed', { error });
|
|
533
|
+
```
|
|
534
|
+
|
|
535
|
+
## API Reference
|
|
536
|
+
|
|
537
|
+
### Client-Side (`@trestleinc/replicate/client`)
|
|
538
|
+
|
|
539
|
+
#### `convexCollectionOptions<T>(config)`
|
|
540
|
+
|
|
541
|
+
Creates collection options for TanStack DB with Yjs CRDT integration.
|
|
542
|
+
|
|
543
|
+
**Config:**
|
|
544
|
+
```typescript
|
|
545
|
+
interface ConvexCollectionOptionsConfig<T> {
|
|
546
|
+
convexClient: ConvexClient;
|
|
547
|
+
api: {
|
|
548
|
+
stream: FunctionReference; // Real-time subscription endpoint
|
|
549
|
+
insertDocument: FunctionReference; // Insert mutation
|
|
550
|
+
updateDocument: FunctionReference; // Update mutation
|
|
551
|
+
deleteDocument: FunctionReference; // Delete mutation
|
|
552
|
+
};
|
|
553
|
+
collectionName: string;
|
|
554
|
+
getKey: (item: T) => string | number;
|
|
555
|
+
initialData?: ReadonlyArray<T>;
|
|
556
|
+
}
|
|
557
|
+
```
|
|
558
|
+
|
|
559
|
+
**Returns:** Collection options for `createCollection()`
|
|
560
|
+
|
|
561
|
+
**Example:**
|
|
562
|
+
```typescript
|
|
563
|
+
const rawCollection = createCollection(
|
|
564
|
+
convexCollectionOptions<Task>({
|
|
565
|
+
convexClient,
|
|
566
|
+
api: {
|
|
567
|
+
stream: api.tasks.stream,
|
|
568
|
+
insertDocument: api.tasks.insertDocument,
|
|
569
|
+
updateDocument: api.tasks.updateDocument,
|
|
570
|
+
deleteDocument: api.tasks.deleteDocument,
|
|
571
|
+
},
|
|
572
|
+
collectionName: 'tasks',
|
|
573
|
+
getKey: (task) => task.id,
|
|
574
|
+
initialData,
|
|
575
|
+
})
|
|
576
|
+
);
|
|
577
|
+
|
|
578
|
+
const collection = createConvexCollection(rawCollection);
|
|
579
|
+
```
|
|
580
|
+
|
|
581
|
+
#### `createConvexCollection<T>(rawCollection)`
|
|
582
|
+
|
|
583
|
+
Wraps a TanStack DB collection with offline support (Yjs + TanStack offline-transactions).
|
|
584
|
+
|
|
585
|
+
**Parameters:**
|
|
586
|
+
- `rawCollection` - Collection created with `createCollection(convexCollectionOptions(...))`
|
|
587
|
+
|
|
588
|
+
**Returns:** `ConvexCollection<T>` (just a type alias for `Collection<T>`)
|
|
589
|
+
|
|
590
|
+
**Example:**
|
|
591
|
+
```typescript
|
|
592
|
+
const collection = createConvexCollection(rawCollection);
|
|
593
|
+
|
|
594
|
+
// Use standard TanStack DB operations
|
|
595
|
+
collection.insert({ id: '1', text: 'Task 1', isCompleted: false });
|
|
596
|
+
collection.update('1', (draft) => { draft.isCompleted = true });
|
|
597
|
+
collection.delete('1');
|
|
598
|
+
```
|
|
599
|
+
|
|
600
|
+
#### `ReplicateStorage<TDocument>`
|
|
601
|
+
|
|
602
|
+
Type-safe API for direct component access (advanced).
|
|
603
|
+
|
|
604
|
+
**Constructor:**
|
|
605
|
+
```typescript
|
|
606
|
+
new ReplicateStorage<TDocument>(component, collectionName)
|
|
607
|
+
```
|
|
608
|
+
|
|
609
|
+
**Methods:**
|
|
610
|
+
|
|
611
|
+
##### `insertDocument(ctx, documentId, crdtBytes, version)`
|
|
612
|
+
Insert a new document with Yjs CRDT bytes.
|
|
613
|
+
|
|
614
|
+
**Parameters:**
|
|
615
|
+
- `ctx` - Convex mutation context
|
|
616
|
+
- `documentId` - Unique document identifier
|
|
617
|
+
- `crdtBytes` - ArrayBuffer containing Yjs CRDT bytes
|
|
618
|
+
- `version` - CRDT version number
|
|
619
|
+
|
|
620
|
+
**Returns:** `Promise<{ success: boolean }>`
|
|
621
|
+
|
|
622
|
+
##### `updateDocument(ctx, documentId, crdtBytes, version)`
|
|
623
|
+
Update an existing document with Yjs CRDT bytes.
|
|
624
|
+
|
|
625
|
+
**Parameters:**
|
|
626
|
+
- `ctx` - Convex mutation context
|
|
627
|
+
- `documentId` - Unique document identifier
|
|
628
|
+
- `crdtBytes` - ArrayBuffer containing Yjs CRDT bytes
|
|
629
|
+
- `version` - CRDT version number
|
|
630
|
+
|
|
631
|
+
**Returns:** `Promise<{ success: boolean }>`
|
|
632
|
+
|
|
633
|
+
##### `deleteDocument(ctx, documentId, crdtBytes, version)`
|
|
634
|
+
Delete a document (appends deletion delta to event log).
|
|
635
|
+
|
|
636
|
+
**Parameters:**
|
|
637
|
+
- `ctx` - Convex mutation context
|
|
638
|
+
- `documentId` - Unique document identifier
|
|
639
|
+
- `crdtBytes` - ArrayBuffer containing Yjs deletion delta
|
|
640
|
+
- `version` - CRDT version number
|
|
641
|
+
|
|
642
|
+
**Returns:** `Promise<{ success: boolean }>`
|
|
643
|
+
|
|
644
|
+
##### `stream(ctx, checkpoint, limit?)`
|
|
645
|
+
Pull document changes for incremental sync.
|
|
646
|
+
|
|
647
|
+
**Parameters:**
|
|
648
|
+
- `ctx` - Convex query context
|
|
649
|
+
- `checkpoint` - Object with `{ lastModified: number }`
|
|
650
|
+
- `limit` - Optional max changes (default: 100)
|
|
651
|
+
|
|
652
|
+
**Returns:**
|
|
653
|
+
```typescript
|
|
654
|
+
Promise<{
|
|
655
|
+
changes: Array<{
|
|
656
|
+
documentId: string;
|
|
657
|
+
crdtBytes: ArrayBuffer;
|
|
658
|
+
version: number;
|
|
659
|
+
timestamp: number;
|
|
660
|
+
}>;
|
|
661
|
+
checkpoint: { lastModified: number };
|
|
662
|
+
hasMore: boolean;
|
|
663
|
+
}>
|
|
664
|
+
```
|
|
665
|
+
|
|
666
|
+
#### `getLogger(category)`
|
|
667
|
+
|
|
668
|
+
Get a logger instance for custom logging.
|
|
669
|
+
|
|
670
|
+
**Parameters:**
|
|
671
|
+
- `category: string | string[]` - Logger category
|
|
672
|
+
|
|
673
|
+
**Returns:** Logger with `debug()`, `info()`, `warn()`, `error()` methods
|
|
674
|
+
|
|
675
|
+
**Examples:**
|
|
676
|
+
```typescript
|
|
677
|
+
const logger = getLogger('my-module');
|
|
678
|
+
const logger = getLogger(['hooks', 'useTasks']);
|
|
679
|
+
|
|
680
|
+
logger.debug('Task created', { id: taskId });
|
|
681
|
+
```
|
|
682
|
+
|
|
683
|
+
### Server-Side (`@trestleinc/replicate/server`)
|
|
684
|
+
|
|
685
|
+
#### `insertDocumentHelper(ctx, components, tableName, args)`
|
|
686
|
+
|
|
687
|
+
Insert a document into both the CRDT component and the main application table.
|
|
688
|
+
|
|
689
|
+
**Parameters:**
|
|
690
|
+
- `ctx` - Convex mutation context
|
|
691
|
+
- `components` - Generated components from Convex
|
|
692
|
+
- `tableName` - Name of the main application table
|
|
693
|
+
- `args` - `{ id: string; crdtBytes: ArrayBuffer; materializedDoc: any; version: number }`
|
|
694
|
+
|
|
695
|
+
**Returns:** `Promise<{ success: boolean; metadata: {...} }>`
|
|
696
|
+
|
|
697
|
+
#### `updateDocumentHelper(ctx, components, tableName, args)`
|
|
698
|
+
|
|
699
|
+
Update a document in both the CRDT component and the main application table.
|
|
700
|
+
|
|
701
|
+
**Parameters:**
|
|
702
|
+
- `ctx` - Convex mutation context
|
|
703
|
+
- `components` - Generated components from Convex
|
|
704
|
+
- `tableName` - Name of the main application table
|
|
705
|
+
- `args` - `{ id: string; crdtBytes: ArrayBuffer; materializedDoc: any; version: number }`
|
|
706
|
+
|
|
707
|
+
**Returns:** `Promise<{ success: boolean; metadata: {...} }>`
|
|
708
|
+
|
|
709
|
+
#### `deleteDocumentHelper(ctx, components, tableName, args)`
|
|
710
|
+
|
|
711
|
+
Hard delete from main table, append deletion delta to component event log.
|
|
712
|
+
|
|
713
|
+
**Parameters:**
|
|
714
|
+
- `ctx` - Convex mutation context
|
|
715
|
+
- `components` - Generated components from Convex
|
|
716
|
+
- `tableName` - Name of the main application table
|
|
717
|
+
- `args` - `{ id: string; crdtBytes: ArrayBuffer; version: number }`
|
|
718
|
+
|
|
719
|
+
**Returns:** `Promise<{ success: boolean; metadata: {...} }>`
|
|
720
|
+
|
|
721
|
+
#### `streamHelper(ctx, components, tableName, args)`
|
|
722
|
+
|
|
723
|
+
Stream CRDT deltas from component storage for incremental sync.
|
|
724
|
+
|
|
725
|
+
**Parameters:**
|
|
726
|
+
- `ctx` - Convex query context
|
|
727
|
+
- `components` - Generated components from Convex
|
|
728
|
+
- `tableName` - Name of the collection
|
|
729
|
+
- `args` - `{ checkpoint: { lastModified: number }; limit?: number }`
|
|
730
|
+
|
|
731
|
+
**Returns:** `Promise<{ changes: Array<...>; checkpoint: {...}; hasMore: boolean }>`
|
|
732
|
+
|
|
733
|
+
#### `replicatedTable(userFields, applyIndexes?)`
|
|
734
|
+
|
|
735
|
+
Automatically inject replication metadata fields (`version`, `timestamp`).
|
|
736
|
+
|
|
737
|
+
**Parameters:**
|
|
738
|
+
- `userFields` - User's business logic fields
|
|
739
|
+
- `applyIndexes` - Optional callback to add indexes
|
|
740
|
+
|
|
741
|
+
**Returns:** TableDefinition with replication fields injected
|
|
742
|
+
|
|
743
|
+
**Example:**
|
|
744
|
+
```typescript
|
|
745
|
+
tasks: replicatedTable(
|
|
746
|
+
{
|
|
747
|
+
id: v.string(),
|
|
748
|
+
text: v.string(),
|
|
749
|
+
},
|
|
750
|
+
(table) => table
|
|
751
|
+
.index('by_user_id', ['id'])
|
|
752
|
+
.index('by_timestamp', ['timestamp'])
|
|
753
|
+
)
|
|
754
|
+
```
|
|
755
|
+
|
|
756
|
+
### SSR (`@trestleinc/replicate/ssr`)
|
|
757
|
+
|
|
758
|
+
#### `loadCollection<T>(httpClient, config)`
|
|
759
|
+
|
|
760
|
+
Load collection data during SSR for instant page loads.
|
|
761
|
+
|
|
762
|
+
**Note:** This function is deprecated. For most SSR use cases, create a dedicated query that reads from your main table.
|
|
763
|
+
|
|
764
|
+
**Parameters:**
|
|
765
|
+
- `httpClient` - ConvexHttpClient instance
|
|
766
|
+
- `config` - `{ api: CollectionAPI; collection: string; limit?: number }`
|
|
767
|
+
|
|
768
|
+
**Returns:** `Promise<ReadonlyArray<T>>`
|
|
769
|
+
|
|
770
|
+
## Performance
|
|
771
|
+
|
|
772
|
+
### Storage Performance
|
|
773
|
+
|
|
774
|
+
- **IndexedDB** via TanStack DB provides efficient local storage
|
|
775
|
+
- **Yjs** CRDT operations are extremely fast (96% smaller than Automerge)
|
|
776
|
+
- **TanStack offline-transactions** provides batching and retry logic
|
|
777
|
+
- **Indexed queries** in Convex for fast incremental sync
|
|
778
|
+
|
|
779
|
+
### Sync Performance
|
|
780
|
+
|
|
781
|
+
- **Real-time updates** - WebSocket-based change notifications
|
|
782
|
+
- **Delta encoding** - Only send what changed (< 1KB per change vs 100KB+ full state)
|
|
783
|
+
- **Event sourcing** - Append-only writes, no update conflicts
|
|
784
|
+
- **Optimistic UI** - Instant updates without waiting for server
|
|
785
|
+
|
|
786
|
+
### Multi-Tab Sync
|
|
787
|
+
|
|
788
|
+
- **TanStack coordination** - Built-in multi-tab sync via BroadcastChannel
|
|
789
|
+
- **Yjs shared state** - Single source of truth per browser
|
|
790
|
+
- **Offline executor** - Only one tab runs sync operations
|
|
791
|
+
|
|
792
|
+
## Offline Behavior
|
|
793
|
+
|
|
794
|
+
### How It Works
|
|
795
|
+
|
|
796
|
+
- **Writes** - Queue locally in Yjs CRDT, sync when online via TanStack outbox
|
|
797
|
+
- **Reads** - Always work from local TanStack DB cache (instant!)
|
|
798
|
+
- **UI** - Fully functional with optimistic updates
|
|
799
|
+
- **Conflicts** - Auto-resolved by Yjs CRDTs (conflict-free!)
|
|
800
|
+
|
|
801
|
+
### Network Resilience
|
|
802
|
+
|
|
803
|
+
- Automatic retry with exponential backoff
|
|
804
|
+
- Network error detection (fetch errors, connection issues)
|
|
805
|
+
- Queue changes while offline
|
|
806
|
+
- Graceful degradation
|
|
807
|
+
|
|
808
|
+
## Examples
|
|
809
|
+
|
|
810
|
+
Complete working example: `examples/tanstack-start/`
|
|
811
|
+
|
|
812
|
+
**Files to explore:**
|
|
813
|
+
- `src/useTasks.ts` - Hook with TanStack DB integration
|
|
814
|
+
- `src/routes/index.tsx` - Component usage with SSR
|
|
815
|
+
- `src/routes/__root.tsx` - Logging configuration
|
|
816
|
+
- `convex/tasks.ts` - Replication functions using dual-storage helpers
|
|
817
|
+
- `convex/schema.ts` - Schema with `replicatedTable` helper
|
|
818
|
+
|
|
819
|
+
**Run the example:**
|
|
820
|
+
```bash
|
|
821
|
+
cd examples/tanstack-start
|
|
822
|
+
pnpm install
|
|
823
|
+
pnpm run dev
|
|
824
|
+
```
|
|
825
|
+
|
|
826
|
+
## Development
|
|
827
|
+
|
|
828
|
+
### Building
|
|
829
|
+
|
|
830
|
+
```bash
|
|
831
|
+
pnpm run build # Build package using Rslib
|
|
832
|
+
pnpm run clean # Remove build artifacts
|
|
833
|
+
pnpm run typecheck # Type check
|
|
834
|
+
```
|
|
835
|
+
|
|
836
|
+
### Code Quality
|
|
837
|
+
|
|
838
|
+
```bash
|
|
839
|
+
pnpm run check # Lint + format check (dry run)
|
|
840
|
+
pnpm run check:fix # Auto-fix all issues (run before committing)
|
|
841
|
+
pnpm run lint # Lint only
|
|
842
|
+
pnpm run lint:fix # Auto-fix lint issues
|
|
843
|
+
pnpm run format # Format only
|
|
844
|
+
pnpm run format:check # Check formatting
|
|
845
|
+
```
|
|
846
|
+
|
|
847
|
+
### Running Example
|
|
848
|
+
|
|
849
|
+
```bash
|
|
850
|
+
pnpm run dev:example # Start example app + Convex dev environment
|
|
851
|
+
```
|
|
852
|
+
|
|
853
|
+
## Roadmap
|
|
854
|
+
|
|
855
|
+
- [ ] Partial sync (sync subset of collection)
|
|
856
|
+
- [ ] Delta sync (only sync changed fields)
|
|
857
|
+
- [ ] Encryption at rest
|
|
858
|
+
- [ ] Attachment support (files, images)
|
|
859
|
+
- [x] React Native support (works with Yjs v0.3.0+)
|
|
860
|
+
- [ ] Advanced Yjs features (rich text editing, shared cursors)
|
|
861
|
+
- [ ] Recovery features (restore deleted items from event log)
|
|
862
|
+
|
|
863
|
+
## Contributing
|
|
864
|
+
|
|
865
|
+
Contributions welcome! Please see `CLAUDE.md` for coding standards.
|
|
866
|
+
|
|
867
|
+
## License
|
|
868
|
+
|
|
869
|
+
Apache-2.0 License - see [LICENSE](./LICENSE) file for details.
|
|
870
|
+
|
|
871
|
+
Copyright 2025 Trestle Inc
|