@terreno/mcp 0.31.0 → 56.0.0-beta.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/bootstrap.d.ts.map +1 -1
- package/dist/bootstrap.js +8 -9
- package/dist/bootstrap.js.map +1 -1
- package/dist/docs/templates/bootstrap/frontend/app/_layout.template.tsx +1 -1
- package/dist/docs/upgrades/56.0.0-beta.2.md +59 -0
- package/dist/docs/versioned/next/README.md +2 -0
- package/dist/docs/versioned/next/explanation/roadmap-seed-issues.md +21 -0
- package/dist/docs/versioned/next/how-to/README.md +2 -0
- package/dist/docs/versioned/next/how-to/migrate-rtk-to-syncdb.md +445 -0
- package/dist/docs/versioned/next/implementationPlans/PLAN_INDEX.md +28 -50
- package/dist/docs/versioned/next/implementationPlans/b2b-platform-program.md +3 -2
- package/dist/docs/versioned/next/implementationPlans/comms-abstraction.md +98 -15
- package/dist/docs/versioned/next/implementationPlans/comms-adapter-sendgrid.md +63 -13
- package/dist/docs/versioned/next/implementationPlans/comms-adapter-twilio-sms.md +65 -12
- package/dist/docs/versioned/next/implementationPlans/comms-adapter-twilio-verify.md +9 -3
- package/dist/docs/versioned/next/implementationPlans/comms-admin-dashboard.md +180 -0
- package/dist/docs/versioned/next/implementationPlans/migrate-cicd-to-circleci-research.md +126 -0
- package/dist/docs/versioned/next/implementationPlans/migrate-cicd-to-circleci.md +269 -0
- package/dist/docs/versioned/next/implementationPlans/rtk-to-syncdb-migration-docs.md +52 -47
- package/dist/docs/versioned/next/implementationPlans/syncdb-api-inventory.md +303 -0
- package/dist/docs/versioned/next/implementationPlans/syncdb-codegen.md +252 -0
- package/dist/docs/versioned/next/implementationPlans/syncdb-local-first.md +468 -420
- package/dist/docs/versioned/next/implementationPlans/syncdb-phase-c-design.md +425 -0
- package/dist/docs/versioned/next/implementationPlans/terreno-syncdb-2.md +800 -0
- package/dist/docs/versioned/next/reference/README.md +2 -0
- package/dist/docs/versioned/next/reference/comms.md +132 -0
- package/dist/docs/versioned/next/reference/environment-variables.md +9 -0
- package/dist/docs/versioned/next/reference/syncdb.md +456 -0
- package/dist/docs/versioned/next/superpowers/plans/2026-07-28-admin-mobile-navigation.md +97 -0
- package/dist/docs/versioned/next/superpowers/specs/2026-07-28-admin-mobile-navigation-design.md +36 -0
- package/dist/docs/versioned/next/tasks/comms-abstraction.md +29 -16
- package/dist/docs/versioned/next/tasks/comms-adapter-sendgrid.md +17 -7
- package/dist/docs/versioned/next/tasks/comms-adapter-twilio-sms.md +17 -7
- package/dist/docs/versioned/next/tasks/comms-adapter-twilio-verify.md +3 -3
- package/dist/docs/versioned/next/tasks/comms-admin-dashboard.md +52 -0
- package/dist/docs/versioned/next/tasks/migrate-cicd-to-circleci.md +215 -0
- package/dist/docs/versioned/next/tasks/rtk-to-syncdb-migration-docs.md +10 -10
- package/dist/docs/versioned/next/tasks/syncdb-codegen.md +65 -0
- package/dist/docs/versioned/next/tasks/syncdb-local-first.md +434 -117
- package/package.json +3 -3
- package/src/docs/templates/bootstrap/frontend/app/_layout.template.tsx +1 -1
- package/src/docs/upgrades/56.0.0-beta.2.md +59 -0
- package/src/docs/versioned/next/README.md +2 -0
- package/src/docs/versioned/next/explanation/roadmap-seed-issues.md +21 -0
- package/src/docs/versioned/next/how-to/README.md +2 -0
- package/src/docs/versioned/next/how-to/migrate-rtk-to-syncdb.md +445 -0
- package/src/docs/versioned/next/implementationPlans/PLAN_INDEX.md +28 -50
- package/src/docs/versioned/next/implementationPlans/b2b-platform-program.md +3 -2
- package/src/docs/versioned/next/implementationPlans/comms-abstraction.md +98 -15
- package/src/docs/versioned/next/implementationPlans/comms-adapter-sendgrid.md +63 -13
- package/src/docs/versioned/next/implementationPlans/comms-adapter-twilio-sms.md +65 -12
- package/src/docs/versioned/next/implementationPlans/comms-adapter-twilio-verify.md +9 -3
- package/src/docs/versioned/next/implementationPlans/comms-admin-dashboard.md +180 -0
- package/src/docs/versioned/next/implementationPlans/migrate-cicd-to-circleci-research.md +126 -0
- package/src/docs/versioned/next/implementationPlans/migrate-cicd-to-circleci.md +269 -0
- package/src/docs/versioned/next/implementationPlans/rtk-to-syncdb-migration-docs.md +52 -47
- package/src/docs/versioned/next/implementationPlans/syncdb-api-inventory.md +303 -0
- package/src/docs/versioned/next/implementationPlans/syncdb-codegen.md +252 -0
- package/src/docs/versioned/next/implementationPlans/syncdb-local-first.md +468 -420
- package/src/docs/versioned/next/implementationPlans/syncdb-phase-c-design.md +425 -0
- package/src/docs/versioned/next/implementationPlans/terreno-syncdb-2.md +800 -0
- package/src/docs/versioned/next/reference/README.md +2 -0
- package/src/docs/versioned/next/reference/comms.md +132 -0
- package/src/docs/versioned/next/reference/environment-variables.md +9 -0
- package/src/docs/versioned/next/reference/syncdb.md +456 -0
- package/src/docs/versioned/next/superpowers/plans/2026-07-28-admin-mobile-navigation.md +97 -0
- package/src/docs/versioned/next/superpowers/specs/2026-07-28-admin-mobile-navigation-design.md +36 -0
- package/src/docs/versioned/next/tasks/comms-abstraction.md +29 -16
- package/src/docs/versioned/next/tasks/comms-adapter-sendgrid.md +17 -7
- package/src/docs/versioned/next/tasks/comms-adapter-twilio-sms.md +17 -7
- package/src/docs/versioned/next/tasks/comms-adapter-twilio-verify.md +3 -3
- package/src/docs/versioned/next/tasks/comms-admin-dashboard.md +52 -0
- package/src/docs/versioned/next/tasks/migrate-cicd-to-circleci.md +215 -0
- package/src/docs/versioned/next/tasks/rtk-to-syncdb-migration-docs.md +10 -10
- package/src/docs/versioned/next/tasks/syncdb-codegen.md +65 -0
- package/src/docs/versioned/next/tasks/syncdb-local-first.md +434 -117
- package/dist/__tests__/bootstrap.test.d.ts +0 -2
- package/dist/__tests__/bootstrap.test.d.ts.map +0 -1
- package/dist/__tests__/bootstrap.test.js +0 -299
- package/dist/__tests__/bootstrap.test.js.map +0 -1
- package/dist/__tests__/databaseQuery.test.d.ts +0 -2
- package/dist/__tests__/databaseQuery.test.d.ts.map +0 -1
- package/dist/__tests__/databaseQuery.test.js +0 -83
- package/dist/__tests__/databaseQuery.test.js.map +0 -1
- package/dist/__tests__/docsRoot.test.d.ts +0 -2
- package/dist/__tests__/docsRoot.test.d.ts.map +0 -1
- package/dist/__tests__/docsRoot.test.js +0 -59
- package/dist/__tests__/docsRoot.test.js.map +0 -1
- package/dist/__tests__/inferPackages.test.d.ts +0 -2
- package/dist/__tests__/inferPackages.test.d.ts.map +0 -1
- package/dist/__tests__/inferPackages.test.js +0 -35
- package/dist/__tests__/inferPackages.test.js.map +0 -1
- package/dist/__tests__/preload.d.ts +0 -2
- package/dist/__tests__/preload.d.ts.map +0 -1
- package/dist/__tests__/preload.js +0 -12
- package/dist/__tests__/preload.js.map +0 -1
- package/dist/__tests__/prompts.test.d.ts +0 -2
- package/dist/__tests__/prompts.test.d.ts.map +0 -1
- package/dist/__tests__/prompts.test.js +0 -295
- package/dist/__tests__/prompts.test.js.map +0 -1
- package/dist/__tests__/resources.test.d.ts +0 -2
- package/dist/__tests__/resources.test.d.ts.map +0 -1
- package/dist/__tests__/resources.test.js +0 -352
- package/dist/__tests__/resources.test.js.map +0 -1
- package/dist/__tests__/search.test.d.ts +0 -2
- package/dist/__tests__/search.test.d.ts.map +0 -1
- package/dist/__tests__/search.test.js +0 -163
- package/dist/__tests__/search.test.js.map +0 -1
- package/dist/__tests__/tools.test.d.ts +0 -2
- package/dist/__tests__/tools.test.d.ts.map +0 -1
- package/dist/__tests__/tools.test.js +0 -540
- package/dist/__tests__/tools.test.js.map +0 -1
- package/dist/__tests__/upgradeGuide.test.d.ts +0 -2
- package/dist/__tests__/upgradeGuide.test.d.ts.map +0 -1
- package/dist/__tests__/upgradeGuide.test.js +0 -22
- package/dist/__tests__/upgradeGuide.test.js.map +0 -1
|
@@ -1,591 +1,639 @@
|
|
|
1
|
-
# Implementation Plan: SyncDB Local-First Data Layer
|
|
1
|
+
# Implementation Plan: SyncDB Local-First Data Layer (v2)
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
*When an engineer is assigned to a project but before you begin coding, you should fill in the implementation plan and get feedback from the engineering team. Once you have finished or you make any changes, tag Josh with the @ symbol so he can review. Also tag anyone else that needs to be notified, has conflicting work, etc.*
|
|
4
|
+
|
|
5
|
+
**Status:** Shaped, ready for review
|
|
4
6
|
**Priority:** High
|
|
5
7
|
**Effort:** Large
|
|
8
|
+
**Supersedes:** the prior v1 plan (PR #739, previous content of this file) and the foundation implementation in PR #835 (to be closed with a comment linking here). Proven pieces of #835 — the outbox state machine, AES-GCM codec, delta applier, and type contracts — are harvested and adapted, not reused wholesale.
|
|
9
|
+
**Research:** investigation + decision log kept in the IP session's `research.md` (gitignored by repo convention); key findings and decisions are inlined below.
|
|
6
10
|
|
|
7
|
-
##
|
|
11
|
+
## Core Concept
|
|
8
12
|
|
|
9
|
-
`@terreno/
|
|
13
|
+
`@terreno/syncdb` is a local-first data layer: a TinyBase `MergeableStore` (encrypted IndexedDB on web, expo-sqlite on native) is the UI's source of truth. Mutations apply locally and enqueue in a durable outbox, then replay over a `sync:mutate` ack/nack channel that executes the existing modelRouter write path (permissions, hooks, validation). Inbound changes flow from the existing change-stream watcher as `sync:delta` events carrying monotonic per-stream cursors, with HTTP snapshot/catch-up reconciliation. The server remains the source of truth for validation and authorization — but not for immediate UI responsiveness.
|
|
10
14
|
|
|
11
|
-
|
|
15
|
+
Key architectural decisions (rationale in `research.md`):
|
|
12
16
|
|
|
13
|
-
-
|
|
14
|
-
-
|
|
15
|
-
-
|
|
16
|
-
-
|
|
17
|
-
-
|
|
17
|
+
- **TinyBase `MergeableStore` from day one.** It is a per-cell LWW CRDT (hybrid logical clocks) with a different on-disk format from plain `Store`; adopting it now avoids a per-device data migration later and keeps the Yjs door open without implementing Yjs.
|
|
18
|
+
- **Custom delta protocol over the existing `RealtimeApp`** (Socket.io + Mongo change streams), not TinyBase's `WsSynchronizer` (a client-relay with no server-authoritative validation hook) and not an off-the-shelf engine (infra dependency, bypasses modelRouter permissions).
|
|
19
|
+
- **Simple cursors:** per-doc `_syncSeq` + reconcile-on-reconnect; no event log collection, no strict gap proof. Synced models must use soft delete (`isDeletedPlugin`) so tombstones remain queryable.
|
|
20
|
+
- **Multi-tenant by default:** streams are scoped `{collection}|{scope}` where scope is owner, tenant (configurable field), broadcast, or custom.
|
|
21
|
+
- **Auth lives in Better Auth.** Syncdb consumes a narrow `AuthProvider` interface; a Better Auth adapter ships in the package.
|
|
22
|
+
- **Encryption default-on for web** via Web Crypto AES-GCM in a custom IndexedDB persister. Key management is a configurable `KeyProvider`: default derives via HKDF from server-provided per-user key material (`GET /sync/key`); an alternative purely-local non-extractable CryptoKey provider ships too. Native relies on the OS sandbox (plaintext sqlite).
|
|
18
23
|
|
|
19
|
-
##
|
|
24
|
+
## **Models**
|
|
20
25
|
|
|
21
|
-
|
|
26
|
+
### Server (`@terreno/api`)
|
|
22
27
|
|
|
23
|
-
|
|
28
|
+
**`sync` option on modelRouter** (parallel to the existing `realtime` option):
|
|
29
|
+
|
|
30
|
+
```typescript
|
|
31
|
+
const router = modelRouter("/todos", Todo, {
|
|
32
|
+
permissions: {...},
|
|
33
|
+
sync: {
|
|
34
|
+
// Which stream a doc belongs to — multi-tenant by default
|
|
35
|
+
scope: {type: "owner"}, // stream = todos|owner:{ownerId}
|
|
36
|
+
// or {type: "tenant", field: "organizationId"} // stream = todos|tenant:{orgId}
|
|
37
|
+
// or {type: "broadcast"} // stream = todos|all
|
|
38
|
+
// or (doc) => string // custom
|
|
39
|
+
responseHandler?: (doc) => unknown, // sanitize, like realtimeResponseHandler
|
|
40
|
+
},
|
|
41
|
+
});
|
|
42
|
+
```
|
|
24
43
|
|
|
25
|
-
|
|
44
|
+
Registering `sync`:
|
|
26
45
|
|
|
27
|
-
|
|
46
|
+
- **requires soft delete** on the model (`isDeletedPlugin`, `deleted: true` tombstones) — validated at startup by the sync registry; hard-delete models cannot enable sync. This is what makes cursor catch-up correct without an event log. Note: `isDeletedPlugin` auto-injects `{deleted: {$ne: true}}` into `find`/`findOne` (`api/src/plugins.ts:52-57`) — sync's snapshot/catch-up queries **must explicitly bypass it** with `deleted: {$in: [true, false]}` (the plugin skips injection when the query already mentions `deleted`).
|
|
47
|
+
- **requires `syncPlugin` on the schema** (developer-applied, like `isDeletedPlugin` — Mongoose middleware cannot be attached after a model is compiled, so registration validates the plugin is present rather than applying it). The plugin's hooks consult the sync registry at write time, so un-registered models with the plugin no-op.
|
|
48
|
+
- custom resolver scopes additionally require a `snapshotFilter` in the sync config (a stream resolver function cannot be inverted into a Mongo query for the snapshot endpoint) — validated at registration.
|
|
49
|
+
- **restricts write paths**: the plugin hooks `save`, `insertMany`, `updateOne`, `findOneAndUpdate`, `replaceOne`, and `findOneAndReplace`; `updateMany`, `deleteMany`, `deleteOne`, and `findOneAndDelete` **throw on synced models** (multi-doc updates cannot stamp per-document seqs; hard deletes are invisible to tombstone catch-up — callers loop per doc and use soft delete). `Model.bulkWrite` bypasses Mongoose middleware entirely and is a documented restriction. The existing `bulkComplete` action in example-backend (`Todo.updateMany`, `example-backend/src/api/todos.ts:22`) is refactored accordingly in Phase 7.
|
|
28
50
|
|
|
29
|
-
|
|
51
|
+
**`realtime` vs `sync` coexistence:** a model may enable both. They emit distinct events (`sync` for the legacy RTK cache-patching path, `sync:delta` for syncdb) so clients never double-apply, at the cost of double emission work per change while both are enabled. Recommendation: treat `realtime` as deprecated for a model once `sync` is on; document this in the migration guide.
|
|
52
|
+
|
|
53
|
+
**New/modified Mongoose schemas** (all fields carry `description` per repo convention):
|
|
30
54
|
|
|
31
55
|
```typescript
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
entityId?: string;
|
|
49
|
-
argsCiphertext: ArrayBuffer;
|
|
50
|
-
iv: Uint8Array;
|
|
51
|
-
keyId: string;
|
|
52
|
-
baseVersion?: string;
|
|
53
|
-
createdAt: string;
|
|
54
|
-
lastAttemptAt?: string;
|
|
55
|
-
attemptCount: number;
|
|
56
|
-
status: "queued" | "inFlight" | "acked" | "conflicted" | "failed";
|
|
57
|
-
}
|
|
58
|
-
|
|
59
|
-
interface SyncCursor {
|
|
60
|
-
stream: string; // e.g. collection or workspace stream
|
|
61
|
-
cursor: string; // monotonic sequence/offset from server
|
|
62
|
-
updatedAt: string;
|
|
63
|
-
}
|
|
64
|
-
|
|
65
|
-
interface SyncConflict {
|
|
66
|
-
conflictId: string;
|
|
67
|
-
mutationId: string;
|
|
68
|
-
collection: string;
|
|
69
|
-
entityId: string;
|
|
70
|
-
localCiphertext: ArrayBuffer;
|
|
71
|
-
serverCiphertext: ArrayBuffer;
|
|
72
|
-
iv: Uint8Array;
|
|
73
|
-
keyId: string;
|
|
74
|
-
createdAt: string;
|
|
75
|
-
dismissed: boolean;
|
|
76
|
-
}
|
|
77
|
-
|
|
78
|
-
interface KeyringEntry {
|
|
79
|
-
keyId: string;
|
|
80
|
-
wrappedDek: ArrayBuffer;
|
|
81
|
-
wrapAlg: "AES-KW";
|
|
82
|
-
createdAt: string;
|
|
83
|
-
}
|
|
56
|
+
// Plugin applied to every synced model at registration:
|
|
57
|
+
{_syncSeq: number} // compound index: {<scopeField>: 1, _syncSeq: 1}
|
|
58
|
+
|
|
59
|
+
// SyncCounter — atomic per-stream monotonic counter
|
|
60
|
+
// (findOneAndUpdate + $inc + upsert; Redis upgrade path documented as future work)
|
|
61
|
+
{stream: string /* unique index */, seq: number}
|
|
62
|
+
|
|
63
|
+
// SyncMutation — idempotency ledger; a re-sent mutation (lost ack) is never double-applied
|
|
64
|
+
{mutationId: string /* unique index */, userId: string,
|
|
65
|
+
status: "applied" | "conflicted" | "failed",
|
|
66
|
+
resultId?: string, resultSeq?: number, error?: string,
|
|
67
|
+
created: Date /* TTL index, 30 days */}
|
|
68
|
+
|
|
69
|
+
// SyncKey — per-user key material for the default encryption KeyProvider
|
|
70
|
+
{userId: string /* unique index */, keyMaterial: string /* 32 random bytes, base64, server-generated */,
|
|
71
|
+
created: Date}
|
|
84
72
|
```
|
|
85
73
|
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
- normalized entity map by `collection:id`
|
|
89
|
-
- derived query subscriptions (similar to RTK query selectors, but local-first)
|
|
90
|
-
- optimistic patch history for rollback/rebase on conflict
|
|
74
|
+
**Cursor semantics:** every synced write claims the next `seq` for its stream (`SyncCounter` `$inc`) and stamps it on the doc as `_syncSeq`. Validation failures never burn a seq (Mongoose runs validation before user pre-save hooks, so the claim happens post-validation); the claim **joins the caller's session** when one is present, giving caller-managed transactions counter+write atomicity. The plugin does not open its own transaction per write — a hot counter doc inside per-write transactions produces WriteConflict retry storms under concurrency — so a rare post-validation write failure (e.g. duplicate key) burns a seq, which the client already treats as a benign gap (see Gap handling). The client's cursor per stream is the highest seq it has applied. Catch-up = `find({...scopeFilter, deleted: {$in: [true, false]}, _syncSeq: {$gt: cursor}})`, paginated by `_syncSeq` ascending — safe under concurrent writes because a doc's seq only ever increases, moving it *ahead of* the scan pointer, never behind it. Deletes remain visible as soft-delete tombstones.
|
|
91
75
|
|
|
92
|
-
|
|
76
|
+
**Why a counter and not change-stream resume tokens:** resume tokens are opaque, oplog-scoped, and expire with oplog retention — they cannot answer "give me every doc in my stream changed since X" as a plain indexed query, which is what makes the HTTP catch-up trivial and durable. The counter's costs (one transactional `$inc` per synced write) are accepted; Redis is the documented upgrade path.
|
|
93
77
|
|
|
94
|
-
|
|
78
|
+
**Gap handling:** stream seqs are **not** guaranteed contiguous from any one client's perspective — permission-filtered deltas legitimately skip numbers. The seq-jump heuristic is therefore a *hint*, not a proof: jumps trigger an HTTP reconcile **rate-limited** (at most once per 30s per stream), and a low-frequency periodic reconcile (on visibility change / every few minutes while connected) guarantees convergence even for deltas missed with no observable jump.
|
|
95
79
|
|
|
96
|
-
|
|
80
|
+
**Scope changes** (doc moves tenant / owner changes): change streams run with `fullDocumentBeforeChange: "off"` (`changeStreamWatcher.ts:420`), so the old scope is **not available from the change event**. Instead, the `_syncSeq` plugin detects scope-field changes at write time and stamps `_syncPrevStream` on the doc; the watcher reads it from the post-image and emits a tombstone delta to the previous stream plus a create delta to the new stream. No Mongo pre-image configuration required.
|
|
97
81
|
|
|
98
|
-
|
|
99
|
-
- `createSyncDbReduxBridge(client)` - optional bridge for Redux apps migrating from RTK
|
|
100
|
-
- `withSyncDbProvider(...)` / hooks for React integration
|
|
82
|
+
### Client (`@terreno/syncdb` — no Mongoose, no Redux)
|
|
101
83
|
|
|
102
|
-
|
|
84
|
+
TinyBase `MergeableStore` layout, one store per `{app, userId}`:
|
|
103
85
|
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
86
|
+
```
|
|
87
|
+
tables:
|
|
88
|
+
{collection} → rowId = doc _id; cells: data (JSON string), seq, deleted, pendingMutationId?
|
|
89
|
+
_outbox → rowId = mutationId; cells: collection, operation, entityId, args (JSON),
|
|
90
|
+
baseVersion, status (queued|inFlight|acked|conflicted|failed),
|
|
91
|
+
attemptCount, userId, createdAt
|
|
92
|
+
_cursors → rowId = stream; cells: seq, updatedAt
|
|
93
|
+
_conflicts → rowId = mutationId; cells: collection, entityId, localData, serverData,
|
|
94
|
+
serverSeq, dismissed
|
|
95
|
+
values: schemaVersion, lastUserId
|
|
96
|
+
```
|
|
107
97
|
|
|
108
|
-
|
|
98
|
+
**Persistence:**
|
|
109
99
|
|
|
110
|
-
- `
|
|
111
|
-
- `
|
|
112
|
-
-
|
|
100
|
+
- Native: `ExpoSqlitePersister` (JSON-serialization mode — required for MergeableStore), plaintext (OS sandbox is sufficient).
|
|
101
|
+
- Web: custom `EncryptedIndexedDbPersister` (`createCustomPersister`): serialize mergeable content → AES-GCM encrypt via Web Crypto → single blob in IndexedDB. Encryption is **default-on**. (TinyBase's stock `IndexedDbPersister` does not support MergeableStore, so a custom persister is required regardless.)
|
|
102
|
+
- Tests/SSR: in-memory persister.
|
|
113
103
|
|
|
114
|
-
|
|
104
|
+
**Key providers (configurable):**
|
|
115
105
|
|
|
116
|
-
- `
|
|
117
|
-
- `
|
|
118
|
-
- `client.replayOutbox()` for manual replay triggers
|
|
106
|
+
- `serverKeyProvider` (**default**): fetch `GET /sync/key` once per login, derive the AES-256-GCM key via HKDF (salt = app name + userId), cache as a **non-extractable CryptoKey in IndexedDB** so offline cold start still decrypts. Server rotation of key material → client detects decrypt failure, wipes, re-bootstraps.
|
|
107
|
+
- `localKeyProvider`: random non-extractable CryptoKey generated and stored locally; no server dependency (and no server-side copy of the material — strictly stronger for the at-rest case, at the cost of no server-driven rotation/revocation).
|
|
119
108
|
|
|
120
|
-
|
|
109
|
+
**Encryption threat model (stated explicitly):** the web encryption defends against **at-rest disk inspection of IndexedDB** (stolen/imaged device, backup scraping) and against a **stale store being readable after user switch** (per-user keys + wipe). It does **not** defend against XSS or any code executing on the origin (a non-extractable CryptoKey can still be *used* to decrypt in place), and with `serverKeyProvider` the server can reconstruct any user's key (that is the trade for rotation/revocation — choose `localKeyProvider` where that is unacceptable). Server-side data is protected by MongoDB/infra controls, not this layer.
|
|
121
110
|
|
|
122
|
-
|
|
111
|
+
## **APIs**
|
|
123
112
|
|
|
124
|
-
-
|
|
125
|
-
- Delta events with monotonic cursor/sequence:
|
|
126
|
-
- `sync:delta`
|
|
127
|
-
- `sync:ack`
|
|
128
|
-
- `sync:nack`
|
|
129
|
-
- Snapshot/bootstrap API for initial hydration (`since=cursor` or full snapshot)
|
|
130
|
-
- Conflict responses include canonical server document/version metadata
|
|
113
|
+
### HTTP endpoints (mounted by the sync plugin; all `authenticateMiddleware()` + model permissions + server-enforced scope filters, documented via `createOpenApiBuilder`)
|
|
131
114
|
|
|
132
|
-
|
|
115
|
+
| Method & Path | Purpose | Permissions |
|
|
116
|
+
|---|---|---|
|
|
117
|
+
| `GET /sync/snapshot?collection=&cursor=&limit=` | Bootstrap + catch-up. Returns `{entities: [{id, data, seq, deleted}], cursor, hasMore}`. `cursor=0` = full snapshot. Paginated (default 500). | Auth + model `list` permissions + scope filter (owner/tenant) applied server-side |
|
|
118
|
+
| `POST /sync/mutate` | HTTP fallback for outbox replay (same handler as the socket channel). Body: `{mutationId, collection, operation, id?, data?, baseVersion?}` | Auth + model's create/update/delete permissions via modelRouter write path |
|
|
119
|
+
| `GET /sync/key` | Returns caller's `keyMaterial` (creates on first call) for the default KeyProvider | Auth (own key only) |
|
|
133
120
|
|
|
134
|
-
|
|
135
|
-
- Input remains backend OpenAPI
|
|
136
|
-
- Output should generate typed syncdb query/mutation descriptors instead of RTK hooks
|
|
121
|
+
### Socket protocol (new events on the existing authenticated Socket.io server)
|
|
137
122
|
|
|
138
|
-
|
|
123
|
+
| Event | Direction | Payload |
|
|
124
|
+
|---|---|---|
|
|
125
|
+
| `sync:subscribe` / `sync:unsubscribe` | client → server | `{collections: string[]}` — server resolves the user's streams from scope config (+ `getUserStreams(user)` for tenant membership) and joins rooms |
|
|
126
|
+
| `sync:delta` | server → client | `{collection, id, method, data?, seq, stream, deleted?}` — emitted by the change-stream watcher with per-socket permission checks (reuses `emitToAuthorizedRoom`) |
|
|
127
|
+
| `sync:mutate` | client → server | `{mutationId, collection, operation, id?, data?, baseVersion?}` — executes the modelRouter write path |
|
|
128
|
+
| `sync:ack` | server → client | `{mutationId, id, seq}` |
|
|
129
|
+
| `sync:nack` | server → client | `{mutationId, code: "conflict"\|"unauthorized"\|"validation"\|"error", serverDoc?, serverSeq?, message}` — conflict nacks carry the canonical server doc |
|
|
139
130
|
|
|
140
|
-
|
|
131
|
+
**Conflict rule:** client sends `baseVersion` = the `_syncSeq` it last saw for the doc; mismatch with the current `_syncSeq` → nack `conflict`. Same LWW semantics as the rtk offline queue's `If-Unmodified-Since`, on a monotonic integer. The extracted update executor (see below) accepts either check — REST passes the timestamp header, sync passes `baseVersion` — so the two LWW modes share one code path instead of diverging.
|
|
141
132
|
|
|
142
|
-
|
|
133
|
+
**A prerequisite refactor this plan owns explicitly:** modelRouter's create/update/delete logic is currently inline inside Express `asyncHandler` closures with permissions and validation as middleware (`api/src/api.ts:648-745` and onward) — there is **no callable write path today**. Phase 2 therefore begins by extracting transport-agnostic executors (`executeCreate/executeUpdate/executeDelete({model, options, user, body, id, ...})`) and migrating the REST handlers onto them with the entire existing API test suite kept green. `applySyncMutation` builds on those executors; it does not synthesize fake `req`/`res` objects.
|
|
143
134
|
|
|
144
|
-
-
|
|
145
|
-
- syncing queued changes,
|
|
146
|
-
- auth-blocked replay,
|
|
147
|
-
- unresolved conflict count.
|
|
135
|
+
**Idempotency (atomic claim):** the handler **inserts** a `SyncMutation` row with status `pending` (unique index on `mutationId`) *before* applying; a duplicate-key error means another delivery owns or completed it — the handler waits/reads back the recorded outcome instead of re-applying. This closes the race where socket retry and HTTP fallback deliver the same mutation concurrently.
|
|
148
136
|
|
|
149
|
-
|
|
137
|
+
**Socket auth:** a pluggable socket authenticator on `RealtimeApp` — legacy JWT (current `@thream/socketio-jwt` path) continues to work; a Better Auth session validator is added alongside.
|
|
150
138
|
|
|
151
|
-
|
|
139
|
+
### Client package API surface
|
|
152
140
|
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
141
|
+
```typescript
|
|
142
|
+
createSyncDb({name, collections, authProvider, keyProvider?, transportFactory?, persisterFactory?})
|
|
143
|
+
// authProvider: {getToken(), getUserId(), onAuthChange(cb)} — betterAuthAdapter ships in the package
|
|
144
|
+
|
|
145
|
+
client.start() / client.stop() // load persister, hydrate, connect socket
|
|
146
|
+
client.bootstrap({collections?}) // HTTP snapshot pull (initial or forced)
|
|
147
|
+
client.mutate({collection, operation, id?, data}) // local apply + outbox enqueue + async flush
|
|
148
|
+
client.resolveConflict({mutationId, strategy: "useServer" | "keepMine"})
|
|
149
|
+
client.getSyncStatus() // {isOnline, queuedCount, conflictCount, isSyncing, streams}
|
|
150
|
+
|
|
151
|
+
// React (TinyBase reactive listeners, no Redux):
|
|
152
|
+
<SyncDbProvider client={client}>
|
|
153
|
+
useEntity(collection, id)
|
|
154
|
+
useQuery(collection, {filter?, sort?})
|
|
155
|
+
useMutate(collection)
|
|
156
|
+
useSyncStatus()
|
|
157
|
+
useConflicts()
|
|
158
|
+
```
|
|
156
159
|
|
|
157
|
-
|
|
160
|
+
Outbox replays FIFO per collection over the socket, falling back to `POST /sync/mutate`; per-user replay isolation is ported from `rtk/src/offlineMiddleware.ts` (queued mutations record `userId`; replay skips on mismatch).
|
|
158
161
|
|
|
159
|
-
|
|
160
|
-
- optimistic CRUD while offline,
|
|
161
|
-
- conflict resolution controls.
|
|
162
|
+
## **Notifications**
|
|
162
163
|
|
|
163
|
-
|
|
164
|
+
None (no push/email/SMS). In-app sync state surfaces via `useSyncStatus()`: offline indicator, queued-mutation count, syncing state, unresolved-conflict count. The example app renders a status banner and a conflict resolution sheet.
|
|
164
165
|
|
|
165
|
-
|
|
166
|
-
- Allow side-by-side mode:
|
|
167
|
-
- existing RTK path remains functional,
|
|
168
|
-
- syncdb path enabled per screen/domain.
|
|
169
|
-
- Migration order:
|
|
170
|
-
1. todos/read-only,
|
|
171
|
-
2. todos with mutations,
|
|
172
|
-
3. additional resources.
|
|
173
|
-
- No data migration on backend required for v1.
|
|
174
|
-
- Client-side IndexedDB schema versioning and migration handlers required.
|
|
166
|
+
## **UI**
|
|
175
167
|
|
|
176
|
-
|
|
168
|
+
Syncdb is a library; the UI work is the example-frontend proof surface:
|
|
177
169
|
|
|
178
|
-
|
|
170
|
+
- **Todos screen on syncdb** behind a `USE_SYNCDB` feature flag (existing OpenFeature infra). The RTK path remains the default until parity is verified.
|
|
171
|
+
- **`SyncStatusBanner`**: offline / N queued / syncing / conflict count. testIDs: `sync-status-banner`, `sync-queued-count`, `sync-conflict-badge`.
|
|
172
|
+
- **Conflict resolution sheet**: local vs server values side by side, "Keep mine" / "Use server". testIDs: `conflict-sheet`, `conflict-item-{id}`, `conflict-keep-mine-button`, `conflict-use-server-button`.
|
|
173
|
+
- **Dev testing panel** (dev-only): offline toggle, force reconnect, wipe local store. testIDs: `syncdb-dev-panel`, `syncdb-offline-toggle`, `syncdb-wipe-button`.
|
|
179
174
|
|
|
180
|
-
|
|
175
|
+
All components built from @terreno/ui primitives (Box, Text, Button, Modal/ModalSheet).
|
|
181
176
|
|
|
182
177
|
## Phases
|
|
183
178
|
|
|
184
|
-
1. **
|
|
185
|
-
|
|
186
|
-
|
|
179
|
+
1. **Server sync foundation** — `sync` modelRouter option + registry + soft-delete validation, `_syncSeq` plugin (transactional seq claim, all write paths, `updateMany`/`bulkWrite` guard, `_syncPrevStream`) + `SyncCounter`, `GET /sync/snapshot` (tombstone-inclusive), `SyncKey` (race-safe upsert) + `GET /sync/key`.
|
|
180
|
+
2. **Server mutation channel + deltas** — **CRUD executor extraction from modelRouter (Task 2.0, the load-bearing refactor)**, then shared mutation handler (`sync:mutate` + `POST /sync/mutate`) with atomic idempotency claim, ack/nack with conflict payloads, `sync:delta` emission with seq, `sync:subscribe` + tenant rooms, pluggable socket authenticator (Better Auth).
|
|
181
|
+
3. **Client core** — package scaffold, `MergeableStore` schema + typed accessors, persister factories (expo-sqlite / memory), outbox state machine + cursor store + idempotent delta applier (harvested from #835).
|
|
182
|
+
4. **Client crypto** — `EncryptedIndexedDbPersister`, AES-GCM codec (harvested), `serverKeyProvider` (default) + `localKeyProvider`, wipe-on-user-change.
|
|
183
|
+
5. **Client sync engine + transport** — Socket.io transport, HTTP fallback + bootstrap, replay coordinator, conflict store/resolver, reconcile heuristics; integration tests against a real example-backend.
|
|
184
|
+
6. **React layer + Better Auth adapter** — `SyncDbProvider`, hooks, `betterAuthAdapter`.
|
|
185
|
+
7. **Example integration + docs** — example-backend `sync` enablement (todos + a tenant-scoped model), example-frontend behind `USE_SYNCDB` with banner/conflict UI, Playwright e2e, migration guide, close #835.
|
|
187
186
|
|
|
188
|
-
|
|
189
|
-
- Implement IndexedDB adapter with encrypted entity/outbox storage.
|
|
190
|
-
- Add key management and login/logout key lifecycle.
|
|
187
|
+
Each phase is one PR. Phases 1–2 (server) and 3–4 (client foundations) can proceed in parallel once Phase 1's shared protocol types are agreed.
|
|
191
188
|
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
189
|
+
## Feature Flags & Migrations
|
|
190
|
+
|
|
191
|
+
- `USE_SYNCDB` feature flag in example-frontend only; the packages themselves need no flag (adoption is opt-in by construction).
|
|
192
|
+
- No data migrations: local stores are new; server collections (`SyncCounter`, `SyncMutation`, `SyncKey`) are additive. Docs without `_syncSeq` sort before any cursor and are delivered by the first snapshot; the plugin stamps them on their next write.
|
|
193
|
+
- `@terreno/rtk` is **not removed**: README deprecation notice for data-sync concerns + migration guide at `docs/how-to/migrate-rtk-to-syncdb.md`. rtk remains for non-synced RPC endpoints and legacy JWT auth until Better Auth migration completes.
|
|
194
|
+
- PR #835 is closed with a comment linking this plan once the IP PR is up.
|
|
195
195
|
|
|
196
|
-
|
|
197
|
-
- Handshake, cursor resume, delta apply pipeline, ack/nack integration.
|
|
198
|
-
- Reconnect handling and idempotent delta application.
|
|
196
|
+
## Activity Log & User Updates
|
|
199
197
|
|
|
200
|
-
|
|
201
|
-
- Query/mutation hooks and optional Redux bridge for existing apps.
|
|
202
|
-
- Implement `example-frontend` todos flow using syncdb behind a flag.
|
|
198
|
+
None user-facing. Server-side `logger` instrumentation: applied mutations at `info` (mutationId, collection, seq), nacks at `info` with code, replay/idempotency anomalies at `warn`.
|
|
203
199
|
|
|
204
|
-
|
|
205
|
-
- Create `@terreno/syncdb-codegen` prototype.
|
|
206
|
-
- Generate typed operation descriptors and migrate one domain.
|
|
200
|
+
## Permissions & Access
|
|
207
201
|
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
202
|
+
- Every mutation — socket or HTTP — executes the modelRouter write path: identical permissions, pre/post hooks, and validation as REST CRUD. No parallel authorization system.
|
|
203
|
+
- Snapshot enforces model `list` permissions plus a server-side scope filter; clients can never read another scope's stream.
|
|
204
|
+
- Delta emission reuses the per-socket permission checks in `emitToAuthorizedRoom` (`api/src/realtime/changeStreamWatcher.ts:214-249`).
|
|
205
|
+
- Local isolation: store per `{app, userId}`; on auth user change syncdb wipes and re-bootstraps. Per-user encryption keys make a stale store unreadable to the next user.
|
|
211
206
|
|
|
212
|
-
## Not
|
|
207
|
+
## **Not included/Future work**
|
|
213
208
|
|
|
214
|
-
- Bundling
|
|
215
|
-
-
|
|
216
|
-
-
|
|
217
|
-
- Field-level
|
|
218
|
-
-
|
|
209
|
+
- **Bundling/chunking optimization** (future speed work, explicitly no tasks in this IP): lazy-load persisters per platform, code-split the sync engine so non-syncdb apps pay nothing, web chunking for faster initial load.
|
|
210
|
+
- Yjs CRDT backend for collaborative structures (rich text); the door stays open via the MergeableStore + persister/transport abstractions.
|
|
211
|
+
- Redis-based per-stream counters for high-throughput multi-instance deployments (documented upgrade path).
|
|
212
|
+
- Field-level conflict merge UI (v1 is whole-doc keep-mine/use-server).
|
|
213
|
+
- `@terreno/syncdb-codegen` typed collection descriptors from OpenAPI.
|
|
214
|
+
- Removing `@terreno/rtk`.
|
|
215
|
+
- Admin sync-inspection UI (outbox/cursor/conflict visibility in admin panel).
|
|
216
|
+
- Native sqlite encryption (SQLCipher) — sandbox deemed sufficient.
|
|
219
217
|
|
|
220
218
|
## Risks & Mitigations
|
|
221
219
|
|
|
222
|
-
- **
|
|
223
|
-
- **
|
|
224
|
-
- **
|
|
225
|
-
- **
|
|
226
|
-
- **
|
|
220
|
+
- **modelRouter executor extraction (Task 2.0) is the highest-risk item in the plan**: it refactors the framework's most load-bearing file (`api/src/api.ts`, ~1000 lines of inline Express handlers). Mitigation: it is its own PR-sized task with the hard gate that the entire existing API test suite passes unchanged before anything sync-specific builds on it.
|
|
221
|
+
- **Whole-store serialize+encrypt per save** (TinyBase persisters persist the full store): debounced/throttled saves (500ms trailing); per-table blob sharding is the documented follow-up if profiling demands it. Encryption cost scales with store size, not change size — bounded in practice by scoping which collections sync.
|
|
222
|
+
- **TinyBase version behavior unverified in-repo** (`tinybase` is not yet a dependency anywhere): MergeableStore persister-mode constraints were verified against tinybase.org v8 docs, not an installed version. Task 3.1 pins the version; Task 3.5/4.2 acceptance tests are the tripwire.
|
|
223
|
+
- **Better Auth socket authentication**: `RealtimeApp` validates legacy JWTs only today. Pluggable authenticator added in Phase 2; legacy JWT path unchanged, so nothing regresses.
|
|
224
|
+
- **Seq counter write amplification** (one transactional `$inc` + doc write per synced write): acceptable at current scale; Redis path documented. Transactions require the replica set, which change streams already mandate.
|
|
225
|
+
- **Missed deltas while connected**: no strict gap proof by design — rate-limited seq-jump reconcile + reconcile-on-reconnect + periodic reconcile give eventual convergence (accepted in shaping). Permission-filtered deltas make seq gaps legitimate, hence the rate limiting.
|
|
226
|
+
- **Scope changes**: `_syncPrevStream` stamped at write time (no Mongo pre-image config needed); watcher emits old-stream tombstone + new-stream create; explicit test case.
|
|
227
|
+
- **Multi-tab same-user on web**: two tabs share one IndexedDB blob; concurrent persister saves are last-writer-wins at the blob level, which can drop the other tab's queued outbox rows. V1 mitigation: single-writer persistence via the Web Locks API (the lock-holding tab owns persister saves; others operate in-memory and re-load on lock acquisition). Full multi-tab coordination (BroadcastChannel state sharing) is future work; AC-8 remains valid because each tab holds its own socket.
|
|
228
|
+
- **`sync:mutate` flood**: per-socket rate limit on the mutation channel (consistent with existing subscription caps) — each mutation costs a transaction + ledger write.
|
|
229
|
+
- **MergeableStore ~2× storage overhead**: accepted for CRDT-readiness; documented.
|
|
227
230
|
|
|
228
231
|
## Acceptance Criteria
|
|
229
232
|
|
|
230
|
-
|
|
233
|
+
*All UI criteria run against example-frontend (web) with `USE_SYNCDB` on unless stated otherwise. Server-only behaviors (scope isolation, idempotency, permission enforcement) are additionally covered by bun API/integration tests per the Task List; the criteria below are the user-observable surface.*
|
|
231
234
|
|
|
232
|
-
|
|
235
|
+
### Feature: Local-First Reads & Bootstrap
|
|
236
|
+
|
|
237
|
+
#### AC-1: Todos load from the local store after login
|
|
233
238
|
**Priority:** P0
|
|
234
|
-
**Screen:** Todos
|
|
239
|
+
**Screen:** Todos
|
|
235
240
|
**Preconditions:**
|
|
236
|
-
- `USE_SYNCDB`
|
|
237
|
-
- User is authenticated.
|
|
238
|
-
- Network is disabled (browser offline mode).
|
|
241
|
+
- `USE_SYNCDB` flag on; test user has ≥3 todos on the server; fresh browser profile (empty local store)
|
|
239
242
|
|
|
240
243
|
**Steps:**
|
|
241
|
-
1.
|
|
242
|
-
2.
|
|
243
|
-
3. Tap `todos-button-save`.
|
|
244
|
-
4. Observe `todos-item-offline-task` and `todos-sync-queue-count`.
|
|
244
|
+
1. Log in via `loginAs()` and wait for `todos-screen` to be visible
|
|
245
|
+
2. Observe the todos list after bootstrap completes
|
|
245
246
|
|
|
246
247
|
**Expected results:**
|
|
247
|
-
- [ ] `
|
|
248
|
-
- [ ] `
|
|
249
|
-
- [ ] `todos-sync-queue-count` increments by 1.
|
|
248
|
+
- [ ] All server todos render as `todo-item-{id}` elements
|
|
249
|
+
- [ ] `sync-status-banner` shows no queued count and no offline indicator
|
|
250
250
|
|
|
251
|
-
**testIDs needed:** `todos-screen
|
|
251
|
+
**testIDs needed:** `todos-screen`, `todo-item-{id}`, `sync-status-banner`
|
|
252
252
|
|
|
253
253
|
---
|
|
254
254
|
|
|
255
|
-
#### AC-2:
|
|
255
|
+
#### AC-2: Data persists across reload without network
|
|
256
256
|
**Priority:** P0
|
|
257
|
-
**Screen:** Todos
|
|
257
|
+
**Screen:** Todos
|
|
258
258
|
**Preconditions:**
|
|
259
|
-
- AC-1 completed
|
|
260
|
-
- Network remains disabled.
|
|
259
|
+
- AC-1 completed (local store populated)
|
|
261
260
|
|
|
262
261
|
**Steps:**
|
|
263
|
-
1.
|
|
264
|
-
2.
|
|
265
|
-
3. Observe `todos-item-offline-task` and `todos-sync-queue-count`.
|
|
262
|
+
1. Toggle offline via `syncdb-offline-toggle` (dev panel)
|
|
263
|
+
2. Reload the page and wait for `todos-screen`
|
|
266
264
|
|
|
267
265
|
**Expected results:**
|
|
268
|
-
- [ ]
|
|
269
|
-
- [ ] `
|
|
270
|
-
- [ ] No login redirect occurs while session remains valid.
|
|
266
|
+
- [ ] Previously synced todos render from the local store with no network
|
|
267
|
+
- [ ] `sync-status-banner` shows the offline indicator (`sync-offline-indicator` visible)
|
|
271
268
|
|
|
272
|
-
**testIDs needed:** `
|
|
269
|
+
**testIDs needed:** `syncdb-dev-panel`, `syncdb-offline-toggle`, `todos-screen`, `todo-item-{id}`, `sync-status-banner`, `sync-offline-indicator`
|
|
273
270
|
|
|
274
271
|
---
|
|
275
272
|
|
|
276
|
-
#### AC-3:
|
|
277
|
-
**Priority:**
|
|
278
|
-
**Screen:** Todos
|
|
273
|
+
#### AC-3: Empty state renders for a user with no data
|
|
274
|
+
**Priority:** P1
|
|
275
|
+
**Screen:** Todos
|
|
279
276
|
**Preconditions:**
|
|
280
|
-
-
|
|
281
|
-
- User remains authenticated.
|
|
277
|
+
- Fresh user with zero todos
|
|
282
278
|
|
|
283
279
|
**Steps:**
|
|
284
|
-
1.
|
|
285
|
-
2. Wait for `todos-sync-status-syncing`.
|
|
286
|
-
3. Wait for `todos-sync-status-online`.
|
|
287
|
-
4. Refresh the todos list using `todos-button-refresh`.
|
|
280
|
+
1. Log in and wait for `todos-screen`
|
|
288
281
|
|
|
289
282
|
**Expected results:**
|
|
290
|
-
- [ ]
|
|
291
|
-
- [ ] `todos-sync-queue-count` reaches `0`.
|
|
292
|
-
- [ ] `todos-item-offline-task` persists in the server-backed list after refresh.
|
|
283
|
+
- [ ] `todos-empty-state` is visible; no `todo-item-*` elements exist
|
|
293
284
|
|
|
294
|
-
**testIDs needed:** `todos-
|
|
285
|
+
**testIDs needed:** `todos-empty-state`
|
|
295
286
|
|
|
296
287
|
---
|
|
297
288
|
|
|
298
|
-
### Feature:
|
|
289
|
+
### Feature: Offline Mutations & Durable Outbox
|
|
299
290
|
|
|
300
|
-
#### AC-4:
|
|
291
|
+
#### AC-4: Offline create applies instantly and syncs on reconnect
|
|
301
292
|
**Priority:** P0
|
|
302
|
-
**Screen:** Todos
|
|
293
|
+
**Screen:** Todos
|
|
303
294
|
**Preconditions:**
|
|
304
|
-
-
|
|
305
|
-
- Both clients have `USE_SYNCDB` enabled and are online.
|
|
295
|
+
- Logged in, store bootstrapped
|
|
306
296
|
|
|
307
297
|
**Steps:**
|
|
308
|
-
1.
|
|
309
|
-
2.
|
|
310
|
-
3.
|
|
311
|
-
4.
|
|
298
|
+
1. Toggle offline via `syncdb-offline-toggle`
|
|
299
|
+
2. Type "Offline milk run" into `todos-title-input`, click `todos-create-button`
|
|
300
|
+
3. Observe the list and banner
|
|
301
|
+
4. Toggle back online
|
|
302
|
+
5. Wait for the queued count to clear
|
|
312
303
|
|
|
313
304
|
**Expected results:**
|
|
314
|
-
- [ ]
|
|
315
|
-
- [ ]
|
|
316
|
-
- [ ] `
|
|
305
|
+
- [ ] The new todo appears in the list immediately (before any network)
|
|
306
|
+
- [ ] `sync-queued-count` shows "1" while offline
|
|
307
|
+
- [ ] After reconnect, `sync-queued-count` disappears and the todo remains (now server-acked)
|
|
308
|
+
- [ ] The todo is visible in a second browser session for the same user
|
|
317
309
|
|
|
318
|
-
**testIDs needed:** `todos-
|
|
310
|
+
**testIDs needed:** `todos-title-input`, `todos-create-button`, `sync-queued-count`, `syncdb-offline-toggle`
|
|
319
311
|
|
|
320
312
|
---
|
|
321
313
|
|
|
322
|
-
|
|
314
|
+
#### AC-5: Queued mutations survive an app reload
|
|
315
|
+
**Priority:** P0
|
|
316
|
+
**Screen:** Todos
|
|
317
|
+
**Preconditions:**
|
|
318
|
+
- Offline with 2 queued mutations (1 create, 1 update)
|
|
319
|
+
|
|
320
|
+
**Steps:**
|
|
321
|
+
1. Reload the page while still offline; wait for `todos-screen`
|
|
322
|
+
2. Observe banner and list
|
|
323
|
+
3. Toggle online
|
|
324
|
+
|
|
325
|
+
**Expected results:**
|
|
326
|
+
- [ ] `sync-queued-count` still shows "2" after reload (outbox is durable)
|
|
327
|
+
- [ ] Optimistic changes are still visible in the list
|
|
328
|
+
- [ ] After reconnect, both mutations replay in order and the queue clears
|
|
329
|
+
|
|
330
|
+
**testIDs needed:** `sync-queued-count`, `todos-screen`, `todo-item-{id}`
|
|
323
331
|
|
|
324
|
-
|
|
332
|
+
---
|
|
333
|
+
|
|
334
|
+
#### AC-6: Offline update and delete apply locally and replay
|
|
325
335
|
**Priority:** P1
|
|
326
|
-
**Screen:** Todos
|
|
336
|
+
**Screen:** Todos
|
|
327
337
|
**Preconditions:**
|
|
328
|
-
-
|
|
329
|
-
- Server conflict contract (`409`) is enabled.
|
|
338
|
+
- Logged in with ≥2 synced todos
|
|
330
339
|
|
|
331
340
|
**Steps:**
|
|
332
|
-
1.
|
|
333
|
-
2.
|
|
334
|
-
3.
|
|
335
|
-
4. In client A, open `todos-conflict-card-shared` and click `todos-conflict-action-use-server`.
|
|
336
|
-
5. Repeat conflict and click `todos-conflict-action-keep-mine`.
|
|
341
|
+
1. Toggle offline
|
|
342
|
+
2. Toggle completion on one todo via `todo-toggle-{id}`; delete another via `todo-delete-{id}`
|
|
343
|
+
3. Toggle online; wait for the queue to clear
|
|
337
344
|
|
|
338
345
|
**Expected results:**
|
|
339
|
-
- [ ]
|
|
340
|
-
- [ ]
|
|
341
|
-
- [ ] Choosing `keep-mine` retries mutation and final row text is `Mine` after replay.
|
|
346
|
+
- [ ] Toggle and delete reflect immediately while offline
|
|
347
|
+
- [ ] After reconnect the server state matches (verify via second session): toggled todo is completed, deleted todo is gone
|
|
342
348
|
|
|
343
|
-
**testIDs needed:** `
|
|
349
|
+
**testIDs needed:** `todo-toggle-{id}`, `todo-delete-{id}`
|
|
344
350
|
|
|
345
351
|
---
|
|
346
352
|
|
|
347
|
-
|
|
353
|
+
#### AC-7: User switch wipes local data
|
|
354
|
+
**Priority:** P0
|
|
355
|
+
**Screen:** Todos / Login
|
|
356
|
+
**Preconditions:**
|
|
357
|
+
- User A logged in with synced todos and 1 queued (offline) mutation
|
|
348
358
|
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
359
|
+
**Steps:**
|
|
360
|
+
1. Log out; log in as user B
|
|
361
|
+
2. Wait for `todos-screen`
|
|
362
|
+
|
|
363
|
+
**Expected results:**
|
|
364
|
+
- [ ] None of user A's todos are visible to user B
|
|
365
|
+
- [ ] `sync-queued-count` is absent (A's outbox not replayed as B)
|
|
366
|
+
- [ ] Logging back in as A re-bootstraps A's server data
|
|
367
|
+
|
|
368
|
+
**testIDs needed:** `todos-screen`, `sync-queued-count`
|
|
369
|
+
|
|
370
|
+
---
|
|
371
|
+
|
|
372
|
+
### Feature: Live Delta Sync
|
|
373
|
+
|
|
374
|
+
#### AC-8: Changes from another session appear without refresh
|
|
375
|
+
**Priority:** P0
|
|
376
|
+
**Screen:** Todos
|
|
377
|
+
**Preconditions:**
|
|
378
|
+
- Same user logged in from two browser contexts (A and B), both on `todos-screen`
|
|
379
|
+
|
|
380
|
+
**Steps:**
|
|
381
|
+
1. In context B, create a todo "From the other tab"
|
|
382
|
+
2. Observe context A without any interaction
|
|
383
|
+
|
|
384
|
+
**Expected results:**
|
|
385
|
+
- [ ] The new todo appears in context A via `sync:delta` (no reload, no refetch button)
|
|
386
|
+
- [ ] Completing it in B updates A; deleting it in B removes it from A
|
|
387
|
+
|
|
388
|
+
**testIDs needed:** `todo-item-{id}`, `todos-create-button`, `todo-toggle-{id}`, `todo-delete-{id}`
|
|
389
|
+
|
|
390
|
+
---
|
|
391
|
+
|
|
392
|
+
#### AC-9: Reconnect catch-up converges missed changes
|
|
393
|
+
**Priority:** P0
|
|
394
|
+
**Screen:** Todos
|
|
395
|
+
**Preconditions:**
|
|
396
|
+
- Same user in contexts A and B; A toggled offline
|
|
397
|
+
|
|
398
|
+
**Steps:**
|
|
399
|
+
1. In context B (online), create one todo, update another, delete a third
|
|
400
|
+
2. Toggle A back online
|
|
401
|
+
3. Wait for sync to settle
|
|
402
|
+
|
|
403
|
+
**Expected results:**
|
|
404
|
+
- [ ] Context A shows all three changes after reconnect (HTTP catch-up from cursor), with no duplicates
|
|
405
|
+
|
|
406
|
+
**testIDs needed:** `todo-item-{id}`, `syncdb-offline-toggle`
|
|
407
|
+
|
|
408
|
+
---
|
|
409
|
+
|
|
410
|
+
### Feature: Conflict Resolution
|
|
411
|
+
|
|
412
|
+
#### AC-10: Conflicting offline edit surfaces a conflict with both versions
|
|
413
|
+
**Priority:** P0
|
|
414
|
+
**Screen:** Todos / Conflict sheet
|
|
352
415
|
**Preconditions:**
|
|
353
|
-
-
|
|
354
|
-
- `USE_SYNCDB` enabled.
|
|
416
|
+
- Same user in contexts A and B; both see todo T
|
|
355
417
|
|
|
356
418
|
**Steps:**
|
|
357
|
-
1.
|
|
358
|
-
2.
|
|
359
|
-
3.
|
|
419
|
+
1. Toggle A offline; edit T's title to "Local edit" in A
|
|
420
|
+
2. In B (online), edit T's title to "Server edit"
|
|
421
|
+
3. Toggle A online; wait for replay
|
|
422
|
+
4. Click `sync-conflict-badge` to open the conflict sheet
|
|
360
423
|
|
|
361
424
|
**Expected results:**
|
|
362
|
-
- [ ] `
|
|
363
|
-
- [ ]
|
|
364
|
-
- [ ]
|
|
425
|
+
- [ ] `sync-conflict-badge` appears showing "1"
|
|
426
|
+
- [ ] `conflict-sheet` shows the local value ("Local edit") and server value ("Server edit") side by side in `conflict-item-{id}`
|
|
427
|
+
- [ ] T's local display is not silently overwritten while the conflict is unresolved
|
|
365
428
|
|
|
366
|
-
**testIDs needed:** `
|
|
429
|
+
**testIDs needed:** `sync-conflict-badge`, `conflict-sheet`, `conflict-item-{id}`
|
|
367
430
|
|
|
368
431
|
---
|
|
369
432
|
|
|
370
|
-
#### AC-
|
|
433
|
+
#### AC-11: "Use server" resolves to the server version
|
|
434
|
+
**Priority:** P0
|
|
435
|
+
**Screen:** Conflict sheet
|
|
436
|
+
**Preconditions:**
|
|
437
|
+
- AC-10 state (open conflict)
|
|
438
|
+
|
|
439
|
+
**Steps:**
|
|
440
|
+
1. Click `conflict-use-server-button`
|
|
441
|
+
|
|
442
|
+
**Expected results:**
|
|
443
|
+
- [ ] The sheet closes (or the item leaves the list); `sync-conflict-badge` disappears
|
|
444
|
+
- [ ] T shows "Server edit" locally; no mutation is re-sent
|
|
445
|
+
|
|
446
|
+
**testIDs needed:** `conflict-use-server-button`, `sync-conflict-badge`
|
|
447
|
+
|
|
448
|
+
---
|
|
449
|
+
|
|
450
|
+
#### AC-12: "Keep mine" re-applies the local version to the server
|
|
371
451
|
**Priority:** P1
|
|
372
|
-
**Screen:**
|
|
452
|
+
**Screen:** Conflict sheet
|
|
373
453
|
**Preconditions:**
|
|
374
|
-
-
|
|
375
|
-
- Access token is expired.
|
|
376
|
-
- Refresh endpoint fails (network/server unavailable).
|
|
454
|
+
- AC-10 state (open conflict)
|
|
377
455
|
|
|
378
456
|
**Steps:**
|
|
379
|
-
1.
|
|
380
|
-
2. Wait for
|
|
381
|
-
3. Inspect todos list and queued item.
|
|
382
|
-
4. Restore refresh endpoint and re-authenticate if needed.
|
|
457
|
+
1. Click `conflict-keep-mine-button`
|
|
458
|
+
2. Wait for the queue to clear
|
|
383
459
|
|
|
384
460
|
**Expected results:**
|
|
385
|
-
- [ ]
|
|
386
|
-
- [ ]
|
|
387
|
-
- [ ] After refresh succeeds, replay resumes and queue drains.
|
|
461
|
+
- [ ] T shows "Local edit" locally and in context B (mutation re-enqueued with fresh baseVersion and acked)
|
|
462
|
+
- [ ] `sync-conflict-badge` disappears
|
|
388
463
|
|
|
389
|
-
**testIDs needed:** `
|
|
464
|
+
**testIDs needed:** `conflict-keep-mine-button`, `sync-conflict-badge`
|
|
390
465
|
|
|
391
466
|
---
|
|
392
467
|
|
|
393
|
-
### Feature:
|
|
468
|
+
### Feature: Sync Status UI
|
|
394
469
|
|
|
395
|
-
#### AC-
|
|
470
|
+
#### AC-13: Banner reflects offline / queued / syncing states
|
|
396
471
|
**Priority:** P1
|
|
397
|
-
**Screen:** Todos
|
|
472
|
+
**Screen:** Todos
|
|
398
473
|
**Preconditions:**
|
|
399
|
-
-
|
|
400
|
-
- `USE_SYNCDB` enabled.
|
|
474
|
+
- Logged in, bootstrapped
|
|
401
475
|
|
|
402
476
|
**Steps:**
|
|
403
|
-
1.
|
|
404
|
-
2.
|
|
405
|
-
3.
|
|
477
|
+
1. Toggle offline → observe banner
|
|
478
|
+
2. Create 2 todos → observe queued count
|
|
479
|
+
3. Toggle online → observe transition
|
|
406
480
|
|
|
407
481
|
**Expected results:**
|
|
408
|
-
- [ ] `
|
|
409
|
-
- [ ] `
|
|
410
|
-
- [ ]
|
|
482
|
+
- [ ] Offline: `sync-offline-indicator` visible
|
|
483
|
+
- [ ] Queued: `sync-queued-count` shows "2"
|
|
484
|
+
- [ ] On reconnect: count clears; banner returns to idle (no offline indicator, no count)
|
|
411
485
|
|
|
412
|
-
**testIDs needed:** `
|
|
486
|
+
**testIDs needed:** `sync-status-banner`, `sync-offline-indicator`, `sync-queued-count`
|
|
413
487
|
|
|
414
488
|
---
|
|
415
489
|
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
|
|
490
|
+
### Feature: Encryption at Rest (web)
|
|
491
|
+
|
|
492
|
+
#### AC-14: IndexedDB contains no plaintext entity data
|
|
493
|
+
**Priority:** P0
|
|
494
|
+
**Screen:** N/A (storage assertion)
|
|
419
495
|
**Preconditions:**
|
|
420
|
-
-
|
|
421
|
-
- One queued item exists while offline.
|
|
496
|
+
- Logged in; a todo titled "SECRET_MARKER_XYZ" has synced; persister has saved
|
|
422
497
|
|
|
423
498
|
**Steps:**
|
|
424
|
-
1.
|
|
425
|
-
2.
|
|
426
|
-
3. Navigate back using `tabs-button-todos`.
|
|
427
|
-
4. Observe status and queue count again.
|
|
499
|
+
1. Via `page.evaluate`, read all records from the syncdb IndexedDB database
|
|
500
|
+
2. Serialize the raw stored values and search for "SECRET_MARKER_XYZ"
|
|
428
501
|
|
|
429
502
|
**Expected results:**
|
|
430
|
-
- [ ]
|
|
431
|
-
- [ ]
|
|
432
|
-
- [ ] No duplicate list rows appear after navigation.
|
|
503
|
+
- [ ] The marker string does not appear anywhere in raw IndexedDB contents (payload is AES-GCM ciphertext)
|
|
504
|
+
- [ ] After reload (same session), the todo still renders with the correct title (decrypt works)
|
|
433
505
|
|
|
434
|
-
**testIDs needed:** `
|
|
506
|
+
**testIDs needed:** `todo-item-{id}` (post-reload assertion)
|
|
435
507
|
|
|
436
508
|
---
|
|
437
509
|
|
|
438
|
-
### Feature:
|
|
510
|
+
### Feature: Flag & Migration Safety
|
|
439
511
|
|
|
440
|
-
#### AC-
|
|
441
|
-
**Priority:**
|
|
442
|
-
**Screen:**
|
|
512
|
+
#### AC-15: USE_SYNCDB off leaves the RTK path unchanged
|
|
513
|
+
**Priority:** P0
|
|
514
|
+
**Screen:** Todos
|
|
443
515
|
**Preconditions:**
|
|
444
|
-
-
|
|
445
|
-
- User A logs out; User B logs in on same device/browser profile.
|
|
516
|
+
- `USE_SYNCDB` flag off
|
|
446
517
|
|
|
447
518
|
**Steps:**
|
|
448
|
-
1.
|
|
449
|
-
2.
|
|
450
|
-
3. Log in as User B with `login-input-email`, `login-input-password`, `login-button-submit`.
|
|
451
|
-
4. Re-enable network and open todos.
|
|
519
|
+
1. Log in; wait for `todos-screen`
|
|
520
|
+
2. Create, toggle, and delete a todo
|
|
452
521
|
|
|
453
522
|
**Expected results:**
|
|
454
|
-
- [ ]
|
|
455
|
-
- [ ]
|
|
456
|
-
- [ ] `todos-sync-queue-count` is 0 after User B login unless User B creates new local mutations.
|
|
523
|
+
- [ ] All CRUD works exactly as before (RTK Query path)
|
|
524
|
+
- [ ] No syncdb UI (`sync-status-banner`, `syncdb-dev-panel`) is rendered
|
|
457
525
|
|
|
458
|
-
**testIDs needed:** `
|
|
526
|
+
**testIDs needed:** `todos-screen`, `sync-status-banner`, `syncdb-dev-panel`
|
|
459
527
|
|
|
460
528
|
---
|
|
461
529
|
|
|
462
|
-
###
|
|
530
|
+
### Feature: Server Contract (API-level, verified by bun tests — listed for completeness)
|
|
531
|
+
|
|
532
|
+
#### AC-16: Scope isolation, idempotency, and authoritative validation
|
|
533
|
+
**Priority:** P0
|
|
534
|
+
**Screen:** N/A (API)
|
|
535
|
+
**Preconditions:** integration test environment (Phase 5.5)
|
|
536
|
+
|
|
537
|
+
**Expected results:**
|
|
538
|
+
- [ ] `GET /sync/snapshot` never returns another owner's/tenant's documents regardless of query params
|
|
539
|
+
- [ ] Re-sending a `sync:mutate` with the same `mutationId` returns the recorded outcome without double-applying
|
|
540
|
+
- [ ] A mutation violating model permissions is nacked `unauthorized`; one failing validation is nacked `validation`; neither modifies data
|
|
541
|
+
- [ ] A mutation with a stale `baseVersion` is nacked `conflict` and includes the canonical server doc + seq
|
|
463
542
|
|
|
464
|
-
|
|
465
|
-
- Validation: AC-6
|
|
466
|
-
- Error handling: AC-7
|
|
467
|
-
- Empty states: AC-8
|
|
468
|
-
- Permissions/access: AC-10
|
|
469
|
-
- Navigation: AC-9
|
|
470
|
-
- Data persistence: AC-2, AC-3
|
|
471
|
-
- Out of scope confirmation: bundling/chunking optimization remains intentionally excluded from acceptance validation for this plan.
|
|
543
|
+
**testIDs needed:** none (API tests)
|
|
472
544
|
|
|
473
545
|
---
|
|
474
546
|
|
|
475
547
|
## Task List (Bot Consumption)
|
|
476
548
|
|
|
477
|
-
*Structured task breakdown for automated implementation.
|
|
478
|
-
|
|
479
|
-
|
|
480
|
-
|
|
481
|
-
|
|
482
|
-
|
|
483
|
-
|
|
484
|
-
|
|
485
|
-
|
|
486
|
-
|
|
487
|
-
|
|
488
|
-
|
|
489
|
-
|
|
490
|
-
|
|
491
|
-
|
|
492
|
-
|
|
493
|
-
|
|
494
|
-
|
|
495
|
-
-
|
|
496
|
-
|
|
497
|
-
|
|
498
|
-
|
|
499
|
-
|
|
500
|
-
|
|
501
|
-
-
|
|
502
|
-
|
|
503
|
-
|
|
504
|
-
-
|
|
505
|
-
|
|
506
|
-
|
|
507
|
-
|
|
508
|
-
|
|
509
|
-
|
|
510
|
-
|
|
511
|
-
|
|
512
|
-
|
|
513
|
-
|
|
514
|
-
|
|
515
|
-
|
|
516
|
-
|
|
517
|
-
|
|
518
|
-
|
|
519
|
-
|
|
520
|
-
|
|
521
|
-
|
|
522
|
-
|
|
523
|
-
|
|
524
|
-
|
|
525
|
-
|
|
526
|
-
|
|
527
|
-
|
|
528
|
-
-
|
|
529
|
-
-
|
|
530
|
-
|
|
531
|
-
|
|
532
|
-
|
|
533
|
-
|
|
534
|
-
|
|
535
|
-
-
|
|
536
|
-
|
|
537
|
-
-
|
|
538
|
-
|
|
539
|
-
|
|
540
|
-
|
|
541
|
-
|
|
542
|
-
|
|
543
|
-
|
|
544
|
-
|
|
545
|
-
|
|
546
|
-
|
|
547
|
-
|
|
548
|
-
|
|
549
|
-
|
|
550
|
-
|
|
551
|
-
|
|
552
|
-
|
|
553
|
-
|
|
554
|
-
|
|
555
|
-
-
|
|
556
|
-
|
|
557
|
-
|
|
558
|
-
|
|
559
|
-
|
|
560
|
-
|
|
561
|
-
|
|
562
|
-
|
|
563
|
-
|
|
564
|
-
|
|
565
|
-
|
|
566
|
-
|
|
567
|
-
|
|
568
|
-
- Description: Integrate syncdb into `example-frontend` for one domain (todos) with `USE_SYNCDB` toggle.
|
|
569
|
-
- Files: `example-frontend/store/index.ts`, `example-frontend/store/sdk.ts`, `example-frontend/app/(tabs)/index.tsx`, `example-frontend/app/_layout.tsx`
|
|
570
|
-
- Depends on: Task 5.1, Task 5.2
|
|
571
|
-
- Acceptance: toggling flag changes data path to syncdb and preserves todo CRUD behavior.
|
|
572
|
-
|
|
573
|
-
### Phase 6: Codegen + Documentation
|
|
574
|
-
|
|
575
|
-
- [ ] **Task 6.1**: Prototype syncdb codegen package
|
|
576
|
-
- Description: Build `@terreno/syncdb-codegen` to emit typed operation descriptors from OpenAPI.
|
|
577
|
-
- Files: `syncdb-codegen/package.json` (new), `syncdb-codegen/src/index.ts` (new), tests under `syncdb-codegen/src/*.test.ts` (new)
|
|
578
|
-
- Depends on: Task 1.1
|
|
579
|
-
- Acceptance: generator emits descriptor file for at least one example endpoint group.
|
|
580
|
-
|
|
581
|
-
- [ ] **Task 6.2**: Add consumer migration documentation
|
|
582
|
-
- Description: Document install/configuration/migration steps from `@terreno/rtk` to `@terreno/syncdb`.
|
|
583
|
-
- Files: `syncdb/README.md`, `docs/implementationPlans/syncdb-local-first.md`
|
|
584
|
-
- Depends on: Task 5.3
|
|
585
|
-
- Acceptance: docs include side-by-side migration example and rollout checklist.
|
|
586
|
-
|
|
587
|
-
- [ ] **Task 6.3**: Validation suite run and stabilization
|
|
588
|
-
- Description: Run compile/lint/tests for new packages and touched example app flows, fixing regressions.
|
|
589
|
-
- Files: no dedicated source file; any fixes in touched files
|
|
590
|
-
- Depends on: Task 6.2
|
|
591
|
-
- Acceptance: relevant package compile/test/lint commands pass with documented outputs.
|
|
549
|
+
*Structured task breakdown for automated implementation. The canonical copy lives at `docs/tasks/syncdb-local-first.md`; keep the two in sync.*
|
|
550
|
+
|
|
551
|
+
See `docs/tasks/syncdb-local-first.md` for the full 26-task breakdown across the 7 phases above (including Task 2.0, the modelRouter executor extraction).
|
|
552
|
+
|
|
553
|
+
---
|
|
554
|
+
|
|
555
|
+
## Phase C — Sync protocol correctness (updated contract)
|
|
556
|
+
|
|
557
|
+
Phase C hardened the wire protocol. The rules below OVERRIDE the earlier descriptions of
|
|
558
|
+
the snapshot endpoint, cursor advancement, and scope-move handling.
|
|
559
|
+
|
|
560
|
+
### C1 — Stable frontier (seq commit fence)
|
|
561
|
+
|
|
562
|
+
A client cursor for a stream may advance to seq N **only when the server reports N ≤ that
|
|
563
|
+
stream's *stable frontier***: every seq ≤ N in the stream is committed (its owning write
|
|
564
|
+
durably landed, or its claim was reclaimed after the crash lease expired).
|
|
565
|
+
|
|
566
|
+
- The `SyncCounter` carries an in-flight `pending: [{seq, claimedAt}]` registry. A claim
|
|
567
|
+
registers its seq(s); the post-write hook (`confirmSyncSeqs`) clears them. The frontier
|
|
568
|
+
is `min(live pending.seq) − 1`, or the head `seq` when no live pending entries remain.
|
|
569
|
+
A pending entry older than `PENDING_CLAIM_LEASE_MS` (60s) is treated as abandoned
|
|
570
|
+
(crashed writer) and excluded — a crash can never freeze the frontier forever.
|
|
571
|
+
- A **session-backed** claim commits atomically with its write, so it skips the pending
|
|
572
|
+
registry entirely (nothing to confirm; frontier = head).
|
|
573
|
+
- **Snapshots** never return a `cursor` above `frontierSeq`. `hasMore` is true when a full
|
|
574
|
+
page returned, extra scope-move markers remain, OR the frontier sits below the stream
|
|
575
|
+
head (more committed seqs are coming). A client at the frontier with `hasMore: true`
|
|
576
|
+
and no new entities polls/reconciles (rate-limited) until the frontier moves.
|
|
577
|
+
- **Deltas** carry `frontierSeq`; the client advances its cursor to
|
|
578
|
+
`min(delta.seq, delta.frontierSeq)`, closing the delta-side commit-order inversion with
|
|
579
|
+
no server round-trip.
|
|
580
|
+
- **Consequence:** no committed document is ever permanently skipped by cursor catch-up.
|
|
581
|
+
|
|
582
|
+
Write-path guards landed with C1: a query-write on a synced model MUST target a single
|
|
583
|
+
document by `_id` (m9 — a non-`_id` filter could stamp the wrong stream's seq); a save
|
|
584
|
+
whose only modified paths are auto-managed metadata (`_syncSeq`, `_syncPrevStream`,
|
|
585
|
+
`updated`) claims no seq (m10 — no-op saves do not burn seqs).
|
|
586
|
+
|
|
587
|
+
### C2 — Per-stream snapshots + stream discovery
|
|
588
|
+
|
|
589
|
+
- `GET /sync/streams` → `{streams: [{stream, collection}]}` — the authoritative set of
|
|
590
|
+
streams the caller currently belongs to (resolved against the full user so tenant
|
|
591
|
+
memberships reflect current `organizationIds`).
|
|
592
|
+
- `GET /sync/snapshot?stream={streamKey}&cursor={n}&limit={n}&legacyCursor={opaque}` →
|
|
593
|
+
`{stream, entities, cursor, hasMore, frontierSeq, oldestRetainedSeq, legacyCursor?}`.
|
|
594
|
+
One stream per request. The old `?collection=` param and the single flattened cursor are
|
|
595
|
+
**removed**. The server verifies the caller belongs to the requested stream (403
|
|
596
|
+
otherwise) and filters to the single scope value (`{field: value}`, never `$in`).
|
|
597
|
+
- The client keeps `_cursors` rows keyed by the **real stream key** (the same key deltas
|
|
598
|
+
use). It persists a `_knownStreams` set and, on `start()`/`reconcile()`, diffs
|
|
599
|
+
`GET /sync/streams` against it:
|
|
600
|
+
- **New stream** → bootstrap from cursor 0 (tenant-join backfill).
|
|
601
|
+
- **Removed stream** → purge that stream's local entities + cursor + known-stream entry —
|
|
602
|
+
**only when `GET /sync/streams` returned HTTP 200** (INV-2). A 401 (AuthRequiredError)
|
|
603
|
+
enters auth-pause and leaves everything intact; a transport error rethrows without
|
|
604
|
+
purging. A 401/403/transport error is NEVER a membership change.
|
|
605
|
+
- Each local entity row records the `stream` it was written under, so leave-purge is
|
|
606
|
+
O(stream).
|
|
607
|
+
- **Migration:** a client holding legacy `snapshot:{collection}` cursors deletes them and
|
|
608
|
+
clears `_knownStreams` on start, then re-bootstraps every stream from 0 (idempotent
|
|
609
|
+
upserts + seq guards make this cheap and non-destructive; local entities are not wiped).
|
|
610
|
+
|
|
611
|
+
### C3 — Legacy-doc pagination
|
|
612
|
+
|
|
613
|
+
Documents predating the seq plugin (no `_syncSeq`, or a literal 0) are paged as a separate
|
|
614
|
+
stratum by `_id`. While the stratum has more, the snapshot returns `cursor: 0` plus an
|
|
615
|
+
opaque `legacyCursor` (the last `_id` seen); the client echoes it back verbatim. When the
|
|
616
|
+
stratum is exhausted the server omits `legacyCursor` and paging proceeds by seq. This
|
|
617
|
+
terminates deterministically — no infinite loop when `> limit` unstamped docs exist.
|
|
618
|
+
|
|
619
|
+
### C4 — Scope-move markers (durable, not `_syncPrevStream`)
|
|
620
|
+
|
|
621
|
+
A scope move (owner/tenant change) writes a durable `SyncScopeMove` marker
|
|
622
|
+
`{collectionTag, entityId, fromStream, toStream, seq, created}` in the same op-scope as the
|
|
623
|
+
move, with the `seq` claimed from the OLD stream's counter. The change-stream watcher and
|
|
624
|
+
old-stream snapshot catch-up emit the old-stream tombstone **from the marker**, not from
|
|
625
|
+
the racy `_syncPrevStream` post-image — so a racing second write that resets
|
|
626
|
+
`_syncPrevStream` can no longer erase the tombstone. Markers share the tombstone retention
|
|
627
|
+
window (TTL). An offline old-stream client learns of the move because the marker is merged
|
|
628
|
+
into its snapshot page as a `{deleted: true, data: null}` entity.
|
|
629
|
+
|
|
630
|
+
### C7 — Retention + re-bootstrap
|
|
631
|
+
|
|
632
|
+
Tombstone deltas and snapshot tombstones carry **no data** (only id/seq/deleted). The sync
|
|
633
|
+
bookkeeping collections (`synccounters`, `syncmutations`, `syncscopemoves`, `synckeys`) are
|
|
634
|
+
in `DEFAULT_IGNORED_COLLECTIONS` so their own change events drive no fan-out. Tombstones and
|
|
635
|
+
markers older than the model's `retentionDays` (default 90) may be hard-deleted by the
|
|
636
|
+
`compactTombstones` maintenance script. The snapshot response reports `oldestRetainedSeq`;
|
|
637
|
+
a client whose stored cursor for a stream is **below** it may have missed compacted
|
|
638
|
+
tombstones → it purges that stream and re-bootstraps from 0 (a sanctioned retention-gap
|
|
639
|
+
wipe, distinct from an auth wipe — INV-2).
|