@terreno/mcp 0.31.0 → 56.0.0-beta.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (116) hide show
  1. package/dist/bootstrap.d.ts.map +1 -1
  2. package/dist/bootstrap.js +8 -9
  3. package/dist/bootstrap.js.map +1 -1
  4. package/dist/docs/templates/bootstrap/frontend/app/_layout.template.tsx +1 -1
  5. package/dist/docs/upgrades/56.0.0-beta.2.md +59 -0
  6. package/dist/docs/versioned/next/README.md +2 -0
  7. package/dist/docs/versioned/next/explanation/roadmap-seed-issues.md +21 -0
  8. package/dist/docs/versioned/next/how-to/README.md +2 -0
  9. package/dist/docs/versioned/next/how-to/migrate-rtk-to-syncdb.md +445 -0
  10. package/dist/docs/versioned/next/implementationPlans/PLAN_INDEX.md +28 -50
  11. package/dist/docs/versioned/next/implementationPlans/b2b-platform-program.md +3 -2
  12. package/dist/docs/versioned/next/implementationPlans/comms-abstraction.md +98 -15
  13. package/dist/docs/versioned/next/implementationPlans/comms-adapter-sendgrid.md +63 -13
  14. package/dist/docs/versioned/next/implementationPlans/comms-adapter-twilio-sms.md +65 -12
  15. package/dist/docs/versioned/next/implementationPlans/comms-adapter-twilio-verify.md +9 -3
  16. package/dist/docs/versioned/next/implementationPlans/comms-admin-dashboard.md +180 -0
  17. package/dist/docs/versioned/next/implementationPlans/migrate-cicd-to-circleci-research.md +126 -0
  18. package/dist/docs/versioned/next/implementationPlans/migrate-cicd-to-circleci.md +269 -0
  19. package/dist/docs/versioned/next/implementationPlans/rtk-to-syncdb-migration-docs.md +52 -47
  20. package/dist/docs/versioned/next/implementationPlans/syncdb-api-inventory.md +303 -0
  21. package/dist/docs/versioned/next/implementationPlans/syncdb-codegen.md +252 -0
  22. package/dist/docs/versioned/next/implementationPlans/syncdb-local-first.md +468 -420
  23. package/dist/docs/versioned/next/implementationPlans/syncdb-phase-c-design.md +425 -0
  24. package/dist/docs/versioned/next/implementationPlans/terreno-syncdb-2.md +800 -0
  25. package/dist/docs/versioned/next/reference/README.md +2 -0
  26. package/dist/docs/versioned/next/reference/comms.md +132 -0
  27. package/dist/docs/versioned/next/reference/environment-variables.md +9 -0
  28. package/dist/docs/versioned/next/reference/syncdb.md +456 -0
  29. package/dist/docs/versioned/next/superpowers/plans/2026-07-28-admin-mobile-navigation.md +97 -0
  30. package/dist/docs/versioned/next/superpowers/specs/2026-07-28-admin-mobile-navigation-design.md +36 -0
  31. package/dist/docs/versioned/next/tasks/comms-abstraction.md +29 -16
  32. package/dist/docs/versioned/next/tasks/comms-adapter-sendgrid.md +17 -7
  33. package/dist/docs/versioned/next/tasks/comms-adapter-twilio-sms.md +17 -7
  34. package/dist/docs/versioned/next/tasks/comms-adapter-twilio-verify.md +3 -3
  35. package/dist/docs/versioned/next/tasks/comms-admin-dashboard.md +52 -0
  36. package/dist/docs/versioned/next/tasks/migrate-cicd-to-circleci.md +215 -0
  37. package/dist/docs/versioned/next/tasks/rtk-to-syncdb-migration-docs.md +10 -10
  38. package/dist/docs/versioned/next/tasks/syncdb-codegen.md +65 -0
  39. package/dist/docs/versioned/next/tasks/syncdb-local-first.md +434 -117
  40. package/package.json +3 -3
  41. package/src/docs/templates/bootstrap/frontend/app/_layout.template.tsx +1 -1
  42. package/src/docs/upgrades/56.0.0-beta.2.md +59 -0
  43. package/src/docs/versioned/next/README.md +2 -0
  44. package/src/docs/versioned/next/explanation/roadmap-seed-issues.md +21 -0
  45. package/src/docs/versioned/next/how-to/README.md +2 -0
  46. package/src/docs/versioned/next/how-to/migrate-rtk-to-syncdb.md +445 -0
  47. package/src/docs/versioned/next/implementationPlans/PLAN_INDEX.md +28 -50
  48. package/src/docs/versioned/next/implementationPlans/b2b-platform-program.md +3 -2
  49. package/src/docs/versioned/next/implementationPlans/comms-abstraction.md +98 -15
  50. package/src/docs/versioned/next/implementationPlans/comms-adapter-sendgrid.md +63 -13
  51. package/src/docs/versioned/next/implementationPlans/comms-adapter-twilio-sms.md +65 -12
  52. package/src/docs/versioned/next/implementationPlans/comms-adapter-twilio-verify.md +9 -3
  53. package/src/docs/versioned/next/implementationPlans/comms-admin-dashboard.md +180 -0
  54. package/src/docs/versioned/next/implementationPlans/migrate-cicd-to-circleci-research.md +126 -0
  55. package/src/docs/versioned/next/implementationPlans/migrate-cicd-to-circleci.md +269 -0
  56. package/src/docs/versioned/next/implementationPlans/rtk-to-syncdb-migration-docs.md +52 -47
  57. package/src/docs/versioned/next/implementationPlans/syncdb-api-inventory.md +303 -0
  58. package/src/docs/versioned/next/implementationPlans/syncdb-codegen.md +252 -0
  59. package/src/docs/versioned/next/implementationPlans/syncdb-local-first.md +468 -420
  60. package/src/docs/versioned/next/implementationPlans/syncdb-phase-c-design.md +425 -0
  61. package/src/docs/versioned/next/implementationPlans/terreno-syncdb-2.md +800 -0
  62. package/src/docs/versioned/next/reference/README.md +2 -0
  63. package/src/docs/versioned/next/reference/comms.md +132 -0
  64. package/src/docs/versioned/next/reference/environment-variables.md +9 -0
  65. package/src/docs/versioned/next/reference/syncdb.md +456 -0
  66. package/src/docs/versioned/next/superpowers/plans/2026-07-28-admin-mobile-navigation.md +97 -0
  67. package/src/docs/versioned/next/superpowers/specs/2026-07-28-admin-mobile-navigation-design.md +36 -0
  68. package/src/docs/versioned/next/tasks/comms-abstraction.md +29 -16
  69. package/src/docs/versioned/next/tasks/comms-adapter-sendgrid.md +17 -7
  70. package/src/docs/versioned/next/tasks/comms-adapter-twilio-sms.md +17 -7
  71. package/src/docs/versioned/next/tasks/comms-adapter-twilio-verify.md +3 -3
  72. package/src/docs/versioned/next/tasks/comms-admin-dashboard.md +52 -0
  73. package/src/docs/versioned/next/tasks/migrate-cicd-to-circleci.md +215 -0
  74. package/src/docs/versioned/next/tasks/rtk-to-syncdb-migration-docs.md +10 -10
  75. package/src/docs/versioned/next/tasks/syncdb-codegen.md +65 -0
  76. package/src/docs/versioned/next/tasks/syncdb-local-first.md +434 -117
  77. package/dist/__tests__/bootstrap.test.d.ts +0 -2
  78. package/dist/__tests__/bootstrap.test.d.ts.map +0 -1
  79. package/dist/__tests__/bootstrap.test.js +0 -299
  80. package/dist/__tests__/bootstrap.test.js.map +0 -1
  81. package/dist/__tests__/databaseQuery.test.d.ts +0 -2
  82. package/dist/__tests__/databaseQuery.test.d.ts.map +0 -1
  83. package/dist/__tests__/databaseQuery.test.js +0 -83
  84. package/dist/__tests__/databaseQuery.test.js.map +0 -1
  85. package/dist/__tests__/docsRoot.test.d.ts +0 -2
  86. package/dist/__tests__/docsRoot.test.d.ts.map +0 -1
  87. package/dist/__tests__/docsRoot.test.js +0 -59
  88. package/dist/__tests__/docsRoot.test.js.map +0 -1
  89. package/dist/__tests__/inferPackages.test.d.ts +0 -2
  90. package/dist/__tests__/inferPackages.test.d.ts.map +0 -1
  91. package/dist/__tests__/inferPackages.test.js +0 -35
  92. package/dist/__tests__/inferPackages.test.js.map +0 -1
  93. package/dist/__tests__/preload.d.ts +0 -2
  94. package/dist/__tests__/preload.d.ts.map +0 -1
  95. package/dist/__tests__/preload.js +0 -12
  96. package/dist/__tests__/preload.js.map +0 -1
  97. package/dist/__tests__/prompts.test.d.ts +0 -2
  98. package/dist/__tests__/prompts.test.d.ts.map +0 -1
  99. package/dist/__tests__/prompts.test.js +0 -295
  100. package/dist/__tests__/prompts.test.js.map +0 -1
  101. package/dist/__tests__/resources.test.d.ts +0 -2
  102. package/dist/__tests__/resources.test.d.ts.map +0 -1
  103. package/dist/__tests__/resources.test.js +0 -352
  104. package/dist/__tests__/resources.test.js.map +0 -1
  105. package/dist/__tests__/search.test.d.ts +0 -2
  106. package/dist/__tests__/search.test.d.ts.map +0 -1
  107. package/dist/__tests__/search.test.js +0 -163
  108. package/dist/__tests__/search.test.js.map +0 -1
  109. package/dist/__tests__/tools.test.d.ts +0 -2
  110. package/dist/__tests__/tools.test.d.ts.map +0 -1
  111. package/dist/__tests__/tools.test.js +0 -540
  112. package/dist/__tests__/tools.test.js.map +0 -1
  113. package/dist/__tests__/upgradeGuide.test.d.ts +0 -2
  114. package/dist/__tests__/upgradeGuide.test.d.ts.map +0 -1
  115. package/dist/__tests__/upgradeGuide.test.js +0 -22
  116. package/dist/__tests__/upgradeGuide.test.js.map +0 -1
@@ -0,0 +1,425 @@
1
+ # SyncDB Phase C Design (syncdb-2)
2
+
3
+ Design authority doc for Phase C (server protocol correctness). Hand this verbatim to a
4
+ Sonnet implementer. Read alongside `terreno-syncdb-2.md` (INV-1..4, C1-C8). C5 (mutation
5
+ ledger lease) is owned/implemented elsewhere — this doc designs *around* it and never
6
+ edits `mutationHandler.ts` beyond what C6/C8 already touch.
7
+
8
+ Global invariants restated in Phase-C terms: INV-1 ordering is per-stream monotonic;
9
+ INV-2 never purge/wipe on 401 — only on confirmed membership change or explicit wipe;
10
+ INV-3 idempotency ledger owns exactly-once; INV-4 repo conventions.
11
+
12
+ Line refs are anchors, not truth — locate by described code.
13
+
14
+ ---
15
+
16
+ ## 1. C1 — Seq commit fence
17
+
18
+ ### Decision: **Option 2 (stable frontier) with a hardened optimistic claim.** Reject Option 1 and 3.
19
+
20
+ **Why not Option 1 (transactional counter+write).** Every synced write currently flows
21
+ through Mongoose `save()` (executors AND arbitrary consumer `.save()` in pre/post hooks
22
+ and app code) or the query-write hooks. A transaction requires a `ClientSession` plumbed
23
+ through *every* one of those call sites. `syncSeqPlugin.claimSyncSeqs` already joins
24
+ `this.$session()` when present, but the executors do NOT open a session, and we cannot
25
+ force a session onto consumer code that calls `doc.save()` directly (the plugin's own
26
+ doc comment admits "without a caller session the claim is a plain atomic `$inc`"). A
27
+ transaction that only wraps the executor path leaves every direct-`.save()` write
28
+ un-fenced — a partial fix that hides the bug. Honest assessment: full transactional
29
+ coverage is **not achievable** without a breaking API change forcing sessions on all
30
+ consumers. Rejected.
31
+
32
+ **Why not Option 3 (post-commit sequencing singleton).** Adds an ordering singleton
33
+ (SPOF + horizontal-scale hazard) and a second write per mutation; makes the ack path
34
+ async w.r.t. seq assignment, breaking the current synchronous `ack.seq` contract that
35
+ the client's `baseVersion` chaining (A2) depends on. Rejected.
36
+
37
+ **Chosen: stable frontier.** Keep the cheap optimistic `$inc` claim (no transaction, no
38
+ per-write session plumbing — zero new cost on the hot path). Add a *frontier* that
39
+ consumers (snapshot + delta cursor advancement) never cross until the claim is known
40
+ committed. A client cursor may advance to seq N **only when every seq ≤ N in that stream
41
+ is committed** (its owning write durably landed or its claim was reclaimed).
42
+
43
+ #### Write-path algorithm (`syncSeqPlugin.ts` + `models.ts`)
44
+
45
+ Extend `SyncCounter` with an in-flight registry so the frontier is computable:
46
+
47
+ ```
48
+ SyncCounterDocument {
49
+ stream: string; // unchanged
50
+ seq: number; // last claimed (unchanged)
51
+ pending: Array<{seq: number; claimedAt: Date}>; // NEW — uncommitted claims
52
+ }
53
+ ```
54
+
55
+ `claimSyncSeqs({stream, count, session})` becomes two atomic phases:
56
+
57
+ 1. **Claim + register (pre-write).** One `findOneAndUpdate`:
58
+ `{$inc: {seq: count}, $push: {pending: {$each: [{seq, claimedAt: now}, ...]}}}`
59
+ returning `new: true`. The claimed range is `[seq-count+1, seq]`. Stamp `_syncSeq`
60
+ as today. (Still joins `this.$session()` when a caller opened one — that path gets
61
+ true atomicity for free and skips the pending registry via a fast path below.)
62
+ 2. **Confirm (post-write).** After the document write commits, remove the claim:
63
+ `{$pull: {pending: {seq: {$in: claimedRange}}}}`. This runs in `schema.post("save")`
64
+ and in a `post` hook on the query-writes. A crash between phases leaves a stale
65
+ `pending` entry — handled by the reclaim lease below.
66
+
67
+ Fast path: when a caller session is present, the write and `$inc` are already atomic;
68
+ skip the pending registry entirely (register nothing, confirm nothing). Frontier logic
69
+ treats a stream with no `pending` entries as fully committed up to `seq`.
70
+
71
+ **Frontier computation** (`models.ts`, `computeStableFrontier({stream})`):
72
+ `frontier = (min pending.seq) - 1`, or `seq` (the head) when `pending` is empty. A stale
73
+ pending entry (claimedAt older than `PENDING_CLAIM_LEASE_MS`, default 60s) is treated as
74
+ abandoned: it is `$pull`ed opportunistically at read time and excluded from the min, so a
75
+ crashed writer cannot freeze the frontier forever. (Liveness cost of the frontier: a live
76
+ write in flight briefly holds the frontier at its seq-1; snapshots see the gap fill within
77
+ one write latency, or one lease on crash.)
78
+
79
+ #### Where the frontier gates cursor advancement
80
+
81
+ - **Snapshot (`routes.ts`):** the page's `seqFilter` upper-bounds at the frontier:
82
+ `{_syncSeq: {$gt: cursor, $lte: frontier}}`. `hasMore` is true when either the frontier
83
+ is below `seq` (more coming, not yet committed) OR a full page was returned. The response
84
+ `cursor` is `min(lastEntitySeq, frontier)` — never advance a client past an uncommitted
85
+ hole. A client at the frontier with `hasMore:true` and zero new entities polls/reconciles
86
+ (rate-limited) until the frontier moves.
87
+ - **Delta (`changeStreamWatcher.ts`):** deltas are only observed via the change stream
88
+ *after* commit, so a delta's seq is always ≤ committed. BUT a delta can arrive for seq N
89
+ while N-1's claim is still pending (out-of-commit-order). The watcher already emits deltas
90
+ as they arrive and the client's `applyDelta` advances the per-stream cursor to `delta.seq`
91
+ unconditionally. **Fix:** the watcher stamps each delta with the *stream frontier at emit
92
+ time* (`frontierSeq` field on `SyncDelta`); the client advances its cursor to
93
+ `min(delta.seq, delta.frontierSeq)` (see §5). This closes the delta-side inversion with
94
+ no server round-trip.
95
+
96
+ #### Multi-doc lookup mis-stamp (m9)
97
+
98
+ `preQueryWrite` does `model.find(filter).limit(1)` — a non-`_id` filter can match a
99
+ different doc than the one the update targets, stamping the wrong stream's seq. **Fix:**
100
+ refuse to sequence a query-write whose filter is not a single-document identity filter.
101
+ Require the filter to contain `_id` (or `{_id: {$eq}}`); if absent, throw the same loud
102
+ `unsupportedWrite`-style error ("query-write on a synced model must target a single
103
+ document by _id; use findByIdAndUpdate or loop"). This is the same contract stance the
104
+ plugin already takes for `updateMany`/`upsert`. Consumer code that did ad-hoc
105
+ `updateOne({field: x})` on a synced model must migrate to `_id`-targeted writes — a loud,
106
+ one-time break beats silent duplicate seqs.
107
+
108
+ #### No-op save seq burn (m10)
109
+
110
+ `schema.pre("save")` claims a seq even when nothing changed. **Fix:** in `pre("save")`,
111
+ if `!this.isNew && this.modifiedPaths().filter(p => p !== "_syncSeq" && p !== "_syncPrevStream").length === 0`,
112
+ skip the claim and leave `_syncSeq` untouched (no delta emitted because the doc write is a
113
+ no-op Mongoose skips anyway). For query-writes, if the effective `$set`/replacement is
114
+ empty, skip.
115
+
116
+ #### Failure / retry semantics
117
+
118
+ - Claim registered but write throws → the confirm never runs; the pending entry ages out
119
+ in one lease and is reclaimed. The burned seq is a benign gap (client rate-limited
120
+ reconcile, as today). No transaction to roll back.
121
+ - Confirm `$pull` throws after a successful write → log `logger.error`; the entry ages out
122
+ via lease. Never fail the user write for a confirm error.
123
+ - `TransientTransactionError` — N/A (no transactions).
124
+
125
+ #### Client-visible contract (write into `syncdb-local-first.md`)
126
+
127
+ > A client cursor for a stream may advance to seq N only when the server reports N ≤ that
128
+ > stream's *stable frontier*: every seq ≤ N in the stream is committed. Snapshots never
129
+ > return a cursor above the frontier; deltas carry the frontier so the client clamps.
130
+ > Consequence: no committed document is ever permanently skipped by cursor catch-up.
131
+
132
+ #### Performance
133
+
134
+ Hot path adds one array `$push` on claim and one `$pull` on confirm per write (both on the
135
+ already-fetched counter doc, indexed by `stream`). `pending` stays tiny (bounded by
136
+ concurrent in-flight writes per stream, seconds-scale). Session-backed writes pay nothing.
137
+
138
+ #### Migration of existing stamped data
139
+
140
+ None required for stamped docs — they are already committed (no pending entries exist
141
+ pre-deploy; `pending` defaults to `[]`, frontier = head). The counter's new `pending`
142
+ field is additive and `strict:"throw"`-safe (defined in schema).
143
+
144
+ ---
145
+
146
+ ## 2. C2 — Per-stream snapshot cursors
147
+
148
+ ### Decision: **one stream per request**, with a discovery endpoint.
149
+
150
+ Simpler than `cursors: Record<stream,number>` (no partial-progress bookkeeping in one
151
+ response; each request is independently retryable and rate-limited; maps 1:1 onto the
152
+ existing `_cursors` rows). The collection-flattened cursor is removed.
153
+
154
+ #### Stream discovery — `GET /sync/streams`
155
+
156
+ ```
157
+ GET /sync/streams -> { streams: Array<{stream: string; collection: string}> }
158
+ ```
159
+
160
+ Implementation reuses `resolveUserStreams` (already in `socketHandlers.ts`, extract to
161
+ `streams.ts` as `resolveUserStreamsForEntry`): for each registered entry the user can
162
+ `list`, resolve owner/tenant/broadcast/custom stream keys. This is the authoritative set
163
+ of streams the user currently belongs to. Runs against the **full user** (D2) so tenant
164
+ memberships resolve.
165
+
166
+ #### Snapshot request/response
167
+
168
+ ```
169
+ GET /sync/snapshot?stream={streamKey}&cursor={n}&limit={n}
170
+ -> SyncStreamSnapshotResponse {
171
+ stream: string;
172
+ entities: SyncEntityPayload[]; // unchanged shape
173
+ cursor: number; // clamped to frontier (§1)
174
+ hasMore: boolean;
175
+ frontierSeq: number; // stream stable frontier (§1)
176
+ oldestRetainedSeq: number; // C7 retention floor (§ interactions)
177
+ }
178
+ ```
179
+
180
+ Server decodes `stream` → `{collectionTag, scopeValue}`, looks up the entry, verifies the
181
+ stream is in the user's `resolveUserStreamsForEntry` set (403 otherwise — a client must
182
+ not snapshot a stream it does not belong to), builds the scope filter as
183
+ `{[scopeField]: scopeValue}` (single value, not `$in`), and pages with the frontier-bounded
184
+ `seqFilter` from §1. Keep the C3 legacy stratum (§3) and C6 per-doc read checks.
185
+
186
+ The old `?collection=` param is removed. `snapshotFilter` for custom scopes still applies
187
+ but is now parameterized by the single stream value.
188
+
189
+ #### Client stream discovery, join, leave (INV-2)
190
+
191
+ Client keeps `_cursors` rows keyed by the **real stream key** (delta plumbing already does
192
+ this). The `snapshot:{collection}` pseudo-cursors are deleted.
193
+
194
+ - On `start()` and each `reconcile()`: call `GET /sync/streams`. Diff against the streams
195
+ present in `_cursors` and a persisted `_knownStreams` set:
196
+ - **New stream** (in server set, not in `_knownStreams`) → cursor 0, bootstrap it
197
+ incrementally (cheap: idempotent upserts). Add to `_knownStreams`. This is the
198
+ tenant-join backfill.
199
+ - **Removed stream** (in `_knownStreams`, absent from server set) → **only when the
200
+ `GET /sync/streams` call succeeded (HTTP 200)** purge that stream's local entities and
201
+ its cursor + `_knownStreams` entry. A 401/403/transport error is NOT a membership
202
+ change (INV-2): leave everything intact, enter/keep auth-pause if 401. Purge = delete
203
+ all entities whose stream resolves to the removed key (client stores `collection`; it
204
+ must be able to map entity→stream — see §5 client work item).
205
+ - Bootstrap iterates `GET /sync/streams` result, not `config.collections`. `config.collections`
206
+ becomes the *subscribe* list only.
207
+
208
+ #### Migration for deployed clients holding legacy `snapshot:{collection}` cursors
209
+
210
+ On `start()`, if any `_cursors` row key matches `/^snapshot:/`, delete all of them and
211
+ clear `_knownStreams`, then let the normal discovery path bootstrap every stream from
212
+ cursor 0. Idempotent upserts + seq guards make re-bootstrap cheap and non-destructive
213
+ (existing entities keep their data; only stale/lower seqs are re-fetched). Gate this behind
214
+ the E2 schema-version bump so it runs exactly once. Legacy entities already local are NOT
215
+ wiped — this is a cursor migration, not a data wipe.
216
+
217
+ ---
218
+
219
+ ## 3. C3 — Legacy-doc pagination
220
+
221
+ ### Decision: **`_id`-paged legacy stratum with an opaque `legacyCursor`.** Reject the migration-script-only option.
222
+
223
+ A back-stamp migration must run per stream across all deployments and races live writes;
224
+ the `_id` stratum is self-contained in the route and needs no ops coordination. (A
225
+ back-stamp maintenance script MAY still be offered later as an optimization, but the route
226
+ must be correct without it.)
227
+
228
+ #### Wire change
229
+
230
+ Snapshot request gains optional `legacyCursor` (opaque string = last `_id` seen in the
231
+ seq-0 stratum). Response gains optional `legacyCursor` (echo-forward token; absent when the
232
+ legacy stratum is exhausted).
233
+
234
+ Algorithm in `routes.ts`:
235
+
236
+ - When `cursor === 0` AND no `legacyCursor` yet, OR `legacyCursor` present: page the
237
+ **seq-0 stratum** with `{_syncSeq: {$exists: false}}` (or `{$in: [null, 0]}` for legacy
238
+ zeros), sorted `{_id: 1}`, filter `_id > legacyCursor`, `limit+1`. Return
239
+ `cursor: 0`, `legacyCursor: <last _id>`, `hasMore: true` while the stratum has more.
240
+ - When the seq-0 stratum is exhausted, return `legacyCursor: undefined` and switch to
241
+ normal seq paging (`{_syncSeq: {$gt: 0, $lte: frontier}}`) starting at `cursor: 0`.
242
+ - Client (`bootstrap.ts`) echoes `legacyCursor` back until absent, then proceeds by seq.
243
+
244
+ Loud failure: none needed — the stratum terminates deterministically by `_id`. Keep the
245
+ existing client loop-guard (`hasMore && page.cursor > cursor`) but extend it to also treat
246
+ `legacyCursor` advancement as progress (so a page of seq-0 docs with an advancing
247
+ `legacyCursor` does not trip the guard).
248
+
249
+ ---
250
+
251
+ ## 4. C4 — Scope-move tombstones from the write path
252
+
253
+ ### Decision: explicit `SyncScopeMove` marker doc written in the same op as the move.
254
+
255
+ The current post-image read of `_syncPrevStream` (`changeStreamWatcher.ts` ~576) is racy:
256
+ a second write resets the field before the first change event is processed. Move the signal
257
+ to the write.
258
+
259
+ #### Schema (`models.ts`, new `SyncScopeMove`)
260
+
261
+ ```
262
+ SyncScopeMoveDocument {
263
+ _id;
264
+ collection: string; // collectionTag
265
+ entityId: string; // moved doc _id
266
+ fromStream: string; // old stream (the one to tombstone)
267
+ toStream: string; // new stream
268
+ seq: number; // claimed from the OLD stream's counter
269
+ created: Date; // TTL-indexed, same retention as tombstones (C7)
270
+ }
271
+ ```
272
+
273
+ Index `{fromStream: 1, seq: 1}` for old-stream snapshot catch-up. TTL index on `created`
274
+ with the model's tombstone retention (default 90d, C7).
275
+
276
+ #### When written
277
+
278
+ In `syncSeqPlugin` (both `pre("save")` and `preQueryWrite`) when a scope move is detected
279
+ (`prevStream !== null`): claim a seq from the **old** stream's counter and insert a
280
+ `SyncScopeMove` marker, joining the caller session when present so it commits with the
281
+ move. Still stamp `_syncPrevStream` on the doc (harmless; the marker is now the source of
282
+ truth). The marker's seq participates in the old stream's frontier (§1) exactly like a
283
+ tombstone.
284
+
285
+ #### Watcher emission
286
+
287
+ Add `synccounters`, `syncmutations`, `syncscopemoves`, `synckeys` to
288
+ `DEFAULT_IGNORED_COLLECTIONS` (C7) so the marker's own change event does NOT drive fan-out.
289
+ Instead, when the watcher processes the *moved document's* change event (or a dedicated
290
+ watch on `SyncScopeMove` inserts — pick the document-event path to reuse existing plumbing),
291
+ it emits the old-stream tombstone from the marker's `fromStream` + `seq` rather than from
292
+ the racy `_syncPrevStream` post-image. Concretely: change `emitSyncDelta` to, on any synced
293
+ doc change, look up any `SyncScopeMove` markers for `{collection, entityId}` with
294
+ `seq > lastEmittedForOldStream` and emit a `{deleted:true}` tombstone to each `fromStream`
295
+ room, then emit the create/normal delta to the new stream. Because the marker is durable,
296
+ a racing second write cannot erase it.
297
+
298
+ #### Snapshot catch-up on the old stream
299
+
300
+ `GET /sync/snapshot?stream={oldStream}` must merge `SyncScopeMove` markers into its page:
301
+ markers with `fromStream === stream` and `seq in (cursor, frontier]` become tombstone
302
+ entities (`{id: entityId, deleted: true, seq, data: null}`). Union with the normal entity
303
+ page, sort by seq, page by frontier. This is how a client that was offline during the move
304
+ learns the doc left its stream.
305
+
306
+ #### TTL / retention
307
+
308
+ Same window as tombstones (C7 `retentionDays`, default 90). A cursor older than
309
+ `oldestRetainedSeq` triggers full re-bootstrap (§ interactions), which re-derives current
310
+ membership and drops the moved-away doc naturally.
311
+
312
+ ---
313
+
314
+ ## 5. Interactions & client work items
315
+
316
+ **C1 fence ↔ C2 cursor.** C2's per-stream `cursor` is upper-bounded by C1's `frontierSeq`.
317
+ The snapshot response carries `frontierSeq`; the client stores it but advances its
318
+ `_cursors` seq only to the returned `cursor` (already clamped). Deltas carry `frontierSeq`;
319
+ `applyDelta` advances the cursor to `min(delta.seq, delta.frontierSeq)`.
320
+
321
+ **C7 retention ↔ C2 response.** `oldestRetainedSeq` (the lowest seq still retained for the
322
+ stream after tombstone compaction) is added to the snapshot response. On reconcile, if the
323
+ client's stored cursor for a stream `< oldestRetainedSeq`, it may have missed compacted
324
+ tombstones → drop that stream's local entities + cursor and re-bootstrap from 0 (sanctioned
325
+ wipe: retention gap, not auth).
326
+
327
+ ### Exact syncdb client work items
328
+
329
+ 1. `types.ts` — add `frontierSeq` and (optional) `legacyCursor` to `SyncSnapshotResponse`;
330
+ add `SyncStreamInfo`/`GET /sync/streams` response type; add `frontierSeq` to `SyncDelta`.
331
+ 2. `httpChannel.ts` — `fetchSnapshotPage` sends `stream` (not `collection`) + optional
332
+ `legacyCursor`, returns the new fields; add `fetchStreams(): Promise<SyncStreamInfo[]>`.
333
+ 3. `cursor.ts` — unchanged API; document that keys are now real stream keys only.
334
+ 4. `bootstrap.ts` — rewrite to page **per stream** (loop streams from `fetchStreams`),
335
+ echo `legacyCursor`, clamp cursor to `frontierSeq`, honor `oldestRetainedSeq`
336
+ re-bootstrap. Remove `snapshotCursorStream`/`snapshot:` pseudo-cursors.
337
+ 5. `deltaApplier.ts` — advance cursor to `min(delta.seq, delta.frontierSeq)`.
338
+ 6. `client.ts` — `start()`/`reconcile()` call `fetchStreams`, run join-backfill /
339
+ leave-purge diff against persisted `_knownStreams` (new store table), gated by HTTP-200
340
+ success (INV-2); legacy `snapshot:` cursor migration on start (behind E2 version bump);
341
+ entity→stream mapping helper for purge (store `stream` on each entity row OR recompute
342
+ from `collection`+scope — prefer storing `stream` at apply time).
343
+ 7. Store schema — add `_knownStreams` table and (recommended) a `stream` column on entity
344
+ rows to make leave-purge O(stream). Bump `SYNC_SCHEMA_VERSION` (triggers E2 wipe path
345
+ for old stores — acceptable, re-bootstrap is cheap).
346
+
347
+ ---
348
+
349
+ ## 6. Test matrix
350
+
351
+ **C1 (concurrency, `api/src/sync/sync.test.ts` + new `syncFrontier.test.ts`):**
352
+ - Forced commit inversion: two writers claim seq 5 and 6 on one stream; a `pre("save")`
353
+ test hook delays writer-5's commit. Assert `computeStableFrontier` returns 4 while 5 is
354
+ pending; snapshot at cursor 4 returns nothing (hasMore true); after 5 commits, frontier
355
+ jumps to 6 and snapshot returns both docs. A client looping catch-up at every intermediate
356
+ cursor eventually sees BOTH.
357
+ - Property-style: N writers, random commit interleavings + random crashes (skip confirm);
358
+ after lease expiry, every committed doc appears in a full catch-up; no committed doc
359
+ skipped. Loop 100 seeds.
360
+ - m9: `updateOne({nonIdField: x})` on a synced model throws the loud error.
361
+ - m10: `doc.save()` with no modified paths claims no new seq (counter unchanged, no delta).
362
+ - Migration: pre-existing stamped docs (empty `pending`) → frontier = head immediately.
363
+
364
+ **C2 (`syncRoutes.test.ts` + integration):**
365
+ - Two-tenant skew: user in org1 (seq 1000) + org2 (seq 5). Per-stream snapshot catches org2
366
+ to 5 and org1 to 1000 independently; the old flattened cursor bug (org2 stuck at 1000)
367
+ does NOT reproduce.
368
+ - `GET /sync/streams` returns owner + both tenant streams; reflects `organizationIds`.
369
+ - Join: add org3 membership → `fetchStreams` includes it → client bootstraps org3 from 0.
370
+ - Leave: remove org2 → `fetchStreams` omits it (200) → client purges org2 entities.
371
+ - Leave under 401: `fetchStreams` 401 → NO purge, auth-pause, data intact (INV-2 regression).
372
+ - Legacy migration: store seeded with `snapshot:todos` cursor → start() deletes it,
373
+ re-bootstraps per stream, entities converge, no data loss.
374
+
375
+ **C3 (`syncRoutes.test.ts` + `bootstrap.test.ts`):**
376
+ - 1,201 legacy docs (no `_syncSeq`), limit 500 → bootstrap terminates with all 1,201 via
377
+ `legacyCursor` paging, then continues into seq stratum. Loop-guard not tripped.
378
+ - Mixed legacy + stamped: legacy stratum fully drained before seq paging begins; no doc
379
+ fetched twice, none missed.
380
+
381
+ **C4 (`changeStreamWatcher.test.ts` / integration):**
382
+ - Move + immediate second write with artificially delayed change-event processing → old
383
+ stream still receives the tombstone (from the marker, not the overwritten
384
+ `_syncPrevStream`). Assert tombstone delivered to `fromStream` room.
385
+ - Offline old-stream client: snapshot from a pre-move cursor returns the move as a tombstone
386
+ entity (marker merged into the page).
387
+ - Marker TTL/retention: marker older than `retentionDays` is absent; cursor older than
388
+ `oldestRetainedSeq` triggers re-bootstrap.
389
+
390
+ ---
391
+
392
+ ## 7. Implementation task list (ordered, Sonnet-executable)
393
+
394
+ 1. **`models.ts`** — add `pending` to `SyncCounter`; rewrite `claimSyncSeqs` (claim+register
395
+ / session fast path); add `confirmSyncSeqs({stream, seqs, session})` (`$pull`) and
396
+ `computeStableFrontier({stream})` (min-pending-1, opportunistic stale `$pull`,
397
+ `PENDING_CLAIM_LEASE_MS`). Add `SyncScopeMove` model + indexes. **[Opus-care: the
398
+ frontier + lease math and the session fast path — get the pending-registry invariants
399
+ reviewed.]**
400
+ 2. **`syncSeqPlugin.ts`** — no-op skip (m10); `_id`-only guard for query-writes (m9);
401
+ call `confirmSyncSeqs` in `post("save")` + query-write post hooks; write `SyncScopeMove`
402
+ marker on detected scope move (claim from old stream). **[Opus-care: post-hook wiring —
403
+ the `this` context differs between save and query middleware.]**
404
+ 3. **`streams.ts`** — extract `resolveUserStreamsForEntry` from `socketHandlers.ts` (shared
405
+ by socket subscribe + `GET /sync/streams` + snapshot stream-membership check).
406
+ 4. **`routes.ts`** — `GET /sync/streams`; rewrite `GET /sync/snapshot` to per-stream
407
+ (membership check, single-value scope filter, frontier-bounded seqFilter, `legacyCursor`
408
+ stratum (C3), `frontierSeq` + `oldestRetainedSeq` in response, `SyncScopeMove` merge for
409
+ old-stream catch-up). Remove `?collection=` path.
410
+ 5. **`changeStreamWatcher.ts`** — add sync bookkeeping collections to
411
+ `DEFAULT_IGNORED_COLLECTIONS`; emit old-stream tombstones from `SyncScopeMove` markers
412
+ instead of `_syncPrevStream`; stamp `frontierSeq` on emitted deltas. **[Opus-care:
413
+ change-stream/marker interleaving.]**
414
+ 6. **`types.ts` (api + syncdb)** — add `frontierSeq`, `legacyCursor`, `oldestRetainedSeq`,
415
+ `SyncScopeMove`/streams types, per-stream snapshot response.
416
+ 7. **syncdb client** — items 1-7 of §5 (httpChannel, bootstrap, deltaApplier, client,
417
+ store schema + version bump, `_knownStreams`, entity `stream` column). **[Opus-care:
418
+ the join/leave diff under INV-2 — the 200-gate is the whole ballgame.]**
419
+ 8. **Tests** — the full §6 matrix, one focused test per fix.
420
+ 9. **Docs** — `syncdb-local-first.md`: the C1 cursor contract, per-stream snapshot protocol,
421
+ `GET /sync/streams`, `SyncScopeMove`, retention/`oldestRetainedSeq` re-bootstrap rule.
422
+
423
+ Sequencing: 1→2 (write path) can land before 3→6 (read path); 7 (client) depends on 6; C3
424
+ (within 4) and C4 (2/4/5) are independent slices. C1 fence must land before C2 client rollout
425
+ so cursors never advance past uncommitted seqs.