@terreno/mcp 57.13.0 → 57.15.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,43 @@
1
+ # Upgrading to 57.14.0
2
+
3
+ **Action required:** No
4
+ **Affected packages:** @terreno/syncdb
5
+
6
+ ## Summary
7
+
8
+ `@terreno/syncdb` pages snapshot streams in parallel (up to 6 at once by default) instead of one at a time, so cold start, reconcile, subscribe catch-up, `forceResync`, and wipe re-bootstrap finish faster when an app has many collections or scopes. Each stream now runs one snapshot pass at a time, which stops a concurrent pass from marking a purged stream bootstrapped over rows it never paged. No code change is required.
9
+
10
+ ## Breaking changes
11
+
12
+ None
13
+
14
+ ## Deprecations
15
+
16
+ None
17
+
18
+ ## New capabilities
19
+
20
+ ### `snapshotConcurrency` on `createSyncDb`
21
+
22
+ The client now sends up to `snapshotConcurrency` snapshot requests at once (default `DEFAULT_SNAPSHOT_CONCURRENCY`, 6). If your backend cannot take the extra parallel load at cold start, lower it. `1` restores the old one-stream-at-a-time behavior.
23
+
24
+ ```typescript
25
+ // Before
26
+ export const syncDb = createSyncDb({authProvider, baseUrl, collections, name: "my-app"});
27
+
28
+ // After (only if you need to limit parallel snapshot requests)
29
+ export const syncDb = createSyncDb({
30
+ authProvider,
31
+ baseUrl,
32
+ collections,
33
+ name: "my-app",
34
+ snapshotConcurrency: 2,
35
+ });
36
+ ```
37
+
38
+ A new how-to, `docs/how-to/use-syncdb-correctly.md`, lists library behavior apps have to handle.
39
+
40
+ ## Verification
41
+
42
+ - `bun run compile` and `bun run lint` at the app root
43
+ - Log in on a fresh device or after `wipeLocalData`; synced screens load, and the backend logs show several `/sync/snapshot` requests in flight together
@@ -0,0 +1,44 @@
1
+ # Upgrading to 57.15.0
2
+
3
+ **Action required:** No
4
+ **Affected packages:** @terreno/api, @terreno/syncdb
5
+
6
+ ## Summary
7
+
8
+ SyncDB pages every stream through one `POST /sync/snapshot/batch` request per round instead of
9
+ one `GET /sync/snapshot` per stream. With many synced collections, cold start is faster and a
10
+ reconnect with no changes costs 2 requests (`GET /sync/streams` plus one batch) whatever the
11
+ collection count. The web IndexedDB persister now serializes the store once per write instead of
12
+ once per transaction, so bootstrap no longer leaves a long save backlog.
13
+
14
+ ## Breaking changes
15
+
16
+ None
17
+
18
+ ## Deprecations
19
+
20
+ None
21
+
22
+ ## New capabilities
23
+
24
+ ### `POST /sync/snapshot/batch`
25
+
26
+ `SyncApp` mounts the route automatically. The body is
27
+ `{streams: [{stream, cursor?, legacyCursor?}], limit?}`, with at most 500 streams.
28
+ `limit` is the row budget for the whole response (default 5,000, max 10,000). The response is
29
+ `{pages}`, one `GET /sync/snapshot`-shaped page per stream. `GET /sync/snapshot` is unchanged.
30
+
31
+ Upgrade `@terreno/api` and `@terreno/syncdb` together for the speedup. A 57.15.0 client talking
32
+ to an older server sees a 404 from the batch route and falls back to per-stream paging
33
+ (`snapshotConcurrency` streams at a time). An older client keeps using `GET /sync/snapshot`.
34
+
35
+ ### `snapshotConcurrency` scope
36
+
37
+ `createSyncDb({snapshotConcurrency})` now applies only to per-stream paging: `forceResync()` and
38
+ the fallback against servers without the batch route.
39
+
40
+ ## Verification
41
+
42
+ - `bun run compile` and `bun run lint` at the app root
43
+ - After deploying the backend, open the web app with dev tools and confirm a reconnect sends one
44
+ `POST /sync/snapshot/batch` instead of one `GET /sync/snapshot` per collection
@@ -159,13 +159,13 @@ Ensures preview environments don't accumulate and waste resources.
159
159
 
160
160
  #### Dependabot Auto-merge
161
161
  **Trigger:** Dependabot pull request opened
162
- **Purpose:** Automatically merges dependency updates after CI passes
162
+ **Purpose:** Automatically merges Dependabot security updates after CI passes. Dependabot opens no version-update PRs; routine upgrades run through the `update-dependencies` skill.
163
163
 
164
164
  **Requirements for auto-merge:**
165
165
  - All CI checks pass
166
166
  - Dependabot PR (created by Dependabot)
167
+ - Not a semver-major bump (for a grouped PR, no bump in the group is major); majors wait for a human review
167
168
  - No conflicts
168
- - 7-day cooldown for security (see Dependency Management)
169
169
 
170
170
  **Learn more:** [Dependency Management](dependency-management.md)
171
171
 
@@ -208,7 +208,7 @@ Ensures preview environments don't accumulate and waste resources.
208
208
  ### Continuous Sync
209
209
  - Documentation syncs on every master push
210
210
  - Rules sync on every master push
211
- - Dependencies update daily (Dependabot)
211
+ - Dependencies update daily (`update-dependencies` skill); Dependabot opens security fixes only
212
212
  - Quality improvements run daily (JSDoc, tests)
213
213
 
214
214
  ## Working with AI Workflows
@@ -280,8 +280,8 @@ The docs-audit workflow flags a missing README or reference page and obvious int
280
280
 
281
281
  ### Dependabot PR doesn't auto-merge
282
282
  - Verify all CI checks pass (green checkmarks)
283
- - Check 7-day cooldown hasn't been violated
284
283
  - Ensure no merge conflicts exist
284
+ - If the security fix needs code changes, fold it into the rolling `chore/update-dependencies` PR instead of pushing to the Dependabot branch
285
285
 
286
286
  ### AI workflow creates poor quality PR
287
287
  - Review the workflow definition in `.github/workflows/*.md`
@@ -290,7 +290,7 @@ The docs-audit workflow flags a missing README or reference page and obvious int
290
290
 
291
291
  ## Related Documentation
292
292
 
293
- - [Dependency Management](dependency-management.md) — Dependabot configuration and auto-merge
293
+ - [Dependency Management](dependency-management.md) — Dependabot security-only configuration, auto-merge, and the update skill
294
294
  - [Deploy to GCP](../how-to/deploy-to-gcp.md) — Manual deployment guide
295
295
  - [GCP Hosting Architecture](gcp-hosting-architecture.md) — Static site hosting design
296
296
  - [Environment Variables](../reference/environment-variables.md) — Configuration reference
@@ -1,16 +1,16 @@
1
1
  # Dependency Management
2
2
 
3
- Terreno pins shared versions in the Bun catalog, lands JavaScript updates through
4
- an agent skill that proves each bump with a test, and keeps Expo native
5
- fingerprints frozen until a release.
3
+ Terreno pins shared versions in the Bun catalog, lands routine JavaScript updates
4
+ through an agent skill that proves each bump with a test, limits Dependabot to
5
+ security fixes, and keeps Expo native fingerprints frozen until a release.
6
6
 
7
7
  ## Layers
8
8
 
9
9
  | Layer | Job |
10
10
  | --- | --- |
11
11
  | **Bun catalog** | One version for every dependency shared by two or more workspace packages (`catalog:` in manifests) |
12
- | **`update-dependencies` skill** | Daily rolling PR on `chore/update-dependencies`: cooldown, exercise test, fingerprint freeze, ledger of landed/failed |
13
- | **Dependabot** | Backup signal for bun + GitHub Actions. Native/Expo catalog names are ignored so auto-merge cannot change the fingerprint |
12
+ | **`update-dependencies` skill** | The only path for routine version upgrades. Daily rolling PR on `chore/update-dependencies`: cooldown, exercise test, local gate, one push per run, fingerprint freeze, ledger of landed/failed |
13
+ | **Dependabot** | Security updates only for bun + GitHub Actions. Native/Expo catalog names are ignored so auto-merge cannot change the fingerprint |
14
14
  | **Release + `upgrading-expo`** | Expo SDK, React Native, and other native-module bumps |
15
15
 
16
16
  Operator steps: [Update dependencies](../how-to/update-dependencies.md). Catalog rule: workspace `package.json` files use `"catalog:"` for shared pins.
@@ -41,16 +41,18 @@ Change a shared version in the catalog once. Do not pin a second copy in a works
41
41
 
42
42
  ## Agent updates vs Dependabot
43
43
 
44
- Dependabot groups many packages and merges when CI is green. That misses two Terreno constraints:
44
+ Dependabot version updates group many packages and merge when CI is green. That misses Terreno constraints, so Dependabot now opens security PRs only:
45
45
 
46
46
  1. **Exercise.** CI can pass without any test importing the bumped module. The skill requires a named test file that imports the package and that passed **after** the bump. If none exists, write a tracer-bullet test first.
47
47
  2. **Fingerprint.** Expo `runtimeVersion.policy: "fingerprint"` ties OTA runtime and EAS dev builds to native hashes of `example-frontend` and `demo`. Native module bumps belong in a release, not a weekday dependency PR.
48
48
 
49
+ 3. **CI is not the test runner.** A red grouped Dependabot PR invites an agent to push fix after fix to the Dependabot branch. Each push starts a full CircleCI pipeline that is attributed to, and emails, whoever owns the pushing token. The skill runs the CI mirror (`bun run prepush`) locally, pushes once per run, never edits CI config to make a bump pass, and stops after two red CI rounds.
50
+
49
51
  `scripts/planning/fingerprintRisk.ts` (`isFingerprintSkip`) is the skip matcher. After every allowed bump the skill still recomputes hashes; a changed hash is reverted even if the matcher allowed the name.
50
52
 
51
53
  ## Cooldown
52
54
 
53
- Non-security npm versions wait **7 days** after publish before a bump (`npm view <name> time`). Most malicious publishes are yanked inside a day. GitHub Dependabot uses the same `cooldown.default-days: 7`. CVE/advisory bumps skip the wait and still need an exercise test and a fingerprint freeze.
55
+ Non-security npm versions wait **7 days** after publish before a bump (`npm view <name> time`). Most malicious publishes are yanked inside a day. CVE/advisory bumps skip the wait and still need an exercise test and a fingerprint freeze. Dependabot security updates are advisory bumps, so they have no cooldown.
54
56
 
55
57
  ## Fingerprint freeze
56
58
 
@@ -67,16 +69,16 @@ JavaScript packages, GitHub Actions, and a few Expo JS-only names may update. Me
67
69
 
68
70
  ## Dependabot configuration
69
71
 
70
- `.github/dependabot.yml` still opens grouped PRs:
72
+ `.github/dependabot.yml` sets `open-pull-requests-limit: 0` on every block. That turns off version-update PRs; security updates ignore the limit and still honor `ignore`. Each block's group uses `applies-to: security-updates`, so fixes arrive grouped:
71
73
 
72
- | Group | Ecosystem | Schedule |
73
- | --- | --- | --- |
74
- | GitHub Actions | `github-actions` at `/` | Weekly |
75
- | Root catalog and root tools | `bun` at `/` | Monthly |
76
- | Backend workspaces | `bun` under api, ai, backends, … | Monthly |
77
- | Frontend workspaces | `bun` under ui, rtk, demo, apps, … | Monthly |
74
+ | Group | Ecosystem |
75
+ | --- | --- |
76
+ | `github-actions` | `github-actions` at `/` |
77
+ | `root` | `bun` at `/` (root catalog and root tools) |
78
+ | `backend` | `bun` under api, ai, backends, … |
79
+ | `frontend` | `bun` under ui, rtk, demo, apps, … |
78
80
 
79
- Frontend ignores catalog-managed names so those PRs do not duplicate the root catalog. Root ignores fingerprint-skip names. `dependabot-auto-merge.yml` still squash-merges Dependabot PRs when required checks pass. Do not merge a Dependabot PR that includes a skip-list package; redo it with [Update dependencies](../how-to/update-dependencies.md).
81
+ Frontend ignores catalog-managed names so those fixes land once, in the root catalog. Root ignores fingerprint-skip names, so a security fix for a native/Expo package never opens a PR; handle it by hand in a release. `dependabot-auto-merge.yml` approves patch and minor Dependabot PRs and enables squash auto-merge, so those security PRs merge when required checks pass. A semver-major security PR (or a group containing one) gets no approval or auto-merge and waits for a human. When one fails because the fix needs code changes, fold the bump into the rolling PR ([Update dependencies](../how-to/update-dependencies.md)); never push to the `dependabot/*` branch.
80
82
 
81
83
  ## Daily rolling PR
82
84
 
@@ -91,10 +93,12 @@ Each package is still proven alone (exercise test + fingerprint) before it stays
91
93
  - One rolling PR; many proven commits on that branch.
92
94
  - Majors: exercise test + fingerprint freeze; no auto-merge.
93
95
  - Security: skip cooldown; do not skip tests or fingerprints.
96
+ - Verify locally, push once; never change CI config to make a bump pass.
94
97
  - New feature dependencies belong in that feature's slice, not this maintenance path.
95
98
 
96
99
  ## Related
97
100
 
98
101
  - [Update dependencies](../how-to/update-dependencies.md)
102
+ - [Run tests locally](../how-to/run-tests-locally.md)
99
103
  - [Install agent skills](../how-to/install-agent-skills.md)
100
104
  - [Bun catalogs](https://bun.sh/docs/install/catalogs)
@@ -41,6 +41,15 @@ Terreno 57.
41
41
  5. **Backend sync registration** — Every synced collection needs `syncPlugin`, `isDeletedPlugin`, and a `sync` block on `modelRouter`, plus `SyncApp` and `RealtimeApp` on the server. Optional `sync.adminBroadcast: true` also emits `sync:delta` to `{collection}|admin`. Admin clients subscribe with `mode: "window"` (`createSyncDb({windowCollections})`) so they do not snapshot or reconcile that collection; they call `hydrateWindow` (REST membership ids + `GET /sync/entities`). Pass the same `accessControl` into `SyncApp` that `AdminApp` uses so `{collection}|admin` requires `admin:access`. `AdminApp.register` also installs that model's admin list/read/`queryFilter` on the admin window so hydrate and live `|admin` deltas match `/admin` REST. Without RBAC, `user.admin` is enough for the shell gate. Live deltas on that stream apply only to ids already in the window; Refresh is a REST re-query plus `hydrateWindow`.
42
42
  6. **Collections too large to sync whole** — List them in `createSyncDb({queryCollections})`. They skip snapshot bootstrap; screens load server-filtered *query windows* (`useWindowQuery`) from the `modelRouter` list endpoint. A window stores only membership (ordered ids); rows live once in the shared entity table. Overlapping windows therefore share rows — one delta updates both views — while each view lists only its own query's members, plus cached rows that newly match its `where` inside the loaded range. Live deltas still arrive on the user's normal streams; windows on screen refetch on reconnect to cover deltas missed offline.
43
43
 
44
+ ## First load and reconnects
45
+
46
+ A fresh device has to download every row the user can see before the UI is complete. The
47
+ client asks for all of the user's streams in one batched request per round, writes each
48
+ round while the next one downloads, and writes rows in small chunks so the screen stays
49
+ responsive. Reconnects use the same path: when nothing changed, catching up costs two
50
+ requests however many collections the app syncs. Collections too large to download whole
51
+ belong in `queryCollections` instead.
52
+
44
53
  ## Tradeoffs
45
54
 
46
55
  **Gains:** instant UI, offline writes with a durable outbox, simpler screen code, and one write path on the server (sync and REST share `modelRouter`).
@@ -47,6 +47,7 @@ Problem-oriented, practical steps. Use these when you know what you want to do.
47
47
  - [Compose charts in a dashboard grid](charts-and-dashboards.md) — `LineChart` / `BarChart` / `AreaChart` / `DonutChart` inside `Card`s in `DashboardGrid` (ops-dashboard scorecards and spans: [dashboard-chart-parity](../implementationPlans/dashboard-chart-parity.md))
48
48
  - [Compare rendered chart snapshots](compare-chart-rendered-snapshots.md) — PNG goldens for the chart visual gallery, not JSON snapshots
49
49
  - [Migrate from @terreno/rtk to @terreno/syncdb](migrate-rtk-to-syncdb.md) — Move data sync to the local-first layer
50
+ - [Use syncdb correctly](use-syncdb-correctly.md) — Serial snapshot startup, outbox-halting 403s, phantom creates, conflicts, and other library behavior apps must handle
50
51
  - [Generate a REST CLI from OpenAPI](generate-a-rest-cli.md) — Scaffold `terreno generate rest-cli` for an app API
51
52
  - [In-app notifications](in-app-notifications.md) — Register `NotificationsApp`, syncdb inbox, and UI bell
52
53
  - [Add organizations](add-organizations.md) — Opt into Membership-backed tenancy, RBAC, admin, and seeded examples
@@ -260,8 +260,10 @@ Caching is off (200 credits per job). The docs Netlify target builds only the
260
260
  current docs tree, disables Docusaurus minification, and pins SSG to one worker
261
261
  thread so the build peaks near 3 GB and fits the 4 GB `medium` executor.
262
262
  Released versions are not rebuilt: `archive-docs-version` (in `publish-release`,
263
- `X.Y.0` tags only) builds each one once and uploads it to its GitHub release, and
264
- `deploy-docs` unpacks those archives under `/<version>/`. See `website/README.md`.
263
+ `X.Y.0` tags only) builds each version once, uploads it to its **draft** GitHub
264
+ release, publishes the release, and then commits the docs snapshot to master, so
265
+ the `deploy-docs` run that commit starts can unpack the archive under
266
+ `/<version>/`. See `website/README.md`.
265
267
 
266
268
  `terreno-gcp` uses CircleCI OIDC (`CIRCLE_OIDC_TOKEN_V2`), never a JSON service
267
269
  account key. Set `circleci_org_id`, `circleci_project_id`, and
@@ -379,6 +381,17 @@ dependencies and runs `test:ci` when that script exists, not `test`.
379
381
  tag version is already on npm, `publish-package.sh` skips it so a recut of
380
382
  the same tag can finish the rest.
381
383
 
384
+ The release skill pushes the tag and creates the GitHub release with
385
+ `gh release create --draft`. The repository has immutable releases on: a
386
+ published release cannot take new assets (GitHub answers `422`). So CI publishes
387
+ the draft last with `scripts/ci/github-release.sh publish`.
388
+ `archive-docs-version` does it right after the upload on `X.Y.0` tags.
389
+ `publish-github-release` does it after `publish-release` on every other tag.
390
+ Both wait up to 10 minutes for the draft and are safe to rerun: a rerun skips
391
+ an upload that already landed on a published release and a snapshot already on
392
+ master. A
393
+ release still in draft after the pipeline means one of them failed.
394
+
382
395
  Only stable tags (`57.3.0`) run `deploy-demo`. It builds the demo from source,
383
396
  so it runs alongside `publish-release` instead of waiting for it. Use
384
397
  `{"run-demo-deploy":true}` on `master` if a prerelease must also refresh the
@@ -1,6 +1,6 @@
1
1
  # Update dependencies
2
2
 
3
- This runs **daily**. Use `/update-dependencies`. There is one rolling PR; later days push to it and rewrite its ledger instead of opening new PRs.
3
+ Routine version upgrades run through the `update-dependencies` skill (`/update-dependencies`). Dependabot opens **security PRs only**. The skill keeps one rolling PR; later runs push to it and rewrite its ledger instead of opening new PRs.
4
4
 
5
5
  ```bash
6
6
  REPOSITORY="$(gh repo view --json nameWithOwner --jq .nameWithOwner)"
@@ -19,7 +19,7 @@ gh pr list --state open --head chore/update-dependencies \
19
19
  bun outdated
20
20
  ```
21
21
 
22
- Prove each bump with an **exercise test** that imports that package. Keep Expo fingerprints for `example-frontend` and `demo` unchanged. Skip `isFingerprintSkip` names until a release.
22
+ Prove each bump with an **exercise test** that imports that package. Verify the whole batch locally, then push **once**. Keep Expo fingerprints for `example-frontend` and `demo` unchanged. Skip `isFingerprintSkip` names until a release.
23
23
 
24
24
  Why: [Dependency management](../explanation/dependency-management.md). Native Expo/React Native bumps wait for `upgrading-expo`.
25
25
 
@@ -27,9 +27,10 @@ Why: [Dependency management](../explanation/dependency-management.md). Native Ex
27
27
 
28
28
  1. Find the same-repository PR with exact head `chore/update-dependencies`, base `master`, and `isCrossRepository: false`. Marker/title matches are not trust signals; ignore forks. Merge `origin/master` without rewriting history. If no trusted PR exists, create the canonical branch on this run.
29
29
  2. Inventory with `bun outdated`. Retry **Failed** rows from the PR body first. Skip `isFingerprintSkip` in `scripts/planning/fingerprintRisk.ts`.
30
- 3. For each candidate: confirm a test imports the package (add a tracer if needed), bump the catalog pin or single-use version, `bun install`, re-run that test, recompute fingerprints. Revert only that bump on failure.
31
- 4. Commit each success on the same branch. Push. Update the **same** PR body: Last run, Landed, Failed, Skipped, Deferred to release. Do not `gh pr create` when that PR is already open.
32
- 5. Read the ledger tomorrow; do not start a new PR.
30
+ 3. For each candidate: confirm a test imports the package (add a tracer if needed), bump the catalog pin or single-use version, `bun install`, re-run that test, recompute fingerprints. Revert only that bump on failure. Commit each success as `Bump <name> from <old> to <new>`.
31
+ 4. Run the batch gate on the final head: `bun run compile`, `bun run lint`, `bun run prepush`. Run by hand any infra job `prepush` lists for a package you bumped (Playwright, Maestro, admin-spa integration). See [Run tests locally](run-tests-locally.md).
32
+ 5. Push once. Update the **same** PR body: Last run, local gate result, CI rounds, Landed, Failed, Skipped, Deferred to release. Do not `gh pr create` when that PR is already open.
33
+ 6. Read CI on the pushed head. Check each failed CircleCI job's `messages` before its step logs. Revert a bump that caused a failure and push a second time at most; stop and report anything else.
33
34
 
34
35
  ## Guardrails
35
36
 
@@ -41,7 +42,11 @@ Why: [Dependency management](../explanation/dependency-management.md). Native Ex
41
42
  | No test imports the package | Write the tracer test first. Do not bump on `test:ci` alone |
42
43
  | Published < 7 days ago (non-CVE) | Skip; log under Skipped |
43
44
  | Major version | May land on the rolling PR if tests + fingerprints pass; no auto-merge |
44
- | Dependabot extra PRs | Do not merge skip-list packages; fold JS bumps into the rolling PR |
45
+ | Local gate red | Fix or revert before pushing. CI is not the test runner |
46
+ | CI job has `messages` (for example `resource-class-not-in-plan`) | Config/plan error, not a dependency failure. Log it and stop |
47
+ | A bump seems to need CI config changes | Never edit `.circleci/`, workflows, executors, or resource classes from this path. Stop and report |
48
+ | CI red twice on heads this run pushed | Stop and report; do not push a third time |
49
+ | Dependabot security PR is red | Do not push to `dependabot/*`. Fold the bump into the rolling PR |
45
50
 
46
51
  ## Fingerprints
47
52
 
@@ -49,6 +54,10 @@ Why: [Dependency management](../explanation/dependency-management.md). Native Ex
49
54
 
50
55
  `eas.json` Bun pins feed the fingerprint. Do not bump them here.
51
56
 
52
- ## Dependabot still runs
57
+ ## Who CI emails
53
58
 
54
- GitHub Dependabot remains in `.github/dependabot.yml`. Root catalog ignores native/Expo packages. Treat Dependabot as a signal; land updates on the rolling PR from this guide.
59
+ CircleCI attributes a pipeline to the GitHub user whose token pushed it and emails that user on each failed workflow. Run the scheduled agent with a bot identity when one is configured. If it runs as a person, the one-push rule caps their mail at one round per run.
60
+
61
+ ## Dependabot security PRs
62
+
63
+ `.github/dependabot.yml` sets `open-pull-requests-limit: 0` on every block, so Dependabot opens no version-update PRs. Security updates still open, grouped per block, and skip ignored native/Expo names. `dependabot-auto-merge.yml` approves them and enables squash auto-merge, so a green security PR merges without this skill. A red one gets its bump folded into the rolling PR. A security fix for an ignored native package is a manual release task (`release` + `upgrading-expo`).
@@ -0,0 +1,111 @@
1
+ # How to Use syncdb Correctly
2
+
3
+ This page lists `@terreno/syncdb` behavior that a working app has to account for. Each entry says what the library does today, what breaks if you ignore it, and what to do instead. It was checked against `syncdb/src` and `api/src/sync` at 57.13.0. Several entries are library gaps rather than intended behavior. Delete an entry when its gap is fixed.
4
+
5
+ Read [Local-first data](../explanation/local-first-data.md) and [Migrate from @terreno/rtk to @terreno/syncdb](migrate-rtk-to-syncdb.md) first.
6
+
7
+ ## Startup and snapshots
8
+
9
+ ### Cold start costs one round trip per round, not per stream
10
+
11
+ A stream is one collection in one scope, so a user has roughly (synced collections) × (tenants they belong to) streams. Reconcile, subscribe catch-up and wipe re-bootstrap page every stream through `POST /sync/snapshot/batch`: one round trip per round, with up to 5,000 rows per response shared across the streams. Empty streams cost nothing extra. `forceResync`, and every pass against a server without the batch route, still page with one `GET /sync/snapshot` per stream, `snapshotConcurrency` streams side by side (default 6). Each stream runs one pass at a time, and a pass that purges (force resync, wipe) waits for the stream's in-flight pass to finish first.
12
+
13
+ Measured with a fake HTTP channel at 150 ms latency, 13 collections and 2 workspaces (26 streams): 4.1 s to the first full reconcile when paging one stream at a time, 0.95 s at the default concurrency. In a browser with 100 collections and 10k rows, batching cut a reconnect with no changes from 101 requests to 2.
14
+
15
+ | Do | Why |
16
+ | --- | --- |
17
+ | Sync only the collections the client reads. | Each collection adds one stream per tenant to cold start. |
18
+ | Gate each screen on its **own** stream's bootstrap (see [No loading flag on synced reads](#no-loading-flag-on-synced-reads)), not on "sync finished". | The stream a screen needs may land after others. |
19
+ | Don't call `forceResync()` on a hot path. | It purges and re-pages every stream. |
20
+ | Raise `snapshotConcurrency` only if the backend can take it. | Without the batch route, each in-flight stream is one concurrent snapshot query on the server. |
21
+
22
+ The startup reconcile and each collection's `sync:subscribed` catch-up both page the same streams. A catch-up queues behind a pass that started before the room join, because that pass may have missed a write. So a cold start still pages each stream twice, one pass after the other.
23
+
24
+ ### No loading flag on synced reads
25
+
26
+ `useQuery` returns a plain array. `useListQuery` returns `{data}`. The `isPending` on `useEntity`/`useReadQuery` means an outbox write is pending, not that data is loading. An empty array before bootstrap looks the same as an empty collection, so a screen that renders its empty state from `[]` flashes "nothing here" on every cold start.
27
+
28
+ The per-stream `bootstrapped` mark is the right signal, but `isStreamBootstrapped` is not exported. Read the cell directly:
29
+
30
+ ```typescript
31
+ const isBootstrapped = (stream: string): boolean =>
32
+ syncDb.store.raw.getCell("_cursors", stream, "bootstrapped") === true;
33
+ ```
34
+
35
+ Don't use the cursor (`useSyncStatus().streams`) for this. A live delta can move the cursor to the head before the snapshot below it has been paged. `useWindowQuery` (query collections only) does expose `isLoading`.
36
+
37
+ A stream key is `<collection>|<scopeType>:<value>`, for example `documents|tenant:<workspaceId>`. A user with several tenants has one stream per tenant. Treat a collection as loaded only once every known stream for it is bootstrapped (`syncDb.store.getKnownStreams()`).
38
+
39
+ ## Writes
40
+
41
+ ### Mutations return an id, not an outcome
42
+
43
+ `create`, `update` and `remove` from `useMutate` (and from the generated `useCreate*`/`useUpdate*`/`useDelete*` hooks) return `{mutationId, id}` synchronously. There is no promise for the server ack. The client also drops the ack's `warning`. A `warning` means a server post-hook threw after the write committed.
44
+
45
+ If a screen needs to know the outcome (close a modal only once the create landed, toast a refusal), wrap the transport and HTTP channel and record ack and nack outcomes by `mutationId`. Then await that `mutationId`. Don't poll the store.
46
+
47
+ ### A 403 from a write halts the whole outbox
48
+
49
+ The server maps 403/405 to an `unauthorized` nack and 400/404/422 to `validation`. They behave very differently:
50
+
51
+ | Nack | Client behavior |
52
+ | --- | --- |
53
+ | `unauthorized` | The mutation is re-queued and the **whole drain stops** (auth-pause). On re-auth the same `mutationId` replays, reads the same stored nack, and pauses again. It never settles. |
54
+ | `validation` | The mutation is marked `failed` and the drain continues. That entity's later writes stay blocked until `retryFailed({entityId})`. |
55
+ | `conflict` | That entity's later writes stay blocked (or every write is blocked, with `haltQueueOnConflict: true`) until `resolveConflict`. |
56
+ | `error` | Retried up to the retry budget, then handled like `validation`. |
57
+
58
+ So a router hook that refuses a synced write for a business reason must throw **422**, not 403. That includes `preCreate`/`preUpdate` membership checks. Keep 403 for real auth failures.
59
+
60
+ ### Don't send a stale tenant field on create
61
+
62
+ For a tenant-scoped collection, `enforceWriteScope` checks a client-sent scope field (for example `workspaceId`) against the user's memberships **before** `preCreate` runs. A value the user does not belong to is a 403, which halts the outbox (see above). If the field is missing or null, the check is skipped and `preCreate` can fill it in.
63
+
64
+ Either omit the tenant field and let `preCreate` set it, or send it only from the currently synced row of the parent tenant, never from a persisted id that may be stale.
65
+
66
+ ### The local row never gets server-computed fields from the create
67
+
68
+ After an ack, the client stamps the row with the ack's seq and keeps the optimistic data. The server's create delta has the same seq, so the client skips it. Whatever `preCreate`, schema defaults or hooks set on the server does not reach the local row until that row changes again or is repaired. If the delta arrives before the ack, the row is marked for repair, but repair only runs on the next `reconcile()` (reconnect, seq jump, or every 5 minutes).
69
+
70
+ Do one of these:
71
+
72
+ - Send every field the UI reads in the create payload, matching what the server would compute (defaults, `order`, ownership fields).
73
+ - After the ack, patch the local row yourself.
74
+
75
+ The local row's `data` is exactly what you passed. Include `_id` in the payload if anything reads `row._id`. Server-written rows carry `_id` and `id`.
76
+
77
+ ### A refused create leaves a phantom row
78
+
79
+ When a create is nacked as `validation` (or runs out of `error` retries), the repair fetch finds no server row and only clears the repair mark. The optimistic row stays in the store with `stream: ""` and shows up in queries. Only `forceResync` purges such rows.
80
+
81
+ On a refused create, tombstone the local row yourself and call `retryFailed({entityId})` so later writes to that id are released.
82
+
83
+ ### Conflicts are never resolved for you
84
+
85
+ A `conflict` nack blocks every later write to that row, and any queued write whose args mention its id, until someone calls `client.resolveConflict({mutationId, strategy: "keepMine" | "useServer"})` (or `useConflicts().resolve`). Local edits keep applying optimistically, so the UI looks fine while nothing reaches the server.
86
+
87
+ If the app has no conflict UI, run a watcher that resolves each conflict as it is recorded. Use `keepMine` for fields the user types, and `useServer` for everything else.
88
+
89
+ ## Server side
90
+
91
+ ### Write synced models one document at a time, by `_id`
92
+
93
+ On a synced model, `updateMany`, `deleteMany`, `deleteOne`, `findOneAndDelete` and `bulkWrite` throw. Single-document writes by `_id` without upsert are allowed: `save()`, `updateOne`, `findOneAndUpdate`, `findByIdAndUpdate`, `replaceOne`. Delete by setting `deleted: true`, because a hard delete emits no delta.
94
+
95
+ In tests, clean up with `Model.collection.deleteMany(...)`, which bypasses Mongoose.
96
+
97
+ ### Deltas don't apply `queryFilter`
98
+
99
+ Snapshots and `/sync/entities` apply the router's `queryFilter`. Live deltas check only `permissions.read` (or RBAC) per recipient. A router that restricts visibility only through `queryFilter` leaks rows over the socket. Every synced router's `read` permission must enforce the same restriction itself.
100
+
101
+ ### Membership grants arrive late
102
+
103
+ A user's new tenant is picked up by the socket's session revalidation sweep (`sessionRevalidationIntervalMs`, 60 s by default) or a resubscribe. The client finds the new stream on its next `reconcile()` (`reconcileIntervalMs`, 5 minutes by default, plus reconnects). Until then the new tenant's rows don't appear.
104
+
105
+ When the product grants membership, push a content-free event to the user's sockets and have the client reconnect syncdb. Reconnecting resubscribes and reconciles. Grants written another way (admin, scripts) still wait for the timers.
106
+
107
+ ## Auth
108
+
109
+ ### Logging out does not wipe local data
110
+
111
+ The client's own `onAuthChange` handling treats a logout as a pause and **never wipes**. A different user logging in does wipe. To clear one user's data on sign-out, set `wipeOnSignOut: true` and call `client.signOut()` explicitly when the app's user clears.
@@ -70,7 +70,7 @@ Expo autolinking walks the app's own dependencies; a nested copy leaves the nati
70
70
  │ sync:mutate ─────────────────────────────►│ │
71
71
  │ sync:ack / sync:nack ◄────────────────────┘ │
72
72
  │ sync:delta ◄──────────────────────────────────────────────┘
73
- │ GET /sync/snapshot (bootstrap + catch-up, HTTP)
73
+ │ POST /sync/snapshot/batch (bootstrap + catch-up, HTTP)
74
74
  └── POST /sync/mutate (fallback while the socket is down)
75
75
  ```
76
76
 
@@ -185,6 +185,7 @@ be passed to `betterAuthAdapter` directly.
185
185
  | `wipeOnSignOut` | `boolean` | `false` | `signOut()` also wipes local data |
186
186
  | `batchSize` | `number` | `50` (server caps at 100) | Max mutations per batched drain send |
187
187
  | `haltQueueOnConflict` | `boolean` | `false` | `true` = conflict halts entire drain; `false` = per-entity blocking |
188
+ | `snapshotConcurrency` | `number` | `6` | Streams paged at once with `GET /sync/snapshot`: by `forceResync`, and by every other pass against a server without `POST /sync/snapshot/batch`. Each stream runs one pass at a time. `1` pages one stream at a time |
188
189
  | `onDecryptFailure` | `() => void` | wipe + re-bootstrap (with `console.warn`) | Override web decrypt-failure recovery |
189
190
  | `tombstoneRetentionMs` | `number` | `7776000000` (90 days); `0` disables | Client tombstone compaction after successful reconcile |
190
191
  | `startAuthRetryAttempts` | `number` | `3`; `1` disables | `start()` attempts to resolve `getUserId()` |
@@ -201,7 +202,7 @@ be passed to `betterAuthAdapter` directly.
201
202
  | `goOffline()` | Simulated outage: disconnect transport, pause replay/reconcile/timer; local mutations keep queueing. |
202
203
  | `goOnline()` | End simulated outage; reconnect triggers reconcile + outbox replay. |
203
204
  | `mutate({collection, operation, id?, data?})` | Optimistic local write + durable outbox enqueue + fire-and-forget replay. Returns `{mutationId, id}`. |
204
- | `reconcile()` | HTTP snapshot catch-up for every known stream; runs tombstone compaction on success. Also runs automatically on (re)connect, on a rate-limited seq-jump hint, and on the periodic timer; each `sync:subscribed` confirmation additionally pages just the streams it names. |
205
+ | `reconcile()` | HTTP snapshot catch-up for every known stream, batched into `POST /sync/snapshot/batch` rounds (see [Snapshot bootstrap](#snapshot-bootstrap)); runs tombstone compaction on success. Also runs automatically on (re)connect, on a rate-limited seq-jump hint, and on the periodic timer; `sync:subscribed` confirmations additionally page just the streams they name, and confirmations arriving while that catch-up request is in flight are paged together in the next one. |
205
206
  | `hydrateWindow({collection, ids, restRows?})` | Admin window upsert: REST rows (optional) land immediately; every requested id is also fetched from `GET /sync/entities` before this resolves so seq/deleted metadata is canonical for immediate update/delete. A `{collection}|admin` delta that lands while the fetch is in flight wins, so hydration never rewinds a row to an older seq. Unknown ids are ignored. |
206
207
  | `queryWindows` | Server query windows for `queryCollections`: `fetchWindow({query, nextPage?})`, `getWindow(query)`, `retain(query)` → release, `refetchRetained()`, `pruneUnretained()`, `subscribe(cb)`, `keyFor(query)`. See [Query windows](#query-windows). |
207
208
  | `forceResync()` | Purge every known stream locally and re-bootstrap from cursor 0 (outbox/conflicts untouched). Returns `{ok, reason?, streams, purged, repaired}`. After discovery and each stream bootstrap it waits for in-flight `start`/`stop`/auth-change work, then abandons with `reason: "superseded"` when that work switched users or generations. |
@@ -587,11 +588,36 @@ sync: {
587
588
 
588
589
  ## Sync protocol
589
590
 
591
+ ### Snapshot bootstrap
592
+
593
+ The client pages every stream it belongs to through `POST /sync/snapshot/batch`, not one
594
+ `GET /sync/snapshot` per stream. With 100 collections, a reconnect that finds nothing new
595
+ is 2 requests (`GET /sync/streams` plus one batch) instead of 101.
596
+
597
+ - **Rounds:** each round asks for every stream that still has pages. Streams that reached
598
+ their head drop out, so later rounds give the remaining streams a larger share of the
599
+ 5,000-entity budget.
600
+ - **Pipelining:** the next round is sent as soon as a response arrives, because its
601
+ cursors come from the response. The client writes the current round while the next
602
+ one downloads.
603
+ - **Chunked apply:** each page is written in store transactions of 500 entities, with a
604
+ yield to the event loop between them, so a large page never freezes the UI in one block.
605
+ A stream's cursor advances in the transaction holding its last entity, so a page cut
606
+ short by a crash is fetched again and re-applied as a no-op.
607
+ - **Older servers:** a 404 without an APIError body (the route does not exist) switches
608
+ the client to per-stream `GET /sync/snapshot` paging, `snapshotConcurrency` streams at
609
+ a time, for the rest of the session. An APIError 404, such as an unknown collection,
610
+ fails that pass and keeps batching on.
611
+ - **One pass per stream:** a batch pass owns each of its streams the way a per-stream
612
+ pass does, so a purge (`forceResync`, wipe) waits for it, and a pass that is superseded
613
+ by a sign-out or user switch stops between chunks without writing the rest.
614
+
590
615
  ### HTTP routes (`SyncApp`, all authenticated)
591
616
 
592
617
  | Endpoint | Purpose |
593
618
  |----------|---------|
594
- | `GET /sync/snapshot?collection=&stream=&cursor=&limit=` | Bootstrap + catch-up per stream |
619
+ | `GET /sync/snapshot?collection=&stream=&cursor=&limit=` | Bootstrap + catch-up for one stream (`limit` max 100) |
620
+ | `POST /sync/snapshot/batch` | Bootstrap + catch-up for many streams in one round trip. Body `{streams: [{stream, cursor?, legacyCursor?}], limit?}`, at most 500 streams, no repeats. `limit` (default 5,000, max 10,000) is the entity budget for the whole response, split evenly across the streams. Returns `{pages}`, one `GET /sync/snapshot`-shaped page per stream in request order. Any stream the caller may not read fails the whole request |
595
621
  | `GET /sync/streams` | Current stream membership for the user |
596
622
  | `GET /sync/entities` | Point lookup for entity repair and admin window hydrate. Admin-window callers (`admin:access` with RBAC, else `user.admin`) on `adminBroadcast` collections receive requested ids across product streams. When AdminApp registered a scope, list/read/`queryFilter` still apply; unknown or out-of-scope ids are omitted |
597
623
  | `POST /sync/mutate` | Single mutation (HTTP fallback). Optional `mutationMode: "adminWindow"` validates `adminBroadcast`, admin-window access, and a registered AdminApp write scope, then runs the shared sync executor with **AdminApp** pre/post hooks (not product `modelRouter` hooks), plus the same permission (including org membership), stripping, and audit semantics as `/admin` REST. HTTP mutate binds `X-Organization-Id` into org context |
@@ -635,6 +661,11 @@ const syncDb = createSyncDb({
635
661
  });
636
662
  ```
637
663
 
664
+ Saves are debounced (500 ms). The store is serialized once per write, not once per
665
+ transaction: autosave compares TinyBase's content hashes (O(1)) to skip saves the last
666
+ write already covered, so a bootstrap of many pages costs a few writes instead of one
667
+ full-store serialization per page.
668
+
638
669
  Undecryptable persisted data triggers wipe + re-bootstrap by default (`onDecryptFailure` to override). Storage read errors surface `persistence: "error"` without overwriting the blob.
639
670
 
640
671
  **Native:** plaintext SQLite in the OS sandbox (by design).
package/package.json CHANGED
@@ -11,8 +11,8 @@
11
11
  "@modelcontextprotocol/node": "^2.0.0",
12
12
  "@modelcontextprotocol/server": "^2.0.0",
13
13
  "@sentry/bun": "^10.39.0",
14
- "@terreno/api": "57.13.0",
15
- "@terreno/blocks": "57.13.0",
14
+ "@terreno/api": "57.15.0",
15
+ "@terreno/blocks": "57.15.0",
16
16
  "express": "^5.2.1",
17
17
  "luxon": "^3.7.2",
18
18
  "minisearch": "^7.2.0",
@@ -105,5 +105,5 @@
105
105
  "test:coverage": "bun run ../scripts/check-coverage.ts"
106
106
  },
107
107
  "type": "module",
108
- "version": "57.13.0"
108
+ "version": "57.15.0"
109
109
  }
@@ -0,0 +1,43 @@
1
+ # Upgrading to 57.14.0
2
+
3
+ **Action required:** No
4
+ **Affected packages:** @terreno/syncdb
5
+
6
+ ## Summary
7
+
8
+ `@terreno/syncdb` pages snapshot streams in parallel (up to 6 at once by default) instead of one at a time, so cold start, reconcile, subscribe catch-up, `forceResync`, and wipe re-bootstrap finish faster when an app has many collections or scopes. Each stream now runs one snapshot pass at a time, which stops a concurrent pass from marking a purged stream bootstrapped over rows it never paged. No code change is required.
9
+
10
+ ## Breaking changes
11
+
12
+ None
13
+
14
+ ## Deprecations
15
+
16
+ None
17
+
18
+ ## New capabilities
19
+
20
+ ### `snapshotConcurrency` on `createSyncDb`
21
+
22
+ The client now sends up to `snapshotConcurrency` snapshot requests at once (default `DEFAULT_SNAPSHOT_CONCURRENCY`, 6). If your backend cannot take the extra parallel load at cold start, lower it. `1` restores the old one-stream-at-a-time behavior.
23
+
24
+ ```typescript
25
+ // Before
26
+ export const syncDb = createSyncDb({authProvider, baseUrl, collections, name: "my-app"});
27
+
28
+ // After (only if you need to limit parallel snapshot requests)
29
+ export const syncDb = createSyncDb({
30
+ authProvider,
31
+ baseUrl,
32
+ collections,
33
+ name: "my-app",
34
+ snapshotConcurrency: 2,
35
+ });
36
+ ```
37
+
38
+ A new how-to, `docs/how-to/use-syncdb-correctly.md`, lists library behavior apps have to handle.
39
+
40
+ ## Verification
41
+
42
+ - `bun run compile` and `bun run lint` at the app root
43
+ - Log in on a fresh device or after `wipeLocalData`; synced screens load, and the backend logs show several `/sync/snapshot` requests in flight together
@@ -0,0 +1,44 @@
1
+ # Upgrading to 57.15.0
2
+
3
+ **Action required:** No
4
+ **Affected packages:** @terreno/api, @terreno/syncdb
5
+
6
+ ## Summary
7
+
8
+ SyncDB pages every stream through one `POST /sync/snapshot/batch` request per round instead of
9
+ one `GET /sync/snapshot` per stream. With many synced collections, cold start is faster and a
10
+ reconnect with no changes costs 2 requests (`GET /sync/streams` plus one batch) whatever the
11
+ collection count. The web IndexedDB persister now serializes the store once per write instead of
12
+ once per transaction, so bootstrap no longer leaves a long save backlog.
13
+
14
+ ## Breaking changes
15
+
16
+ None
17
+
18
+ ## Deprecations
19
+
20
+ None
21
+
22
+ ## New capabilities
23
+
24
+ ### `POST /sync/snapshot/batch`
25
+
26
+ `SyncApp` mounts the route automatically. The body is
27
+ `{streams: [{stream, cursor?, legacyCursor?}], limit?}`, with at most 500 streams.
28
+ `limit` is the row budget for the whole response (default 5,000, max 10,000). The response is
29
+ `{pages}`, one `GET /sync/snapshot`-shaped page per stream. `GET /sync/snapshot` is unchanged.
30
+
31
+ Upgrade `@terreno/api` and `@terreno/syncdb` together for the speedup. A 57.15.0 client talking
32
+ to an older server sees a 404 from the batch route and falls back to per-stream paging
33
+ (`snapshotConcurrency` streams at a time). An older client keeps using `GET /sync/snapshot`.
34
+
35
+ ### `snapshotConcurrency` scope
36
+
37
+ `createSyncDb({snapshotConcurrency})` now applies only to per-stream paging: `forceResync()` and
38
+ the fallback against servers without the batch route.
39
+
40
+ ## Verification
41
+
42
+ - `bun run compile` and `bun run lint` at the app root
43
+ - After deploying the backend, open the web app with dev tools and confirm a reconnect sends one
44
+ `POST /sync/snapshot/batch` instead of one `GET /sync/snapshot` per collection
@@ -159,13 +159,13 @@ Ensures preview environments don't accumulate and waste resources.
159
159
 
160
160
  #### Dependabot Auto-merge
161
161
  **Trigger:** Dependabot pull request opened
162
- **Purpose:** Automatically merges dependency updates after CI passes
162
+ **Purpose:** Automatically merges Dependabot security updates after CI passes. Dependabot opens no version-update PRs; routine upgrades run through the `update-dependencies` skill.
163
163
 
164
164
  **Requirements for auto-merge:**
165
165
  - All CI checks pass
166
166
  - Dependabot PR (created by Dependabot)
167
+ - Not a semver-major bump (for a grouped PR, no bump in the group is major); majors wait for a human review
167
168
  - No conflicts
168
- - 7-day cooldown for security (see Dependency Management)
169
169
 
170
170
  **Learn more:** [Dependency Management](dependency-management.md)
171
171
 
@@ -208,7 +208,7 @@ Ensures preview environments don't accumulate and waste resources.
208
208
  ### Continuous Sync
209
209
  - Documentation syncs on every master push
210
210
  - Rules sync on every master push
211
- - Dependencies update daily (Dependabot)
211
+ - Dependencies update daily (`update-dependencies` skill); Dependabot opens security fixes only
212
212
  - Quality improvements run daily (JSDoc, tests)
213
213
 
214
214
  ## Working with AI Workflows
@@ -280,8 +280,8 @@ The docs-audit workflow flags a missing README or reference page and obvious int
280
280
 
281
281
  ### Dependabot PR doesn't auto-merge
282
282
  - Verify all CI checks pass (green checkmarks)
283
- - Check 7-day cooldown hasn't been violated
284
283
  - Ensure no merge conflicts exist
284
+ - If the security fix needs code changes, fold it into the rolling `chore/update-dependencies` PR instead of pushing to the Dependabot branch
285
285
 
286
286
  ### AI workflow creates poor quality PR
287
287
  - Review the workflow definition in `.github/workflows/*.md`
@@ -290,7 +290,7 @@ The docs-audit workflow flags a missing README or reference page and obvious int
290
290
 
291
291
  ## Related Documentation
292
292
 
293
- - [Dependency Management](dependency-management.md) — Dependabot configuration and auto-merge
293
+ - [Dependency Management](dependency-management.md) — Dependabot security-only configuration, auto-merge, and the update skill
294
294
  - [Deploy to GCP](../how-to/deploy-to-gcp.md) — Manual deployment guide
295
295
  - [GCP Hosting Architecture](gcp-hosting-architecture.md) — Static site hosting design
296
296
  - [Environment Variables](../reference/environment-variables.md) — Configuration reference
@@ -1,16 +1,16 @@
1
1
  # Dependency Management
2
2
 
3
- Terreno pins shared versions in the Bun catalog, lands JavaScript updates through
4
- an agent skill that proves each bump with a test, and keeps Expo native
5
- fingerprints frozen until a release.
3
+ Terreno pins shared versions in the Bun catalog, lands routine JavaScript updates
4
+ through an agent skill that proves each bump with a test, limits Dependabot to
5
+ security fixes, and keeps Expo native fingerprints frozen until a release.
6
6
 
7
7
  ## Layers
8
8
 
9
9
  | Layer | Job |
10
10
  | --- | --- |
11
11
  | **Bun catalog** | One version for every dependency shared by two or more workspace packages (`catalog:` in manifests) |
12
- | **`update-dependencies` skill** | Daily rolling PR on `chore/update-dependencies`: cooldown, exercise test, fingerprint freeze, ledger of landed/failed |
13
- | **Dependabot** | Backup signal for bun + GitHub Actions. Native/Expo catalog names are ignored so auto-merge cannot change the fingerprint |
12
+ | **`update-dependencies` skill** | The only path for routine version upgrades. Daily rolling PR on `chore/update-dependencies`: cooldown, exercise test, local gate, one push per run, fingerprint freeze, ledger of landed/failed |
13
+ | **Dependabot** | Security updates only for bun + GitHub Actions. Native/Expo catalog names are ignored so auto-merge cannot change the fingerprint |
14
14
  | **Release + `upgrading-expo`** | Expo SDK, React Native, and other native-module bumps |
15
15
 
16
16
  Operator steps: [Update dependencies](../how-to/update-dependencies.md). Catalog rule: workspace `package.json` files use `"catalog:"` for shared pins.
@@ -41,16 +41,18 @@ Change a shared version in the catalog once. Do not pin a second copy in a works
41
41
 
42
42
  ## Agent updates vs Dependabot
43
43
 
44
- Dependabot groups many packages and merges when CI is green. That misses two Terreno constraints:
44
+ Dependabot version updates group many packages and merge when CI is green. That misses Terreno constraints, so Dependabot now opens security PRs only:
45
45
 
46
46
  1. **Exercise.** CI can pass without any test importing the bumped module. The skill requires a named test file that imports the package and that passed **after** the bump. If none exists, write a tracer-bullet test first.
47
47
  2. **Fingerprint.** Expo `runtimeVersion.policy: "fingerprint"` ties OTA runtime and EAS dev builds to native hashes of `example-frontend` and `demo`. Native module bumps belong in a release, not a weekday dependency PR.
48
48
 
49
+ 3. **CI is not the test runner.** A red grouped Dependabot PR invites an agent to push fix after fix to the Dependabot branch. Each push starts a full CircleCI pipeline that is attributed to, and emails, whoever owns the pushing token. The skill runs the CI mirror (`bun run prepush`) locally, pushes once per run, never edits CI config to make a bump pass, and stops after two red CI rounds.
50
+
49
51
  `scripts/planning/fingerprintRisk.ts` (`isFingerprintSkip`) is the skip matcher. After every allowed bump the skill still recomputes hashes; a changed hash is reverted even if the matcher allowed the name.
50
52
 
51
53
  ## Cooldown
52
54
 
53
- Non-security npm versions wait **7 days** after publish before a bump (`npm view <name> time`). Most malicious publishes are yanked inside a day. GitHub Dependabot uses the same `cooldown.default-days: 7`. CVE/advisory bumps skip the wait and still need an exercise test and a fingerprint freeze.
55
+ Non-security npm versions wait **7 days** after publish before a bump (`npm view <name> time`). Most malicious publishes are yanked inside a day. CVE/advisory bumps skip the wait and still need an exercise test and a fingerprint freeze. Dependabot security updates are advisory bumps, so they have no cooldown.
54
56
 
55
57
  ## Fingerprint freeze
56
58
 
@@ -67,16 +69,16 @@ JavaScript packages, GitHub Actions, and a few Expo JS-only names may update. Me
67
69
 
68
70
  ## Dependabot configuration
69
71
 
70
- `.github/dependabot.yml` still opens grouped PRs:
72
+ `.github/dependabot.yml` sets `open-pull-requests-limit: 0` on every block. That turns off version-update PRs; security updates ignore the limit and still honor `ignore`. Each block's group uses `applies-to: security-updates`, so fixes arrive grouped:
71
73
 
72
- | Group | Ecosystem | Schedule |
73
- | --- | --- | --- |
74
- | GitHub Actions | `github-actions` at `/` | Weekly |
75
- | Root catalog and root tools | `bun` at `/` | Monthly |
76
- | Backend workspaces | `bun` under api, ai, backends, … | Monthly |
77
- | Frontend workspaces | `bun` under ui, rtk, demo, apps, … | Monthly |
74
+ | Group | Ecosystem |
75
+ | --- | --- |
76
+ | `github-actions` | `github-actions` at `/` |
77
+ | `root` | `bun` at `/` (root catalog and root tools) |
78
+ | `backend` | `bun` under api, ai, backends, … |
79
+ | `frontend` | `bun` under ui, rtk, demo, apps, … |
78
80
 
79
- Frontend ignores catalog-managed names so those PRs do not duplicate the root catalog. Root ignores fingerprint-skip names. `dependabot-auto-merge.yml` still squash-merges Dependabot PRs when required checks pass. Do not merge a Dependabot PR that includes a skip-list package; redo it with [Update dependencies](../how-to/update-dependencies.md).
81
+ Frontend ignores catalog-managed names so those fixes land once, in the root catalog. Root ignores fingerprint-skip names, so a security fix for a native/Expo package never opens a PR; handle it by hand in a release. `dependabot-auto-merge.yml` approves patch and minor Dependabot PRs and enables squash auto-merge, so those security PRs merge when required checks pass. A semver-major security PR (or a group containing one) gets no approval or auto-merge and waits for a human. When one fails because the fix needs code changes, fold the bump into the rolling PR ([Update dependencies](../how-to/update-dependencies.md)); never push to the `dependabot/*` branch.
80
82
 
81
83
  ## Daily rolling PR
82
84
 
@@ -91,10 +93,12 @@ Each package is still proven alone (exercise test + fingerprint) before it stays
91
93
  - One rolling PR; many proven commits on that branch.
92
94
  - Majors: exercise test + fingerprint freeze; no auto-merge.
93
95
  - Security: skip cooldown; do not skip tests or fingerprints.
96
+ - Verify locally, push once; never change CI config to make a bump pass.
94
97
  - New feature dependencies belong in that feature's slice, not this maintenance path.
95
98
 
96
99
  ## Related
97
100
 
98
101
  - [Update dependencies](../how-to/update-dependencies.md)
102
+ - [Run tests locally](../how-to/run-tests-locally.md)
99
103
  - [Install agent skills](../how-to/install-agent-skills.md)
100
104
  - [Bun catalogs](https://bun.sh/docs/install/catalogs)
@@ -41,6 +41,15 @@ Terreno 57.
41
41
  5. **Backend sync registration** — Every synced collection needs `syncPlugin`, `isDeletedPlugin`, and a `sync` block on `modelRouter`, plus `SyncApp` and `RealtimeApp` on the server. Optional `sync.adminBroadcast: true` also emits `sync:delta` to `{collection}|admin`. Admin clients subscribe with `mode: "window"` (`createSyncDb({windowCollections})`) so they do not snapshot or reconcile that collection; they call `hydrateWindow` (REST membership ids + `GET /sync/entities`). Pass the same `accessControl` into `SyncApp` that `AdminApp` uses so `{collection}|admin` requires `admin:access`. `AdminApp.register` also installs that model's admin list/read/`queryFilter` on the admin window so hydrate and live `|admin` deltas match `/admin` REST. Without RBAC, `user.admin` is enough for the shell gate. Live deltas on that stream apply only to ids already in the window; Refresh is a REST re-query plus `hydrateWindow`.
42
42
  6. **Collections too large to sync whole** — List them in `createSyncDb({queryCollections})`. They skip snapshot bootstrap; screens load server-filtered *query windows* (`useWindowQuery`) from the `modelRouter` list endpoint. A window stores only membership (ordered ids); rows live once in the shared entity table. Overlapping windows therefore share rows — one delta updates both views — while each view lists only its own query's members, plus cached rows that newly match its `where` inside the loaded range. Live deltas still arrive on the user's normal streams; windows on screen refetch on reconnect to cover deltas missed offline.
43
43
 
44
+ ## First load and reconnects
45
+
46
+ A fresh device has to download every row the user can see before the UI is complete. The
47
+ client asks for all of the user's streams in one batched request per round, writes each
48
+ round while the next one downloads, and writes rows in small chunks so the screen stays
49
+ responsive. Reconnects use the same path: when nothing changed, catching up costs two
50
+ requests however many collections the app syncs. Collections too large to download whole
51
+ belong in `queryCollections` instead.
52
+
44
53
  ## Tradeoffs
45
54
 
46
55
  **Gains:** instant UI, offline writes with a durable outbox, simpler screen code, and one write path on the server (sync and REST share `modelRouter`).
@@ -47,6 +47,7 @@ Problem-oriented, practical steps. Use these when you know what you want to do.
47
47
  - [Compose charts in a dashboard grid](charts-and-dashboards.md) — `LineChart` / `BarChart` / `AreaChart` / `DonutChart` inside `Card`s in `DashboardGrid` (ops-dashboard scorecards and spans: [dashboard-chart-parity](../implementationPlans/dashboard-chart-parity.md))
48
48
  - [Compare rendered chart snapshots](compare-chart-rendered-snapshots.md) — PNG goldens for the chart visual gallery, not JSON snapshots
49
49
  - [Migrate from @terreno/rtk to @terreno/syncdb](migrate-rtk-to-syncdb.md) — Move data sync to the local-first layer
50
+ - [Use syncdb correctly](use-syncdb-correctly.md) — Serial snapshot startup, outbox-halting 403s, phantom creates, conflicts, and other library behavior apps must handle
50
51
  - [Generate a REST CLI from OpenAPI](generate-a-rest-cli.md) — Scaffold `terreno generate rest-cli` for an app API
51
52
  - [In-app notifications](in-app-notifications.md) — Register `NotificationsApp`, syncdb inbox, and UI bell
52
53
  - [Add organizations](add-organizations.md) — Opt into Membership-backed tenancy, RBAC, admin, and seeded examples
@@ -260,8 +260,10 @@ Caching is off (200 credits per job). The docs Netlify target builds only the
260
260
  current docs tree, disables Docusaurus minification, and pins SSG to one worker
261
261
  thread so the build peaks near 3 GB and fits the 4 GB `medium` executor.
262
262
  Released versions are not rebuilt: `archive-docs-version` (in `publish-release`,
263
- `X.Y.0` tags only) builds each one once and uploads it to its GitHub release, and
264
- `deploy-docs` unpacks those archives under `/<version>/`. See `website/README.md`.
263
+ `X.Y.0` tags only) builds each version once, uploads it to its **draft** GitHub
264
+ release, publishes the release, and then commits the docs snapshot to master, so
265
+ the `deploy-docs` run that commit starts can unpack the archive under
266
+ `/<version>/`. See `website/README.md`.
265
267
 
266
268
  `terreno-gcp` uses CircleCI OIDC (`CIRCLE_OIDC_TOKEN_V2`), never a JSON service
267
269
  account key. Set `circleci_org_id`, `circleci_project_id`, and
@@ -379,6 +381,17 @@ dependencies and runs `test:ci` when that script exists, not `test`.
379
381
  tag version is already on npm, `publish-package.sh` skips it so a recut of
380
382
  the same tag can finish the rest.
381
383
 
384
+ The release skill pushes the tag and creates the GitHub release with
385
+ `gh release create --draft`. The repository has immutable releases on: a
386
+ published release cannot take new assets (GitHub answers `422`). So CI publishes
387
+ the draft last with `scripts/ci/github-release.sh publish`.
388
+ `archive-docs-version` does it right after the upload on `X.Y.0` tags.
389
+ `publish-github-release` does it after `publish-release` on every other tag.
390
+ Both wait up to 10 minutes for the draft and are safe to rerun: a rerun skips
391
+ an upload that already landed on a published release and a snapshot already on
392
+ master. A
393
+ release still in draft after the pipeline means one of them failed.
394
+
382
395
  Only stable tags (`57.3.0`) run `deploy-demo`. It builds the demo from source,
383
396
  so it runs alongside `publish-release` instead of waiting for it. Use
384
397
  `{"run-demo-deploy":true}` on `master` if a prerelease must also refresh the
@@ -1,6 +1,6 @@
1
1
  # Update dependencies
2
2
 
3
- This runs **daily**. Use `/update-dependencies`. There is one rolling PR; later days push to it and rewrite its ledger instead of opening new PRs.
3
+ Routine version upgrades run through the `update-dependencies` skill (`/update-dependencies`). Dependabot opens **security PRs only**. The skill keeps one rolling PR; later runs push to it and rewrite its ledger instead of opening new PRs.
4
4
 
5
5
  ```bash
6
6
  REPOSITORY="$(gh repo view --json nameWithOwner --jq .nameWithOwner)"
@@ -19,7 +19,7 @@ gh pr list --state open --head chore/update-dependencies \
19
19
  bun outdated
20
20
  ```
21
21
 
22
- Prove each bump with an **exercise test** that imports that package. Keep Expo fingerprints for `example-frontend` and `demo` unchanged. Skip `isFingerprintSkip` names until a release.
22
+ Prove each bump with an **exercise test** that imports that package. Verify the whole batch locally, then push **once**. Keep Expo fingerprints for `example-frontend` and `demo` unchanged. Skip `isFingerprintSkip` names until a release.
23
23
 
24
24
  Why: [Dependency management](../explanation/dependency-management.md). Native Expo/React Native bumps wait for `upgrading-expo`.
25
25
 
@@ -27,9 +27,10 @@ Why: [Dependency management](../explanation/dependency-management.md). Native Ex
27
27
 
28
28
  1. Find the same-repository PR with exact head `chore/update-dependencies`, base `master`, and `isCrossRepository: false`. Marker/title matches are not trust signals; ignore forks. Merge `origin/master` without rewriting history. If no trusted PR exists, create the canonical branch on this run.
29
29
  2. Inventory with `bun outdated`. Retry **Failed** rows from the PR body first. Skip `isFingerprintSkip` in `scripts/planning/fingerprintRisk.ts`.
30
- 3. For each candidate: confirm a test imports the package (add a tracer if needed), bump the catalog pin or single-use version, `bun install`, re-run that test, recompute fingerprints. Revert only that bump on failure.
31
- 4. Commit each success on the same branch. Push. Update the **same** PR body: Last run, Landed, Failed, Skipped, Deferred to release. Do not `gh pr create` when that PR is already open.
32
- 5. Read the ledger tomorrow; do not start a new PR.
30
+ 3. For each candidate: confirm a test imports the package (add a tracer if needed), bump the catalog pin or single-use version, `bun install`, re-run that test, recompute fingerprints. Revert only that bump on failure. Commit each success as `Bump <name> from <old> to <new>`.
31
+ 4. Run the batch gate on the final head: `bun run compile`, `bun run lint`, `bun run prepush`. Run by hand any infra job `prepush` lists for a package you bumped (Playwright, Maestro, admin-spa integration). See [Run tests locally](run-tests-locally.md).
32
+ 5. Push once. Update the **same** PR body: Last run, local gate result, CI rounds, Landed, Failed, Skipped, Deferred to release. Do not `gh pr create` when that PR is already open.
33
+ 6. Read CI on the pushed head. Check each failed CircleCI job's `messages` before its step logs. Revert a bump that caused a failure and push a second time at most; stop and report anything else.
33
34
 
34
35
  ## Guardrails
35
36
 
@@ -41,7 +42,11 @@ Why: [Dependency management](../explanation/dependency-management.md). Native Ex
41
42
  | No test imports the package | Write the tracer test first. Do not bump on `test:ci` alone |
42
43
  | Published < 7 days ago (non-CVE) | Skip; log under Skipped |
43
44
  | Major version | May land on the rolling PR if tests + fingerprints pass; no auto-merge |
44
- | Dependabot extra PRs | Do not merge skip-list packages; fold JS bumps into the rolling PR |
45
+ | Local gate red | Fix or revert before pushing. CI is not the test runner |
46
+ | CI job has `messages` (for example `resource-class-not-in-plan`) | Config/plan error, not a dependency failure. Log it and stop |
47
+ | A bump seems to need CI config changes | Never edit `.circleci/`, workflows, executors, or resource classes from this path. Stop and report |
48
+ | CI red twice on heads this run pushed | Stop and report; do not push a third time |
49
+ | Dependabot security PR is red | Do not push to `dependabot/*`. Fold the bump into the rolling PR |
45
50
 
46
51
  ## Fingerprints
47
52
 
@@ -49,6 +54,10 @@ Why: [Dependency management](../explanation/dependency-management.md). Native Ex
49
54
 
50
55
  `eas.json` Bun pins feed the fingerprint. Do not bump them here.
51
56
 
52
- ## Dependabot still runs
57
+ ## Who CI emails
53
58
 
54
- GitHub Dependabot remains in `.github/dependabot.yml`. Root catalog ignores native/Expo packages. Treat Dependabot as a signal; land updates on the rolling PR from this guide.
59
+ CircleCI attributes a pipeline to the GitHub user whose token pushed it and emails that user on each failed workflow. Run the scheduled agent with a bot identity when one is configured. If it runs as a person, the one-push rule caps their mail at one round per run.
60
+
61
+ ## Dependabot security PRs
62
+
63
+ `.github/dependabot.yml` sets `open-pull-requests-limit: 0` on every block, so Dependabot opens no version-update PRs. Security updates still open, grouped per block, and skip ignored native/Expo names. `dependabot-auto-merge.yml` approves them and enables squash auto-merge, so a green security PR merges without this skill. A red one gets its bump folded into the rolling PR. A security fix for an ignored native package is a manual release task (`release` + `upgrading-expo`).
@@ -0,0 +1,111 @@
1
+ # How to Use syncdb Correctly
2
+
3
+ This page lists `@terreno/syncdb` behavior that a working app has to account for. Each entry says what the library does today, what breaks if you ignore it, and what to do instead. It was checked against `syncdb/src` and `api/src/sync` at 57.13.0. Several entries are library gaps rather than intended behavior. Delete an entry when its gap is fixed.
4
+
5
+ Read [Local-first data](../explanation/local-first-data.md) and [Migrate from @terreno/rtk to @terreno/syncdb](migrate-rtk-to-syncdb.md) first.
6
+
7
+ ## Startup and snapshots
8
+
9
+ ### Cold start costs one round trip per round, not per stream
10
+
11
+ A stream is one collection in one scope, so a user has roughly (synced collections) × (tenants they belong to) streams. Reconcile, subscribe catch-up and wipe re-bootstrap page every stream through `POST /sync/snapshot/batch`: one round trip per round, with up to 5,000 rows per response shared across the streams. Empty streams cost nothing extra. `forceResync`, and every pass against a server without the batch route, still page with one `GET /sync/snapshot` per stream, `snapshotConcurrency` streams side by side (default 6). Each stream runs one pass at a time, and a pass that purges (force resync, wipe) waits for the stream's in-flight pass to finish first.
12
+
13
+ Measured with a fake HTTP channel at 150 ms latency, 13 collections and 2 workspaces (26 streams): 4.1 s to the first full reconcile when paging one stream at a time, 0.95 s at the default concurrency. In a browser with 100 collections and 10k rows, batching cut a reconnect with no changes from 101 requests to 2.
14
+
15
+ | Do | Why |
16
+ | --- | --- |
17
+ | Sync only the collections the client reads. | Each collection adds one stream per tenant to cold start. |
18
+ | Gate each screen on its **own** stream's bootstrap (see [No loading flag on synced reads](#no-loading-flag-on-synced-reads)), not on "sync finished". | The stream a screen needs may land after others. |
19
+ | Don't call `forceResync()` on a hot path. | It purges and re-pages every stream. |
20
+ | Raise `snapshotConcurrency` only if the backend can take it. | Without the batch route, each in-flight stream is one concurrent snapshot query on the server. |
21
+
22
+ The startup reconcile and each collection's `sync:subscribed` catch-up both page the same streams. A catch-up queues behind a pass that started before the room join, because that pass may have missed a write. So a cold start still pages each stream twice, one pass after the other.
23
+
24
+ ### No loading flag on synced reads
25
+
26
+ `useQuery` returns a plain array. `useListQuery` returns `{data}`. The `isPending` on `useEntity`/`useReadQuery` means an outbox write is pending, not that data is loading. An empty array before bootstrap looks the same as an empty collection, so a screen that renders its empty state from `[]` flashes "nothing here" on every cold start.
27
+
28
+ The per-stream `bootstrapped` mark is the right signal, but `isStreamBootstrapped` is not exported. Read the cell directly:
29
+
30
+ ```typescript
31
+ const isBootstrapped = (stream: string): boolean =>
32
+ syncDb.store.raw.getCell("_cursors", stream, "bootstrapped") === true;
33
+ ```
34
+
35
+ Don't use the cursor (`useSyncStatus().streams`) for this. A live delta can move the cursor to the head before the snapshot below it has been paged. `useWindowQuery` (query collections only) does expose `isLoading`.
36
+
37
+ A stream key is `<collection>|<scopeType>:<value>`, for example `documents|tenant:<workspaceId>`. A user with several tenants has one stream per tenant. Treat a collection as loaded only once every known stream for it is bootstrapped (`syncDb.store.getKnownStreams()`).
38
+
39
+ ## Writes
40
+
41
+ ### Mutations return an id, not an outcome
42
+
43
+ `create`, `update` and `remove` from `useMutate` (and from the generated `useCreate*`/`useUpdate*`/`useDelete*` hooks) return `{mutationId, id}` synchronously. There is no promise for the server ack. The client also drops the ack's `warning`. A `warning` means a server post-hook threw after the write committed.
44
+
45
+ If a screen needs to know the outcome (close a modal only once the create landed, toast a refusal), wrap the transport and HTTP channel and record ack and nack outcomes by `mutationId`. Then await that `mutationId`. Don't poll the store.
46
+
47
+ ### A 403 from a write halts the whole outbox
48
+
49
+ The server maps 403/405 to an `unauthorized` nack and 400/404/422 to `validation`. They behave very differently:
50
+
51
+ | Nack | Client behavior |
52
+ | --- | --- |
53
+ | `unauthorized` | The mutation is re-queued and the **whole drain stops** (auth-pause). On re-auth the same `mutationId` replays, reads the same stored nack, and pauses again. It never settles. |
54
+ | `validation` | The mutation is marked `failed` and the drain continues. That entity's later writes stay blocked until `retryFailed({entityId})`. |
55
+ | `conflict` | That entity's later writes stay blocked (or every write is blocked, with `haltQueueOnConflict: true`) until `resolveConflict`. |
56
+ | `error` | Retried up to the retry budget, then handled like `validation`. |
57
+
58
+ So a router hook that refuses a synced write for a business reason must throw **422**, not 403. That includes `preCreate`/`preUpdate` membership checks. Keep 403 for real auth failures.
59
+
60
+ ### Don't send a stale tenant field on create
61
+
62
+ For a tenant-scoped collection, `enforceWriteScope` checks a client-sent scope field (for example `workspaceId`) against the user's memberships **before** `preCreate` runs. A value the user does not belong to is a 403, which halts the outbox (see above). If the field is missing or null, the check is skipped and `preCreate` can fill it in.
63
+
64
+ Either omit the tenant field and let `preCreate` set it, or send it only from the currently synced row of the parent tenant, never from a persisted id that may be stale.
65
+
66
+ ### The local row never gets server-computed fields from the create
67
+
68
+ After an ack, the client stamps the row with the ack's seq and keeps the optimistic data. The server's create delta has the same seq, so the client skips it. Whatever `preCreate`, schema defaults or hooks set on the server does not reach the local row until that row changes again or is repaired. If the delta arrives before the ack, the row is marked for repair, but repair only runs on the next `reconcile()` (reconnect, seq jump, or every 5 minutes).
69
+
70
+ Do one of these:
71
+
72
+ - Send every field the UI reads in the create payload, matching what the server would compute (defaults, `order`, ownership fields).
73
+ - After the ack, patch the local row yourself.
74
+
75
+ The local row's `data` is exactly what you passed. Include `_id` in the payload if anything reads `row._id`. Server-written rows carry `_id` and `id`.
76
+
77
+ ### A refused create leaves a phantom row
78
+
79
+ When a create is nacked as `validation` (or runs out of `error` retries), the repair fetch finds no server row and only clears the repair mark. The optimistic row stays in the store with `stream: ""` and shows up in queries. Only `forceResync` purges such rows.
80
+
81
+ On a refused create, tombstone the local row yourself and call `retryFailed({entityId})` so later writes to that id are released.
82
+
83
+ ### Conflicts are never resolved for you
84
+
85
+ A `conflict` nack blocks every later write to that row, and any queued write whose args mention its id, until someone calls `client.resolveConflict({mutationId, strategy: "keepMine" | "useServer"})` (or `useConflicts().resolve`). Local edits keep applying optimistically, so the UI looks fine while nothing reaches the server.
86
+
87
+ If the app has no conflict UI, run a watcher that resolves each conflict as it is recorded. Use `keepMine` for fields the user types, and `useServer` for everything else.
88
+
89
+ ## Server side
90
+
91
+ ### Write synced models one document at a time, by `_id`
92
+
93
+ On a synced model, `updateMany`, `deleteMany`, `deleteOne`, `findOneAndDelete` and `bulkWrite` throw. Single-document writes by `_id` without upsert are allowed: `save()`, `updateOne`, `findOneAndUpdate`, `findByIdAndUpdate`, `replaceOne`. Delete by setting `deleted: true`, because a hard delete emits no delta.
94
+
95
+ In tests, clean up with `Model.collection.deleteMany(...)`, which bypasses Mongoose.
96
+
97
+ ### Deltas don't apply `queryFilter`
98
+
99
+ Snapshots and `/sync/entities` apply the router's `queryFilter`. Live deltas check only `permissions.read` (or RBAC) per recipient. A router that restricts visibility only through `queryFilter` leaks rows over the socket. Every synced router's `read` permission must enforce the same restriction itself.
100
+
101
+ ### Membership grants arrive late
102
+
103
+ A user's new tenant is picked up by the socket's session revalidation sweep (`sessionRevalidationIntervalMs`, 60 s by default) or a resubscribe. The client finds the new stream on its next `reconcile()` (`reconcileIntervalMs`, 5 minutes by default, plus reconnects). Until then the new tenant's rows don't appear.
104
+
105
+ When the product grants membership, push a content-free event to the user's sockets and have the client reconnect syncdb. Reconnecting resubscribes and reconciles. Grants written another way (admin, scripts) still wait for the timers.
106
+
107
+ ## Auth
108
+
109
+ ### Logging out does not wipe local data
110
+
111
+ The client's own `onAuthChange` handling treats a logout as a pause and **never wipes**. A different user logging in does wipe. To clear one user's data on sign-out, set `wipeOnSignOut: true` and call `client.signOut()` explicitly when the app's user clears.
@@ -70,7 +70,7 @@ Expo autolinking walks the app's own dependencies; a nested copy leaves the nati
70
70
  │ sync:mutate ─────────────────────────────►│ │
71
71
  │ sync:ack / sync:nack ◄────────────────────┘ │
72
72
  │ sync:delta ◄──────────────────────────────────────────────┘
73
- │ GET /sync/snapshot (bootstrap + catch-up, HTTP)
73
+ │ POST /sync/snapshot/batch (bootstrap + catch-up, HTTP)
74
74
  └── POST /sync/mutate (fallback while the socket is down)
75
75
  ```
76
76
 
@@ -185,6 +185,7 @@ be passed to `betterAuthAdapter` directly.
185
185
  | `wipeOnSignOut` | `boolean` | `false` | `signOut()` also wipes local data |
186
186
  | `batchSize` | `number` | `50` (server caps at 100) | Max mutations per batched drain send |
187
187
  | `haltQueueOnConflict` | `boolean` | `false` | `true` = conflict halts entire drain; `false` = per-entity blocking |
188
+ | `snapshotConcurrency` | `number` | `6` | Streams paged at once with `GET /sync/snapshot`: by `forceResync`, and by every other pass against a server without `POST /sync/snapshot/batch`. Each stream runs one pass at a time. `1` pages one stream at a time |
188
189
  | `onDecryptFailure` | `() => void` | wipe + re-bootstrap (with `console.warn`) | Override web decrypt-failure recovery |
189
190
  | `tombstoneRetentionMs` | `number` | `7776000000` (90 days); `0` disables | Client tombstone compaction after successful reconcile |
190
191
  | `startAuthRetryAttempts` | `number` | `3`; `1` disables | `start()` attempts to resolve `getUserId()` |
@@ -201,7 +202,7 @@ be passed to `betterAuthAdapter` directly.
201
202
  | `goOffline()` | Simulated outage: disconnect transport, pause replay/reconcile/timer; local mutations keep queueing. |
202
203
  | `goOnline()` | End simulated outage; reconnect triggers reconcile + outbox replay. |
203
204
  | `mutate({collection, operation, id?, data?})` | Optimistic local write + durable outbox enqueue + fire-and-forget replay. Returns `{mutationId, id}`. |
204
- | `reconcile()` | HTTP snapshot catch-up for every known stream; runs tombstone compaction on success. Also runs automatically on (re)connect, on a rate-limited seq-jump hint, and on the periodic timer; each `sync:subscribed` confirmation additionally pages just the streams it names. |
205
+ | `reconcile()` | HTTP snapshot catch-up for every known stream, batched into `POST /sync/snapshot/batch` rounds (see [Snapshot bootstrap](#snapshot-bootstrap)); runs tombstone compaction on success. Also runs automatically on (re)connect, on a rate-limited seq-jump hint, and on the periodic timer; `sync:subscribed` confirmations additionally page just the streams they name, and confirmations arriving while that catch-up request is in flight are paged together in the next one. |
205
206
  | `hydrateWindow({collection, ids, restRows?})` | Admin window upsert: REST rows (optional) land immediately; every requested id is also fetched from `GET /sync/entities` before this resolves so seq/deleted metadata is canonical for immediate update/delete. A `{collection}|admin` delta that lands while the fetch is in flight wins, so hydration never rewinds a row to an older seq. Unknown ids are ignored. |
206
207
  | `queryWindows` | Server query windows for `queryCollections`: `fetchWindow({query, nextPage?})`, `getWindow(query)`, `retain(query)` → release, `refetchRetained()`, `pruneUnretained()`, `subscribe(cb)`, `keyFor(query)`. See [Query windows](#query-windows). |
207
208
  | `forceResync()` | Purge every known stream locally and re-bootstrap from cursor 0 (outbox/conflicts untouched). Returns `{ok, reason?, streams, purged, repaired}`. After discovery and each stream bootstrap it waits for in-flight `start`/`stop`/auth-change work, then abandons with `reason: "superseded"` when that work switched users or generations. |
@@ -587,11 +588,36 @@ sync: {
587
588
 
588
589
  ## Sync protocol
589
590
 
591
+ ### Snapshot bootstrap
592
+
593
+ The client pages every stream it belongs to through `POST /sync/snapshot/batch`, not one
594
+ `GET /sync/snapshot` per stream. With 100 collections, a reconnect that finds nothing new
595
+ is 2 requests (`GET /sync/streams` plus one batch) instead of 101.
596
+
597
+ - **Rounds:** each round asks for every stream that still has pages. Streams that reached
598
+ their head drop out, so later rounds give the remaining streams a larger share of the
599
+ 5,000-entity budget.
600
+ - **Pipelining:** the next round is sent as soon as a response arrives, because its
601
+ cursors come from the response. The client writes the current round while the next
602
+ one downloads.
603
+ - **Chunked apply:** each page is written in store transactions of 500 entities, with a
604
+ yield to the event loop between them, so a large page never freezes the UI in one block.
605
+ A stream's cursor advances in the transaction holding its last entity, so a page cut
606
+ short by a crash is fetched again and re-applied as a no-op.
607
+ - **Older servers:** a 404 without an APIError body (the route does not exist) switches
608
+ the client to per-stream `GET /sync/snapshot` paging, `snapshotConcurrency` streams at
609
+ a time, for the rest of the session. An APIError 404, such as an unknown collection,
610
+ fails that pass and keeps batching on.
611
+ - **One pass per stream:** a batch pass owns each of its streams the way a per-stream
612
+ pass does, so a purge (`forceResync`, wipe) waits for it, and a pass that is superseded
613
+ by a sign-out or user switch stops between chunks without writing the rest.
614
+
590
615
  ### HTTP routes (`SyncApp`, all authenticated)
591
616
 
592
617
  | Endpoint | Purpose |
593
618
  |----------|---------|
594
- | `GET /sync/snapshot?collection=&stream=&cursor=&limit=` | Bootstrap + catch-up per stream |
619
+ | `GET /sync/snapshot?collection=&stream=&cursor=&limit=` | Bootstrap + catch-up for one stream (`limit` max 100) |
620
+ | `POST /sync/snapshot/batch` | Bootstrap + catch-up for many streams in one round trip. Body `{streams: [{stream, cursor?, legacyCursor?}], limit?}`, at most 500 streams, no repeats. `limit` (default 5,000, max 10,000) is the entity budget for the whole response, split evenly across the streams. Returns `{pages}`, one `GET /sync/snapshot`-shaped page per stream in request order. Any stream the caller may not read fails the whole request |
595
621
  | `GET /sync/streams` | Current stream membership for the user |
596
622
  | `GET /sync/entities` | Point lookup for entity repair and admin window hydrate. Admin-window callers (`admin:access` with RBAC, else `user.admin`) on `adminBroadcast` collections receive requested ids across product streams. When AdminApp registered a scope, list/read/`queryFilter` still apply; unknown or out-of-scope ids are omitted |
597
623
  | `POST /sync/mutate` | Single mutation (HTTP fallback). Optional `mutationMode: "adminWindow"` validates `adminBroadcast`, admin-window access, and a registered AdminApp write scope, then runs the shared sync executor with **AdminApp** pre/post hooks (not product `modelRouter` hooks), plus the same permission (including org membership), stripping, and audit semantics as `/admin` REST. HTTP mutate binds `X-Organization-Id` into org context |
@@ -635,6 +661,11 @@ const syncDb = createSyncDb({
635
661
  });
636
662
  ```
637
663
 
664
+ Saves are debounced (500 ms). The store is serialized once per write, not once per
665
+ transaction: autosave compares TinyBase's content hashes (O(1)) to skip saves the last
666
+ write already covered, so a bootstrap of many pages costs a few writes instead of one
667
+ full-store serialization per page.
668
+
638
669
  Undecryptable persisted data triggers wipe + re-bootstrap by default (`onDecryptFailure` to override). Storage read errors surface `persistence: "error"` without overwriting the blob.
639
670
 
640
671
  **Native:** plaintext SQLite in the OS sandbox (by design).