@terreno/mcp 57.13.0 → 57.15.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/docs/upgrades/57.14.0.md +43 -0
- package/dist/docs/upgrades/57.15.0.md +44 -0
- package/dist/docs/versioned/next/explanation/ai-workflows.md +5 -5
- package/dist/docs/versioned/next/explanation/dependency-management.md +19 -15
- package/dist/docs/versioned/next/explanation/local-first-data.md +9 -0
- package/dist/docs/versioned/next/how-to/README.md +1 -0
- package/dist/docs/versioned/next/how-to/circleci.md +15 -2
- package/dist/docs/versioned/next/how-to/update-dependencies.md +17 -8
- package/dist/docs/versioned/next/how-to/use-syncdb-correctly.md +111 -0
- package/dist/docs/versioned/next/reference/syncdb.md +34 -3
- package/package.json +3 -3
- package/src/docs/upgrades/57.14.0.md +43 -0
- package/src/docs/upgrades/57.15.0.md +44 -0
- package/src/docs/versioned/next/explanation/ai-workflows.md +5 -5
- package/src/docs/versioned/next/explanation/dependency-management.md +19 -15
- package/src/docs/versioned/next/explanation/local-first-data.md +9 -0
- package/src/docs/versioned/next/how-to/README.md +1 -0
- package/src/docs/versioned/next/how-to/circleci.md +15 -2
- package/src/docs/versioned/next/how-to/update-dependencies.md +17 -8
- package/src/docs/versioned/next/how-to/use-syncdb-correctly.md +111 -0
- package/src/docs/versioned/next/reference/syncdb.md +34 -3
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# Upgrading to 57.14.0
|
|
2
|
+
|
|
3
|
+
**Action required:** No
|
|
4
|
+
**Affected packages:** @terreno/syncdb
|
|
5
|
+
|
|
6
|
+
## Summary
|
|
7
|
+
|
|
8
|
+
`@terreno/syncdb` pages snapshot streams in parallel (up to 6 at once by default) instead of one at a time, so cold start, reconcile, subscribe catch-up, `forceResync`, and wipe re-bootstrap finish faster when an app has many collections or scopes. Each stream now runs one snapshot pass at a time, which stops a concurrent pass from marking a purged stream bootstrapped over rows it never paged. No code change is required.
|
|
9
|
+
|
|
10
|
+
## Breaking changes
|
|
11
|
+
|
|
12
|
+
None
|
|
13
|
+
|
|
14
|
+
## Deprecations
|
|
15
|
+
|
|
16
|
+
None
|
|
17
|
+
|
|
18
|
+
## New capabilities
|
|
19
|
+
|
|
20
|
+
### `snapshotConcurrency` on `createSyncDb`
|
|
21
|
+
|
|
22
|
+
The client now sends up to `snapshotConcurrency` snapshot requests at once (default `DEFAULT_SNAPSHOT_CONCURRENCY`, 6). If your backend cannot take the extra parallel load at cold start, lower it. `1` restores the old one-stream-at-a-time behavior.
|
|
23
|
+
|
|
24
|
+
```typescript
|
|
25
|
+
// Before
|
|
26
|
+
export const syncDb = createSyncDb({authProvider, baseUrl, collections, name: "my-app"});
|
|
27
|
+
|
|
28
|
+
// After (only if you need to limit parallel snapshot requests)
|
|
29
|
+
export const syncDb = createSyncDb({
|
|
30
|
+
authProvider,
|
|
31
|
+
baseUrl,
|
|
32
|
+
collections,
|
|
33
|
+
name: "my-app",
|
|
34
|
+
snapshotConcurrency: 2,
|
|
35
|
+
});
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
A new how-to, `docs/how-to/use-syncdb-correctly.md`, lists library behavior apps have to handle.
|
|
39
|
+
|
|
40
|
+
## Verification
|
|
41
|
+
|
|
42
|
+
- `bun run compile` and `bun run lint` at the app root
|
|
43
|
+
- Log in on a fresh device or after `wipeLocalData`; synced screens load, and the backend logs show several `/sync/snapshot` requests in flight together
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
# Upgrading to 57.15.0
|
|
2
|
+
|
|
3
|
+
**Action required:** No
|
|
4
|
+
**Affected packages:** @terreno/api, @terreno/syncdb
|
|
5
|
+
|
|
6
|
+
## Summary
|
|
7
|
+
|
|
8
|
+
SyncDB pages every stream through one `POST /sync/snapshot/batch` request per round instead of
|
|
9
|
+
one `GET /sync/snapshot` per stream. With many synced collections, cold start is faster and a
|
|
10
|
+
reconnect with no changes costs 2 requests (`GET /sync/streams` plus one batch) whatever the
|
|
11
|
+
collection count. The web IndexedDB persister now serializes the store once per write instead of
|
|
12
|
+
once per transaction, so bootstrap no longer leaves a long save backlog.
|
|
13
|
+
|
|
14
|
+
## Breaking changes
|
|
15
|
+
|
|
16
|
+
None
|
|
17
|
+
|
|
18
|
+
## Deprecations
|
|
19
|
+
|
|
20
|
+
None
|
|
21
|
+
|
|
22
|
+
## New capabilities
|
|
23
|
+
|
|
24
|
+
### `POST /sync/snapshot/batch`
|
|
25
|
+
|
|
26
|
+
`SyncApp` mounts the route automatically. The body is
|
|
27
|
+
`{streams: [{stream, cursor?, legacyCursor?}], limit?}`, with at most 500 streams.
|
|
28
|
+
`limit` is the row budget for the whole response (default 5,000, max 10,000). The response is
|
|
29
|
+
`{pages}`, one `GET /sync/snapshot`-shaped page per stream. `GET /sync/snapshot` is unchanged.
|
|
30
|
+
|
|
31
|
+
Upgrade `@terreno/api` and `@terreno/syncdb` together for the speedup. A 57.15.0 client talking
|
|
32
|
+
to an older server sees a 404 from the batch route and falls back to per-stream paging
|
|
33
|
+
(`snapshotConcurrency` streams at a time). An older client keeps using `GET /sync/snapshot`.
|
|
34
|
+
|
|
35
|
+
### `snapshotConcurrency` scope
|
|
36
|
+
|
|
37
|
+
`createSyncDb({snapshotConcurrency})` now applies only to per-stream paging: `forceResync()` and
|
|
38
|
+
the fallback against servers without the batch route.
|
|
39
|
+
|
|
40
|
+
## Verification
|
|
41
|
+
|
|
42
|
+
- `bun run compile` and `bun run lint` at the app root
|
|
43
|
+
- After deploying the backend, open the web app with dev tools and confirm a reconnect sends one
|
|
44
|
+
`POST /sync/snapshot/batch` instead of one `GET /sync/snapshot` per collection
|
|
@@ -159,13 +159,13 @@ Ensures preview environments don't accumulate and waste resources.
|
|
|
159
159
|
|
|
160
160
|
#### Dependabot Auto-merge
|
|
161
161
|
**Trigger:** Dependabot pull request opened
|
|
162
|
-
**Purpose:** Automatically merges
|
|
162
|
+
**Purpose:** Automatically merges Dependabot security updates after CI passes. Dependabot opens no version-update PRs; routine upgrades run through the `update-dependencies` skill.
|
|
163
163
|
|
|
164
164
|
**Requirements for auto-merge:**
|
|
165
165
|
- All CI checks pass
|
|
166
166
|
- Dependabot PR (created by Dependabot)
|
|
167
|
+
- Not a semver-major bump (for a grouped PR, no bump in the group is major); majors wait for a human review
|
|
167
168
|
- No conflicts
|
|
168
|
-
- 7-day cooldown for security (see Dependency Management)
|
|
169
169
|
|
|
170
170
|
**Learn more:** [Dependency Management](dependency-management.md)
|
|
171
171
|
|
|
@@ -208,7 +208,7 @@ Ensures preview environments don't accumulate and waste resources.
|
|
|
208
208
|
### Continuous Sync
|
|
209
209
|
- Documentation syncs on every master push
|
|
210
210
|
- Rules sync on every master push
|
|
211
|
-
- Dependencies update daily (Dependabot
|
|
211
|
+
- Dependencies update daily (`update-dependencies` skill); Dependabot opens security fixes only
|
|
212
212
|
- Quality improvements run daily (JSDoc, tests)
|
|
213
213
|
|
|
214
214
|
## Working with AI Workflows
|
|
@@ -280,8 +280,8 @@ The docs-audit workflow flags a missing README or reference page and obvious int
|
|
|
280
280
|
|
|
281
281
|
### Dependabot PR doesn't auto-merge
|
|
282
282
|
- Verify all CI checks pass (green checkmarks)
|
|
283
|
-
- Check 7-day cooldown hasn't been violated
|
|
284
283
|
- Ensure no merge conflicts exist
|
|
284
|
+
- If the security fix needs code changes, fold it into the rolling `chore/update-dependencies` PR instead of pushing to the Dependabot branch
|
|
285
285
|
|
|
286
286
|
### AI workflow creates poor quality PR
|
|
287
287
|
- Review the workflow definition in `.github/workflows/*.md`
|
|
@@ -290,7 +290,7 @@ The docs-audit workflow flags a missing README or reference page and obvious int
|
|
|
290
290
|
|
|
291
291
|
## Related Documentation
|
|
292
292
|
|
|
293
|
-
- [Dependency Management](dependency-management.md) — Dependabot configuration
|
|
293
|
+
- [Dependency Management](dependency-management.md) — Dependabot security-only configuration, auto-merge, and the update skill
|
|
294
294
|
- [Deploy to GCP](../how-to/deploy-to-gcp.md) — Manual deployment guide
|
|
295
295
|
- [GCP Hosting Architecture](gcp-hosting-architecture.md) — Static site hosting design
|
|
296
296
|
- [Environment Variables](../reference/environment-variables.md) — Configuration reference
|
|
@@ -1,16 +1,16 @@
|
|
|
1
1
|
# Dependency Management
|
|
2
2
|
|
|
3
|
-
Terreno pins shared versions in the Bun catalog, lands JavaScript updates
|
|
4
|
-
an agent skill that proves each bump with a test,
|
|
5
|
-
fingerprints frozen until a release.
|
|
3
|
+
Terreno pins shared versions in the Bun catalog, lands routine JavaScript updates
|
|
4
|
+
through an agent skill that proves each bump with a test, limits Dependabot to
|
|
5
|
+
security fixes, and keeps Expo native fingerprints frozen until a release.
|
|
6
6
|
|
|
7
7
|
## Layers
|
|
8
8
|
|
|
9
9
|
| Layer | Job |
|
|
10
10
|
| --- | --- |
|
|
11
11
|
| **Bun catalog** | One version for every dependency shared by two or more workspace packages (`catalog:` in manifests) |
|
|
12
|
-
| **`update-dependencies` skill** | Daily rolling PR on `chore/update-dependencies`: cooldown, exercise test, fingerprint freeze, ledger of landed/failed |
|
|
13
|
-
| **Dependabot** |
|
|
12
|
+
| **`update-dependencies` skill** | The only path for routine version upgrades. Daily rolling PR on `chore/update-dependencies`: cooldown, exercise test, local gate, one push per run, fingerprint freeze, ledger of landed/failed |
|
|
13
|
+
| **Dependabot** | Security updates only for bun + GitHub Actions. Native/Expo catalog names are ignored so auto-merge cannot change the fingerprint |
|
|
14
14
|
| **Release + `upgrading-expo`** | Expo SDK, React Native, and other native-module bumps |
|
|
15
15
|
|
|
16
16
|
Operator steps: [Update dependencies](../how-to/update-dependencies.md). Catalog rule: workspace `package.json` files use `"catalog:"` for shared pins.
|
|
@@ -41,16 +41,18 @@ Change a shared version in the catalog once. Do not pin a second copy in a works
|
|
|
41
41
|
|
|
42
42
|
## Agent updates vs Dependabot
|
|
43
43
|
|
|
44
|
-
Dependabot
|
|
44
|
+
Dependabot version updates group many packages and merge when CI is green. That misses Terreno constraints, so Dependabot now opens security PRs only:
|
|
45
45
|
|
|
46
46
|
1. **Exercise.** CI can pass without any test importing the bumped module. The skill requires a named test file that imports the package and that passed **after** the bump. If none exists, write a tracer-bullet test first.
|
|
47
47
|
2. **Fingerprint.** Expo `runtimeVersion.policy: "fingerprint"` ties OTA runtime and EAS dev builds to native hashes of `example-frontend` and `demo`. Native module bumps belong in a release, not a weekday dependency PR.
|
|
48
48
|
|
|
49
|
+
3. **CI is not the test runner.** A red grouped Dependabot PR invites an agent to push fix after fix to the Dependabot branch. Each push starts a full CircleCI pipeline that is attributed to, and emails, whoever owns the pushing token. The skill runs the CI mirror (`bun run prepush`) locally, pushes once per run, never edits CI config to make a bump pass, and stops after two red CI rounds.
|
|
50
|
+
|
|
49
51
|
`scripts/planning/fingerprintRisk.ts` (`isFingerprintSkip`) is the skip matcher. After every allowed bump the skill still recomputes hashes; a changed hash is reverted even if the matcher allowed the name.
|
|
50
52
|
|
|
51
53
|
## Cooldown
|
|
52
54
|
|
|
53
|
-
Non-security npm versions wait **7 days** after publish before a bump (`npm view <name> time`). Most malicious publishes are yanked inside a day.
|
|
55
|
+
Non-security npm versions wait **7 days** after publish before a bump (`npm view <name> time`). Most malicious publishes are yanked inside a day. CVE/advisory bumps skip the wait and still need an exercise test and a fingerprint freeze. Dependabot security updates are advisory bumps, so they have no cooldown.
|
|
54
56
|
|
|
55
57
|
## Fingerprint freeze
|
|
56
58
|
|
|
@@ -67,16 +69,16 @@ JavaScript packages, GitHub Actions, and a few Expo JS-only names may update. Me
|
|
|
67
69
|
|
|
68
70
|
## Dependabot configuration
|
|
69
71
|
|
|
70
|
-
`.github/dependabot.yml` still
|
|
72
|
+
`.github/dependabot.yml` sets `open-pull-requests-limit: 0` on every block. That turns off version-update PRs; security updates ignore the limit and still honor `ignore`. Each block's group uses `applies-to: security-updates`, so fixes arrive grouped:
|
|
71
73
|
|
|
72
|
-
| Group | Ecosystem |
|
|
73
|
-
| --- | --- |
|
|
74
|
-
|
|
|
75
|
-
|
|
|
76
|
-
|
|
|
77
|
-
|
|
|
74
|
+
| Group | Ecosystem |
|
|
75
|
+
| --- | --- |
|
|
76
|
+
| `github-actions` | `github-actions` at `/` |
|
|
77
|
+
| `root` | `bun` at `/` (root catalog and root tools) |
|
|
78
|
+
| `backend` | `bun` under api, ai, backends, … |
|
|
79
|
+
| `frontend` | `bun` under ui, rtk, demo, apps, … |
|
|
78
80
|
|
|
79
|
-
Frontend ignores catalog-managed names so those
|
|
81
|
+
Frontend ignores catalog-managed names so those fixes land once, in the root catalog. Root ignores fingerprint-skip names, so a security fix for a native/Expo package never opens a PR; handle it by hand in a release. `dependabot-auto-merge.yml` approves patch and minor Dependabot PRs and enables squash auto-merge, so those security PRs merge when required checks pass. A semver-major security PR (or a group containing one) gets no approval or auto-merge and waits for a human. When one fails because the fix needs code changes, fold the bump into the rolling PR ([Update dependencies](../how-to/update-dependencies.md)); never push to the `dependabot/*` branch.
|
|
80
82
|
|
|
81
83
|
## Daily rolling PR
|
|
82
84
|
|
|
@@ -91,10 +93,12 @@ Each package is still proven alone (exercise test + fingerprint) before it stays
|
|
|
91
93
|
- One rolling PR; many proven commits on that branch.
|
|
92
94
|
- Majors: exercise test + fingerprint freeze; no auto-merge.
|
|
93
95
|
- Security: skip cooldown; do not skip tests or fingerprints.
|
|
96
|
+
- Verify locally, push once; never change CI config to make a bump pass.
|
|
94
97
|
- New feature dependencies belong in that feature's slice, not this maintenance path.
|
|
95
98
|
|
|
96
99
|
## Related
|
|
97
100
|
|
|
98
101
|
- [Update dependencies](../how-to/update-dependencies.md)
|
|
102
|
+
- [Run tests locally](../how-to/run-tests-locally.md)
|
|
99
103
|
- [Install agent skills](../how-to/install-agent-skills.md)
|
|
100
104
|
- [Bun catalogs](https://bun.sh/docs/install/catalogs)
|
|
@@ -41,6 +41,15 @@ Terreno 57.
|
|
|
41
41
|
5. **Backend sync registration** — Every synced collection needs `syncPlugin`, `isDeletedPlugin`, and a `sync` block on `modelRouter`, plus `SyncApp` and `RealtimeApp` on the server. Optional `sync.adminBroadcast: true` also emits `sync:delta` to `{collection}|admin`. Admin clients subscribe with `mode: "window"` (`createSyncDb({windowCollections})`) so they do not snapshot or reconcile that collection; they call `hydrateWindow` (REST membership ids + `GET /sync/entities`). Pass the same `accessControl` into `SyncApp` that `AdminApp` uses so `{collection}|admin` requires `admin:access`. `AdminApp.register` also installs that model's admin list/read/`queryFilter` on the admin window so hydrate and live `|admin` deltas match `/admin` REST. Without RBAC, `user.admin` is enough for the shell gate. Live deltas on that stream apply only to ids already in the window; Refresh is a REST re-query plus `hydrateWindow`.
|
|
42
42
|
6. **Collections too large to sync whole** — List them in `createSyncDb({queryCollections})`. They skip snapshot bootstrap; screens load server-filtered *query windows* (`useWindowQuery`) from the `modelRouter` list endpoint. A window stores only membership (ordered ids); rows live once in the shared entity table. Overlapping windows therefore share rows — one delta updates both views — while each view lists only its own query's members, plus cached rows that newly match its `where` inside the loaded range. Live deltas still arrive on the user's normal streams; windows on screen refetch on reconnect to cover deltas missed offline.
|
|
43
43
|
|
|
44
|
+
## First load and reconnects
|
|
45
|
+
|
|
46
|
+
A fresh device has to download every row the user can see before the UI is complete. The
|
|
47
|
+
client asks for all of the user's streams in one batched request per round, writes each
|
|
48
|
+
round while the next one downloads, and writes rows in small chunks so the screen stays
|
|
49
|
+
responsive. Reconnects use the same path: when nothing changed, catching up costs two
|
|
50
|
+
requests however many collections the app syncs. Collections too large to download whole
|
|
51
|
+
belong in `queryCollections` instead.
|
|
52
|
+
|
|
44
53
|
## Tradeoffs
|
|
45
54
|
|
|
46
55
|
**Gains:** instant UI, offline writes with a durable outbox, simpler screen code, and one write path on the server (sync and REST share `modelRouter`).
|
|
@@ -47,6 +47,7 @@ Problem-oriented, practical steps. Use these when you know what you want to do.
|
|
|
47
47
|
- [Compose charts in a dashboard grid](charts-and-dashboards.md) — `LineChart` / `BarChart` / `AreaChart` / `DonutChart` inside `Card`s in `DashboardGrid` (ops-dashboard scorecards and spans: [dashboard-chart-parity](../implementationPlans/dashboard-chart-parity.md))
|
|
48
48
|
- [Compare rendered chart snapshots](compare-chart-rendered-snapshots.md) — PNG goldens for the chart visual gallery, not JSON snapshots
|
|
49
49
|
- [Migrate from @terreno/rtk to @terreno/syncdb](migrate-rtk-to-syncdb.md) — Move data sync to the local-first layer
|
|
50
|
+
- [Use syncdb correctly](use-syncdb-correctly.md) — Serial snapshot startup, outbox-halting 403s, phantom creates, conflicts, and other library behavior apps must handle
|
|
50
51
|
- [Generate a REST CLI from OpenAPI](generate-a-rest-cli.md) — Scaffold `terreno generate rest-cli` for an app API
|
|
51
52
|
- [In-app notifications](in-app-notifications.md) — Register `NotificationsApp`, syncdb inbox, and UI bell
|
|
52
53
|
- [Add organizations](add-organizations.md) — Opt into Membership-backed tenancy, RBAC, admin, and seeded examples
|
|
@@ -260,8 +260,10 @@ Caching is off (200 credits per job). The docs Netlify target builds only the
|
|
|
260
260
|
current docs tree, disables Docusaurus minification, and pins SSG to one worker
|
|
261
261
|
thread so the build peaks near 3 GB and fits the 4 GB `medium` executor.
|
|
262
262
|
Released versions are not rebuilt: `archive-docs-version` (in `publish-release`,
|
|
263
|
-
`X.Y.0` tags only) builds each
|
|
264
|
-
|
|
263
|
+
`X.Y.0` tags only) builds each version once, uploads it to its **draft** GitHub
|
|
264
|
+
release, publishes the release, and then commits the docs snapshot to master, so
|
|
265
|
+
the `deploy-docs` run that commit starts can unpack the archive under
|
|
266
|
+
`/<version>/`. See `website/README.md`.
|
|
265
267
|
|
|
266
268
|
`terreno-gcp` uses CircleCI OIDC (`CIRCLE_OIDC_TOKEN_V2`), never a JSON service
|
|
267
269
|
account key. Set `circleci_org_id`, `circleci_project_id`, and
|
|
@@ -379,6 +381,17 @@ dependencies and runs `test:ci` when that script exists, not `test`.
|
|
|
379
381
|
tag version is already on npm, `publish-package.sh` skips it so a recut of
|
|
380
382
|
the same tag can finish the rest.
|
|
381
383
|
|
|
384
|
+
The release skill pushes the tag and creates the GitHub release with
|
|
385
|
+
`gh release create --draft`. The repository has immutable releases on: a
|
|
386
|
+
published release cannot take new assets (GitHub answers `422`). So CI publishes
|
|
387
|
+
the draft last with `scripts/ci/github-release.sh publish`.
|
|
388
|
+
`archive-docs-version` does it right after the upload on `X.Y.0` tags.
|
|
389
|
+
`publish-github-release` does it after `publish-release` on every other tag.
|
|
390
|
+
Both wait up to 10 minutes for the draft and are safe to rerun: a rerun skips
|
|
391
|
+
an upload that already landed on a published release and a snapshot already on
|
|
392
|
+
master. A
|
|
393
|
+
release still in draft after the pipeline means one of them failed.
|
|
394
|
+
|
|
382
395
|
Only stable tags (`57.3.0`) run `deploy-demo`. It builds the demo from source,
|
|
383
396
|
so it runs alongside `publish-release` instead of waiting for it. Use
|
|
384
397
|
`{"run-demo-deploy":true}` on `master` if a prerelease must also refresh the
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Update dependencies
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Routine version upgrades run through the `update-dependencies` skill (`/update-dependencies`). Dependabot opens **security PRs only**. The skill keeps one rolling PR; later runs push to it and rewrite its ledger instead of opening new PRs.
|
|
4
4
|
|
|
5
5
|
```bash
|
|
6
6
|
REPOSITORY="$(gh repo view --json nameWithOwner --jq .nameWithOwner)"
|
|
@@ -19,7 +19,7 @@ gh pr list --state open --head chore/update-dependencies \
|
|
|
19
19
|
bun outdated
|
|
20
20
|
```
|
|
21
21
|
|
|
22
|
-
Prove each bump with an **exercise test** that imports that package. Keep Expo fingerprints for `example-frontend` and `demo` unchanged. Skip `isFingerprintSkip` names until a release.
|
|
22
|
+
Prove each bump with an **exercise test** that imports that package. Verify the whole batch locally, then push **once**. Keep Expo fingerprints for `example-frontend` and `demo` unchanged. Skip `isFingerprintSkip` names until a release.
|
|
23
23
|
|
|
24
24
|
Why: [Dependency management](../explanation/dependency-management.md). Native Expo/React Native bumps wait for `upgrading-expo`.
|
|
25
25
|
|
|
@@ -27,9 +27,10 @@ Why: [Dependency management](../explanation/dependency-management.md). Native Ex
|
|
|
27
27
|
|
|
28
28
|
1. Find the same-repository PR with exact head `chore/update-dependencies`, base `master`, and `isCrossRepository: false`. Marker/title matches are not trust signals; ignore forks. Merge `origin/master` without rewriting history. If no trusted PR exists, create the canonical branch on this run.
|
|
29
29
|
2. Inventory with `bun outdated`. Retry **Failed** rows from the PR body first. Skip `isFingerprintSkip` in `scripts/planning/fingerprintRisk.ts`.
|
|
30
|
-
3. For each candidate: confirm a test imports the package (add a tracer if needed), bump the catalog pin or single-use version, `bun install`, re-run that test, recompute fingerprints. Revert only that bump on failure.
|
|
31
|
-
4.
|
|
32
|
-
5.
|
|
30
|
+
3. For each candidate: confirm a test imports the package (add a tracer if needed), bump the catalog pin or single-use version, `bun install`, re-run that test, recompute fingerprints. Revert only that bump on failure. Commit each success as `Bump <name> from <old> to <new>`.
|
|
31
|
+
4. Run the batch gate on the final head: `bun run compile`, `bun run lint`, `bun run prepush`. Run by hand any infra job `prepush` lists for a package you bumped (Playwright, Maestro, admin-spa integration). See [Run tests locally](run-tests-locally.md).
|
|
32
|
+
5. Push once. Update the **same** PR body: Last run, local gate result, CI rounds, Landed, Failed, Skipped, Deferred to release. Do not `gh pr create` when that PR is already open.
|
|
33
|
+
6. Read CI on the pushed head. Check each failed CircleCI job's `messages` before its step logs. Revert a bump that caused a failure and push a second time at most; stop and report anything else.
|
|
33
34
|
|
|
34
35
|
## Guardrails
|
|
35
36
|
|
|
@@ -41,7 +42,11 @@ Why: [Dependency management](../explanation/dependency-management.md). Native Ex
|
|
|
41
42
|
| No test imports the package | Write the tracer test first. Do not bump on `test:ci` alone |
|
|
42
43
|
| Published < 7 days ago (non-CVE) | Skip; log under Skipped |
|
|
43
44
|
| Major version | May land on the rolling PR if tests + fingerprints pass; no auto-merge |
|
|
44
|
-
|
|
|
45
|
+
| Local gate red | Fix or revert before pushing. CI is not the test runner |
|
|
46
|
+
| CI job has `messages` (for example `resource-class-not-in-plan`) | Config/plan error, not a dependency failure. Log it and stop |
|
|
47
|
+
| A bump seems to need CI config changes | Never edit `.circleci/`, workflows, executors, or resource classes from this path. Stop and report |
|
|
48
|
+
| CI red twice on heads this run pushed | Stop and report; do not push a third time |
|
|
49
|
+
| Dependabot security PR is red | Do not push to `dependabot/*`. Fold the bump into the rolling PR |
|
|
45
50
|
|
|
46
51
|
## Fingerprints
|
|
47
52
|
|
|
@@ -49,6 +54,10 @@ Why: [Dependency management](../explanation/dependency-management.md). Native Ex
|
|
|
49
54
|
|
|
50
55
|
`eas.json` Bun pins feed the fingerprint. Do not bump them here.
|
|
51
56
|
|
|
52
|
-
##
|
|
57
|
+
## Who CI emails
|
|
53
58
|
|
|
54
|
-
GitHub
|
|
59
|
+
CircleCI attributes a pipeline to the GitHub user whose token pushed it and emails that user on each failed workflow. Run the scheduled agent with a bot identity when one is configured. If it runs as a person, the one-push rule caps their mail at one round per run.
|
|
60
|
+
|
|
61
|
+
## Dependabot security PRs
|
|
62
|
+
|
|
63
|
+
`.github/dependabot.yml` sets `open-pull-requests-limit: 0` on every block, so Dependabot opens no version-update PRs. Security updates still open, grouped per block, and skip ignored native/Expo names. `dependabot-auto-merge.yml` approves them and enables squash auto-merge, so a green security PR merges without this skill. A red one gets its bump folded into the rolling PR. A security fix for an ignored native package is a manual release task (`release` + `upgrading-expo`).
|
|
@@ -0,0 +1,111 @@
|
|
|
1
|
+
# How to Use syncdb Correctly
|
|
2
|
+
|
|
3
|
+
This page lists `@terreno/syncdb` behavior that a working app has to account for. Each entry says what the library does today, what breaks if you ignore it, and what to do instead. It was checked against `syncdb/src` and `api/src/sync` at 57.13.0. Several entries are library gaps rather than intended behavior. Delete an entry when its gap is fixed.
|
|
4
|
+
|
|
5
|
+
Read [Local-first data](../explanation/local-first-data.md) and [Migrate from @terreno/rtk to @terreno/syncdb](migrate-rtk-to-syncdb.md) first.
|
|
6
|
+
|
|
7
|
+
## Startup and snapshots
|
|
8
|
+
|
|
9
|
+
### Cold start costs one round trip per round, not per stream
|
|
10
|
+
|
|
11
|
+
A stream is one collection in one scope, so a user has roughly (synced collections) × (tenants they belong to) streams. Reconcile, subscribe catch-up and wipe re-bootstrap page every stream through `POST /sync/snapshot/batch`: one round trip per round, with up to 5,000 rows per response shared across the streams. Empty streams cost nothing extra. `forceResync`, and every pass against a server without the batch route, still page with one `GET /sync/snapshot` per stream, `snapshotConcurrency` streams side by side (default 6). Each stream runs one pass at a time, and a pass that purges (force resync, wipe) waits for the stream's in-flight pass to finish first.
|
|
12
|
+
|
|
13
|
+
Measured with a fake HTTP channel at 150 ms latency, 13 collections and 2 workspaces (26 streams): 4.1 s to the first full reconcile when paging one stream at a time, 0.95 s at the default concurrency. In a browser with 100 collections and 10k rows, batching cut a reconnect with no changes from 101 requests to 2.
|
|
14
|
+
|
|
15
|
+
| Do | Why |
|
|
16
|
+
| --- | --- |
|
|
17
|
+
| Sync only the collections the client reads. | Each collection adds one stream per tenant to cold start. |
|
|
18
|
+
| Gate each screen on its **own** stream's bootstrap (see [No loading flag on synced reads](#no-loading-flag-on-synced-reads)), not on "sync finished". | The stream a screen needs may land after others. |
|
|
19
|
+
| Don't call `forceResync()` on a hot path. | It purges and re-pages every stream. |
|
|
20
|
+
| Raise `snapshotConcurrency` only if the backend can take it. | Without the batch route, each in-flight stream is one concurrent snapshot query on the server. |
|
|
21
|
+
|
|
22
|
+
The startup reconcile and each collection's `sync:subscribed` catch-up both page the same streams. A catch-up queues behind a pass that started before the room join, because that pass may have missed a write. So a cold start still pages each stream twice, one pass after the other.
|
|
23
|
+
|
|
24
|
+
### No loading flag on synced reads
|
|
25
|
+
|
|
26
|
+
`useQuery` returns a plain array. `useListQuery` returns `{data}`. The `isPending` on `useEntity`/`useReadQuery` means an outbox write is pending, not that data is loading. An empty array before bootstrap looks the same as an empty collection, so a screen that renders its empty state from `[]` flashes "nothing here" on every cold start.
|
|
27
|
+
|
|
28
|
+
The per-stream `bootstrapped` mark is the right signal, but `isStreamBootstrapped` is not exported. Read the cell directly:
|
|
29
|
+
|
|
30
|
+
```typescript
|
|
31
|
+
const isBootstrapped = (stream: string): boolean =>
|
|
32
|
+
syncDb.store.raw.getCell("_cursors", stream, "bootstrapped") === true;
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
Don't use the cursor (`useSyncStatus().streams`) for this. A live delta can move the cursor to the head before the snapshot below it has been paged. `useWindowQuery` (query collections only) does expose `isLoading`.
|
|
36
|
+
|
|
37
|
+
A stream key is `<collection>|<scopeType>:<value>`, for example `documents|tenant:<workspaceId>`. A user with several tenants has one stream per tenant. Treat a collection as loaded only once every known stream for it is bootstrapped (`syncDb.store.getKnownStreams()`).
|
|
38
|
+
|
|
39
|
+
## Writes
|
|
40
|
+
|
|
41
|
+
### Mutations return an id, not an outcome
|
|
42
|
+
|
|
43
|
+
`create`, `update` and `remove` from `useMutate` (and from the generated `useCreate*`/`useUpdate*`/`useDelete*` hooks) return `{mutationId, id}` synchronously. There is no promise for the server ack. The client also drops the ack's `warning`. A `warning` means a server post-hook threw after the write committed.
|
|
44
|
+
|
|
45
|
+
If a screen needs to know the outcome (close a modal only once the create landed, toast a refusal), wrap the transport and HTTP channel and record ack and nack outcomes by `mutationId`. Then await that `mutationId`. Don't poll the store.
|
|
46
|
+
|
|
47
|
+
### A 403 from a write halts the whole outbox
|
|
48
|
+
|
|
49
|
+
The server maps 403/405 to an `unauthorized` nack and 400/404/422 to `validation`. They behave very differently:
|
|
50
|
+
|
|
51
|
+
| Nack | Client behavior |
|
|
52
|
+
| --- | --- |
|
|
53
|
+
| `unauthorized` | The mutation is re-queued and the **whole drain stops** (auth-pause). On re-auth the same `mutationId` replays, reads the same stored nack, and pauses again. It never settles. |
|
|
54
|
+
| `validation` | The mutation is marked `failed` and the drain continues. That entity's later writes stay blocked until `retryFailed({entityId})`. |
|
|
55
|
+
| `conflict` | That entity's later writes stay blocked (or every write is blocked, with `haltQueueOnConflict: true`) until `resolveConflict`. |
|
|
56
|
+
| `error` | Retried up to the retry budget, then handled like `validation`. |
|
|
57
|
+
|
|
58
|
+
So a router hook that refuses a synced write for a business reason must throw **422**, not 403. That includes `preCreate`/`preUpdate` membership checks. Keep 403 for real auth failures.
|
|
59
|
+
|
|
60
|
+
### Don't send a stale tenant field on create
|
|
61
|
+
|
|
62
|
+
For a tenant-scoped collection, `enforceWriteScope` checks a client-sent scope field (for example `workspaceId`) against the user's memberships **before** `preCreate` runs. A value the user does not belong to is a 403, which halts the outbox (see above). If the field is missing or null, the check is skipped and `preCreate` can fill it in.
|
|
63
|
+
|
|
64
|
+
Either omit the tenant field and let `preCreate` set it, or send it only from the currently synced row of the parent tenant, never from a persisted id that may be stale.
|
|
65
|
+
|
|
66
|
+
### The local row never gets server-computed fields from the create
|
|
67
|
+
|
|
68
|
+
After an ack, the client stamps the row with the ack's seq and keeps the optimistic data. The server's create delta has the same seq, so the client skips it. Whatever `preCreate`, schema defaults or hooks set on the server does not reach the local row until that row changes again or is repaired. If the delta arrives before the ack, the row is marked for repair, but repair only runs on the next `reconcile()` (reconnect, seq jump, or every 5 minutes).
|
|
69
|
+
|
|
70
|
+
Do one of these:
|
|
71
|
+
|
|
72
|
+
- Send every field the UI reads in the create payload, matching what the server would compute (defaults, `order`, ownership fields).
|
|
73
|
+
- After the ack, patch the local row yourself.
|
|
74
|
+
|
|
75
|
+
The local row's `data` is exactly what you passed. Include `_id` in the payload if anything reads `row._id`. Server-written rows carry `_id` and `id`.
|
|
76
|
+
|
|
77
|
+
### A refused create leaves a phantom row
|
|
78
|
+
|
|
79
|
+
When a create is nacked as `validation` (or runs out of `error` retries), the repair fetch finds no server row and only clears the repair mark. The optimistic row stays in the store with `stream: ""` and shows up in queries. Only `forceResync` purges such rows.
|
|
80
|
+
|
|
81
|
+
On a refused create, tombstone the local row yourself and call `retryFailed({entityId})` so later writes to that id are released.
|
|
82
|
+
|
|
83
|
+
### Conflicts are never resolved for you
|
|
84
|
+
|
|
85
|
+
A `conflict` nack blocks every later write to that row, and any queued write whose args mention its id, until someone calls `client.resolveConflict({mutationId, strategy: "keepMine" | "useServer"})` (or `useConflicts().resolve`). Local edits keep applying optimistically, so the UI looks fine while nothing reaches the server.
|
|
86
|
+
|
|
87
|
+
If the app has no conflict UI, run a watcher that resolves each conflict as it is recorded. Use `keepMine` for fields the user types, and `useServer` for everything else.
|
|
88
|
+
|
|
89
|
+
## Server side
|
|
90
|
+
|
|
91
|
+
### Write synced models one document at a time, by `_id`
|
|
92
|
+
|
|
93
|
+
On a synced model, `updateMany`, `deleteMany`, `deleteOne`, `findOneAndDelete` and `bulkWrite` throw. Single-document writes by `_id` without upsert are allowed: `save()`, `updateOne`, `findOneAndUpdate`, `findByIdAndUpdate`, `replaceOne`. Delete by setting `deleted: true`, because a hard delete emits no delta.
|
|
94
|
+
|
|
95
|
+
In tests, clean up with `Model.collection.deleteMany(...)`, which bypasses Mongoose.
|
|
96
|
+
|
|
97
|
+
### Deltas don't apply `queryFilter`
|
|
98
|
+
|
|
99
|
+
Snapshots and `/sync/entities` apply the router's `queryFilter`. Live deltas check only `permissions.read` (or RBAC) per recipient. A router that restricts visibility only through `queryFilter` leaks rows over the socket. Every synced router's `read` permission must enforce the same restriction itself.
|
|
100
|
+
|
|
101
|
+
### Membership grants arrive late
|
|
102
|
+
|
|
103
|
+
A user's new tenant is picked up by the socket's session revalidation sweep (`sessionRevalidationIntervalMs`, 60 s by default) or a resubscribe. The client finds the new stream on its next `reconcile()` (`reconcileIntervalMs`, 5 minutes by default, plus reconnects). Until then the new tenant's rows don't appear.
|
|
104
|
+
|
|
105
|
+
When the product grants membership, push a content-free event to the user's sockets and have the client reconnect syncdb. Reconnecting resubscribes and reconciles. Grants written another way (admin, scripts) still wait for the timers.
|
|
106
|
+
|
|
107
|
+
## Auth
|
|
108
|
+
|
|
109
|
+
### Logging out does not wipe local data
|
|
110
|
+
|
|
111
|
+
The client's own `onAuthChange` handling treats a logout as a pause and **never wipes**. A different user logging in does wipe. To clear one user's data on sign-out, set `wipeOnSignOut: true` and call `client.signOut()` explicitly when the app's user clears.
|
|
@@ -70,7 +70,7 @@ Expo autolinking walks the app's own dependencies; a nested copy leaves the nati
|
|
|
70
70
|
│ sync:mutate ─────────────────────────────►│ │
|
|
71
71
|
│ sync:ack / sync:nack ◄────────────────────┘ │
|
|
72
72
|
│ sync:delta ◄──────────────────────────────────────────────┘
|
|
73
|
-
│
|
|
73
|
+
│ POST /sync/snapshot/batch (bootstrap + catch-up, HTTP)
|
|
74
74
|
└── POST /sync/mutate (fallback while the socket is down)
|
|
75
75
|
```
|
|
76
76
|
|
|
@@ -185,6 +185,7 @@ be passed to `betterAuthAdapter` directly.
|
|
|
185
185
|
| `wipeOnSignOut` | `boolean` | `false` | `signOut()` also wipes local data |
|
|
186
186
|
| `batchSize` | `number` | `50` (server caps at 100) | Max mutations per batched drain send |
|
|
187
187
|
| `haltQueueOnConflict` | `boolean` | `false` | `true` = conflict halts entire drain; `false` = per-entity blocking |
|
|
188
|
+
| `snapshotConcurrency` | `number` | `6` | Streams paged at once with `GET /sync/snapshot`: by `forceResync`, and by every other pass against a server without `POST /sync/snapshot/batch`. Each stream runs one pass at a time. `1` pages one stream at a time |
|
|
188
189
|
| `onDecryptFailure` | `() => void` | wipe + re-bootstrap (with `console.warn`) | Override web decrypt-failure recovery |
|
|
189
190
|
| `tombstoneRetentionMs` | `number` | `7776000000` (90 days); `0` disables | Client tombstone compaction after successful reconcile |
|
|
190
191
|
| `startAuthRetryAttempts` | `number` | `3`; `1` disables | `start()` attempts to resolve `getUserId()` |
|
|
@@ -201,7 +202,7 @@ be passed to `betterAuthAdapter` directly.
|
|
|
201
202
|
| `goOffline()` | Simulated outage: disconnect transport, pause replay/reconcile/timer; local mutations keep queueing. |
|
|
202
203
|
| `goOnline()` | End simulated outage; reconnect triggers reconcile + outbox replay. |
|
|
203
204
|
| `mutate({collection, operation, id?, data?})` | Optimistic local write + durable outbox enqueue + fire-and-forget replay. Returns `{mutationId, id}`. |
|
|
204
|
-
| `reconcile()` | HTTP snapshot catch-up for every known stream; runs tombstone compaction on success. Also runs automatically on (re)connect, on a rate-limited seq-jump hint, and on the periodic timer;
|
|
205
|
+
| `reconcile()` | HTTP snapshot catch-up for every known stream, batched into `POST /sync/snapshot/batch` rounds (see [Snapshot bootstrap](#snapshot-bootstrap)); runs tombstone compaction on success. Also runs automatically on (re)connect, on a rate-limited seq-jump hint, and on the periodic timer; `sync:subscribed` confirmations additionally page just the streams they name, and confirmations arriving while that catch-up request is in flight are paged together in the next one. |
|
|
205
206
|
| `hydrateWindow({collection, ids, restRows?})` | Admin window upsert: REST rows (optional) land immediately; every requested id is also fetched from `GET /sync/entities` before this resolves so seq/deleted metadata is canonical for immediate update/delete. A `{collection}|admin` delta that lands while the fetch is in flight wins, so hydration never rewinds a row to an older seq. Unknown ids are ignored. |
|
|
206
207
|
| `queryWindows` | Server query windows for `queryCollections`: `fetchWindow({query, nextPage?})`, `getWindow(query)`, `retain(query)` → release, `refetchRetained()`, `pruneUnretained()`, `subscribe(cb)`, `keyFor(query)`. See [Query windows](#query-windows). |
|
|
207
208
|
| `forceResync()` | Purge every known stream locally and re-bootstrap from cursor 0 (outbox/conflicts untouched). Returns `{ok, reason?, streams, purged, repaired}`. After discovery and each stream bootstrap it waits for in-flight `start`/`stop`/auth-change work, then abandons with `reason: "superseded"` when that work switched users or generations. |
|
|
@@ -587,11 +588,36 @@ sync: {
|
|
|
587
588
|
|
|
588
589
|
## Sync protocol
|
|
589
590
|
|
|
591
|
+
### Snapshot bootstrap
|
|
592
|
+
|
|
593
|
+
The client pages every stream it belongs to through `POST /sync/snapshot/batch`, not one
|
|
594
|
+
`GET /sync/snapshot` per stream. With 100 collections, a reconnect that finds nothing new
|
|
595
|
+
is 2 requests (`GET /sync/streams` plus one batch) instead of 101.
|
|
596
|
+
|
|
597
|
+
- **Rounds:** each round asks for every stream that still has pages. Streams that reached
|
|
598
|
+
their head drop out, so later rounds give the remaining streams a larger share of the
|
|
599
|
+
5,000-entity budget.
|
|
600
|
+
- **Pipelining:** the next round is sent as soon as a response arrives, because its
|
|
601
|
+
cursors come from the response. The client writes the current round while the next
|
|
602
|
+
one downloads.
|
|
603
|
+
- **Chunked apply:** each page is written in store transactions of 500 entities, with a
|
|
604
|
+
yield to the event loop between them, so a large page never freezes the UI in one block.
|
|
605
|
+
A stream's cursor advances in the transaction holding its last entity, so a page cut
|
|
606
|
+
short by a crash is fetched again and re-applied as a no-op.
|
|
607
|
+
- **Older servers:** a 404 without an APIError body (the route does not exist) switches
|
|
608
|
+
the client to per-stream `GET /sync/snapshot` paging, `snapshotConcurrency` streams at
|
|
609
|
+
a time, for the rest of the session. An APIError 404, such as an unknown collection,
|
|
610
|
+
fails that pass and keeps batching on.
|
|
611
|
+
- **One pass per stream:** a batch pass owns each of its streams the way a per-stream
|
|
612
|
+
pass does, so a purge (`forceResync`, wipe) waits for it, and a pass that is superseded
|
|
613
|
+
by a sign-out or user switch stops between chunks without writing the rest.
|
|
614
|
+
|
|
590
615
|
### HTTP routes (`SyncApp`, all authenticated)
|
|
591
616
|
|
|
592
617
|
| Endpoint | Purpose |
|
|
593
618
|
|----------|---------|
|
|
594
|
-
| `GET /sync/snapshot?collection=&stream=&cursor=&limit=` | Bootstrap + catch-up
|
|
619
|
+
| `GET /sync/snapshot?collection=&stream=&cursor=&limit=` | Bootstrap + catch-up for one stream (`limit` max 100) |
|
|
620
|
+
| `POST /sync/snapshot/batch` | Bootstrap + catch-up for many streams in one round trip. Body `{streams: [{stream, cursor?, legacyCursor?}], limit?}`, at most 500 streams, no repeats. `limit` (default 5,000, max 10,000) is the entity budget for the whole response, split evenly across the streams. Returns `{pages}`, one `GET /sync/snapshot`-shaped page per stream in request order. Any stream the caller may not read fails the whole request |
|
|
595
621
|
| `GET /sync/streams` | Current stream membership for the user |
|
|
596
622
|
| `GET /sync/entities` | Point lookup for entity repair and admin window hydrate. Admin-window callers (`admin:access` with RBAC, else `user.admin`) on `adminBroadcast` collections receive requested ids across product streams. When AdminApp registered a scope, list/read/`queryFilter` still apply; unknown or out-of-scope ids are omitted |
|
|
597
623
|
| `POST /sync/mutate` | Single mutation (HTTP fallback). Optional `mutationMode: "adminWindow"` validates `adminBroadcast`, admin-window access, and a registered AdminApp write scope, then runs the shared sync executor with **AdminApp** pre/post hooks (not product `modelRouter` hooks), plus the same permission (including org membership), stripping, and audit semantics as `/admin` REST. HTTP mutate binds `X-Organization-Id` into org context |
|
|
@@ -635,6 +661,11 @@ const syncDb = createSyncDb({
|
|
|
635
661
|
});
|
|
636
662
|
```
|
|
637
663
|
|
|
664
|
+
Saves are debounced (500 ms). The store is serialized once per write, not once per
|
|
665
|
+
transaction: autosave compares TinyBase's content hashes (O(1)) to skip saves the last
|
|
666
|
+
write already covered, so a bootstrap of many pages costs a few writes instead of one
|
|
667
|
+
full-store serialization per page.
|
|
668
|
+
|
|
638
669
|
Undecryptable persisted data triggers wipe + re-bootstrap by default (`onDecryptFailure` to override). Storage read errors surface `persistence: "error"` without overwriting the blob.
|
|
639
670
|
|
|
640
671
|
**Native:** plaintext SQLite in the OS sandbox (by design).
|
package/package.json
CHANGED
|
@@ -11,8 +11,8 @@
|
|
|
11
11
|
"@modelcontextprotocol/node": "^2.0.0",
|
|
12
12
|
"@modelcontextprotocol/server": "^2.0.0",
|
|
13
13
|
"@sentry/bun": "^10.39.0",
|
|
14
|
-
"@terreno/api": "57.
|
|
15
|
-
"@terreno/blocks": "57.
|
|
14
|
+
"@terreno/api": "57.15.0",
|
|
15
|
+
"@terreno/blocks": "57.15.0",
|
|
16
16
|
"express": "^5.2.1",
|
|
17
17
|
"luxon": "^3.7.2",
|
|
18
18
|
"minisearch": "^7.2.0",
|
|
@@ -105,5 +105,5 @@
|
|
|
105
105
|
"test:coverage": "bun run ../scripts/check-coverage.ts"
|
|
106
106
|
},
|
|
107
107
|
"type": "module",
|
|
108
|
-
"version": "57.
|
|
108
|
+
"version": "57.15.0"
|
|
109
109
|
}
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# Upgrading to 57.14.0
|
|
2
|
+
|
|
3
|
+
**Action required:** No
|
|
4
|
+
**Affected packages:** @terreno/syncdb
|
|
5
|
+
|
|
6
|
+
## Summary
|
|
7
|
+
|
|
8
|
+
`@terreno/syncdb` pages snapshot streams in parallel (up to 6 at once by default) instead of one at a time, so cold start, reconcile, subscribe catch-up, `forceResync`, and wipe re-bootstrap finish faster when an app has many collections or scopes. Each stream now runs one snapshot pass at a time, which stops a concurrent pass from marking a purged stream bootstrapped over rows it never paged. No code change is required.
|
|
9
|
+
|
|
10
|
+
## Breaking changes
|
|
11
|
+
|
|
12
|
+
None
|
|
13
|
+
|
|
14
|
+
## Deprecations
|
|
15
|
+
|
|
16
|
+
None
|
|
17
|
+
|
|
18
|
+
## New capabilities
|
|
19
|
+
|
|
20
|
+
### `snapshotConcurrency` on `createSyncDb`
|
|
21
|
+
|
|
22
|
+
The client now sends up to `snapshotConcurrency` snapshot requests at once (default `DEFAULT_SNAPSHOT_CONCURRENCY`, 6). If your backend cannot take the extra parallel load at cold start, lower it. `1` restores the old one-stream-at-a-time behavior.
|
|
23
|
+
|
|
24
|
+
```typescript
|
|
25
|
+
// Before
|
|
26
|
+
export const syncDb = createSyncDb({authProvider, baseUrl, collections, name: "my-app"});
|
|
27
|
+
|
|
28
|
+
// After (only if you need to limit parallel snapshot requests)
|
|
29
|
+
export const syncDb = createSyncDb({
|
|
30
|
+
authProvider,
|
|
31
|
+
baseUrl,
|
|
32
|
+
collections,
|
|
33
|
+
name: "my-app",
|
|
34
|
+
snapshotConcurrency: 2,
|
|
35
|
+
});
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
A new how-to, `docs/how-to/use-syncdb-correctly.md`, lists library behavior apps have to handle.
|
|
39
|
+
|
|
40
|
+
## Verification
|
|
41
|
+
|
|
42
|
+
- `bun run compile` and `bun run lint` at the app root
|
|
43
|
+
- Log in on a fresh device or after `wipeLocalData`; synced screens load, and the backend logs show several `/sync/snapshot` requests in flight together
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
# Upgrading to 57.15.0
|
|
2
|
+
|
|
3
|
+
**Action required:** No
|
|
4
|
+
**Affected packages:** @terreno/api, @terreno/syncdb
|
|
5
|
+
|
|
6
|
+
## Summary
|
|
7
|
+
|
|
8
|
+
SyncDB pages every stream through one `POST /sync/snapshot/batch` request per round instead of
|
|
9
|
+
one `GET /sync/snapshot` per stream. With many synced collections, cold start is faster and a
|
|
10
|
+
reconnect with no changes costs 2 requests (`GET /sync/streams` plus one batch) whatever the
|
|
11
|
+
collection count. The web IndexedDB persister now serializes the store once per write instead of
|
|
12
|
+
once per transaction, so bootstrap no longer leaves a long save backlog.
|
|
13
|
+
|
|
14
|
+
## Breaking changes
|
|
15
|
+
|
|
16
|
+
None
|
|
17
|
+
|
|
18
|
+
## Deprecations
|
|
19
|
+
|
|
20
|
+
None
|
|
21
|
+
|
|
22
|
+
## New capabilities
|
|
23
|
+
|
|
24
|
+
### `POST /sync/snapshot/batch`
|
|
25
|
+
|
|
26
|
+
`SyncApp` mounts the route automatically. The body is
|
|
27
|
+
`{streams: [{stream, cursor?, legacyCursor?}], limit?}`, with at most 500 streams.
|
|
28
|
+
`limit` is the row budget for the whole response (default 5,000, max 10,000). The response is
|
|
29
|
+
`{pages}`, one `GET /sync/snapshot`-shaped page per stream. `GET /sync/snapshot` is unchanged.
|
|
30
|
+
|
|
31
|
+
Upgrade `@terreno/api` and `@terreno/syncdb` together for the speedup. A 57.15.0 client talking
|
|
32
|
+
to an older server sees a 404 from the batch route and falls back to per-stream paging
|
|
33
|
+
(`snapshotConcurrency` streams at a time). An older client keeps using `GET /sync/snapshot`.
|
|
34
|
+
|
|
35
|
+
### `snapshotConcurrency` scope
|
|
36
|
+
|
|
37
|
+
`createSyncDb({snapshotConcurrency})` now applies only to per-stream paging: `forceResync()` and
|
|
38
|
+
the fallback against servers without the batch route.
|
|
39
|
+
|
|
40
|
+
## Verification
|
|
41
|
+
|
|
42
|
+
- `bun run compile` and `bun run lint` at the app root
|
|
43
|
+
- After deploying the backend, open the web app with dev tools and confirm a reconnect sends one
|
|
44
|
+
`POST /sync/snapshot/batch` instead of one `GET /sync/snapshot` per collection
|
|
@@ -159,13 +159,13 @@ Ensures preview environments don't accumulate and waste resources.
|
|
|
159
159
|
|
|
160
160
|
#### Dependabot Auto-merge
|
|
161
161
|
**Trigger:** Dependabot pull request opened
|
|
162
|
-
**Purpose:** Automatically merges
|
|
162
|
+
**Purpose:** Automatically merges Dependabot security updates after CI passes. Dependabot opens no version-update PRs; routine upgrades run through the `update-dependencies` skill.
|
|
163
163
|
|
|
164
164
|
**Requirements for auto-merge:**
|
|
165
165
|
- All CI checks pass
|
|
166
166
|
- Dependabot PR (created by Dependabot)
|
|
167
|
+
- Not a semver-major bump (for a grouped PR, no bump in the group is major); majors wait for a human review
|
|
167
168
|
- No conflicts
|
|
168
|
-
- 7-day cooldown for security (see Dependency Management)
|
|
169
169
|
|
|
170
170
|
**Learn more:** [Dependency Management](dependency-management.md)
|
|
171
171
|
|
|
@@ -208,7 +208,7 @@ Ensures preview environments don't accumulate and waste resources.
|
|
|
208
208
|
### Continuous Sync
|
|
209
209
|
- Documentation syncs on every master push
|
|
210
210
|
- Rules sync on every master push
|
|
211
|
-
- Dependencies update daily (Dependabot
|
|
211
|
+
- Dependencies update daily (`update-dependencies` skill); Dependabot opens security fixes only
|
|
212
212
|
- Quality improvements run daily (JSDoc, tests)
|
|
213
213
|
|
|
214
214
|
## Working with AI Workflows
|
|
@@ -280,8 +280,8 @@ The docs-audit workflow flags a missing README or reference page and obvious int
|
|
|
280
280
|
|
|
281
281
|
### Dependabot PR doesn't auto-merge
|
|
282
282
|
- Verify all CI checks pass (green checkmarks)
|
|
283
|
-
- Check 7-day cooldown hasn't been violated
|
|
284
283
|
- Ensure no merge conflicts exist
|
|
284
|
+
- If the security fix needs code changes, fold it into the rolling `chore/update-dependencies` PR instead of pushing to the Dependabot branch
|
|
285
285
|
|
|
286
286
|
### AI workflow creates poor quality PR
|
|
287
287
|
- Review the workflow definition in `.github/workflows/*.md`
|
|
@@ -290,7 +290,7 @@ The docs-audit workflow flags a missing README or reference page and obvious int
|
|
|
290
290
|
|
|
291
291
|
## Related Documentation
|
|
292
292
|
|
|
293
|
-
- [Dependency Management](dependency-management.md) — Dependabot configuration
|
|
293
|
+
- [Dependency Management](dependency-management.md) — Dependabot security-only configuration, auto-merge, and the update skill
|
|
294
294
|
- [Deploy to GCP](../how-to/deploy-to-gcp.md) — Manual deployment guide
|
|
295
295
|
- [GCP Hosting Architecture](gcp-hosting-architecture.md) — Static site hosting design
|
|
296
296
|
- [Environment Variables](../reference/environment-variables.md) — Configuration reference
|
|
@@ -1,16 +1,16 @@
|
|
|
1
1
|
# Dependency Management
|
|
2
2
|
|
|
3
|
-
Terreno pins shared versions in the Bun catalog, lands JavaScript updates
|
|
4
|
-
an agent skill that proves each bump with a test,
|
|
5
|
-
fingerprints frozen until a release.
|
|
3
|
+
Terreno pins shared versions in the Bun catalog, lands routine JavaScript updates
|
|
4
|
+
through an agent skill that proves each bump with a test, limits Dependabot to
|
|
5
|
+
security fixes, and keeps Expo native fingerprints frozen until a release.
|
|
6
6
|
|
|
7
7
|
## Layers
|
|
8
8
|
|
|
9
9
|
| Layer | Job |
|
|
10
10
|
| --- | --- |
|
|
11
11
|
| **Bun catalog** | One version for every dependency shared by two or more workspace packages (`catalog:` in manifests) |
|
|
12
|
-
| **`update-dependencies` skill** | Daily rolling PR on `chore/update-dependencies`: cooldown, exercise test, fingerprint freeze, ledger of landed/failed |
|
|
13
|
-
| **Dependabot** |
|
|
12
|
+
| **`update-dependencies` skill** | The only path for routine version upgrades. Daily rolling PR on `chore/update-dependencies`: cooldown, exercise test, local gate, one push per run, fingerprint freeze, ledger of landed/failed |
|
|
13
|
+
| **Dependabot** | Security updates only for bun + GitHub Actions. Native/Expo catalog names are ignored so auto-merge cannot change the fingerprint |
|
|
14
14
|
| **Release + `upgrading-expo`** | Expo SDK, React Native, and other native-module bumps |
|
|
15
15
|
|
|
16
16
|
Operator steps: [Update dependencies](../how-to/update-dependencies.md). Catalog rule: workspace `package.json` files use `"catalog:"` for shared pins.
|
|
@@ -41,16 +41,18 @@ Change a shared version in the catalog once. Do not pin a second copy in a works
|
|
|
41
41
|
|
|
42
42
|
## Agent updates vs Dependabot
|
|
43
43
|
|
|
44
|
-
Dependabot
|
|
44
|
+
Dependabot version updates group many packages and merge when CI is green. That misses Terreno constraints, so Dependabot now opens security PRs only:
|
|
45
45
|
|
|
46
46
|
1. **Exercise.** CI can pass without any test importing the bumped module. The skill requires a named test file that imports the package and that passed **after** the bump. If none exists, write a tracer-bullet test first.
|
|
47
47
|
2. **Fingerprint.** Expo `runtimeVersion.policy: "fingerprint"` ties OTA runtime and EAS dev builds to native hashes of `example-frontend` and `demo`. Native module bumps belong in a release, not a weekday dependency PR.
|
|
48
48
|
|
|
49
|
+
3. **CI is not the test runner.** A red grouped Dependabot PR invites an agent to push fix after fix to the Dependabot branch. Each push starts a full CircleCI pipeline that is attributed to, and emails, whoever owns the pushing token. The skill runs the CI mirror (`bun run prepush`) locally, pushes once per run, never edits CI config to make a bump pass, and stops after two red CI rounds.
|
|
50
|
+
|
|
49
51
|
`scripts/planning/fingerprintRisk.ts` (`isFingerprintSkip`) is the skip matcher. After every allowed bump the skill still recomputes hashes; a changed hash is reverted even if the matcher allowed the name.
|
|
50
52
|
|
|
51
53
|
## Cooldown
|
|
52
54
|
|
|
53
|
-
Non-security npm versions wait **7 days** after publish before a bump (`npm view <name> time`). Most malicious publishes are yanked inside a day.
|
|
55
|
+
Non-security npm versions wait **7 days** after publish before a bump (`npm view <name> time`). Most malicious publishes are yanked inside a day. CVE/advisory bumps skip the wait and still need an exercise test and a fingerprint freeze. Dependabot security updates are advisory bumps, so they have no cooldown.
|
|
54
56
|
|
|
55
57
|
## Fingerprint freeze
|
|
56
58
|
|
|
@@ -67,16 +69,16 @@ JavaScript packages, GitHub Actions, and a few Expo JS-only names may update. Me
|
|
|
67
69
|
|
|
68
70
|
## Dependabot configuration
|
|
69
71
|
|
|
70
|
-
`.github/dependabot.yml` still
|
|
72
|
+
`.github/dependabot.yml` sets `open-pull-requests-limit: 0` on every block. That turns off version-update PRs; security updates ignore the limit and still honor `ignore`. Each block's group uses `applies-to: security-updates`, so fixes arrive grouped:
|
|
71
73
|
|
|
72
|
-
| Group | Ecosystem |
|
|
73
|
-
| --- | --- |
|
|
74
|
-
|
|
|
75
|
-
|
|
|
76
|
-
|
|
|
77
|
-
|
|
|
74
|
+
| Group | Ecosystem |
|
|
75
|
+
| --- | --- |
|
|
76
|
+
| `github-actions` | `github-actions` at `/` |
|
|
77
|
+
| `root` | `bun` at `/` (root catalog and root tools) |
|
|
78
|
+
| `backend` | `bun` under api, ai, backends, … |
|
|
79
|
+
| `frontend` | `bun` under ui, rtk, demo, apps, … |
|
|
78
80
|
|
|
79
|
-
Frontend ignores catalog-managed names so those
|
|
81
|
+
Frontend ignores catalog-managed names so those fixes land once, in the root catalog. Root ignores fingerprint-skip names, so a security fix for a native/Expo package never opens a PR; handle it by hand in a release. `dependabot-auto-merge.yml` approves patch and minor Dependabot PRs and enables squash auto-merge, so those security PRs merge when required checks pass. A semver-major security PR (or a group containing one) gets no approval or auto-merge and waits for a human. When one fails because the fix needs code changes, fold the bump into the rolling PR ([Update dependencies](../how-to/update-dependencies.md)); never push to the `dependabot/*` branch.
|
|
80
82
|
|
|
81
83
|
## Daily rolling PR
|
|
82
84
|
|
|
@@ -91,10 +93,12 @@ Each package is still proven alone (exercise test + fingerprint) before it stays
|
|
|
91
93
|
- One rolling PR; many proven commits on that branch.
|
|
92
94
|
- Majors: exercise test + fingerprint freeze; no auto-merge.
|
|
93
95
|
- Security: skip cooldown; do not skip tests or fingerprints.
|
|
96
|
+
- Verify locally, push once; never change CI config to make a bump pass.
|
|
94
97
|
- New feature dependencies belong in that feature's slice, not this maintenance path.
|
|
95
98
|
|
|
96
99
|
## Related
|
|
97
100
|
|
|
98
101
|
- [Update dependencies](../how-to/update-dependencies.md)
|
|
102
|
+
- [Run tests locally](../how-to/run-tests-locally.md)
|
|
99
103
|
- [Install agent skills](../how-to/install-agent-skills.md)
|
|
100
104
|
- [Bun catalogs](https://bun.sh/docs/install/catalogs)
|
|
@@ -41,6 +41,15 @@ Terreno 57.
|
|
|
41
41
|
5. **Backend sync registration** — Every synced collection needs `syncPlugin`, `isDeletedPlugin`, and a `sync` block on `modelRouter`, plus `SyncApp` and `RealtimeApp` on the server. Optional `sync.adminBroadcast: true` also emits `sync:delta` to `{collection}|admin`. Admin clients subscribe with `mode: "window"` (`createSyncDb({windowCollections})`) so they do not snapshot or reconcile that collection; they call `hydrateWindow` (REST membership ids + `GET /sync/entities`). Pass the same `accessControl` into `SyncApp` that `AdminApp` uses so `{collection}|admin` requires `admin:access`. `AdminApp.register` also installs that model's admin list/read/`queryFilter` on the admin window so hydrate and live `|admin` deltas match `/admin` REST. Without RBAC, `user.admin` is enough for the shell gate. Live deltas on that stream apply only to ids already in the window; Refresh is a REST re-query plus `hydrateWindow`.
|
|
42
42
|
6. **Collections too large to sync whole** — List them in `createSyncDb({queryCollections})`. They skip snapshot bootstrap; screens load server-filtered *query windows* (`useWindowQuery`) from the `modelRouter` list endpoint. A window stores only membership (ordered ids); rows live once in the shared entity table. Overlapping windows therefore share rows — one delta updates both views — while each view lists only its own query's members, plus cached rows that newly match its `where` inside the loaded range. Live deltas still arrive on the user's normal streams; windows on screen refetch on reconnect to cover deltas missed offline.
|
|
43
43
|
|
|
44
|
+
## First load and reconnects
|
|
45
|
+
|
|
46
|
+
A fresh device has to download every row the user can see before the UI is complete. The
|
|
47
|
+
client asks for all of the user's streams in one batched request per round, writes each
|
|
48
|
+
round while the next one downloads, and writes rows in small chunks so the screen stays
|
|
49
|
+
responsive. Reconnects use the same path: when nothing changed, catching up costs two
|
|
50
|
+
requests however many collections the app syncs. Collections too large to download whole
|
|
51
|
+
belong in `queryCollections` instead.
|
|
52
|
+
|
|
44
53
|
## Tradeoffs
|
|
45
54
|
|
|
46
55
|
**Gains:** instant UI, offline writes with a durable outbox, simpler screen code, and one write path on the server (sync and REST share `modelRouter`).
|
|
@@ -47,6 +47,7 @@ Problem-oriented, practical steps. Use these when you know what you want to do.
|
|
|
47
47
|
- [Compose charts in a dashboard grid](charts-and-dashboards.md) — `LineChart` / `BarChart` / `AreaChart` / `DonutChart` inside `Card`s in `DashboardGrid` (ops-dashboard scorecards and spans: [dashboard-chart-parity](../implementationPlans/dashboard-chart-parity.md))
|
|
48
48
|
- [Compare rendered chart snapshots](compare-chart-rendered-snapshots.md) — PNG goldens for the chart visual gallery, not JSON snapshots
|
|
49
49
|
- [Migrate from @terreno/rtk to @terreno/syncdb](migrate-rtk-to-syncdb.md) — Move data sync to the local-first layer
|
|
50
|
+
- [Use syncdb correctly](use-syncdb-correctly.md) — Serial snapshot startup, outbox-halting 403s, phantom creates, conflicts, and other library behavior apps must handle
|
|
50
51
|
- [Generate a REST CLI from OpenAPI](generate-a-rest-cli.md) — Scaffold `terreno generate rest-cli` for an app API
|
|
51
52
|
- [In-app notifications](in-app-notifications.md) — Register `NotificationsApp`, syncdb inbox, and UI bell
|
|
52
53
|
- [Add organizations](add-organizations.md) — Opt into Membership-backed tenancy, RBAC, admin, and seeded examples
|
|
@@ -260,8 +260,10 @@ Caching is off (200 credits per job). The docs Netlify target builds only the
|
|
|
260
260
|
current docs tree, disables Docusaurus minification, and pins SSG to one worker
|
|
261
261
|
thread so the build peaks near 3 GB and fits the 4 GB `medium` executor.
|
|
262
262
|
Released versions are not rebuilt: `archive-docs-version` (in `publish-release`,
|
|
263
|
-
`X.Y.0` tags only) builds each
|
|
264
|
-
|
|
263
|
+
`X.Y.0` tags only) builds each version once, uploads it to its **draft** GitHub
|
|
264
|
+
release, publishes the release, and then commits the docs snapshot to master, so
|
|
265
|
+
the `deploy-docs` run that commit starts can unpack the archive under
|
|
266
|
+
`/<version>/`. See `website/README.md`.
|
|
265
267
|
|
|
266
268
|
`terreno-gcp` uses CircleCI OIDC (`CIRCLE_OIDC_TOKEN_V2`), never a JSON service
|
|
267
269
|
account key. Set `circleci_org_id`, `circleci_project_id`, and
|
|
@@ -379,6 +381,17 @@ dependencies and runs `test:ci` when that script exists, not `test`.
|
|
|
379
381
|
tag version is already on npm, `publish-package.sh` skips it so a recut of
|
|
380
382
|
the same tag can finish the rest.
|
|
381
383
|
|
|
384
|
+
The release skill pushes the tag and creates the GitHub release with
|
|
385
|
+
`gh release create --draft`. The repository has immutable releases on: a
|
|
386
|
+
published release cannot take new assets (GitHub answers `422`). So CI publishes
|
|
387
|
+
the draft last with `scripts/ci/github-release.sh publish`.
|
|
388
|
+
`archive-docs-version` does it right after the upload on `X.Y.0` tags.
|
|
389
|
+
`publish-github-release` does it after `publish-release` on every other tag.
|
|
390
|
+
Both wait up to 10 minutes for the draft and are safe to rerun: a rerun skips
|
|
391
|
+
an upload that already landed on a published release and a snapshot already on
|
|
392
|
+
master. A
|
|
393
|
+
release still in draft after the pipeline means one of them failed.
|
|
394
|
+
|
|
382
395
|
Only stable tags (`57.3.0`) run `deploy-demo`. It builds the demo from source,
|
|
383
396
|
so it runs alongside `publish-release` instead of waiting for it. Use
|
|
384
397
|
`{"run-demo-deploy":true}` on `master` if a prerelease must also refresh the
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Update dependencies
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Routine version upgrades run through the `update-dependencies` skill (`/update-dependencies`). Dependabot opens **security PRs only**. The skill keeps one rolling PR; later runs push to it and rewrite its ledger instead of opening new PRs.
|
|
4
4
|
|
|
5
5
|
```bash
|
|
6
6
|
REPOSITORY="$(gh repo view --json nameWithOwner --jq .nameWithOwner)"
|
|
@@ -19,7 +19,7 @@ gh pr list --state open --head chore/update-dependencies \
|
|
|
19
19
|
bun outdated
|
|
20
20
|
```
|
|
21
21
|
|
|
22
|
-
Prove each bump with an **exercise test** that imports that package. Keep Expo fingerprints for `example-frontend` and `demo` unchanged. Skip `isFingerprintSkip` names until a release.
|
|
22
|
+
Prove each bump with an **exercise test** that imports that package. Verify the whole batch locally, then push **once**. Keep Expo fingerprints for `example-frontend` and `demo` unchanged. Skip `isFingerprintSkip` names until a release.
|
|
23
23
|
|
|
24
24
|
Why: [Dependency management](../explanation/dependency-management.md). Native Expo/React Native bumps wait for `upgrading-expo`.
|
|
25
25
|
|
|
@@ -27,9 +27,10 @@ Why: [Dependency management](../explanation/dependency-management.md). Native Ex
|
|
|
27
27
|
|
|
28
28
|
1. Find the same-repository PR with exact head `chore/update-dependencies`, base `master`, and `isCrossRepository: false`. Marker/title matches are not trust signals; ignore forks. Merge `origin/master` without rewriting history. If no trusted PR exists, create the canonical branch on this run.
|
|
29
29
|
2. Inventory with `bun outdated`. Retry **Failed** rows from the PR body first. Skip `isFingerprintSkip` in `scripts/planning/fingerprintRisk.ts`.
|
|
30
|
-
3. For each candidate: confirm a test imports the package (add a tracer if needed), bump the catalog pin or single-use version, `bun install`, re-run that test, recompute fingerprints. Revert only that bump on failure.
|
|
31
|
-
4.
|
|
32
|
-
5.
|
|
30
|
+
3. For each candidate: confirm a test imports the package (add a tracer if needed), bump the catalog pin or single-use version, `bun install`, re-run that test, recompute fingerprints. Revert only that bump on failure. Commit each success as `Bump <name> from <old> to <new>`.
|
|
31
|
+
4. Run the batch gate on the final head: `bun run compile`, `bun run lint`, `bun run prepush`. Run by hand any infra job `prepush` lists for a package you bumped (Playwright, Maestro, admin-spa integration). See [Run tests locally](run-tests-locally.md).
|
|
32
|
+
5. Push once. Update the **same** PR body: Last run, local gate result, CI rounds, Landed, Failed, Skipped, Deferred to release. Do not `gh pr create` when that PR is already open.
|
|
33
|
+
6. Read CI on the pushed head. Check each failed CircleCI job's `messages` before its step logs. Revert a bump that caused a failure and push a second time at most; stop and report anything else.
|
|
33
34
|
|
|
34
35
|
## Guardrails
|
|
35
36
|
|
|
@@ -41,7 +42,11 @@ Why: [Dependency management](../explanation/dependency-management.md). Native Ex
|
|
|
41
42
|
| No test imports the package | Write the tracer test first. Do not bump on `test:ci` alone |
|
|
42
43
|
| Published < 7 days ago (non-CVE) | Skip; log under Skipped |
|
|
43
44
|
| Major version | May land on the rolling PR if tests + fingerprints pass; no auto-merge |
|
|
44
|
-
|
|
|
45
|
+
| Local gate red | Fix or revert before pushing. CI is not the test runner |
|
|
46
|
+
| CI job has `messages` (for example `resource-class-not-in-plan`) | Config/plan error, not a dependency failure. Log it and stop |
|
|
47
|
+
| A bump seems to need CI config changes | Never edit `.circleci/`, workflows, executors, or resource classes from this path. Stop and report |
|
|
48
|
+
| CI red twice on heads this run pushed | Stop and report; do not push a third time |
|
|
49
|
+
| Dependabot security PR is red | Do not push to `dependabot/*`. Fold the bump into the rolling PR |
|
|
45
50
|
|
|
46
51
|
## Fingerprints
|
|
47
52
|
|
|
@@ -49,6 +54,10 @@ Why: [Dependency management](../explanation/dependency-management.md). Native Ex
|
|
|
49
54
|
|
|
50
55
|
`eas.json` Bun pins feed the fingerprint. Do not bump them here.
|
|
51
56
|
|
|
52
|
-
##
|
|
57
|
+
## Who CI emails
|
|
53
58
|
|
|
54
|
-
GitHub
|
|
59
|
+
CircleCI attributes a pipeline to the GitHub user whose token pushed it and emails that user on each failed workflow. Run the scheduled agent with a bot identity when one is configured. If it runs as a person, the one-push rule caps their mail at one round per run.
|
|
60
|
+
|
|
61
|
+
## Dependabot security PRs
|
|
62
|
+
|
|
63
|
+
`.github/dependabot.yml` sets `open-pull-requests-limit: 0` on every block, so Dependabot opens no version-update PRs. Security updates still open, grouped per block, and skip ignored native/Expo names. `dependabot-auto-merge.yml` approves them and enables squash auto-merge, so a green security PR merges without this skill. A red one gets its bump folded into the rolling PR. A security fix for an ignored native package is a manual release task (`release` + `upgrading-expo`).
|
|
@@ -0,0 +1,111 @@
|
|
|
1
|
+
# How to Use syncdb Correctly
|
|
2
|
+
|
|
3
|
+
This page lists `@terreno/syncdb` behavior that a working app has to account for. Each entry says what the library does today, what breaks if you ignore it, and what to do instead. It was checked against `syncdb/src` and `api/src/sync` at 57.13.0. Several entries are library gaps rather than intended behavior. Delete an entry when its gap is fixed.
|
|
4
|
+
|
|
5
|
+
Read [Local-first data](../explanation/local-first-data.md) and [Migrate from @terreno/rtk to @terreno/syncdb](migrate-rtk-to-syncdb.md) first.
|
|
6
|
+
|
|
7
|
+
## Startup and snapshots
|
|
8
|
+
|
|
9
|
+
### Cold start costs one round trip per round, not per stream
|
|
10
|
+
|
|
11
|
+
A stream is one collection in one scope, so a user has roughly (synced collections) × (tenants they belong to) streams. Reconcile, subscribe catch-up and wipe re-bootstrap page every stream through `POST /sync/snapshot/batch`: one round trip per round, with up to 5,000 rows per response shared across the streams. Empty streams cost nothing extra. `forceResync`, and every pass against a server without the batch route, still page with one `GET /sync/snapshot` per stream, `snapshotConcurrency` streams side by side (default 6). Each stream runs one pass at a time, and a pass that purges (force resync, wipe) waits for the stream's in-flight pass to finish first.
|
|
12
|
+
|
|
13
|
+
Measured with a fake HTTP channel at 150 ms latency, 13 collections and 2 workspaces (26 streams): 4.1 s to the first full reconcile when paging one stream at a time, 0.95 s at the default concurrency. In a browser with 100 collections and 10k rows, batching cut a reconnect with no changes from 101 requests to 2.
|
|
14
|
+
|
|
15
|
+
| Do | Why |
|
|
16
|
+
| --- | --- |
|
|
17
|
+
| Sync only the collections the client reads. | Each collection adds one stream per tenant to cold start. |
|
|
18
|
+
| Gate each screen on its **own** stream's bootstrap (see [No loading flag on synced reads](#no-loading-flag-on-synced-reads)), not on "sync finished". | The stream a screen needs may land after others. |
|
|
19
|
+
| Don't call `forceResync()` on a hot path. | It purges and re-pages every stream. |
|
|
20
|
+
| Raise `snapshotConcurrency` only if the backend can take it. | Without the batch route, each in-flight stream is one concurrent snapshot query on the server. |
|
|
21
|
+
|
|
22
|
+
The startup reconcile and each collection's `sync:subscribed` catch-up both page the same streams. A catch-up queues behind a pass that started before the room join, because that pass may have missed a write. So a cold start still pages each stream twice, one pass after the other.
|
|
23
|
+
|
|
24
|
+
### No loading flag on synced reads
|
|
25
|
+
|
|
26
|
+
`useQuery` returns a plain array. `useListQuery` returns `{data}`. The `isPending` on `useEntity`/`useReadQuery` means an outbox write is pending, not that data is loading. An empty array before bootstrap looks the same as an empty collection, so a screen that renders its empty state from `[]` flashes "nothing here" on every cold start.
|
|
27
|
+
|
|
28
|
+
The per-stream `bootstrapped` mark is the right signal, but `isStreamBootstrapped` is not exported. Read the cell directly:
|
|
29
|
+
|
|
30
|
+
```typescript
|
|
31
|
+
const isBootstrapped = (stream: string): boolean =>
|
|
32
|
+
syncDb.store.raw.getCell("_cursors", stream, "bootstrapped") === true;
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
Don't use the cursor (`useSyncStatus().streams`) for this. A live delta can move the cursor to the head before the snapshot below it has been paged. `useWindowQuery` (query collections only) does expose `isLoading`.
|
|
36
|
+
|
|
37
|
+
A stream key is `<collection>|<scopeType>:<value>`, for example `documents|tenant:<workspaceId>`. A user with several tenants has one stream per tenant. Treat a collection as loaded only once every known stream for it is bootstrapped (`syncDb.store.getKnownStreams()`).
|
|
38
|
+
|
|
39
|
+
## Writes
|
|
40
|
+
|
|
41
|
+
### Mutations return an id, not an outcome
|
|
42
|
+
|
|
43
|
+
`create`, `update` and `remove` from `useMutate` (and from the generated `useCreate*`/`useUpdate*`/`useDelete*` hooks) return `{mutationId, id}` synchronously. There is no promise for the server ack. The client also drops the ack's `warning`. A `warning` means a server post-hook threw after the write committed.
|
|
44
|
+
|
|
45
|
+
If a screen needs to know the outcome (close a modal only once the create landed, toast a refusal), wrap the transport and HTTP channel and record ack and nack outcomes by `mutationId`. Then await that `mutationId`. Don't poll the store.
|
|
46
|
+
|
|
47
|
+
### A 403 from a write halts the whole outbox
|
|
48
|
+
|
|
49
|
+
The server maps 403/405 to an `unauthorized` nack and 400/404/422 to `validation`. They behave very differently:
|
|
50
|
+
|
|
51
|
+
| Nack | Client behavior |
|
|
52
|
+
| --- | --- |
|
|
53
|
+
| `unauthorized` | The mutation is re-queued and the **whole drain stops** (auth-pause). On re-auth the same `mutationId` replays, reads the same stored nack, and pauses again. It never settles. |
|
|
54
|
+
| `validation` | The mutation is marked `failed` and the drain continues. That entity's later writes stay blocked until `retryFailed({entityId})`. |
|
|
55
|
+
| `conflict` | That entity's later writes stay blocked (or every write is blocked, with `haltQueueOnConflict: true`) until `resolveConflict`. |
|
|
56
|
+
| `error` | Retried up to the retry budget, then handled like `validation`. |
|
|
57
|
+
|
|
58
|
+
So a router hook that refuses a synced write for a business reason must throw **422**, not 403. That includes `preCreate`/`preUpdate` membership checks. Keep 403 for real auth failures.
|
|
59
|
+
|
|
60
|
+
### Don't send a stale tenant field on create
|
|
61
|
+
|
|
62
|
+
For a tenant-scoped collection, `enforceWriteScope` checks a client-sent scope field (for example `workspaceId`) against the user's memberships **before** `preCreate` runs. A value the user does not belong to is a 403, which halts the outbox (see above). If the field is missing or null, the check is skipped and `preCreate` can fill it in.
|
|
63
|
+
|
|
64
|
+
Either omit the tenant field and let `preCreate` set it, or send it only from the currently synced row of the parent tenant, never from a persisted id that may be stale.
|
|
65
|
+
|
|
66
|
+
### The local row never gets server-computed fields from the create
|
|
67
|
+
|
|
68
|
+
After an ack, the client stamps the row with the ack's seq and keeps the optimistic data. The server's create delta has the same seq, so the client skips it. Whatever `preCreate`, schema defaults or hooks set on the server does not reach the local row until that row changes again or is repaired. If the delta arrives before the ack, the row is marked for repair, but repair only runs on the next `reconcile()` (reconnect, seq jump, or every 5 minutes).
|
|
69
|
+
|
|
70
|
+
Do one of these:
|
|
71
|
+
|
|
72
|
+
- Send every field the UI reads in the create payload, matching what the server would compute (defaults, `order`, ownership fields).
|
|
73
|
+
- After the ack, patch the local row yourself.
|
|
74
|
+
|
|
75
|
+
The local row's `data` is exactly what you passed. Include `_id` in the payload if anything reads `row._id`. Server-written rows carry `_id` and `id`.
|
|
76
|
+
|
|
77
|
+
### A refused create leaves a phantom row
|
|
78
|
+
|
|
79
|
+
When a create is nacked as `validation` (or runs out of `error` retries), the repair fetch finds no server row and only clears the repair mark. The optimistic row stays in the store with `stream: ""` and shows up in queries. Only `forceResync` purges such rows.
|
|
80
|
+
|
|
81
|
+
On a refused create, tombstone the local row yourself and call `retryFailed({entityId})` so later writes to that id are released.
|
|
82
|
+
|
|
83
|
+
### Conflicts are never resolved for you
|
|
84
|
+
|
|
85
|
+
A `conflict` nack blocks every later write to that row, and any queued write whose args mention its id, until someone calls `client.resolveConflict({mutationId, strategy: "keepMine" | "useServer"})` (or `useConflicts().resolve`). Local edits keep applying optimistically, so the UI looks fine while nothing reaches the server.
|
|
86
|
+
|
|
87
|
+
If the app has no conflict UI, run a watcher that resolves each conflict as it is recorded. Use `keepMine` for fields the user types, and `useServer` for everything else.
|
|
88
|
+
|
|
89
|
+
## Server side
|
|
90
|
+
|
|
91
|
+
### Write synced models one document at a time, by `_id`
|
|
92
|
+
|
|
93
|
+
On a synced model, `updateMany`, `deleteMany`, `deleteOne`, `findOneAndDelete` and `bulkWrite` throw. Single-document writes by `_id` without upsert are allowed: `save()`, `updateOne`, `findOneAndUpdate`, `findByIdAndUpdate`, `replaceOne`. Delete by setting `deleted: true`, because a hard delete emits no delta.
|
|
94
|
+
|
|
95
|
+
In tests, clean up with `Model.collection.deleteMany(...)`, which bypasses Mongoose.
|
|
96
|
+
|
|
97
|
+
### Deltas don't apply `queryFilter`
|
|
98
|
+
|
|
99
|
+
Snapshots and `/sync/entities` apply the router's `queryFilter`. Live deltas check only `permissions.read` (or RBAC) per recipient. A router that restricts visibility only through `queryFilter` leaks rows over the socket. Every synced router's `read` permission must enforce the same restriction itself.
|
|
100
|
+
|
|
101
|
+
### Membership grants arrive late
|
|
102
|
+
|
|
103
|
+
A user's new tenant is picked up by the socket's session revalidation sweep (`sessionRevalidationIntervalMs`, 60 s by default) or a resubscribe. The client finds the new stream on its next `reconcile()` (`reconcileIntervalMs`, 5 minutes by default, plus reconnects). Until then the new tenant's rows don't appear.
|
|
104
|
+
|
|
105
|
+
When the product grants membership, push a content-free event to the user's sockets and have the client reconnect syncdb. Reconnecting resubscribes and reconciles. Grants written another way (admin, scripts) still wait for the timers.
|
|
106
|
+
|
|
107
|
+
## Auth
|
|
108
|
+
|
|
109
|
+
### Logging out does not wipe local data
|
|
110
|
+
|
|
111
|
+
The client's own `onAuthChange` handling treats a logout as a pause and **never wipes**. A different user logging in does wipe. To clear one user's data on sign-out, set `wipeOnSignOut: true` and call `client.signOut()` explicitly when the app's user clears.
|
|
@@ -70,7 +70,7 @@ Expo autolinking walks the app's own dependencies; a nested copy leaves the nati
|
|
|
70
70
|
│ sync:mutate ─────────────────────────────►│ │
|
|
71
71
|
│ sync:ack / sync:nack ◄────────────────────┘ │
|
|
72
72
|
│ sync:delta ◄──────────────────────────────────────────────┘
|
|
73
|
-
│
|
|
73
|
+
│ POST /sync/snapshot/batch (bootstrap + catch-up, HTTP)
|
|
74
74
|
└── POST /sync/mutate (fallback while the socket is down)
|
|
75
75
|
```
|
|
76
76
|
|
|
@@ -185,6 +185,7 @@ be passed to `betterAuthAdapter` directly.
|
|
|
185
185
|
| `wipeOnSignOut` | `boolean` | `false` | `signOut()` also wipes local data |
|
|
186
186
|
| `batchSize` | `number` | `50` (server caps at 100) | Max mutations per batched drain send |
|
|
187
187
|
| `haltQueueOnConflict` | `boolean` | `false` | `true` = conflict halts entire drain; `false` = per-entity blocking |
|
|
188
|
+
| `snapshotConcurrency` | `number` | `6` | Streams paged at once with `GET /sync/snapshot`: by `forceResync`, and by every other pass against a server without `POST /sync/snapshot/batch`. Each stream runs one pass at a time. `1` pages one stream at a time |
|
|
188
189
|
| `onDecryptFailure` | `() => void` | wipe + re-bootstrap (with `console.warn`) | Override web decrypt-failure recovery |
|
|
189
190
|
| `tombstoneRetentionMs` | `number` | `7776000000` (90 days); `0` disables | Client tombstone compaction after successful reconcile |
|
|
190
191
|
| `startAuthRetryAttempts` | `number` | `3`; `1` disables | `start()` attempts to resolve `getUserId()` |
|
|
@@ -201,7 +202,7 @@ be passed to `betterAuthAdapter` directly.
|
|
|
201
202
|
| `goOffline()` | Simulated outage: disconnect transport, pause replay/reconcile/timer; local mutations keep queueing. |
|
|
202
203
|
| `goOnline()` | End simulated outage; reconnect triggers reconcile + outbox replay. |
|
|
203
204
|
| `mutate({collection, operation, id?, data?})` | Optimistic local write + durable outbox enqueue + fire-and-forget replay. Returns `{mutationId, id}`. |
|
|
204
|
-
| `reconcile()` | HTTP snapshot catch-up for every known stream; runs tombstone compaction on success. Also runs automatically on (re)connect, on a rate-limited seq-jump hint, and on the periodic timer;
|
|
205
|
+
| `reconcile()` | HTTP snapshot catch-up for every known stream, batched into `POST /sync/snapshot/batch` rounds (see [Snapshot bootstrap](#snapshot-bootstrap)); runs tombstone compaction on success. Also runs automatically on (re)connect, on a rate-limited seq-jump hint, and on the periodic timer; `sync:subscribed` confirmations additionally page just the streams they name, and confirmations arriving while that catch-up request is in flight are paged together in the next one. |
|
|
205
206
|
| `hydrateWindow({collection, ids, restRows?})` | Admin window upsert: REST rows (optional) land immediately; every requested id is also fetched from `GET /sync/entities` before this resolves so seq/deleted metadata is canonical for immediate update/delete. A `{collection}|admin` delta that lands while the fetch is in flight wins, so hydration never rewinds a row to an older seq. Unknown ids are ignored. |
|
|
206
207
|
| `queryWindows` | Server query windows for `queryCollections`: `fetchWindow({query, nextPage?})`, `getWindow(query)`, `retain(query)` → release, `refetchRetained()`, `pruneUnretained()`, `subscribe(cb)`, `keyFor(query)`. See [Query windows](#query-windows). |
|
|
207
208
|
| `forceResync()` | Purge every known stream locally and re-bootstrap from cursor 0 (outbox/conflicts untouched). Returns `{ok, reason?, streams, purged, repaired}`. After discovery and each stream bootstrap it waits for in-flight `start`/`stop`/auth-change work, then abandons with `reason: "superseded"` when that work switched users or generations. |
|
|
@@ -587,11 +588,36 @@ sync: {
|
|
|
587
588
|
|
|
588
589
|
## Sync protocol
|
|
589
590
|
|
|
591
|
+
### Snapshot bootstrap
|
|
592
|
+
|
|
593
|
+
The client pages every stream it belongs to through `POST /sync/snapshot/batch`, not one
|
|
594
|
+
`GET /sync/snapshot` per stream. With 100 collections, a reconnect that finds nothing new
|
|
595
|
+
is 2 requests (`GET /sync/streams` plus one batch) instead of 101.
|
|
596
|
+
|
|
597
|
+
- **Rounds:** each round asks for every stream that still has pages. Streams that reached
|
|
598
|
+
their head drop out, so later rounds give the remaining streams a larger share of the
|
|
599
|
+
5,000-entity budget.
|
|
600
|
+
- **Pipelining:** the next round is sent as soon as a response arrives, because its
|
|
601
|
+
cursors come from the response. The client writes the current round while the next
|
|
602
|
+
one downloads.
|
|
603
|
+
- **Chunked apply:** each page is written in store transactions of 500 entities, with a
|
|
604
|
+
yield to the event loop between them, so a large page never freezes the UI in one block.
|
|
605
|
+
A stream's cursor advances in the transaction holding its last entity, so a page cut
|
|
606
|
+
short by a crash is fetched again and re-applied as a no-op.
|
|
607
|
+
- **Older servers:** a 404 without an APIError body (the route does not exist) switches
|
|
608
|
+
the client to per-stream `GET /sync/snapshot` paging, `snapshotConcurrency` streams at
|
|
609
|
+
a time, for the rest of the session. An APIError 404, such as an unknown collection,
|
|
610
|
+
fails that pass and keeps batching on.
|
|
611
|
+
- **One pass per stream:** a batch pass owns each of its streams the way a per-stream
|
|
612
|
+
pass does, so a purge (`forceResync`, wipe) waits for it, and a pass that is superseded
|
|
613
|
+
by a sign-out or user switch stops between chunks without writing the rest.
|
|
614
|
+
|
|
590
615
|
### HTTP routes (`SyncApp`, all authenticated)
|
|
591
616
|
|
|
592
617
|
| Endpoint | Purpose |
|
|
593
618
|
|----------|---------|
|
|
594
|
-
| `GET /sync/snapshot?collection=&stream=&cursor=&limit=` | Bootstrap + catch-up
|
|
619
|
+
| `GET /sync/snapshot?collection=&stream=&cursor=&limit=` | Bootstrap + catch-up for one stream (`limit` max 100) |
|
|
620
|
+
| `POST /sync/snapshot/batch` | Bootstrap + catch-up for many streams in one round trip. Body `{streams: [{stream, cursor?, legacyCursor?}], limit?}`, at most 500 streams, no repeats. `limit` (default 5,000, max 10,000) is the entity budget for the whole response, split evenly across the streams. Returns `{pages}`, one `GET /sync/snapshot`-shaped page per stream in request order. Any stream the caller may not read fails the whole request |
|
|
595
621
|
| `GET /sync/streams` | Current stream membership for the user |
|
|
596
622
|
| `GET /sync/entities` | Point lookup for entity repair and admin window hydrate. Admin-window callers (`admin:access` with RBAC, else `user.admin`) on `adminBroadcast` collections receive requested ids across product streams. When AdminApp registered a scope, list/read/`queryFilter` still apply; unknown or out-of-scope ids are omitted |
|
|
597
623
|
| `POST /sync/mutate` | Single mutation (HTTP fallback). Optional `mutationMode: "adminWindow"` validates `adminBroadcast`, admin-window access, and a registered AdminApp write scope, then runs the shared sync executor with **AdminApp** pre/post hooks (not product `modelRouter` hooks), plus the same permission (including org membership), stripping, and audit semantics as `/admin` REST. HTTP mutate binds `X-Organization-Id` into org context |
|
|
@@ -635,6 +661,11 @@ const syncDb = createSyncDb({
|
|
|
635
661
|
});
|
|
636
662
|
```
|
|
637
663
|
|
|
664
|
+
Saves are debounced (500 ms). The store is serialized once per write, not once per
|
|
665
|
+
transaction: autosave compares TinyBase's content hashes (O(1)) to skip saves the last
|
|
666
|
+
write already covered, so a bootstrap of many pages costs a few writes instead of one
|
|
667
|
+
full-store serialization per page.
|
|
668
|
+
|
|
638
669
|
Undecryptable persisted data triggers wipe + re-bootstrap by default (`onDecryptFailure` to override). Storage read errors surface `persistence: "error"` without overwriting the blob.
|
|
639
670
|
|
|
640
671
|
**Native:** plaintext SQLite in the OS sandbox (by design).
|