@terreno/mcp 57.14.0 → 57.15.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/docs/upgrades/57.15.0.md +44 -0
- package/dist/docs/versioned/next/explanation/ai-workflows.md +5 -5
- package/dist/docs/versioned/next/explanation/dependency-management.md +19 -15
- package/dist/docs/versioned/next/explanation/local-first-data.md +9 -0
- package/dist/docs/versioned/next/how-to/circleci.md +15 -2
- package/dist/docs/versioned/next/how-to/update-dependencies.md +17 -8
- package/dist/docs/versioned/next/how-to/use-syncdb-correctly.md +4 -4
- package/dist/docs/versioned/next/reference/syncdb.md +34 -4
- package/package.json +3 -3
- package/src/docs/upgrades/57.15.0.md +44 -0
- package/src/docs/versioned/next/explanation/ai-workflows.md +5 -5
- package/src/docs/versioned/next/explanation/dependency-management.md +19 -15
- package/src/docs/versioned/next/explanation/local-first-data.md +9 -0
- package/src/docs/versioned/next/how-to/circleci.md +15 -2
- package/src/docs/versioned/next/how-to/update-dependencies.md +17 -8
- package/src/docs/versioned/next/how-to/use-syncdb-correctly.md +4 -4
- package/src/docs/versioned/next/reference/syncdb.md +34 -4
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
# Upgrading to 57.15.0
|
|
2
|
+
|
|
3
|
+
**Action required:** No
|
|
4
|
+
**Affected packages:** @terreno/api, @terreno/syncdb
|
|
5
|
+
|
|
6
|
+
## Summary
|
|
7
|
+
|
|
8
|
+
SyncDB pages every stream through one `POST /sync/snapshot/batch` request per round instead of
|
|
9
|
+
one `GET /sync/snapshot` per stream. With many synced collections, cold start is faster and a
|
|
10
|
+
reconnect with no changes costs 2 requests (`GET /sync/streams` plus one batch) whatever the
|
|
11
|
+
collection count. The web IndexedDB persister now serializes the store once per write instead of
|
|
12
|
+
once per transaction, so bootstrap no longer leaves a long save backlog.
|
|
13
|
+
|
|
14
|
+
## Breaking changes
|
|
15
|
+
|
|
16
|
+
None
|
|
17
|
+
|
|
18
|
+
## Deprecations
|
|
19
|
+
|
|
20
|
+
None
|
|
21
|
+
|
|
22
|
+
## New capabilities
|
|
23
|
+
|
|
24
|
+
### `POST /sync/snapshot/batch`
|
|
25
|
+
|
|
26
|
+
`SyncApp` mounts the route automatically. The body is
|
|
27
|
+
`{streams: [{stream, cursor?, legacyCursor?}], limit?}`, with at most 500 streams.
|
|
28
|
+
`limit` is the row budget for the whole response (default 5,000, max 10,000). The response is
|
|
29
|
+
`{pages}`, one `GET /sync/snapshot`-shaped page per stream. `GET /sync/snapshot` is unchanged.
|
|
30
|
+
|
|
31
|
+
Upgrade `@terreno/api` and `@terreno/syncdb` together for the speedup. A 57.15.0 client talking
|
|
32
|
+
to an older server sees a 404 from the batch route and falls back to per-stream paging
|
|
33
|
+
(`snapshotConcurrency` streams at a time). An older client keeps using `GET /sync/snapshot`.
|
|
34
|
+
|
|
35
|
+
### `snapshotConcurrency` scope
|
|
36
|
+
|
|
37
|
+
`createSyncDb({snapshotConcurrency})` now applies only to per-stream paging: `forceResync()` and
|
|
38
|
+
the fallback against servers without the batch route.
|
|
39
|
+
|
|
40
|
+
## Verification
|
|
41
|
+
|
|
42
|
+
- `bun run compile` and `bun run lint` at the app root
|
|
43
|
+
- After deploying the backend, open the web app with dev tools and confirm a reconnect sends one
|
|
44
|
+
`POST /sync/snapshot/batch` instead of one `GET /sync/snapshot` per collection
|
|
@@ -159,13 +159,13 @@ Ensures preview environments don't accumulate and waste resources.
|
|
|
159
159
|
|
|
160
160
|
#### Dependabot Auto-merge
|
|
161
161
|
**Trigger:** Dependabot pull request opened
|
|
162
|
-
**Purpose:** Automatically merges
|
|
162
|
+
**Purpose:** Automatically merges Dependabot security updates after CI passes. Dependabot opens no version-update PRs; routine upgrades run through the `update-dependencies` skill.
|
|
163
163
|
|
|
164
164
|
**Requirements for auto-merge:**
|
|
165
165
|
- All CI checks pass
|
|
166
166
|
- Dependabot PR (created by Dependabot)
|
|
167
|
+
- Not a semver-major bump (for a grouped PR, no bump in the group is major); majors wait for a human review
|
|
167
168
|
- No conflicts
|
|
168
|
-
- 7-day cooldown for security (see Dependency Management)
|
|
169
169
|
|
|
170
170
|
**Learn more:** [Dependency Management](dependency-management.md)
|
|
171
171
|
|
|
@@ -208,7 +208,7 @@ Ensures preview environments don't accumulate and waste resources.
|
|
|
208
208
|
### Continuous Sync
|
|
209
209
|
- Documentation syncs on every master push
|
|
210
210
|
- Rules sync on every master push
|
|
211
|
-
- Dependencies update daily (Dependabot
|
|
211
|
+
- Dependencies update daily (`update-dependencies` skill); Dependabot opens security fixes only
|
|
212
212
|
- Quality improvements run daily (JSDoc, tests)
|
|
213
213
|
|
|
214
214
|
## Working with AI Workflows
|
|
@@ -280,8 +280,8 @@ The docs-audit workflow flags a missing README or reference page and obvious int
|
|
|
280
280
|
|
|
281
281
|
### Dependabot PR doesn't auto-merge
|
|
282
282
|
- Verify all CI checks pass (green checkmarks)
|
|
283
|
-
- Check 7-day cooldown hasn't been violated
|
|
284
283
|
- Ensure no merge conflicts exist
|
|
284
|
+
- If the security fix needs code changes, fold it into the rolling `chore/update-dependencies` PR instead of pushing to the Dependabot branch
|
|
285
285
|
|
|
286
286
|
### AI workflow creates poor quality PR
|
|
287
287
|
- Review the workflow definition in `.github/workflows/*.md`
|
|
@@ -290,7 +290,7 @@ The docs-audit workflow flags a missing README or reference page and obvious int
|
|
|
290
290
|
|
|
291
291
|
## Related Documentation
|
|
292
292
|
|
|
293
|
-
- [Dependency Management](dependency-management.md) — Dependabot configuration
|
|
293
|
+
- [Dependency Management](dependency-management.md) — Dependabot security-only configuration, auto-merge, and the update skill
|
|
294
294
|
- [Deploy to GCP](../how-to/deploy-to-gcp.md) — Manual deployment guide
|
|
295
295
|
- [GCP Hosting Architecture](gcp-hosting-architecture.md) — Static site hosting design
|
|
296
296
|
- [Environment Variables](../reference/environment-variables.md) — Configuration reference
|
|
@@ -1,16 +1,16 @@
|
|
|
1
1
|
# Dependency Management
|
|
2
2
|
|
|
3
|
-
Terreno pins shared versions in the Bun catalog, lands JavaScript updates
|
|
4
|
-
an agent skill that proves each bump with a test,
|
|
5
|
-
fingerprints frozen until a release.
|
|
3
|
+
Terreno pins shared versions in the Bun catalog, lands routine JavaScript updates
|
|
4
|
+
through an agent skill that proves each bump with a test, limits Dependabot to
|
|
5
|
+
security fixes, and keeps Expo native fingerprints frozen until a release.
|
|
6
6
|
|
|
7
7
|
## Layers
|
|
8
8
|
|
|
9
9
|
| Layer | Job |
|
|
10
10
|
| --- | --- |
|
|
11
11
|
| **Bun catalog** | One version for every dependency shared by two or more workspace packages (`catalog:` in manifests) |
|
|
12
|
-
| **`update-dependencies` skill** | Daily rolling PR on `chore/update-dependencies`: cooldown, exercise test, fingerprint freeze, ledger of landed/failed |
|
|
13
|
-
| **Dependabot** |
|
|
12
|
+
| **`update-dependencies` skill** | The only path for routine version upgrades. Daily rolling PR on `chore/update-dependencies`: cooldown, exercise test, local gate, one push per run, fingerprint freeze, ledger of landed/failed |
|
|
13
|
+
| **Dependabot** | Security updates only for bun + GitHub Actions. Native/Expo catalog names are ignored so auto-merge cannot change the fingerprint |
|
|
14
14
|
| **Release + `upgrading-expo`** | Expo SDK, React Native, and other native-module bumps |
|
|
15
15
|
|
|
16
16
|
Operator steps: [Update dependencies](../how-to/update-dependencies.md). Catalog rule: workspace `package.json` files use `"catalog:"` for shared pins.
|
|
@@ -41,16 +41,18 @@ Change a shared version in the catalog once. Do not pin a second copy in a works
|
|
|
41
41
|
|
|
42
42
|
## Agent updates vs Dependabot
|
|
43
43
|
|
|
44
|
-
Dependabot
|
|
44
|
+
Dependabot version updates group many packages and merge when CI is green. That misses Terreno constraints, so Dependabot now opens security PRs only:
|
|
45
45
|
|
|
46
46
|
1. **Exercise.** CI can pass without any test importing the bumped module. The skill requires a named test file that imports the package and that passed **after** the bump. If none exists, write a tracer-bullet test first.
|
|
47
47
|
2. **Fingerprint.** Expo `runtimeVersion.policy: "fingerprint"` ties OTA runtime and EAS dev builds to native hashes of `example-frontend` and `demo`. Native module bumps belong in a release, not a weekday dependency PR.
|
|
48
48
|
|
|
49
|
+
3. **CI is not the test runner.** A red grouped Dependabot PR invites an agent to push fix after fix to the Dependabot branch. Each push starts a full CircleCI pipeline that is attributed to, and emails, whoever owns the pushing token. The skill runs the CI mirror (`bun run prepush`) locally, pushes once per run, never edits CI config to make a bump pass, and stops after two red CI rounds.
|
|
50
|
+
|
|
49
51
|
`scripts/planning/fingerprintRisk.ts` (`isFingerprintSkip`) is the skip matcher. After every allowed bump the skill still recomputes hashes; a changed hash is reverted even if the matcher allowed the name.
|
|
50
52
|
|
|
51
53
|
## Cooldown
|
|
52
54
|
|
|
53
|
-
Non-security npm versions wait **7 days** after publish before a bump (`npm view <name> time`). Most malicious publishes are yanked inside a day.
|
|
55
|
+
Non-security npm versions wait **7 days** after publish before a bump (`npm view <name> time`). Most malicious publishes are yanked inside a day. CVE/advisory bumps skip the wait and still need an exercise test and a fingerprint freeze. Dependabot security updates are advisory bumps, so they have no cooldown.
|
|
54
56
|
|
|
55
57
|
## Fingerprint freeze
|
|
56
58
|
|
|
@@ -67,16 +69,16 @@ JavaScript packages, GitHub Actions, and a few Expo JS-only names may update. Me
|
|
|
67
69
|
|
|
68
70
|
## Dependabot configuration
|
|
69
71
|
|
|
70
|
-
`.github/dependabot.yml` still
|
|
72
|
+
`.github/dependabot.yml` sets `open-pull-requests-limit: 0` on every block. That turns off version-update PRs; security updates ignore the limit and still honor `ignore`. Each block's group uses `applies-to: security-updates`, so fixes arrive grouped:
|
|
71
73
|
|
|
72
|
-
| Group | Ecosystem |
|
|
73
|
-
| --- | --- |
|
|
74
|
-
|
|
|
75
|
-
|
|
|
76
|
-
|
|
|
77
|
-
|
|
|
74
|
+
| Group | Ecosystem |
|
|
75
|
+
| --- | --- |
|
|
76
|
+
| `github-actions` | `github-actions` at `/` |
|
|
77
|
+
| `root` | `bun` at `/` (root catalog and root tools) |
|
|
78
|
+
| `backend` | `bun` under api, ai, backends, … |
|
|
79
|
+
| `frontend` | `bun` under ui, rtk, demo, apps, … |
|
|
78
80
|
|
|
79
|
-
Frontend ignores catalog-managed names so those
|
|
81
|
+
Frontend ignores catalog-managed names so those fixes land once, in the root catalog. Root ignores fingerprint-skip names, so a security fix for a native/Expo package never opens a PR; handle it by hand in a release. `dependabot-auto-merge.yml` approves patch and minor Dependabot PRs and enables squash auto-merge, so those security PRs merge when required checks pass. A semver-major security PR (or a group containing one) gets no approval or auto-merge and waits for a human. When one fails because the fix needs code changes, fold the bump into the rolling PR ([Update dependencies](../how-to/update-dependencies.md)); never push to the `dependabot/*` branch.
|
|
80
82
|
|
|
81
83
|
## Daily rolling PR
|
|
82
84
|
|
|
@@ -91,10 +93,12 @@ Each package is still proven alone (exercise test + fingerprint) before it stays
|
|
|
91
93
|
- One rolling PR; many proven commits on that branch.
|
|
92
94
|
- Majors: exercise test + fingerprint freeze; no auto-merge.
|
|
93
95
|
- Security: skip cooldown; do not skip tests or fingerprints.
|
|
96
|
+
- Verify locally, push once; never change CI config to make a bump pass.
|
|
94
97
|
- New feature dependencies belong in that feature's slice, not this maintenance path.
|
|
95
98
|
|
|
96
99
|
## Related
|
|
97
100
|
|
|
98
101
|
- [Update dependencies](../how-to/update-dependencies.md)
|
|
102
|
+
- [Run tests locally](../how-to/run-tests-locally.md)
|
|
99
103
|
- [Install agent skills](../how-to/install-agent-skills.md)
|
|
100
104
|
- [Bun catalogs](https://bun.sh/docs/install/catalogs)
|
|
@@ -41,6 +41,15 @@ Terreno 57.
|
|
|
41
41
|
5. **Backend sync registration** — Every synced collection needs `syncPlugin`, `isDeletedPlugin`, and a `sync` block on `modelRouter`, plus `SyncApp` and `RealtimeApp` on the server. Optional `sync.adminBroadcast: true` also emits `sync:delta` to `{collection}|admin`. Admin clients subscribe with `mode: "window"` (`createSyncDb({windowCollections})`) so they do not snapshot or reconcile that collection; they call `hydrateWindow` (REST membership ids + `GET /sync/entities`). Pass the same `accessControl` into `SyncApp` that `AdminApp` uses so `{collection}|admin` requires `admin:access`. `AdminApp.register` also installs that model's admin list/read/`queryFilter` on the admin window so hydrate and live `|admin` deltas match `/admin` REST. Without RBAC, `user.admin` is enough for the shell gate. Live deltas on that stream apply only to ids already in the window; Refresh is a REST re-query plus `hydrateWindow`.
|
|
42
42
|
6. **Collections too large to sync whole** — List them in `createSyncDb({queryCollections})`. They skip snapshot bootstrap; screens load server-filtered *query windows* (`useWindowQuery`) from the `modelRouter` list endpoint. A window stores only membership (ordered ids); rows live once in the shared entity table. Overlapping windows therefore share rows — one delta updates both views — while each view lists only its own query's members, plus cached rows that newly match its `where` inside the loaded range. Live deltas still arrive on the user's normal streams; windows on screen refetch on reconnect to cover deltas missed offline.
|
|
43
43
|
|
|
44
|
+
## First load and reconnects
|
|
45
|
+
|
|
46
|
+
A fresh device has to download every row the user can see before the UI is complete. The
|
|
47
|
+
client asks for all of the user's streams in one batched request per round, writes each
|
|
48
|
+
round while the next one downloads, and writes rows in small chunks so the screen stays
|
|
49
|
+
responsive. Reconnects use the same path: when nothing changed, catching up costs two
|
|
50
|
+
requests however many collections the app syncs. Collections too large to download whole
|
|
51
|
+
belong in `queryCollections` instead.
|
|
52
|
+
|
|
44
53
|
## Tradeoffs
|
|
45
54
|
|
|
46
55
|
**Gains:** instant UI, offline writes with a durable outbox, simpler screen code, and one write path on the server (sync and REST share `modelRouter`).
|
|
@@ -260,8 +260,10 @@ Caching is off (200 credits per job). The docs Netlify target builds only the
|
|
|
260
260
|
current docs tree, disables Docusaurus minification, and pins SSG to one worker
|
|
261
261
|
thread so the build peaks near 3 GB and fits the 4 GB `medium` executor.
|
|
262
262
|
Released versions are not rebuilt: `archive-docs-version` (in `publish-release`,
|
|
263
|
-
`X.Y.0` tags only) builds each
|
|
264
|
-
|
|
263
|
+
`X.Y.0` tags only) builds each version once, uploads it to its **draft** GitHub
|
|
264
|
+
release, publishes the release, and then commits the docs snapshot to master, so
|
|
265
|
+
the `deploy-docs` run that commit starts can unpack the archive under
|
|
266
|
+
`/<version>/`. See `website/README.md`.
|
|
265
267
|
|
|
266
268
|
`terreno-gcp` uses CircleCI OIDC (`CIRCLE_OIDC_TOKEN_V2`), never a JSON service
|
|
267
269
|
account key. Set `circleci_org_id`, `circleci_project_id`, and
|
|
@@ -379,6 +381,17 @@ dependencies and runs `test:ci` when that script exists, not `test`.
|
|
|
379
381
|
tag version is already on npm, `publish-package.sh` skips it so a recut of
|
|
380
382
|
the same tag can finish the rest.
|
|
381
383
|
|
|
384
|
+
The release skill pushes the tag and creates the GitHub release with
|
|
385
|
+
`gh release create --draft`. The repository has immutable releases on: a
|
|
386
|
+
published release cannot take new assets (GitHub answers `422`). So CI publishes
|
|
387
|
+
the draft last with `scripts/ci/github-release.sh publish`.
|
|
388
|
+
`archive-docs-version` does it right after the upload on `X.Y.0` tags.
|
|
389
|
+
`publish-github-release` does it after `publish-release` on every other tag.
|
|
390
|
+
Both wait up to 10 minutes for the draft and are safe to rerun: a rerun skips
|
|
391
|
+
an upload that already landed on a published release and a snapshot already on
|
|
392
|
+
master. A
|
|
393
|
+
release still in draft after the pipeline means one of them failed.
|
|
394
|
+
|
|
382
395
|
Only stable tags (`57.3.0`) run `deploy-demo`. It builds the demo from source,
|
|
383
396
|
so it runs alongside `publish-release` instead of waiting for it. Use
|
|
384
397
|
`{"run-demo-deploy":true}` on `master` if a prerelease must also refresh the
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Update dependencies
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Routine version upgrades run through the `update-dependencies` skill (`/update-dependencies`). Dependabot opens **security PRs only**. The skill keeps one rolling PR; later runs push to it and rewrite its ledger instead of opening new PRs.
|
|
4
4
|
|
|
5
5
|
```bash
|
|
6
6
|
REPOSITORY="$(gh repo view --json nameWithOwner --jq .nameWithOwner)"
|
|
@@ -19,7 +19,7 @@ gh pr list --state open --head chore/update-dependencies \
|
|
|
19
19
|
bun outdated
|
|
20
20
|
```
|
|
21
21
|
|
|
22
|
-
Prove each bump with an **exercise test** that imports that package. Keep Expo fingerprints for `example-frontend` and `demo` unchanged. Skip `isFingerprintSkip` names until a release.
|
|
22
|
+
Prove each bump with an **exercise test** that imports that package. Verify the whole batch locally, then push **once**. Keep Expo fingerprints for `example-frontend` and `demo` unchanged. Skip `isFingerprintSkip` names until a release.
|
|
23
23
|
|
|
24
24
|
Why: [Dependency management](../explanation/dependency-management.md). Native Expo/React Native bumps wait for `upgrading-expo`.
|
|
25
25
|
|
|
@@ -27,9 +27,10 @@ Why: [Dependency management](../explanation/dependency-management.md). Native Ex
|
|
|
27
27
|
|
|
28
28
|
1. Find the same-repository PR with exact head `chore/update-dependencies`, base `master`, and `isCrossRepository: false`. Marker/title matches are not trust signals; ignore forks. Merge `origin/master` without rewriting history. If no trusted PR exists, create the canonical branch on this run.
|
|
29
29
|
2. Inventory with `bun outdated`. Retry **Failed** rows from the PR body first. Skip `isFingerprintSkip` in `scripts/planning/fingerprintRisk.ts`.
|
|
30
|
-
3. For each candidate: confirm a test imports the package (add a tracer if needed), bump the catalog pin or single-use version, `bun install`, re-run that test, recompute fingerprints. Revert only that bump on failure.
|
|
31
|
-
4.
|
|
32
|
-
5.
|
|
30
|
+
3. For each candidate: confirm a test imports the package (add a tracer if needed), bump the catalog pin or single-use version, `bun install`, re-run that test, recompute fingerprints. Revert only that bump on failure. Commit each success as `Bump <name> from <old> to <new>`.
|
|
31
|
+
4. Run the batch gate on the final head: `bun run compile`, `bun run lint`, `bun run prepush`. Run by hand any infra job `prepush` lists for a package you bumped (Playwright, Maestro, admin-spa integration). See [Run tests locally](run-tests-locally.md).
|
|
32
|
+
5. Push once. Update the **same** PR body: Last run, local gate result, CI rounds, Landed, Failed, Skipped, Deferred to release. Do not `gh pr create` when that PR is already open.
|
|
33
|
+
6. Read CI on the pushed head. Check each failed CircleCI job's `messages` before its step logs. Revert a bump that caused a failure and push a second time at most; stop and report anything else.
|
|
33
34
|
|
|
34
35
|
## Guardrails
|
|
35
36
|
|
|
@@ -41,7 +42,11 @@ Why: [Dependency management](../explanation/dependency-management.md). Native Ex
|
|
|
41
42
|
| No test imports the package | Write the tracer test first. Do not bump on `test:ci` alone |
|
|
42
43
|
| Published < 7 days ago (non-CVE) | Skip; log under Skipped |
|
|
43
44
|
| Major version | May land on the rolling PR if tests + fingerprints pass; no auto-merge |
|
|
44
|
-
|
|
|
45
|
+
| Local gate red | Fix or revert before pushing. CI is not the test runner |
|
|
46
|
+
| CI job has `messages` (for example `resource-class-not-in-plan`) | Config/plan error, not a dependency failure. Log it and stop |
|
|
47
|
+
| A bump seems to need CI config changes | Never edit `.circleci/`, workflows, executors, or resource classes from this path. Stop and report |
|
|
48
|
+
| CI red twice on heads this run pushed | Stop and report; do not push a third time |
|
|
49
|
+
| Dependabot security PR is red | Do not push to `dependabot/*`. Fold the bump into the rolling PR |
|
|
45
50
|
|
|
46
51
|
## Fingerprints
|
|
47
52
|
|
|
@@ -49,6 +54,10 @@ Why: [Dependency management](../explanation/dependency-management.md). Native Ex
|
|
|
49
54
|
|
|
50
55
|
`eas.json` Bun pins feed the fingerprint. Do not bump them here.
|
|
51
56
|
|
|
52
|
-
##
|
|
57
|
+
## Who CI emails
|
|
53
58
|
|
|
54
|
-
GitHub
|
|
59
|
+
CircleCI attributes a pipeline to the GitHub user whose token pushed it and emails that user on each failed workflow. Run the scheduled agent with a bot identity when one is configured. If it runs as a person, the one-push rule caps their mail at one round per run.
|
|
60
|
+
|
|
61
|
+
## Dependabot security PRs
|
|
62
|
+
|
|
63
|
+
`.github/dependabot.yml` sets `open-pull-requests-limit: 0` on every block, so Dependabot opens no version-update PRs. Security updates still open, grouped per block, and skip ignored native/Expo names. `dependabot-auto-merge.yml` approves them and enables squash auto-merge, so a green security PR merges without this skill. A red one gets its bump folded into the rolling PR. A security fix for an ignored native package is a manual release task (`release` + `upgrading-expo`).
|
|
@@ -6,18 +6,18 @@ Read [Local-first data](../explanation/local-first-data.md) and [Migrate from @t
|
|
|
6
6
|
|
|
7
7
|
## Startup and snapshots
|
|
8
8
|
|
|
9
|
-
### Cold start costs one round trip per stream
|
|
9
|
+
### Cold start costs one round trip per round, not per stream
|
|
10
10
|
|
|
11
|
-
A stream is one collection in one scope, so a user has roughly (synced collections) × (tenants they belong to) streams.
|
|
11
|
+
A stream is one collection in one scope, so a user has roughly (synced collections) × (tenants they belong to) streams. Reconcile, subscribe catch-up and wipe re-bootstrap page every stream through `POST /sync/snapshot/batch`: one round trip per round, with up to 5,000 rows per response shared across the streams. Empty streams cost nothing extra. `forceResync`, and every pass against a server without the batch route, still page with one `GET /sync/snapshot` per stream, `snapshotConcurrency` streams side by side (default 6). Each stream runs one pass at a time, and a pass that purges (force resync, wipe) waits for the stream's in-flight pass to finish first.
|
|
12
12
|
|
|
13
|
-
Measured with a fake HTTP channel at 150 ms latency, 13 collections and 2 workspaces (26 streams): 4.1 s to the first full reconcile when paging one stream at a time, 0.95 s at the default concurrency.
|
|
13
|
+
Measured with a fake HTTP channel at 150 ms latency, 13 collections and 2 workspaces (26 streams): 4.1 s to the first full reconcile when paging one stream at a time, 0.95 s at the default concurrency. In a browser with 100 collections and 10k rows, batching cut a reconnect with no changes from 101 requests to 2.
|
|
14
14
|
|
|
15
15
|
| Do | Why |
|
|
16
16
|
| --- | --- |
|
|
17
17
|
| Sync only the collections the client reads. | Each collection adds one stream per tenant to cold start. |
|
|
18
18
|
| Gate each screen on its **own** stream's bootstrap (see [No loading flag on synced reads](#no-loading-flag-on-synced-reads)), not on "sync finished". | The stream a screen needs may land after others. |
|
|
19
19
|
| Don't call `forceResync()` on a hot path. | It purges and re-pages every stream. |
|
|
20
|
-
| Raise `snapshotConcurrency` only if the backend can take it. |
|
|
20
|
+
| Raise `snapshotConcurrency` only if the backend can take it. | Without the batch route, each in-flight stream is one concurrent snapshot query on the server. |
|
|
21
21
|
|
|
22
22
|
The startup reconcile and each collection's `sync:subscribed` catch-up both page the same streams. A catch-up queues behind a pass that started before the room join, because that pass may have missed a write. So a cold start still pages each stream twice, one pass after the other.
|
|
23
23
|
|
|
@@ -70,7 +70,7 @@ Expo autolinking walks the app's own dependencies; a nested copy leaves the nati
|
|
|
70
70
|
│ sync:mutate ─────────────────────────────►│ │
|
|
71
71
|
│ sync:ack / sync:nack ◄────────────────────┘ │
|
|
72
72
|
│ sync:delta ◄──────────────────────────────────────────────┘
|
|
73
|
-
│
|
|
73
|
+
│ POST /sync/snapshot/batch (bootstrap + catch-up, HTTP)
|
|
74
74
|
└── POST /sync/mutate (fallback while the socket is down)
|
|
75
75
|
```
|
|
76
76
|
|
|
@@ -185,7 +185,7 @@ be passed to `betterAuthAdapter` directly.
|
|
|
185
185
|
| `wipeOnSignOut` | `boolean` | `false` | `signOut()` also wipes local data |
|
|
186
186
|
| `batchSize` | `number` | `50` (server caps at 100) | Max mutations per batched drain send |
|
|
187
187
|
| `haltQueueOnConflict` | `boolean` | `false` | `true` = conflict halts entire drain; `false` = per-entity blocking |
|
|
188
|
-
| `snapshotConcurrency` | `number` | `6` | Streams
|
|
188
|
+
| `snapshotConcurrency` | `number` | `6` | Streams paged at once with `GET /sync/snapshot`: by `forceResync`, and by every other pass against a server without `POST /sync/snapshot/batch`. Each stream runs one pass at a time. `1` pages one stream at a time |
|
|
189
189
|
| `onDecryptFailure` | `() => void` | wipe + re-bootstrap (with `console.warn`) | Override web decrypt-failure recovery |
|
|
190
190
|
| `tombstoneRetentionMs` | `number` | `7776000000` (90 days); `0` disables | Client tombstone compaction after successful reconcile |
|
|
191
191
|
| `startAuthRetryAttempts` | `number` | `3`; `1` disables | `start()` attempts to resolve `getUserId()` |
|
|
@@ -202,7 +202,7 @@ be passed to `betterAuthAdapter` directly.
|
|
|
202
202
|
| `goOffline()` | Simulated outage: disconnect transport, pause replay/reconcile/timer; local mutations keep queueing. |
|
|
203
203
|
| `goOnline()` | End simulated outage; reconnect triggers reconcile + outbox replay. |
|
|
204
204
|
| `mutate({collection, operation, id?, data?})` | Optimistic local write + durable outbox enqueue + fire-and-forget replay. Returns `{mutationId, id}`. |
|
|
205
|
-
| `reconcile()` | HTTP snapshot catch-up for every known stream; runs tombstone compaction on success. Also runs automatically on (re)connect, on a rate-limited seq-jump hint, and on the periodic timer;
|
|
205
|
+
| `reconcile()` | HTTP snapshot catch-up for every known stream, batched into `POST /sync/snapshot/batch` rounds (see [Snapshot bootstrap](#snapshot-bootstrap)); runs tombstone compaction on success. Also runs automatically on (re)connect, on a rate-limited seq-jump hint, and on the periodic timer; `sync:subscribed` confirmations additionally page just the streams they name, and confirmations arriving while that catch-up request is in flight are paged together in the next one. |
|
|
206
206
|
| `hydrateWindow({collection, ids, restRows?})` | Admin window upsert: REST rows (optional) land immediately; every requested id is also fetched from `GET /sync/entities` before this resolves so seq/deleted metadata is canonical for immediate update/delete. A `{collection}|admin` delta that lands while the fetch is in flight wins, so hydration never rewinds a row to an older seq. Unknown ids are ignored. |
|
|
207
207
|
| `queryWindows` | Server query windows for `queryCollections`: `fetchWindow({query, nextPage?})`, `getWindow(query)`, `retain(query)` → release, `refetchRetained()`, `pruneUnretained()`, `subscribe(cb)`, `keyFor(query)`. See [Query windows](#query-windows). |
|
|
208
208
|
| `forceResync()` | Purge every known stream locally and re-bootstrap from cursor 0 (outbox/conflicts untouched). Returns `{ok, reason?, streams, purged, repaired}`. After discovery and each stream bootstrap it waits for in-flight `start`/`stop`/auth-change work, then abandons with `reason: "superseded"` when that work switched users or generations. |
|
|
@@ -588,11 +588,36 @@ sync: {
|
|
|
588
588
|
|
|
589
589
|
## Sync protocol
|
|
590
590
|
|
|
591
|
+
### Snapshot bootstrap
|
|
592
|
+
|
|
593
|
+
The client pages every stream it belongs to through `POST /sync/snapshot/batch`, not one
|
|
594
|
+
`GET /sync/snapshot` per stream. With 100 collections, a reconnect that finds nothing new
|
|
595
|
+
is 2 requests (`GET /sync/streams` plus one batch) instead of 101.
|
|
596
|
+
|
|
597
|
+
- **Rounds:** each round asks for every stream that still has pages. Streams that reached
|
|
598
|
+
their head drop out, so later rounds give the remaining streams a larger share of the
|
|
599
|
+
5,000-entity budget.
|
|
600
|
+
- **Pipelining:** the next round is sent as soon as a response arrives, because its
|
|
601
|
+
cursors come from the response. The client writes the current round while the next
|
|
602
|
+
one downloads.
|
|
603
|
+
- **Chunked apply:** each page is written in store transactions of 500 entities, with a
|
|
604
|
+
yield to the event loop between them, so a large page never freezes the UI in one block.
|
|
605
|
+
A stream's cursor advances in the transaction holding its last entity, so a page cut
|
|
606
|
+
short by a crash is fetched again and re-applied as a no-op.
|
|
607
|
+
- **Older servers:** a 404 without an APIError body (the route does not exist) switches
|
|
608
|
+
the client to per-stream `GET /sync/snapshot` paging, `snapshotConcurrency` streams at
|
|
609
|
+
a time, for the rest of the session. An APIError 404, such as an unknown collection,
|
|
610
|
+
fails that pass and keeps batching on.
|
|
611
|
+
- **One pass per stream:** a batch pass owns each of its streams the way a per-stream
|
|
612
|
+
pass does, so a purge (`forceResync`, wipe) waits for it, and a pass that is superseded
|
|
613
|
+
by a sign-out or user switch stops between chunks without writing the rest.
|
|
614
|
+
|
|
591
615
|
### HTTP routes (`SyncApp`, all authenticated)
|
|
592
616
|
|
|
593
617
|
| Endpoint | Purpose |
|
|
594
618
|
|----------|---------|
|
|
595
|
-
| `GET /sync/snapshot?collection=&stream=&cursor=&limit=` | Bootstrap + catch-up
|
|
619
|
+
| `GET /sync/snapshot?collection=&stream=&cursor=&limit=` | Bootstrap + catch-up for one stream (`limit` max 100) |
|
|
620
|
+
| `POST /sync/snapshot/batch` | Bootstrap + catch-up for many streams in one round trip. Body `{streams: [{stream, cursor?, legacyCursor?}], limit?}`, at most 500 streams, no repeats. `limit` (default 5,000, max 10,000) is the entity budget for the whole response, split evenly across the streams. Returns `{pages}`, one `GET /sync/snapshot`-shaped page per stream in request order. Any stream the caller may not read fails the whole request |
|
|
596
621
|
| `GET /sync/streams` | Current stream membership for the user |
|
|
597
622
|
| `GET /sync/entities` | Point lookup for entity repair and admin window hydrate. Admin-window callers (`admin:access` with RBAC, else `user.admin`) on `adminBroadcast` collections receive requested ids across product streams. When AdminApp registered a scope, list/read/`queryFilter` still apply; unknown or out-of-scope ids are omitted |
|
|
598
623
|
| `POST /sync/mutate` | Single mutation (HTTP fallback). Optional `mutationMode: "adminWindow"` validates `adminBroadcast`, admin-window access, and a registered AdminApp write scope, then runs the shared sync executor with **AdminApp** pre/post hooks (not product `modelRouter` hooks), plus the same permission (including org membership), stripping, and audit semantics as `/admin` REST. HTTP mutate binds `X-Organization-Id` into org context |
|
|
@@ -636,6 +661,11 @@ const syncDb = createSyncDb({
|
|
|
636
661
|
});
|
|
637
662
|
```
|
|
638
663
|
|
|
664
|
+
Saves are debounced (500 ms). The store is serialized once per write, not once per
|
|
665
|
+
transaction: autosave compares TinyBase's content hashes (O(1)) to skip saves the last
|
|
666
|
+
write already covered, so a bootstrap of many pages costs a few writes instead of one
|
|
667
|
+
full-store serialization per page.
|
|
668
|
+
|
|
639
669
|
Undecryptable persisted data triggers wipe + re-bootstrap by default (`onDecryptFailure` to override). Storage read errors surface `persistence: "error"` without overwriting the blob.
|
|
640
670
|
|
|
641
671
|
**Native:** plaintext SQLite in the OS sandbox (by design).
|
package/package.json
CHANGED
|
@@ -11,8 +11,8 @@
|
|
|
11
11
|
"@modelcontextprotocol/node": "^2.0.0",
|
|
12
12
|
"@modelcontextprotocol/server": "^2.0.0",
|
|
13
13
|
"@sentry/bun": "^10.39.0",
|
|
14
|
-
"@terreno/api": "57.
|
|
15
|
-
"@terreno/blocks": "57.
|
|
14
|
+
"@terreno/api": "57.15.0",
|
|
15
|
+
"@terreno/blocks": "57.15.0",
|
|
16
16
|
"express": "^5.2.1",
|
|
17
17
|
"luxon": "^3.7.2",
|
|
18
18
|
"minisearch": "^7.2.0",
|
|
@@ -105,5 +105,5 @@
|
|
|
105
105
|
"test:coverage": "bun run ../scripts/check-coverage.ts"
|
|
106
106
|
},
|
|
107
107
|
"type": "module",
|
|
108
|
-
"version": "57.
|
|
108
|
+
"version": "57.15.0"
|
|
109
109
|
}
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
# Upgrading to 57.15.0
|
|
2
|
+
|
|
3
|
+
**Action required:** No
|
|
4
|
+
**Affected packages:** @terreno/api, @terreno/syncdb
|
|
5
|
+
|
|
6
|
+
## Summary
|
|
7
|
+
|
|
8
|
+
SyncDB pages every stream through one `POST /sync/snapshot/batch` request per round instead of
|
|
9
|
+
one `GET /sync/snapshot` per stream. With many synced collections, cold start is faster and a
|
|
10
|
+
reconnect with no changes costs 2 requests (`GET /sync/streams` plus one batch) whatever the
|
|
11
|
+
collection count. The web IndexedDB persister now serializes the store once per write instead of
|
|
12
|
+
once per transaction, so bootstrap no longer leaves a long save backlog.
|
|
13
|
+
|
|
14
|
+
## Breaking changes
|
|
15
|
+
|
|
16
|
+
None
|
|
17
|
+
|
|
18
|
+
## Deprecations
|
|
19
|
+
|
|
20
|
+
None
|
|
21
|
+
|
|
22
|
+
## New capabilities
|
|
23
|
+
|
|
24
|
+
### `POST /sync/snapshot/batch`
|
|
25
|
+
|
|
26
|
+
`SyncApp` mounts the route automatically. The body is
|
|
27
|
+
`{streams: [{stream, cursor?, legacyCursor?}], limit?}`, with at most 500 streams.
|
|
28
|
+
`limit` is the row budget for the whole response (default 5,000, max 10,000). The response is
|
|
29
|
+
`{pages}`, one `GET /sync/snapshot`-shaped page per stream. `GET /sync/snapshot` is unchanged.
|
|
30
|
+
|
|
31
|
+
Upgrade `@terreno/api` and `@terreno/syncdb` together for the speedup. A 57.15.0 client talking
|
|
32
|
+
to an older server sees a 404 from the batch route and falls back to per-stream paging
|
|
33
|
+
(`snapshotConcurrency` streams at a time). An older client keeps using `GET /sync/snapshot`.
|
|
34
|
+
|
|
35
|
+
### `snapshotConcurrency` scope
|
|
36
|
+
|
|
37
|
+
`createSyncDb({snapshotConcurrency})` now applies only to per-stream paging: `forceResync()` and
|
|
38
|
+
the fallback against servers without the batch route.
|
|
39
|
+
|
|
40
|
+
## Verification
|
|
41
|
+
|
|
42
|
+
- `bun run compile` and `bun run lint` at the app root
|
|
43
|
+
- After deploying the backend, open the web app with dev tools and confirm a reconnect sends one
|
|
44
|
+
`POST /sync/snapshot/batch` instead of one `GET /sync/snapshot` per collection
|
|
@@ -159,13 +159,13 @@ Ensures preview environments don't accumulate and waste resources.
|
|
|
159
159
|
|
|
160
160
|
#### Dependabot Auto-merge
|
|
161
161
|
**Trigger:** Dependabot pull request opened
|
|
162
|
-
**Purpose:** Automatically merges
|
|
162
|
+
**Purpose:** Automatically merges Dependabot security updates after CI passes. Dependabot opens no version-update PRs; routine upgrades run through the `update-dependencies` skill.
|
|
163
163
|
|
|
164
164
|
**Requirements for auto-merge:**
|
|
165
165
|
- All CI checks pass
|
|
166
166
|
- Dependabot PR (created by Dependabot)
|
|
167
|
+
- Not a semver-major bump (for a grouped PR, no bump in the group is major); majors wait for a human review
|
|
167
168
|
- No conflicts
|
|
168
|
-
- 7-day cooldown for security (see Dependency Management)
|
|
169
169
|
|
|
170
170
|
**Learn more:** [Dependency Management](dependency-management.md)
|
|
171
171
|
|
|
@@ -208,7 +208,7 @@ Ensures preview environments don't accumulate and waste resources.
|
|
|
208
208
|
### Continuous Sync
|
|
209
209
|
- Documentation syncs on every master push
|
|
210
210
|
- Rules sync on every master push
|
|
211
|
-
- Dependencies update daily (Dependabot
|
|
211
|
+
- Dependencies update daily (`update-dependencies` skill); Dependabot opens security fixes only
|
|
212
212
|
- Quality improvements run daily (JSDoc, tests)
|
|
213
213
|
|
|
214
214
|
## Working with AI Workflows
|
|
@@ -280,8 +280,8 @@ The docs-audit workflow flags a missing README or reference page and obvious int
|
|
|
280
280
|
|
|
281
281
|
### Dependabot PR doesn't auto-merge
|
|
282
282
|
- Verify all CI checks pass (green checkmarks)
|
|
283
|
-
- Check 7-day cooldown hasn't been violated
|
|
284
283
|
- Ensure no merge conflicts exist
|
|
284
|
+
- If the security fix needs code changes, fold it into the rolling `chore/update-dependencies` PR instead of pushing to the Dependabot branch
|
|
285
285
|
|
|
286
286
|
### AI workflow creates poor quality PR
|
|
287
287
|
- Review the workflow definition in `.github/workflows/*.md`
|
|
@@ -290,7 +290,7 @@ The docs-audit workflow flags a missing README or reference page and obvious int
|
|
|
290
290
|
|
|
291
291
|
## Related Documentation
|
|
292
292
|
|
|
293
|
-
- [Dependency Management](dependency-management.md) — Dependabot configuration
|
|
293
|
+
- [Dependency Management](dependency-management.md) — Dependabot security-only configuration, auto-merge, and the update skill
|
|
294
294
|
- [Deploy to GCP](../how-to/deploy-to-gcp.md) — Manual deployment guide
|
|
295
295
|
- [GCP Hosting Architecture](gcp-hosting-architecture.md) — Static site hosting design
|
|
296
296
|
- [Environment Variables](../reference/environment-variables.md) — Configuration reference
|
|
@@ -1,16 +1,16 @@
|
|
|
1
1
|
# Dependency Management
|
|
2
2
|
|
|
3
|
-
Terreno pins shared versions in the Bun catalog, lands JavaScript updates
|
|
4
|
-
an agent skill that proves each bump with a test,
|
|
5
|
-
fingerprints frozen until a release.
|
|
3
|
+
Terreno pins shared versions in the Bun catalog, lands routine JavaScript updates
|
|
4
|
+
through an agent skill that proves each bump with a test, limits Dependabot to
|
|
5
|
+
security fixes, and keeps Expo native fingerprints frozen until a release.
|
|
6
6
|
|
|
7
7
|
## Layers
|
|
8
8
|
|
|
9
9
|
| Layer | Job |
|
|
10
10
|
| --- | --- |
|
|
11
11
|
| **Bun catalog** | One version for every dependency shared by two or more workspace packages (`catalog:` in manifests) |
|
|
12
|
-
| **`update-dependencies` skill** | Daily rolling PR on `chore/update-dependencies`: cooldown, exercise test, fingerprint freeze, ledger of landed/failed |
|
|
13
|
-
| **Dependabot** |
|
|
12
|
+
| **`update-dependencies` skill** | The only path for routine version upgrades. Daily rolling PR on `chore/update-dependencies`: cooldown, exercise test, local gate, one push per run, fingerprint freeze, ledger of landed/failed |
|
|
13
|
+
| **Dependabot** | Security updates only for bun + GitHub Actions. Native/Expo catalog names are ignored so auto-merge cannot change the fingerprint |
|
|
14
14
|
| **Release + `upgrading-expo`** | Expo SDK, React Native, and other native-module bumps |
|
|
15
15
|
|
|
16
16
|
Operator steps: [Update dependencies](../how-to/update-dependencies.md). Catalog rule: workspace `package.json` files use `"catalog:"` for shared pins.
|
|
@@ -41,16 +41,18 @@ Change a shared version in the catalog once. Do not pin a second copy in a works
|
|
|
41
41
|
|
|
42
42
|
## Agent updates vs Dependabot
|
|
43
43
|
|
|
44
|
-
Dependabot
|
|
44
|
+
Dependabot version updates group many packages and merge when CI is green. That misses Terreno constraints, so Dependabot now opens security PRs only:
|
|
45
45
|
|
|
46
46
|
1. **Exercise.** CI can pass without any test importing the bumped module. The skill requires a named test file that imports the package and that passed **after** the bump. If none exists, write a tracer-bullet test first.
|
|
47
47
|
2. **Fingerprint.** Expo `runtimeVersion.policy: "fingerprint"` ties OTA runtime and EAS dev builds to native hashes of `example-frontend` and `demo`. Native module bumps belong in a release, not a weekday dependency PR.
|
|
48
48
|
|
|
49
|
+
3. **CI is not the test runner.** A red grouped Dependabot PR invites an agent to push fix after fix to the Dependabot branch. Each push starts a full CircleCI pipeline that is attributed to, and emails, whoever owns the pushing token. The skill runs the CI mirror (`bun run prepush`) locally, pushes once per run, never edits CI config to make a bump pass, and stops after two red CI rounds.
|
|
50
|
+
|
|
49
51
|
`scripts/planning/fingerprintRisk.ts` (`isFingerprintSkip`) is the skip matcher. After every allowed bump the skill still recomputes hashes; a changed hash is reverted even if the matcher allowed the name.
|
|
50
52
|
|
|
51
53
|
## Cooldown
|
|
52
54
|
|
|
53
|
-
Non-security npm versions wait **7 days** after publish before a bump (`npm view <name> time`). Most malicious publishes are yanked inside a day.
|
|
55
|
+
Non-security npm versions wait **7 days** after publish before a bump (`npm view <name> time`). Most malicious publishes are yanked inside a day. CVE/advisory bumps skip the wait and still need an exercise test and a fingerprint freeze. Dependabot security updates are advisory bumps, so they have no cooldown.
|
|
54
56
|
|
|
55
57
|
## Fingerprint freeze
|
|
56
58
|
|
|
@@ -67,16 +69,16 @@ JavaScript packages, GitHub Actions, and a few Expo JS-only names may update. Me
|
|
|
67
69
|
|
|
68
70
|
## Dependabot configuration
|
|
69
71
|
|
|
70
|
-
`.github/dependabot.yml` still
|
|
72
|
+
`.github/dependabot.yml` sets `open-pull-requests-limit: 0` on every block. That turns off version-update PRs; security updates ignore the limit and still honor `ignore`. Each block's group uses `applies-to: security-updates`, so fixes arrive grouped:
|
|
71
73
|
|
|
72
|
-
| Group | Ecosystem |
|
|
73
|
-
| --- | --- |
|
|
74
|
-
|
|
|
75
|
-
|
|
|
76
|
-
|
|
|
77
|
-
|
|
|
74
|
+
| Group | Ecosystem |
|
|
75
|
+
| --- | --- |
|
|
76
|
+
| `github-actions` | `github-actions` at `/` |
|
|
77
|
+
| `root` | `bun` at `/` (root catalog and root tools) |
|
|
78
|
+
| `backend` | `bun` under api, ai, backends, … |
|
|
79
|
+
| `frontend` | `bun` under ui, rtk, demo, apps, … |
|
|
78
80
|
|
|
79
|
-
Frontend ignores catalog-managed names so those
|
|
81
|
+
Frontend ignores catalog-managed names so those fixes land once, in the root catalog. Root ignores fingerprint-skip names, so a security fix for a native/Expo package never opens a PR; handle it by hand in a release. `dependabot-auto-merge.yml` approves patch and minor Dependabot PRs and enables squash auto-merge, so those security PRs merge when required checks pass. A semver-major security PR (or a group containing one) gets no approval or auto-merge and waits for a human. When one fails because the fix needs code changes, fold the bump into the rolling PR ([Update dependencies](../how-to/update-dependencies.md)); never push to the `dependabot/*` branch.
|
|
80
82
|
|
|
81
83
|
## Daily rolling PR
|
|
82
84
|
|
|
@@ -91,10 +93,12 @@ Each package is still proven alone (exercise test + fingerprint) before it stays
|
|
|
91
93
|
- One rolling PR; many proven commits on that branch.
|
|
92
94
|
- Majors: exercise test + fingerprint freeze; no auto-merge.
|
|
93
95
|
- Security: skip cooldown; do not skip tests or fingerprints.
|
|
96
|
+
- Verify locally, push once; never change CI config to make a bump pass.
|
|
94
97
|
- New feature dependencies belong in that feature's slice, not this maintenance path.
|
|
95
98
|
|
|
96
99
|
## Related
|
|
97
100
|
|
|
98
101
|
- [Update dependencies](../how-to/update-dependencies.md)
|
|
102
|
+
- [Run tests locally](../how-to/run-tests-locally.md)
|
|
99
103
|
- [Install agent skills](../how-to/install-agent-skills.md)
|
|
100
104
|
- [Bun catalogs](https://bun.sh/docs/install/catalogs)
|
|
@@ -41,6 +41,15 @@ Terreno 57.
|
|
|
41
41
|
5. **Backend sync registration** — Every synced collection needs `syncPlugin`, `isDeletedPlugin`, and a `sync` block on `modelRouter`, plus `SyncApp` and `RealtimeApp` on the server. Optional `sync.adminBroadcast: true` also emits `sync:delta` to `{collection}|admin`. Admin clients subscribe with `mode: "window"` (`createSyncDb({windowCollections})`) so they do not snapshot or reconcile that collection; they call `hydrateWindow` (REST membership ids + `GET /sync/entities`). Pass the same `accessControl` into `SyncApp` that `AdminApp` uses so `{collection}|admin` requires `admin:access`. `AdminApp.register` also installs that model's admin list/read/`queryFilter` on the admin window so hydrate and live `|admin` deltas match `/admin` REST. Without RBAC, `user.admin` is enough for the shell gate. Live deltas on that stream apply only to ids already in the window; Refresh is a REST re-query plus `hydrateWindow`.
|
|
42
42
|
6. **Collections too large to sync whole** — List them in `createSyncDb({queryCollections})`. They skip snapshot bootstrap; screens load server-filtered *query windows* (`useWindowQuery`) from the `modelRouter` list endpoint. A window stores only membership (ordered ids); rows live once in the shared entity table. Overlapping windows therefore share rows — one delta updates both views — while each view lists only its own query's members, plus cached rows that newly match its `where` inside the loaded range. Live deltas still arrive on the user's normal streams; windows on screen refetch on reconnect to cover deltas missed offline.
|
|
43
43
|
|
|
44
|
+
## First load and reconnects
|
|
45
|
+
|
|
46
|
+
A fresh device has to download every row the user can see before the UI is complete. The
|
|
47
|
+
client asks for all of the user's streams in one batched request per round, writes each
|
|
48
|
+
round while the next one downloads, and writes rows in small chunks so the screen stays
|
|
49
|
+
responsive. Reconnects use the same path: when nothing changed, catching up costs two
|
|
50
|
+
requests however many collections the app syncs. Collections too large to download whole
|
|
51
|
+
belong in `queryCollections` instead.
|
|
52
|
+
|
|
44
53
|
## Tradeoffs
|
|
45
54
|
|
|
46
55
|
**Gains:** instant UI, offline writes with a durable outbox, simpler screen code, and one write path on the server (sync and REST share `modelRouter`).
|
|
@@ -260,8 +260,10 @@ Caching is off (200 credits per job). The docs Netlify target builds only the
|
|
|
260
260
|
current docs tree, disables Docusaurus minification, and pins SSG to one worker
|
|
261
261
|
thread so the build peaks near 3 GB and fits the 4 GB `medium` executor.
|
|
262
262
|
Released versions are not rebuilt: `archive-docs-version` (in `publish-release`,
|
|
263
|
-
`X.Y.0` tags only) builds each
|
|
264
|
-
|
|
263
|
+
`X.Y.0` tags only) builds each version once, uploads it to its **draft** GitHub
|
|
264
|
+
release, publishes the release, and then commits the docs snapshot to master, so
|
|
265
|
+
the `deploy-docs` run that commit starts can unpack the archive under
|
|
266
|
+
`/<version>/`. See `website/README.md`.
|
|
265
267
|
|
|
266
268
|
`terreno-gcp` uses CircleCI OIDC (`CIRCLE_OIDC_TOKEN_V2`), never a JSON service
|
|
267
269
|
account key. Set `circleci_org_id`, `circleci_project_id`, and
|
|
@@ -379,6 +381,17 @@ dependencies and runs `test:ci` when that script exists, not `test`.
|
|
|
379
381
|
tag version is already on npm, `publish-package.sh` skips it so a recut of
|
|
380
382
|
the same tag can finish the rest.
|
|
381
383
|
|
|
384
|
+
The release skill pushes the tag and creates the GitHub release with
|
|
385
|
+
`gh release create --draft`. The repository has immutable releases on: a
|
|
386
|
+
published release cannot take new assets (GitHub answers `422`). So CI publishes
|
|
387
|
+
the draft last with `scripts/ci/github-release.sh publish`.
|
|
388
|
+
`archive-docs-version` does it right after the upload on `X.Y.0` tags.
|
|
389
|
+
`publish-github-release` does it after `publish-release` on every other tag.
|
|
390
|
+
Both wait up to 10 minutes for the draft and are safe to rerun: a rerun skips
|
|
391
|
+
an upload that already landed on a published release and a snapshot already on
|
|
392
|
+
master. A
|
|
393
|
+
release still in draft after the pipeline means one of them failed.
|
|
394
|
+
|
|
382
395
|
Only stable tags (`57.3.0`) run `deploy-demo`. It builds the demo from source,
|
|
383
396
|
so it runs alongside `publish-release` instead of waiting for it. Use
|
|
384
397
|
`{"run-demo-deploy":true}` on `master` if a prerelease must also refresh the
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Update dependencies
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Routine version upgrades run through the `update-dependencies` skill (`/update-dependencies`). Dependabot opens **security PRs only**. The skill keeps one rolling PR; later runs push to it and rewrite its ledger instead of opening new PRs.
|
|
4
4
|
|
|
5
5
|
```bash
|
|
6
6
|
REPOSITORY="$(gh repo view --json nameWithOwner --jq .nameWithOwner)"
|
|
@@ -19,7 +19,7 @@ gh pr list --state open --head chore/update-dependencies \
|
|
|
19
19
|
bun outdated
|
|
20
20
|
```
|
|
21
21
|
|
|
22
|
-
Prove each bump with an **exercise test** that imports that package. Keep Expo fingerprints for `example-frontend` and `demo` unchanged. Skip `isFingerprintSkip` names until a release.
|
|
22
|
+
Prove each bump with an **exercise test** that imports that package. Verify the whole batch locally, then push **once**. Keep Expo fingerprints for `example-frontend` and `demo` unchanged. Skip `isFingerprintSkip` names until a release.
|
|
23
23
|
|
|
24
24
|
Why: [Dependency management](../explanation/dependency-management.md). Native Expo/React Native bumps wait for `upgrading-expo`.
|
|
25
25
|
|
|
@@ -27,9 +27,10 @@ Why: [Dependency management](../explanation/dependency-management.md). Native Ex
|
|
|
27
27
|
|
|
28
28
|
1. Find the same-repository PR with exact head `chore/update-dependencies`, base `master`, and `isCrossRepository: false`. Marker/title matches are not trust signals; ignore forks. Merge `origin/master` without rewriting history. If no trusted PR exists, create the canonical branch on this run.
|
|
29
29
|
2. Inventory with `bun outdated`. Retry **Failed** rows from the PR body first. Skip `isFingerprintSkip` in `scripts/planning/fingerprintRisk.ts`.
|
|
30
|
-
3. For each candidate: confirm a test imports the package (add a tracer if needed), bump the catalog pin or single-use version, `bun install`, re-run that test, recompute fingerprints. Revert only that bump on failure.
|
|
31
|
-
4.
|
|
32
|
-
5.
|
|
30
|
+
3. For each candidate: confirm a test imports the package (add a tracer if needed), bump the catalog pin or single-use version, `bun install`, re-run that test, recompute fingerprints. Revert only that bump on failure. Commit each success as `Bump <name> from <old> to <new>`.
|
|
31
|
+
4. Run the batch gate on the final head: `bun run compile`, `bun run lint`, `bun run prepush`. Run by hand any infra job `prepush` lists for a package you bumped (Playwright, Maestro, admin-spa integration). See [Run tests locally](run-tests-locally.md).
|
|
32
|
+
5. Push once. Update the **same** PR body: Last run, local gate result, CI rounds, Landed, Failed, Skipped, Deferred to release. Do not `gh pr create` when that PR is already open.
|
|
33
|
+
6. Read CI on the pushed head. Check each failed CircleCI job's `messages` before its step logs. Revert a bump that caused a failure and push a second time at most; stop and report anything else.
|
|
33
34
|
|
|
34
35
|
## Guardrails
|
|
35
36
|
|
|
@@ -41,7 +42,11 @@ Why: [Dependency management](../explanation/dependency-management.md). Native Ex
|
|
|
41
42
|
| No test imports the package | Write the tracer test first. Do not bump on `test:ci` alone |
|
|
42
43
|
| Published < 7 days ago (non-CVE) | Skip; log under Skipped |
|
|
43
44
|
| Major version | May land on the rolling PR if tests + fingerprints pass; no auto-merge |
|
|
44
|
-
|
|
|
45
|
+
| Local gate red | Fix or revert before pushing. CI is not the test runner |
|
|
46
|
+
| CI job has `messages` (for example `resource-class-not-in-plan`) | Config/plan error, not a dependency failure. Log it and stop |
|
|
47
|
+
| A bump seems to need CI config changes | Never edit `.circleci/`, workflows, executors, or resource classes from this path. Stop and report |
|
|
48
|
+
| CI red twice on heads this run pushed | Stop and report; do not push a third time |
|
|
49
|
+
| Dependabot security PR is red | Do not push to `dependabot/*`. Fold the bump into the rolling PR |
|
|
45
50
|
|
|
46
51
|
## Fingerprints
|
|
47
52
|
|
|
@@ -49,6 +54,10 @@ Why: [Dependency management](../explanation/dependency-management.md). Native Ex
|
|
|
49
54
|
|
|
50
55
|
`eas.json` Bun pins feed the fingerprint. Do not bump them here.
|
|
51
56
|
|
|
52
|
-
##
|
|
57
|
+
## Who CI emails
|
|
53
58
|
|
|
54
|
-
GitHub
|
|
59
|
+
CircleCI attributes a pipeline to the GitHub user whose token pushed it and emails that user on each failed workflow. Run the scheduled agent with a bot identity when one is configured. If it runs as a person, the one-push rule caps their mail at one round per run.
|
|
60
|
+
|
|
61
|
+
## Dependabot security PRs
|
|
62
|
+
|
|
63
|
+
`.github/dependabot.yml` sets `open-pull-requests-limit: 0` on every block, so Dependabot opens no version-update PRs. Security updates still open, grouped per block, and skip ignored native/Expo names. `dependabot-auto-merge.yml` approves them and enables squash auto-merge, so a green security PR merges without this skill. A red one gets its bump folded into the rolling PR. A security fix for an ignored native package is a manual release task (`release` + `upgrading-expo`).
|
|
@@ -6,18 +6,18 @@ Read [Local-first data](../explanation/local-first-data.md) and [Migrate from @t
|
|
|
6
6
|
|
|
7
7
|
## Startup and snapshots
|
|
8
8
|
|
|
9
|
-
### Cold start costs one round trip per stream
|
|
9
|
+
### Cold start costs one round trip per round, not per stream
|
|
10
10
|
|
|
11
|
-
A stream is one collection in one scope, so a user has roughly (synced collections) × (tenants they belong to) streams.
|
|
11
|
+
A stream is one collection in one scope, so a user has roughly (synced collections) × (tenants they belong to) streams. Reconcile, subscribe catch-up and wipe re-bootstrap page every stream through `POST /sync/snapshot/batch`: one round trip per round, with up to 5,000 rows per response shared across the streams. Empty streams cost nothing extra. `forceResync`, and every pass against a server without the batch route, still page with one `GET /sync/snapshot` per stream, `snapshotConcurrency` streams side by side (default 6). Each stream runs one pass at a time, and a pass that purges (force resync, wipe) waits for the stream's in-flight pass to finish first.
|
|
12
12
|
|
|
13
|
-
Measured with a fake HTTP channel at 150 ms latency, 13 collections and 2 workspaces (26 streams): 4.1 s to the first full reconcile when paging one stream at a time, 0.95 s at the default concurrency.
|
|
13
|
+
Measured with a fake HTTP channel at 150 ms latency, 13 collections and 2 workspaces (26 streams): 4.1 s to the first full reconcile when paging one stream at a time, 0.95 s at the default concurrency. In a browser with 100 collections and 10k rows, batching cut a reconnect with no changes from 101 requests to 2.
|
|
14
14
|
|
|
15
15
|
| Do | Why |
|
|
16
16
|
| --- | --- |
|
|
17
17
|
| Sync only the collections the client reads. | Each collection adds one stream per tenant to cold start. |
|
|
18
18
|
| Gate each screen on its **own** stream's bootstrap (see [No loading flag on synced reads](#no-loading-flag-on-synced-reads)), not on "sync finished". | The stream a screen needs may land after others. |
|
|
19
19
|
| Don't call `forceResync()` on a hot path. | It purges and re-pages every stream. |
|
|
20
|
-
| Raise `snapshotConcurrency` only if the backend can take it. |
|
|
20
|
+
| Raise `snapshotConcurrency` only if the backend can take it. | Without the batch route, each in-flight stream is one concurrent snapshot query on the server. |
|
|
21
21
|
|
|
22
22
|
The startup reconcile and each collection's `sync:subscribed` catch-up both page the same streams. A catch-up queues behind a pass that started before the room join, because that pass may have missed a write. So a cold start still pages each stream twice, one pass after the other.
|
|
23
23
|
|
|
@@ -70,7 +70,7 @@ Expo autolinking walks the app's own dependencies; a nested copy leaves the nati
|
|
|
70
70
|
│ sync:mutate ─────────────────────────────►│ │
|
|
71
71
|
│ sync:ack / sync:nack ◄────────────────────┘ │
|
|
72
72
|
│ sync:delta ◄──────────────────────────────────────────────┘
|
|
73
|
-
│
|
|
73
|
+
│ POST /sync/snapshot/batch (bootstrap + catch-up, HTTP)
|
|
74
74
|
└── POST /sync/mutate (fallback while the socket is down)
|
|
75
75
|
```
|
|
76
76
|
|
|
@@ -185,7 +185,7 @@ be passed to `betterAuthAdapter` directly.
|
|
|
185
185
|
| `wipeOnSignOut` | `boolean` | `false` | `signOut()` also wipes local data |
|
|
186
186
|
| `batchSize` | `number` | `50` (server caps at 100) | Max mutations per batched drain send |
|
|
187
187
|
| `haltQueueOnConflict` | `boolean` | `false` | `true` = conflict halts entire drain; `false` = per-entity blocking |
|
|
188
|
-
| `snapshotConcurrency` | `number` | `6` | Streams
|
|
188
|
+
| `snapshotConcurrency` | `number` | `6` | Streams paged at once with `GET /sync/snapshot`: by `forceResync`, and by every other pass against a server without `POST /sync/snapshot/batch`. Each stream runs one pass at a time. `1` pages one stream at a time |
|
|
189
189
|
| `onDecryptFailure` | `() => void` | wipe + re-bootstrap (with `console.warn`) | Override web decrypt-failure recovery |
|
|
190
190
|
| `tombstoneRetentionMs` | `number` | `7776000000` (90 days); `0` disables | Client tombstone compaction after successful reconcile |
|
|
191
191
|
| `startAuthRetryAttempts` | `number` | `3`; `1` disables | `start()` attempts to resolve `getUserId()` |
|
|
@@ -202,7 +202,7 @@ be passed to `betterAuthAdapter` directly.
|
|
|
202
202
|
| `goOffline()` | Simulated outage: disconnect transport, pause replay/reconcile/timer; local mutations keep queueing. |
|
|
203
203
|
| `goOnline()` | End simulated outage; reconnect triggers reconcile + outbox replay. |
|
|
204
204
|
| `mutate({collection, operation, id?, data?})` | Optimistic local write + durable outbox enqueue + fire-and-forget replay. Returns `{mutationId, id}`. |
|
|
205
|
-
| `reconcile()` | HTTP snapshot catch-up for every known stream; runs tombstone compaction on success. Also runs automatically on (re)connect, on a rate-limited seq-jump hint, and on the periodic timer;
|
|
205
|
+
| `reconcile()` | HTTP snapshot catch-up for every known stream, batched into `POST /sync/snapshot/batch` rounds (see [Snapshot bootstrap](#snapshot-bootstrap)); runs tombstone compaction on success. Also runs automatically on (re)connect, on a rate-limited seq-jump hint, and on the periodic timer; `sync:subscribed` confirmations additionally page just the streams they name, and confirmations arriving while that catch-up request is in flight are paged together in the next one. |
|
|
206
206
|
| `hydrateWindow({collection, ids, restRows?})` | Admin window upsert: REST rows (optional) land immediately; every requested id is also fetched from `GET /sync/entities` before this resolves so seq/deleted metadata is canonical for immediate update/delete. A `{collection}|admin` delta that lands while the fetch is in flight wins, so hydration never rewinds a row to an older seq. Unknown ids are ignored. |
|
|
207
207
|
| `queryWindows` | Server query windows for `queryCollections`: `fetchWindow({query, nextPage?})`, `getWindow(query)`, `retain(query)` → release, `refetchRetained()`, `pruneUnretained()`, `subscribe(cb)`, `keyFor(query)`. See [Query windows](#query-windows). |
|
|
208
208
|
| `forceResync()` | Purge every known stream locally and re-bootstrap from cursor 0 (outbox/conflicts untouched). Returns `{ok, reason?, streams, purged, repaired}`. After discovery and each stream bootstrap it waits for in-flight `start`/`stop`/auth-change work, then abandons with `reason: "superseded"` when that work switched users or generations. |
|
|
@@ -588,11 +588,36 @@ sync: {
|
|
|
588
588
|
|
|
589
589
|
## Sync protocol
|
|
590
590
|
|
|
591
|
+
### Snapshot bootstrap
|
|
592
|
+
|
|
593
|
+
The client pages every stream it belongs to through `POST /sync/snapshot/batch`, not one
|
|
594
|
+
`GET /sync/snapshot` per stream. With 100 collections, a reconnect that finds nothing new
|
|
595
|
+
is 2 requests (`GET /sync/streams` plus one batch) instead of 101.
|
|
596
|
+
|
|
597
|
+
- **Rounds:** each round asks for every stream that still has pages. Streams that reached
|
|
598
|
+
their head drop out, so later rounds give the remaining streams a larger share of the
|
|
599
|
+
5,000-entity budget.
|
|
600
|
+
- **Pipelining:** the next round is sent as soon as a response arrives, because its
|
|
601
|
+
cursors come from the response. The client writes the current round while the next
|
|
602
|
+
one downloads.
|
|
603
|
+
- **Chunked apply:** each page is written in store transactions of 500 entities, with a
|
|
604
|
+
yield to the event loop between them, so a large page never freezes the UI in one block.
|
|
605
|
+
A stream's cursor advances in the transaction holding its last entity, so a page cut
|
|
606
|
+
short by a crash is fetched again and re-applied as a no-op.
|
|
607
|
+
- **Older servers:** a 404 without an APIError body (the route does not exist) switches
|
|
608
|
+
the client to per-stream `GET /sync/snapshot` paging, `snapshotConcurrency` streams at
|
|
609
|
+
a time, for the rest of the session. An APIError 404, such as an unknown collection,
|
|
610
|
+
fails that pass and keeps batching on.
|
|
611
|
+
- **One pass per stream:** a batch pass owns each of its streams the way a per-stream
|
|
612
|
+
pass does, so a purge (`forceResync`, wipe) waits for it, and a pass that is superseded
|
|
613
|
+
by a sign-out or user switch stops between chunks without writing the rest.
|
|
614
|
+
|
|
591
615
|
### HTTP routes (`SyncApp`, all authenticated)
|
|
592
616
|
|
|
593
617
|
| Endpoint | Purpose |
|
|
594
618
|
|----------|---------|
|
|
595
|
-
| `GET /sync/snapshot?collection=&stream=&cursor=&limit=` | Bootstrap + catch-up
|
|
619
|
+
| `GET /sync/snapshot?collection=&stream=&cursor=&limit=` | Bootstrap + catch-up for one stream (`limit` max 100) |
|
|
620
|
+
| `POST /sync/snapshot/batch` | Bootstrap + catch-up for many streams in one round trip. Body `{streams: [{stream, cursor?, legacyCursor?}], limit?}`, at most 500 streams, no repeats. `limit` (default 5,000, max 10,000) is the entity budget for the whole response, split evenly across the streams. Returns `{pages}`, one `GET /sync/snapshot`-shaped page per stream in request order. Any stream the caller may not read fails the whole request |
|
|
596
621
|
| `GET /sync/streams` | Current stream membership for the user |
|
|
597
622
|
| `GET /sync/entities` | Point lookup for entity repair and admin window hydrate. Admin-window callers (`admin:access` with RBAC, else `user.admin`) on `adminBroadcast` collections receive requested ids across product streams. When AdminApp registered a scope, list/read/`queryFilter` still apply; unknown or out-of-scope ids are omitted |
|
|
598
623
|
| `POST /sync/mutate` | Single mutation (HTTP fallback). Optional `mutationMode: "adminWindow"` validates `adminBroadcast`, admin-window access, and a registered AdminApp write scope, then runs the shared sync executor with **AdminApp** pre/post hooks (not product `modelRouter` hooks), plus the same permission (including org membership), stripping, and audit semantics as `/admin` REST. HTTP mutate binds `X-Organization-Id` into org context |
|
|
@@ -636,6 +661,11 @@ const syncDb = createSyncDb({
|
|
|
636
661
|
});
|
|
637
662
|
```
|
|
638
663
|
|
|
664
|
+
Saves are debounced (500 ms). The store is serialized once per write, not once per
|
|
665
|
+
transaction: autosave compares TinyBase's content hashes (O(1)) to skip saves the last
|
|
666
|
+
write already covered, so a bootstrap of many pages costs a few writes instead of one
|
|
667
|
+
full-store serialization per page.
|
|
668
|
+
|
|
639
669
|
Undecryptable persisted data triggers wipe + re-bootstrap by default (`onDecryptFailure` to override). Storage read errors surface `persistence: "error"` without overwriting the blob.
|
|
640
670
|
|
|
641
671
|
**Native:** plaintext SQLite in the OS sandbox (by design).
|