@terreno/mcp 58.0.1 → 58.0.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (68) hide show
  1. package/CHANGELOG.md +31 -0
  2. package/README.md +6 -4
  3. package/dist/docs/templates/bootstrap/frontend/app/_layout.template.tsx +2 -0
  4. package/dist/docs/upgrades/58.0.3.md +28 -0
  5. package/dist/docs/versioned/next/explanation/ai-development-loop.md +10 -7
  6. package/dist/docs/versioned/next/explanation/dependency-management.md +2 -9
  7. package/dist/docs/versioned/next/explanation/roadmap-seed-issues.md +13 -0
  8. package/dist/docs/versioned/next/explanation/test-isolation.md +2 -1
  9. package/dist/docs/versioned/next/explanation/test-sharding.md +111 -0
  10. package/dist/docs/versioned/next/how-to/README.md +1 -0
  11. package/dist/docs/versioned/next/how-to/circleci.md +12 -4
  12. package/dist/docs/versioned/next/how-to/debug-with-mcp.md +10 -1
  13. package/dist/docs/versioned/next/how-to/fix-code-scanning-alerts.md +43 -0
  14. package/dist/docs/versioned/next/how-to/run-tests-locally.md +5 -0
  15. package/dist/docs/versioned/next/how-to/set-up-terreno-mcp.md +3 -1
  16. package/dist/docs/versioned/next/how-to/update-dependencies.md +1 -1
  17. package/dist/docs/versioned/next/implementationPlans/mcp-agent-debugger-gaps.md +257 -0
  18. package/dist/docs/versioned/next/implementationPlans/syncdb-high-churn-fields.md +266 -0
  19. package/dist/docs/versioned/next/reference/environment-variables.md +2 -0
  20. package/dist/docs/versioned/next/reference/mcp-server.md +12 -0
  21. package/dist/docs/versioned/next/reference/test.md +4 -1
  22. package/dist/docs/versioned/next/tasks/mcp-agent-debugger-gaps.md +151 -0
  23. package/dist/local/localTools.d.ts.map +1 -1
  24. package/dist/local/localTools.js +94 -1
  25. package/dist/local/localTools.js.map +1 -1
  26. package/dist/local/metro/metroDevSession.d.ts +1 -0
  27. package/dist/local/metro/metroDevSession.d.ts.map +1 -1
  28. package/dist/local/metro/metroDevSession.js +104 -1
  29. package/dist/local/metro/metroDevSession.js.map +1 -1
  30. package/dist/local/native/nativeLogs.d.ts +12 -0
  31. package/dist/local/native/nativeLogs.d.ts.map +1 -0
  32. package/dist/local/native/nativeLogs.js +103 -0
  33. package/dist/local/native/nativeLogs.js.map +1 -0
  34. package/dist/local/tools/clientDev.d.ts +22 -0
  35. package/dist/local/tools/clientDev.d.ts.map +1 -0
  36. package/dist/local/tools/clientDev.isolated.d.ts +2 -0
  37. package/dist/local/tools/clientDev.isolated.d.ts.map +1 -0
  38. package/dist/local/tools/clientDev.isolated.js +55 -0
  39. package/dist/local/tools/clientDev.isolated.js.map +1 -0
  40. package/dist/local/tools/clientDev.js +224 -0
  41. package/dist/local/tools/clientDev.js.map +1 -0
  42. package/dist/local/tools/readLogs.d.ts.map +1 -1
  43. package/dist/local/tools/readLogs.js +22 -3
  44. package/dist/local/tools/readLogs.js.map +1 -1
  45. package/dist/tools.d.ts.map +1 -1
  46. package/dist/tools.js +3 -1
  47. package/dist/tools.js.map +1 -1
  48. package/package.json +7 -7
  49. package/src/docs/templates/bootstrap/frontend/app/_layout.template.tsx +2 -0
  50. package/src/docs/upgrades/58.0.3.md +28 -0
  51. package/src/docs/versioned/next/explanation/ai-development-loop.md +10 -7
  52. package/src/docs/versioned/next/explanation/dependency-management.md +2 -9
  53. package/src/docs/versioned/next/explanation/roadmap-seed-issues.md +13 -0
  54. package/src/docs/versioned/next/explanation/test-isolation.md +2 -1
  55. package/src/docs/versioned/next/explanation/test-sharding.md +111 -0
  56. package/src/docs/versioned/next/how-to/README.md +1 -0
  57. package/src/docs/versioned/next/how-to/circleci.md +12 -4
  58. package/src/docs/versioned/next/how-to/debug-with-mcp.md +10 -1
  59. package/src/docs/versioned/next/how-to/fix-code-scanning-alerts.md +43 -0
  60. package/src/docs/versioned/next/how-to/run-tests-locally.md +5 -0
  61. package/src/docs/versioned/next/how-to/set-up-terreno-mcp.md +3 -1
  62. package/src/docs/versioned/next/how-to/update-dependencies.md +1 -1
  63. package/src/docs/versioned/next/implementationPlans/mcp-agent-debugger-gaps.md +257 -0
  64. package/src/docs/versioned/next/implementationPlans/syncdb-high-churn-fields.md +266 -0
  65. package/src/docs/versioned/next/reference/environment-variables.md +2 -0
  66. package/src/docs/versioned/next/reference/mcp-server.md +12 -0
  67. package/src/docs/versioned/next/reference/test.md +4 -1
  68. package/src/docs/versioned/next/tasks/mcp-agent-debugger-gaps.md +151 -0
@@ -0,0 +1,257 @@
1
+ # Implementation Plan: MCP Agent Debugger Gaps (Network, Dispatch, Storage, Native Logs, Dev Commands)
2
+
3
+ **Status:** Complete — Pick landed 2026-11-06
4
+ **Parent IP:** [mcp-boost-parity.md](mcp-boost-parity.md) (extends the local stdio companion; does not repeat hosted docs/codegen work)
5
+ **Roadmap:** Area=`mcp`, Target=`Next`, Impact=`Feature`
6
+ **Created:** 2026-11-05
7
+
8
+ ## Goal
9
+
10
+ Close the remaining agent-debugging gaps compared to integrated RN debuggers (for example
11
+ Xerlock): **network traffic**, **structured Redux dispatch**, **local key-value storage
12
+ read/write**, **native platform logs**, and **app-registered dev commands**. Agents already get
13
+ merged console/backend logs, Mongo, SyncDB/RTK state, navigation, and web driving via
14
+ `terreno-mcp-local`; this plan adds the missing observe/act surfaces without building a
15
+ human debugger UI.
16
+
17
+ ## Non-Goals
18
+
19
+ - On-device **SQLite** inspection outside SyncDB/TinyBase (partial overlap with
20
+ `get_syncdb_state`; raw `expo-sqlite` is out of scope).
21
+ - **Push notification** simulation, **network throttle**, and **React component tree /
22
+ profiler** MCP tools (human DevTools + screenshots + `get_client_state` remain the path).
23
+ - **Zustand** as a first-class framework integration (apps may expose Zustand via
24
+ `registerTerrenoDevCommand` only).
25
+ - Taking **`metro-mcp`** as a dependency (implement on the existing CDP session in
26
+ `metroDevSession.ts`).
27
+ - New **hosted** (`terreno-mcp`) tools — all work is **`terreno-mcp-local`** plus small
28
+ **`@terreno/syncdb`** / bootstrap dev hooks.
29
+
30
+ ## Decisions
31
+
32
+ | # | Question | Decision |
33
+ |---|----------|----------|
34
+ | D1 | Separate `read_network` tool vs extend `read_logs`? | **Extend `read_logs`** with source `network` (chronological merge with other sources when omitted). |
35
+ | D2 | Depend on `metro-mcp` vs in-house? | **In-house** on `mcp-server/src/local/metro/metroDevSession.ts` (same Hermes CDP connection as `app` / `evaluate`). |
36
+ | D3 | Writable local storage in v1? | **Yes** — `write_local_storage` requires `TERRENO_MCP_EVAL=1`; reads are always allowed. |
37
+ | D4 | Build order | **Network → dispatch + storage → native logs → dev commands → docs**. |
38
+ | D5 | React tree snapshot | **Defer** — not in this plan. |
39
+
40
+ ## Architecture
41
+
42
+ ### Tool surface (after this plan)
43
+
44
+ | Tool | Server | Purpose |
45
+ |------|--------|---------|
46
+ | `read_logs` | local | Existing sources `backend`, `browser`, `metro`, `app`; **add** `network`, `native`. |
47
+ | `last_error` | local | Unchanged; may surface errors from new sources when level is `error`. |
48
+ | `dispatch_client_action` | local | Dispatch a Redux action on `globalThis.__TERRENO_STORE__` via CDP (EVAL-gated). |
49
+ | `read_local_storage` | local | Read AsyncStorage keys (and opt-in SecureStore key names) via dev global adapter. |
50
+ | `write_local_storage` | local | Write one key (EVAL-gated); never writes SecureStore unless app registers an explicit allowlist entry. |
51
+ | `list_dev_commands` | local | List `{name, description, inputSchema}` from `globalThis.__TERRENO_DEV_COMMANDS__`. |
52
+ | `invoke_dev_command` | local | Run a registered handler by name (EVAL-gated). |
53
+
54
+ Existing tools (`get_client_state`, `syncdb_*`, `evaluate`, `navigate`, `browser`, …) are
55
+ unchanged.
56
+
57
+ ### CDP network capture (D1, D2)
58
+
59
+ Extend `metroDevSession.ts`:
60
+
61
+ 1. After attaching to the debug target, enable CDP domains **`Runtime`** (existing) and
62
+ **`Network`**.
63
+ 2. Maintain a ring buffer (same cap as console, default 2000) of normalized entries:
64
+ - `timestamp`, `method`, `url`, `status`, `mimeType`, `durationMs` when available
65
+ - Request/response **header maps** with **`Authorization`**, **`Cookie`**, and
66
+ **`set-auth-token`** redacted
67
+ - Optional truncated **response body** (text only, max 4 KiB per entry; skip binary)
68
+ 3. `read_logs` with `sources: ["network"]` (or default merge) maps the ring into the same
69
+ JSON shape as other sources, with `source: "network"`.
70
+ 4. `connection` metadata reuses the existing CDP status block; document that enabling
71
+ Network shares the single Hermes debugger slot with DevTools contention rules already
72
+ documented for `app`.
73
+
74
+ No separate MCP tool name.
75
+
76
+ ### Redux dispatch (D3, D4 phase 2)
77
+
78
+ `dispatch_client_action`:
79
+
80
+ ```typescript
81
+ // Input (strict JSON schema)
82
+ {
83
+ type: string; // required Redux action type
84
+ payload?: unknown; // optional JSON-serializable payload
85
+ }
86
+ ```
87
+
88
+ Implementation:
89
+
90
+ 1. Refuse unless `TERRENO_MCP_EVAL=1` (same posture as `evaluate` / `navigate`).
91
+ 2. Prefer in-process read of `globalThis.__TERRENO_STORE__` when the MCP process shares the
92
+ app heap (tests); otherwise CDP `Runtime.evaluate` with a fixed snippet that calls
93
+ `globalThis.__TERRENO_STORE__?.dispatch({type, payload})` and returns `{ok, error?}`.
94
+ 3. Validate `type` is a non-empty string; reject non-JSON-serializable payloads before evaluate.
95
+ 4. Tool description tells agents to use `get_client_state` first, then dispatch, then
96
+ `read_logs` / `get_client_state` again.
97
+
98
+ Requires `registerTerrenoDevStore(store)` in consumer apps (already in bootstrap /
99
+ `create-terreno-app`).
100
+
101
+ ### Local storage (D3, D4 phase 2)
102
+
103
+ Dev-only adapter on the app (new **`@terreno/syncdb`** export, wired in bootstrap template and
104
+ `example-frontend`):
105
+
106
+ ```typescript
107
+ registerTerrenoDevStorage({
108
+ asyncStorage: true, // default: wire AsyncStorage getAllKeys/getItem/setItem
109
+ secureStoreKeyAllowlist?: string[]; // optional SecureStore keys readable/writable via MCP
110
+ });
111
+ ```
112
+
113
+ Sets `globalThis.__TERRENO_DEV_STORAGE__` with `{listKeys, get, set}` in `__DEV__` only.
114
+
115
+ MCP tools:
116
+
117
+ - **`read_local_storage`**: `{keys?: string[], prefix?: string}` — returns key/value pairs;
118
+ omit keys to return all keys from `listKeys()` (cap 200 keys, 8 KiB per value in output).
119
+ - **`write_local_storage`**: `{key, value}` — EVAL-gated; rejects keys not allowed (SecureStore
120
+ keys not on allowlist).
121
+
122
+ MMKV and custom stores: apps register via `registerTerrenoDevCommand` or extend the adapter in
123
+ app code (document pattern; no MMKV in framework).
124
+
125
+ ### Native logs (D4 phase 3)
126
+
127
+ Extend `read_logs` with source **`native`**:
128
+
129
+ | Platform | Mechanism | When unavailable |
130
+ |----------|-----------|------------------|
131
+ | Android emulator/device | `adb logcat -d -t <N>` subprocess; optional `TERRENO_ADB_SERIAL` | Clear status: `adb` not on PATH or no device |
132
+ | iOS Simulator | `xcrun simctl spawn booted log stream --style compact` with timeout/read cap | No booted sim or Xcode tools missing |
133
+
134
+ Entries use `source: "native"`, plain-text `raw`, parsed `level` when log line matches common
135
+ patterns. **Not** merged into `last_error` unless line matches error heuristics (document
136
+ behavior). Opt-in env **`TERRENO_NATIVE_LOGS=0`** disables subprocess collection (default on).
137
+
138
+ Web and Metro-only sessions: `native` source returns empty with status note.
139
+
140
+ ### Dev commands (D4 phase 4)
141
+
142
+ App API ( **`@terreno/syncdb`** ):
143
+
144
+ ```typescript
145
+ registerTerrenoDevCommand({
146
+ name: "seedTodos",
147
+ description: "Insert three fixture todos for the signed-in user",
148
+ inputSchema: { type: "object", properties: { count: { type: "number" } } },
149
+ handler: async ({count}) => ({created: count ?? 3}),
150
+ });
151
+ ```
152
+
153
+ Registry: `globalThis.__TERRENO_DEV_COMMANDS__` — map of name → `{description, inputSchema,
154
+ handler}` (handlers run only inside the app via CDP evaluate, same as dispatch).
155
+
156
+ MCP:
157
+
158
+ - **`list_dev_commands`** — no args; JSON list of `{name, description, inputSchema}`.
159
+ - **`invoke_dev_command`** — `{name, arguments?}`; EVAL-gated; returns JSON-serializable
160
+ result or structured error.
161
+
162
+ Example: register one command in `example-frontend` that toggles a feature flag in AsyncStorage
163
+ to prove the loop.
164
+
165
+ ## Notifications
166
+
167
+ None.
168
+
169
+ ## UI
170
+
171
+ No user-visible UI. Dev hooks are `__DEV__` only and no-op in production builds.
172
+
173
+ ## Phases
174
+
175
+ | Phase | Scope | Depends on |
176
+ |-------|-------|------------|
177
+ | 1. Network via CDP | `metroDevSession` Network domain + `read_logs` `network` source | — |
178
+ | 2. Dispatch + storage | `dispatch_client_action`, storage adapter, read/write tools, bootstrap + example | 1 (shares CDP) |
179
+ | 3. Native logs | `read_logs` `native` source, adb/simctl collectors | — (parallel with 2 after 1) |
180
+ | 4. Dev commands | Registry API, `list_dev_commands`, `invoke_dev_command`, example command | 2 (same evaluate path) |
181
+ | 5. Documentation | Reference, how-to, explainer updates; real captured outputs | 1–4 |
182
+
183
+ ## Feature Flags & Migrations
184
+
185
+ - No data migrations.
186
+ - All new app hooks are stripped or no-op when `!__DEV__`.
187
+ - `TERRENO_MCP_EVAL=1` remains opt-in; bootstrap must not enable it by default.
188
+ - New env vars: `TERRENO_NATIVE_LOGS` (default on), `TERRENO_ADB_SERIAL` (optional).
189
+
190
+ ## Risks
191
+
192
+ - **CDP load**: Enabling Network increases traffic on the single Hermes session; keep ring caps
193
+ and body truncation strict.
194
+ - **Secret leakage**: Redact auth headers; never log request bodies that look like JWTs (regex
195
+ heuristic in normalizer).
196
+ - **Storage writes**: Mis-invoked `write_local_storage` can corrupt auth tokens; tool description
197
+ and docs warn to prefer `syncdb_action` for synced data.
198
+ - **Native log volume**: Cap lines per `read_logs` call; default `entries` applies per source.
199
+ - **`invoke_dev_command`**: Arbitrary app code — same trust model as `evaluate`; dev-only registry.
200
+
201
+ ## Not Included / Future Work
202
+
203
+ - SQLite file browser, notification dispatcher, network throttling, React tree/profiler MCP
204
+ tools, first-party MMKV adapter.
205
+
206
+ ## Files to Create / Modify
207
+
208
+ | Area | Files |
209
+ |------|--------|
210
+ | CDP | `mcp-server/src/local/metro/metroDevSession.ts`, tests |
211
+ | Logs | `mcp-server/src/local/tools/readLogs.ts`, tests |
212
+ | Runtime tools | `mcp-server/src/local/tools/runtime.ts` or new `clientDispatch.ts`, `localStorage.ts`, `devCommands.ts` |
213
+ | Registry | `syncdb/src/devMcpRegistry.ts` (or split files), `syncdb/src/index.ts`, tests |
214
+ | Bootstrap | `mcp-server/src/docs/templates/bootstrap/frontend/**`, `create-terreno-app` store bootstrap |
215
+ | Example | `example-frontend/app/_layout.tsx`, optional `devCommands.ts` |
216
+ | Docs | `docs/reference/mcp-server.md`, `docs/how-to/debug-with-mcp.md`, `docs/explanation/ai-development-loop.md`, `docs/reference/environment-variables.md`, `mcp-server/README.md` |
217
+
218
+ ## Task List
219
+
220
+ See [`docs/tasks/mcp-agent-debugger-gaps.md`](../tasks/mcp-agent-debugger-gaps.md).
221
+
222
+ ## E2E tests
223
+
224
+ | ID | Spec or flow | Layer | Proves | Written in |
225
+ | --- | --- | --- | --- | --- |
226
+ | E1 | `mcp-server/src/local/localServer.e2e.test.ts` (extend) | process | `read_logs` returns at least one `network` entry after CDP mock emits `Network.responseReceived` | Task 1.3 |
227
+ | E2 | `example-frontend/e2e/mcp-network-debug.spec.ts` | browser | A failing API call produces a 403/404 visible via agent workflow documented in how-to (page triggers fetch; test asserts backend log + optional network source when Metro attached in CI) | Task 5.2 |
228
+
229
+ If CI cannot attach Metro CDP reliably, Task 5.2 records `None — Metro CDP not in CI` and E2
230
+ becomes a manual evidence step in Task 5.1 only; unit/E1 coverage remains required.
231
+
232
+ **Pick note (2026-11-06):** E2 browser spec not added — Metro CDP is not reliable in CI. E1
233
+ (`localServer.e2e.test.ts`) plus unit tests satisfy automated proof.
234
+
235
+ ## Regression cases
236
+
237
+ None — no hard-to-automate change
238
+
239
+ ## User guide
240
+
241
+ None — operator/agent documentation only; no end-user workflow change
242
+
243
+ ## Acceptance Criteria
244
+
245
+ - [x] `read_logs` documents and returns entries with `source: "network"` when the app performs
246
+ fetch/XHR over CDP, with auth headers redacted in captured output.
247
+ - [x] `read_logs` documents and returns entries with `source: "native"` on Android (adb) and
248
+ iOS Simulator when tools are available, with clear status when not.
249
+ - [x] `dispatch_client_action` dispatches a known action in `example-frontend` under
250
+ `TERRENO_MCP_EVAL=1` and `get_client_state` reflects the new state.
251
+ - [x] `read_local_storage` / `write_local_storage` round-trip a key in `example-frontend` dev
252
+ build; write refused without `TERRENO_MCP_EVAL=1`.
253
+ - [x] `list_dev_commands` / `invoke_dev_command` list and run the example-frontend registered
254
+ command with EVAL gate on invoke only.
255
+ - [x] Reference and how-to pages list every new tool and source with real output captured in
256
+ Task 5.1.
257
+ - [x] `bun run mcp:build`, affected package tests, and `bun run lint` pass.
@@ -0,0 +1,266 @@
1
+ # SyncDB: high-churn fields without a sequence claim per write
2
+
3
+ **Status:** Draft
4
+ **Roadmap:** Area=`api`, Target=`Next`, Impact=`Feature`
5
+ **Branch:** syncdb-high-churn-fields
6
+ **Owner:** Josh Gachnang
7
+ **Created:** 2026-11-09
8
+
9
+ This is an options document. It describes the problem, compares five ways to handle it, and
10
+ recommends an order. Nothing here is implemented yet; the Decisions table lists what needs an
11
+ answer before a task list is written.
12
+
13
+ ## Goal
14
+
15
+ A synced document can carry fields that change every few seconds (a gauge, a heartbeat, a
16
+ progress counter) without each change costing three replicated writes and a round trip through
17
+ the stream's counter document.
18
+
19
+ ## Problem
20
+
21
+ Every write to a synced model outside a caller session does three things
22
+ (`api/src/sync/syncSeqPlugin.ts`, `api/src/sync/models.ts`):
23
+
24
+ 1. `claimSyncSeqs`: a pipeline update on the stream's `synccounters` document that increments
25
+ `seq` and pushes a `pending` entry.
26
+ 2. The document write itself, stamped with `_syncSeq`.
27
+ 3. `confirmSyncSeqs`: a `$pull` of the `pending` entry.
28
+
29
+ That is three oplog entries per write, two of them on one hot document per stream. The cost is
30
+ the same whether the write changes a title a person typed or a memory reading that moved by a
31
+ few kilobytes.
32
+
33
+ ### Measured in a consumer app (Shade, 2026-11-09)
34
+
35
+ Shade projects each agent session's container memory (`memUsedBytes`, `memPercent`) into two
36
+ synced collections every few seconds. Over a 20-hour oplog:
37
+
38
+ | Namespace | Oplog entries | Size | Share |
39
+ |---|---|---|---|
40
+ | `synccounters` | 1,519,392 | 547 MB | 56% |
41
+ | the two projection collections | 443,273 | 167 MB | 17% |
42
+ | everything else | about 395,000 | about 270 MB | 27% |
43
+
44
+ In one hour the sessions collection took 12,596 updates and its counter took 25,094. The
45
+ replication oplog window fell from about 48 hours to 20 hours and tripped the oplog-window
46
+ alert. Shade has since added a deadband in its own projection code, which is Option B below done
47
+ by hand in one app.
48
+
49
+ ### What makes this hard
50
+
51
+ - **Catch-up reads by sequence.** A snapshot pages `_syncSeq` in `(cursor, frontier]`
52
+ (`api/src/sync/routes.ts`). A write with no new sequence is invisible to a client that was
53
+ offline when it happened.
54
+ - **Clients drop deltas that do not raise the sequence.** `applyDelta` ignores a delta whose
55
+ `seq` is not strictly higher than the stored entity's (`syncdb/src/sync/deltaApplier.ts`). An
56
+ unsequenced change cannot reuse the `sync:delta` event as it stands.
57
+ - **The no-op check is all or nothing.** `pre("save")` already skips the claim when the only
58
+ modified path is `updated`. There is no way for a model to add its own fields to that set, and
59
+ the query-write path (`findOneAndUpdate`, `updateOne`) has no such check at all.
60
+
61
+ ## Non-Goals
62
+
63
+ - Changing ordering or frontier guarantees for ordinary fields (C1 stays as it is).
64
+ - Offline edits to high-churn fields. They are server-written values; client mutations on them
65
+ keep going through the normal sequenced path or are rejected.
66
+ - Replacing the per-stream counter with another ordering source. See Option E, listed only for
67
+ completeness.
68
+
69
+ ## Options
70
+
71
+ ### Option A: volatile fields (unsequenced writes, live-only delivery, periodic sequencing)
72
+
73
+ A model declares fields that change often:
74
+
75
+ ```ts
76
+ sync: {
77
+ scope: {type: "broadcast"},
78
+ volatile: {fields: ["memUsedBytes", "memPercent"], sequenceEveryMs: 60_000},
79
+ }
80
+ ```
81
+
82
+ - A write whose meaningful modified paths are all volatile skips the claim and the confirm. The
83
+ document write still happens, so the value is durable and a snapshot of the document returns
84
+ the current value. Cost: one oplog entry instead of three.
85
+ - The change-stream watcher sees the update with an unchanged `_syncSeq` and emits a new
86
+ `sync:volatile` event carrying `{collection, id, stream, seq, fields}`. The client merges the
87
+ fields into the entity it holds when `seq` matches the entity's sequence, and ignores the
88
+ event otherwise (a sequenced delta is on its way or already applied).
89
+ - To bound staleness for clients that were offline, a volatile-only write is sequenced normally
90
+ when the document's last sequenced write is older than `sequenceEveryMs`. Catch-up then sees
91
+ a value at most that old. The plugin needs the time of the last sequenced write, either a new
92
+ `_syncSeqAt` stamp or an in-process map keyed by document id.
93
+ - A write that touches any non-volatile field is sequenced as today and carries the volatile
94
+ fields with it.
95
+
96
+ Cost and risk:
97
+
98
+ - Protocol addition: a new socket event and a client handler. Older clients ignore the event and
99
+ see volatile fields refresh only on sequenced writes, which is a safe degradation.
100
+ - The query-write path has to learn which fields an update touches (`$set`, `$unset`, `$inc`
101
+ keys) to decide whether it is volatile-only. Replacements are always sequenced.
102
+ - The watcher has to distinguish "update with unchanged `_syncSeq`" from a sequenced delta.
103
+ `updateDescription.updatedFields` already says whether `_syncSeq` changed.
104
+ - Permissions and masking apply to `sync:volatile` exactly as to deltas; the event must go
105
+ through the same per-socket serialization.
106
+
107
+ Effect on the measured case: 3 entries per memory write down to 1, with no loss of live
108
+ resolution. The two counter updates behind each projection write were about 320 MB of the
109
+ 980 MB measured, so roughly a third off Shade's total oplog volume.
110
+
111
+ ### Option B: declarative write coalescing (drop writes that do not matter yet)
112
+
113
+ A model declares a tolerance and a minimum interval per field:
114
+
115
+ ```ts
116
+ sync: {
117
+ scope: {type: "broadcast"},
118
+ coalesce: {
119
+ memPercent: {minDelta: 1, minIntervalMs: 15_000},
120
+ memUsedBytes: {follows: "memPercent"},
121
+ },
122
+ }
123
+ ```
124
+
125
+ - In `pre("save")` and the query-write hook, a write whose meaningful modified paths are all
126
+ coalesced fields, and where each change is inside its tolerance or its interval, is dropped
127
+ entirely: no claim, no document write, no oplog entry.
128
+ - A change that exceeds the tolerance, or the first change after the interval, is written and
129
+ sequenced as today.
130
+
131
+ Cost and risk:
132
+
133
+ - No protocol change, no client change. Smallest diff of the five.
134
+ - Lossy by design: clients see the value in steps. Fine for gauges, wrong for anything that must
135
+ be exact at rest. A trailing flush (write the last dropped value after the interval) fixes
136
+ "exact at rest" but needs an in-process timer, which does not survive a restart and misbehaves
137
+ with several API instances unless the timer is per instance and idempotent.
138
+ - Dropping a `save()` silently is surprising. The hook must unmark the paths and leave the
139
+ in-memory document holding the stored value, or callers will read back a value that was never
140
+ written.
141
+ - `minDelta` only means something for numbers. Other types get `minIntervalMs` only.
142
+
143
+ Effect on the measured case: depends on the tolerance. A one-point deadband on `memPercent`
144
+ removes nearly all of the writes for an idle session.
145
+
146
+ ### Option C: cheaper sequencing for every write (batched confirms)
147
+
148
+ Keep the semantics and reduce the cost of the confirm. Today each confirm is its own `$pull`.
149
+ Instead:
150
+
151
+ - Hold confirmed seqs in a per-stream, in-process queue.
152
+ - Fold the queue into the next claim on that stream: the claim's pipeline update already
153
+ rewrites `pending`, so it can filter out confirmed seqs in the same update.
154
+ - Flush the queue with a standalone `$pull` after a short idle delay (for example 50 ms) so a
155
+ quiet stream's frontier still advances promptly.
156
+
157
+ Cost and risk:
158
+
159
+ - Under sustained writes this takes 3 entries per write to 2, and counter writes from 2 to 1.
160
+ A quiet stream gains nothing, since each write still gets its own flush.
161
+ - The frontier lags by up to the flush delay. Deltas already carry `frontierSeq` and clients
162
+ clamp to it, so correctness holds, but cursor advancement and `hasMore` lag by the same
163
+ amount.
164
+ - A crash loses queued confirms. Those entries age out through the existing 60-second lease,
165
+ so the frontier stalls for up to a minute after a crash under load. That is the same failure
166
+ mode as today's crash between claim and confirm, but it would now cover more seqs.
167
+ - Helps every synced model with no configuration. Does nothing about the write to the document
168
+ itself, or about deltas fanned out to every subscriber.
169
+
170
+ Effect on the measured case: up to half of the counter entries, so at most about a quarter
171
+ off the total, with no app changes.
172
+
173
+ ### Option D: ephemeral channel (values that never touch MongoDB)
174
+
175
+ Give apps a first-class way to publish values that are not part of the document at all:
176
+
177
+ ```ts
178
+ syncApp.publishEphemeral({collection: "zergsessions", id, stream, fields: {memPercent: 41.2}});
179
+ ```
180
+
181
+ - The server keeps the latest value per `(collection, id)` in memory and emits `sync:ephemeral`
182
+ to the stream's room. A client joining the room gets the current values in the join
183
+ acknowledgement.
184
+ - The client keeps ephemeral fields in a separate in-memory layer that query hooks merge over
185
+ the stored entity. Nothing is written to the local store or to MongoDB.
186
+
187
+ Cost and risk:
188
+
189
+ - Zero oplog entries. The right model for presence, typing indicators, live gauges and progress.
190
+ - Values are lost on server restart until the next publish, and are not available offline.
191
+ - Several API instances need a shared fan-out (the Socket.IO adapter already in use, or Redis)
192
+ and a shared "latest value" store for joins, or a join can miss values published on another
193
+ instance.
194
+ - Largest surface of the five: a server API, a socket event, a client layer, hook changes and
195
+ codegen typing for the ephemeral fields. Apps must move the fields out of their schema.
196
+
197
+ Effect on the measured case: removes the projection writes and their counter updates, about
198
+ half of Shade's oplog volume.
199
+
200
+ ### Option E: order by cluster time instead of a counter (not recommended)
201
+
202
+ Stamp documents with the server's operation time (`$currentDate` with `$type: "timestamp"`) and
203
+ use that as the cursor. No counter document, so one oplog entry per write for every model.
204
+
205
+ Rejected for now: the stable-frontier problem that C1 solved comes back in a harder form
206
+ (timestamps are assigned before commit, so a reader can see a later timestamp before an earlier
207
+ one lands), stream-local ordering turns into global ordering, every stored client cursor needs a
208
+ migration, and scope-move markers and tombstone compaction are both built on per-stream seqs.
209
+ It is a rewrite of the protocol, listed so the comparison is complete.
210
+
211
+ ## Comparison
212
+
213
+ | | Oplog entries per high-churn write | Live resolution | Offline catch-up | Protocol change | Client change | App change |
214
+ |---|---|---|---|---|---|---|
215
+ | Today | 3 | full | exact | | | |
216
+ | A: volatile fields | 1 | full | stale by at most `sequenceEveryMs` | new event | new handler | config only |
217
+ | B: coalescing | 0 for dropped writes, 3 for kept | stepped | exact for what was written | none | none | config only |
218
+ | C: batched confirms | 2 under load | full | exact | none | none | none |
219
+ | D: ephemeral channel | 0 | full | none | new event and join payload | new layer | move fields, new publish call |
220
+ | E: cluster time | 1 | full | exact | rewrite | rewrite | none |
221
+
222
+ ## Recommendation
223
+
224
+ 1. **Ship B first.** It needs no protocol or client work, it is what the affected app has
225
+ already done by hand, and it fixes the alerting problem on its own. Start with
226
+ `minDelta` and `minIntervalMs`, without a trailing flush.
227
+ 2. **Then A**, for fields where stepping is not acceptable. It keeps full live resolution at a
228
+ third of the cost and degrades safely for older clients.
229
+ 3. **C is independent** and worth doing for every model, but it is an optimisation of the hot
230
+ path with a crash-recovery trade-off, so it should come with a load-harness comparison
231
+ (`api/src/sync/loadHarness.ts`) before it is approved.
232
+ 4. **D only if** presence-style features (typing, cursors, live progress) arrive on the roadmap.
233
+ It is the cleanest answer for values that should never be stored, and too much surface to
234
+ build for gauges alone.
235
+
236
+ ## Decisions
237
+
238
+ | Question | Decision |
239
+ |----------|----------|
240
+ | Which options go ahead, and in what order? | Open. Recommendation: B, then A; C separately after a load test; D deferred. |
241
+ | B: does a dropped write need a trailing flush so the stored value is exact at rest? | Open. Recommendation: no in the first version. |
242
+ | B: should a dropped `save()` be silent, or logged at debug level? | Open. Recommendation: debug log with model and paths. |
243
+ | A: where does "last sequenced at" live, a `_syncSeqAt` field or process memory? | Open. Recommendation: `_syncSeqAt`, so it is correct across instances and restarts. |
244
+ | A: default for `sequenceEveryMs` | Open. Recommendation: 60 seconds. |
245
+ | A: may clients mutate volatile fields? | Open. Recommendation: reject in the mutation handler; they are server-written. |
246
+ | C: flush delay, and is a longer post-crash frontier stall acceptable? | Open. Needs load-harness numbers. |
247
+
248
+ ## Phases
249
+
250
+ Written once the decisions above are made. Each chosen option is its own phase with its own
251
+ task list under `docs/tasks/`.
252
+
253
+ ## Files to Create / Modify
254
+
255
+ For B and A, the likely touch points:
256
+
257
+ - `api/src/sync/types.ts`: `SyncConfig.coalesce`, `SyncConfig.volatile`, the `sync:volatile`
258
+ payload type.
259
+ - `api/src/sync/syncSeqPlugin.ts`: extend the no-op check in `pre("save")`; add the same check
260
+ to `preQueryWrite`.
261
+ - `api/src/realtime/changeStreamWatcher.ts`: emit `sync:volatile` for updates that leave
262
+ `_syncSeq` unchanged on a model with volatile fields.
263
+ - `api/src/sync/mutationHandler.ts`: reject client mutations on volatile fields.
264
+ - `syncdb/src/sync/deltaApplier.ts` and `syncdb/src/client.ts`: apply `sync:volatile`.
265
+ - `docs/explanation/local-first-data.md` and `docs/reference/syncdb.md`: document the options
266
+ and when to use each.
@@ -121,6 +121,8 @@ Hosting platforms scope variables per environment. Vercel has **Production**, **
121
121
  | `TERRENO_PROJECT_ROOT` | `terreno-mcp-local`, `terreno` CLI | ❌ | nearest Terreno project root | No | tooling |
122
122
  | `TERRENO_METRO_URL` | `terreno-mcp-local` | ❌ | frontend script port or `http://localhost:8082` | No | tooling |
123
123
  | `TERRENO_MCP_EVAL` | `terreno-mcp-local` | ❌ | disabled | No | tooling |
124
+ | `TERRENO_NATIVE_LOGS` | `terreno-mcp-local` `read_logs` source `native` | ❌ | enabled (`0` disables) | No | tooling |
125
+ | `TERRENO_ADB_SERIAL` | `terreno-mcp-local` native Android logs | ❌ | default `adb` device | No | tooling |
124
126
  | `TERRENO_WEB_URL` | `terreno` CLI | ❌ | `http://localhost:8082` | No | tooling |
125
127
  | `BUN_CHROME_PATH` | Bun WebView | ❌ | browser auto-discovery | No | tooling |
126
128
  | `TERRENO_OPENAPI` | `terreno` CLI | ❌ | — | No | tooling |
@@ -159,6 +159,18 @@ privately and are not returned. Start the local MCP with
159
159
  See [Debug a Terreno app with MCP](../how-to/debug-with-mcp.md) for examples and
160
160
  safety guidance.
161
161
 
162
+ ### Dev client hooks (`@terreno/syncdb`)
163
+
164
+ | Tool | EVAL | Registration |
165
+ | --- | --- | --- |
166
+ | `read_logs` / `last_error` | no | sources include **network** and **native** |
167
+ | `dispatch_client_action` | yes | `registerTerrenoDevClientDispatch` |
168
+ | `read_local_storage` / `write_local_storage` | write | `registerTerrenoDevStorage` (reads redact auth-like keys) |
169
+ | `list_dev_commands` / `invoke_dev_command` | invoke | `registerTerrenoDevCommand` |
170
+
171
+ Network capture redacts `Authorization`, `Cookie`, `Set-Cookie`, and `Set-Auth-Token`.
172
+ Failed requests include the URL from the matching `requestWillBeSent` `requestId`.
173
+
162
174
  ### terreno_search_docs
163
175
 
164
176
  BM25-style keyword search over markdown bundled with the MCP server: `docs/resources/*.md`, synced Diátaxis docs under `docs/versioned/`, and per-component excerpts derived from `ui-types-documentation.json`. **Prefer this tool before guessing** Terreno APIs (same posture as Laravel Boost `search-docs`).
@@ -361,7 +361,10 @@ exist or has no `smoke` script is an error.
361
361
  ## Coverage gates
362
362
 
363
363
  Package CI uses `scripts/check-coverage.ts` (`bun run test:coverage`, default 95%
364
- functions and lines) as the live gate. Bun 1.4.2+ may exit 1 on bunfig
364
+ functions and lines) as the live gate. It runs the suite as concurrent
365
+ `bun test --shard` processes and gates the merged LCOV; a sharded run below the
366
+ threshold is confirmed with one unsplit pass
367
+ ([Test sharding](../explanation/test-sharding.md)). Bun 1.4.2+ may exit 1 on bunfig
365
368
  `coverageThreshold` before isolated LCOV merges; the script continues when tests
366
369
  reported `0 fail` and then enforces 95% on the merged report. The blocks
367
370
  coverage pass sets `--max-concurrency=1`, because Bun's default 20-way run