@terreno/mcp 58.0.0-beta.3 → 58.0.0

package/src/docs/versioned/next/explanation/ai-observability.md

@@ -1,26 +1,26 @@
 # AI observability
 
-Terreno ships Langfuse-like prompt versioning, nested traces, evaluators, datasets, experiments, and a local human review queue **inside `@terreno/ai`**, with operator UI in **`admin-frontend` only**.
+Terreno ships prompt versioning, nested traces, evaluators, datasets, experiments, and a local human review queue **inside `@terreno/ai`**, with operator UI in **`admin-frontend` only**.
 
-## Why not Langfuse-only
+## Why built in
 
-Apps must iterate prompts without a required vendor account. `LangfuseApp` stays a second plugin. Traces and scores **fan out** to every registered sink (local Mongo, Langfuse, OTLP/OpenInference). Prompts, datasets, experiments, and the review queue have **one writer** each (a primary). The review queue is **local-only**.
+Apps must iterate prompts without a required vendor account, so the prompt registry, traces, and review queue live in the app's own Mongo. Traces and scores **fan out** to every registered sink (local Mongo plus any extra `TraceSink` / `ScoreSink` plugin an app registers). Prompts, datasets, experiments, and the review queue have **one writer** each (a primary). The review queue is **local-only**.
 
 That split is the two planes:
 
 | Plane | Write | Admin read |
 | --- | --- | --- |
-| Telemetry (traces, scores) | All sinks, best-effort | Local if the local plugin is on, else Langfuse |
+| Telemetry (traces, scores) | All sinks, best-effort | Local |
 | Control (prompts, datasets, experiments, queue) | One primary per capability | That primary |
 
-`experiments.primary` must equal `datasets.primary`. Defaults are all `local`.
+`local` is the only control primary, and the default for all four. `experiments.primary` must equal `datasets.primary`.
 
 `ObservabilityApp` fails boot (constructor throws) when:
 
 | Error | Cause |
 | --- | --- |
 | `experiments.primary must equal datasets.primary` | Mixed dataset/experiment writers |
-| `reviewQueue.primary must be local` | `reviewQueue` set to `langfuse` |
+| `reviewQueue.primary must be local` | `reviewQueue` set to anything but `local` |
 | `<capability> primary "<id>" has no plugin` | No registered plugin with that `id`, capability, and store |
 
 Telemetry sinks still fan out even when a control primary is local-only.
@@ -28,8 +28,10 @@ Telemetry sinks still fan out even when a control primary is local-only.
 Admin chrome lives in `admin-frontend`. One sidebar group **AI Observability** holds Prompts,
 Traces, Evaluators, Datasets, Experiments, and Review (Review is omitted when the local plugin is
 off). Existing **AI Requests** stays under **Screens**. Every observability screen shows
-breadcrumbs `Admin / AI Observability / <Section> / <leaf>` and a status chip from
-`GET /ai/observability/status`. Prompt detail is a hub (Overview, Versions, Traces, Experiments);
+breadcrumbs (`Admin / AI Observability / <Section>` on lists, plus `/ <leaf>` on detail
+screens), a quiet status chip from `GET /ai/observability/status`, and one left-aligned page
+header with the screen's single primary action. Prompt detail is a hub (Versions, Traces, Experiments).
+Versions shows the description, folder, tags, and the version editor on one screen;
 versions are immutable in admin (`Save as vN+1`); apps never inline the string.
 
 ## Why this shape for product work
@@ -58,7 +60,7 @@ customized `admin` and `auditor` roles gain only missing observability actions o
 
 ## Prompt hub relationships (phase 4)
 
-Prompts may carry an optional domain-neutral `description` (folder semantics stay consumer-defined). `GET /ai/observability/prompts/:name` composes bounded recent **traces** and **experiments** for that prompt name so the admin hub does not scan full trace or experiment lists on the client. Trace evidence is version-aware: list filters accept `prompt` plus optional `promptVersion`, and relationship trace rows include the matching `promptVersion` from `ObsTrace.prompts[]`. The same optional positive integer `promptVersion` on prompt detail filters hub traces with `$elemMatch` on `prompts` (name and version). Omit it to keep the unfiltered prompt-name match. Invalid or non-positive versions return **400**. Experiment relationships filter server-side by `promptName` on `GET /ai/observability/experiments?promptName=…` (omit or pass an empty value for no filter). The experiment list endpoint returns summary rows without per-item hydration; use experiment detail for full item results. Reading relationships uses the same `aiPrompt:read` grant as prompt detail.
+Prompts may carry an optional domain-neutral `description` (folder semantics stay consumer-defined). `GET /ai/observability/prompts/:name` composes bounded recent **traces** and **experiments** for that prompt name so the admin hub does not scan full trace or experiment lists on the client. Trace evidence is version-aware: list filters accept `prompt` plus optional `promptVersion`, and relationship trace rows include the matching `promptVersion` from `ObsTrace.prompts[]`. The same optional positive integer `promptVersion` on prompt detail filters hub traces with `$elemMatch` on `prompts` (name and version). Omit it to keep the unfiltered prompt-name match. Invalid or non-positive versions return **400**. Experiment relationships filter server-side by `promptName` on `GET /ai/observability/experiments?promptName=…` (omit or pass an empty value for no filter). The experiment list endpoint returns summary rows without per-item hydration; use experiment detail for full item results. Reading relationships uses the same `aiPrompt:read` grant as prompt detail. Detail also returns `usedBy` (`items`, `limit`, `total`): other prompts whose saved text contains `{{> this name}}`, capped at 20 like the other relationships. The editor inserts that tag from the **Insert prompt reference** modal.
 
 ## Phase 1 reference loop
 
@@ -67,7 +69,7 @@ The example backend always registers the local plugin. Its idempotent seed creat
 `correctness-human` evaluator, the automatic `schema-assert` evaluator, and a two-item
 `example-gold` dataset. It also creates `chat-safety-synthetic`, a proofread set of
 two-person chats for routing, toxicity, privacy, and dismissive-reply experiments.
-This makes the review and experiment loops walkable without Langfuse:
+This makes the review and experiment loops walkable with no external service:
 resolve the production prompt → emit a trace → inspect spans and sensitive I/O → send the
 trace to Review → record a human score.
 
@@ -87,11 +89,35 @@ With the local plugin registered and primaries set to `local`, the control plane
 
 - **`llm-judge` and `json-assert` evaluators** — judges call `AIService.generateJsonObject` through a named registry prompt (`judgePromptName`); create/update rejects a judge when the prompt `outputSchema` omits a required dimension (the error names that key). `json-assert` supports path/constraint checks and a built-in mode that validates output against the prompt version `outputSchema`. Parse or generation failures record an error outcome instead of throwing to the caller. Custom creates score a full trace; generation span and dataset item targets are reserved. Seeded templates install `correctness` / `hallucination` / `helpfulness` / `toxicity` as `llm-judge`, `schema-assert` as `json-assert`, and `*-human` variants for the review queue.
 - **Human review evaluators** — sending traces to review requires a `human` evaluator. Its dimensions become the review item's score fields and its instructions tell the reviewer how to apply them. `llm-judge` and `json-assert` evaluators run automatically and are rejected from the human queue.
+- **Review batches and judge agreement** — a human evaluator with several yes/no dimensions (each with a reviewer `question`) acts as a question set. A batch fixes the traces people answer, so an operator can run any number of LLM judges over the same traces and see, per judge and per question, how often the judge gives the human answer. That number is what tells you a new judge can replace or extend human review; disagreements point at the prompt rule to fix.
 - **Datasets** — CRUD plus item provenance (`origin`, `proofread`, `sourceTraceId`, tags, outcome class). Import accepts **JSON** (bare objects or structured rows) and **CSV** (quoted fields; `input.*` / `expectedOutput.*` column prefixes). Rows validate against the bound prompt input schema when configured. Adding from a trace copies I/O; sensitive traces always land `proofread: false`. Deleting an item never mutates the trace.
-- **Experiments** — compare 2–3 prompt versions on a dataset with optional `modelOverride` (requires `aiServiceFactory` on `ObservabilityApp`), evaluator thresholds (defaulting to the SOP gates), **per-version gate tiles** (`gates[].version`), outlier/low-confidence item ids, and version-scoped promote (**409** when the selected version's gates fail). Unproofread items are excluded unless `includeUnproofread` is true. Local runs compile exact prompt versions and call `AIService.generateText` with compiled `prompt` + `systemPrompt` (no `promptName`/`promptLabel`). `ObservabilityApp` wires the local experiment runner from `aiService` at register time.
+- **Experiments** — compare 2–3 prompt versions on a dataset with optional `modelOverride` (requires `aiServiceFactory` on `ObservabilityApp`), evaluator thresholds (defaulting to the SOP gates), **per-version gate tiles** (`gates[].version`; the results screen groups them in one card per version and shows operators as ≥ ≤ > < =, never `gte`/`lte`), outlier/low-confidence item ids, and version-scoped promote (**409** when the selected version's gates fail). Unproofread items are excluded unless `includeUnproofread` is true. Local runs compile exact prompt versions and call `AIService.generateText` with compiled `prompt` + `systemPrompt` (no `promptName`/`promptLabel`). `ObservabilityApp` wires the local experiment runner from `aiService` at register time.
 
 Operator UI for datasets and experiments ships in tasks 2.7–2.9; routes and stores are live for API clients and tests today.
 
+## The eval pipeline: humans first, then calibrated judges
+
+Terreno's evaluation flow has four steps, and each step reuses the one before it:
+
+1. **Build a dataset** of traces (or hand-written items).
+2. **Humans answer a question set** — a `human` evaluator whose dimensions are usually yes/no
+   `question`s — for every dataset item, through review batches built from the dataset. Each
+   submitted answer is written back onto the dataset item (`humanAnswers`).
+3. **Calibrate the judge.** A judge calibration experiment (`kind: "judge"`) runs 2–3 versions of
+   an LLM judge's prompt over the human-answered items and ranks the versions by agreement with
+   the humans. Iterate on the judge prompt until a version clears the target (default 90%), then
+   promote it.
+4. **Compare product prompts** with ordinary prompt experiments, scored by a judge whose
+   calibration (latest agreement, and whether it is stale) shows on the evaluator.
+
+Human answers are expensive, so they are reused: a dataset judged once feeds any number of judge
+experiments. The rule that keeps reuse honest is the **question fingerprint** — a hash of each
+question's key, type, wording, and range. Answers stay *current* while the fingerprint matches;
+rewording a question makes them *stale*, the dataset shows them as stale, new review batches
+re-queue those items, and judge calibrations based on the old wording are flagged stale.
+Judges always see what reviewers saw (the source trace, else the item's input and expected
+output), so agreement measures the judge, not a difference in evidence.
+
 ## GPT tool spans and nested traces
 
 `/gpt/prompt` streams tool rounds to the client **and** records them in observability traces. Each provider `tool-call` / `tool-result` pair becomes a `TOOL` span (input args, cleaned output, timing, `parentSpanId` on the chain root). When tools run, the exported trace uses a `CHAIN` root wrapping those `TOOL` children instead of a lone `LLM` span. Plain `AIService.generate*` calls without tool children still log a single `LLM` root.

package/src/docs/versioned/next/explanation/durable-agent-harness.md

@@ -69,7 +69,7 @@ transaction:
 
 Either all three land or none do. This is why `Harness.open` refuses to start without a
 replica set (transactions need one) or without the local observability models. Planned
-Langfuse and OTel exports will stay best-effort, after the transaction.
+exports to other trace sinks will stay best-effort, after the transaction.
 
 ```
 createTask ──tx──> HarnessTask(pending) + ObsTrace + root CHAIN span

package/src/docs/versioned/next/explanation/roadmap-seed-issues.md

@@ -585,7 +585,7 @@ are a separate item.
 **Title:** `Native module baseline for the next major release`
 
 **Labels:** `area:ui`, `type:chore`, `breaking`
-**Project fields:** Area=`ui`, Target=`Next`, Impact=`Breaking`, IP=`native-module-baseline`, Status=`Shaping`
+**Project fields:** Area=`ui`, Target=`Next`, Impact=`Breaking`, IP=`native-module-baseline`, Status=`In progress`
 
 Adding a native module to a Terreno app forces a new dev-client/store binary, so every
 native dependency the B2B program needs lands in one major release: Stripe payment sheet,
@@ -1158,18 +1158,46 @@ Delete the **legacy RTK cache-patching realtime path** in Terreno 58. Collection
 
 ## ai-observability
 
-**Title:** `AI observability (Langfuse-light, pluggable)`
+**Title:** `AI observability (built in, pluggable)`
 
 **Labels:** `area:ai`, `type:feature`  
 **Project fields:** Area=`ai`, Target=`Future`, Impact=`Feature`, IP=`ai-observability`, Status=`Planned`
 
-Ship **Langfuse-like** prompt versioning, nested traces (user / session / cost), multidimensional evaluators (LLM-as-judge + structured I/O), datasets, experiments, and a **local human review queue** — **inside Terreno**, as **plugins on `@terreno/ai`**, with an operator UI in **`admin-frontend` only**.
+Ship built-in prompt versioning, nested traces (user / session / cost), multidimensional evaluators (LLM-as-judge + structured I/O), datasets, experiments, and a **local human review queue** — **inside Terreno**, as **plugins on `@terreno/ai`**, with an operator UI in **`admin-frontend` only**.
 
 - **Implementation plan:** [ai-observability.md](https://github.com/TerrenoLabs/terreno/blob/master/docs/implementationPlans/ai-observability.md)
 - **Tasks:** [ai-observability.md](https://github.com/TerrenoLabs/terreno/blob/master/docs/tasks/ai-observability.md)
 
 ---
 
+## ai-observability-production-loop
+
+**Title:** `AI observability in production: live evals, alerts, threads, CI suites, prompt optimizer`
+
+**Labels:** `area:ai`, `type:feature`  
+**Project fields:** Area=`ai`, Target=`Next`, Impact=`Feature`, IP=`ai-observability-production-loop`, Status=`Planned`
+
+Terreno's AI observability covers building an AI feature: versioned prompts, traces, evaluators, datasets, experiments, and human review. It stops at production. The evaluator's live-sampling setting is never run. There's no view of quality, cost, or latency over time, nothing alerts on a regression, and a chat conversation is a loose set of traces. Prompt changes can't be gated in CI.
+
+This work adds:
+
+- Online evaluation of sampled production traces.
+- Named trace groups (conversations, or any id the app chooses) with a timeline view.
+- A monitoring dashboard.
+- Threshold alerts, delivered as signed, metadata-only webhooks to Slack or any URL.
+- Stored eval suites that `terreno ai test <suite>` runs from CI with a scoped service token.
+- A prompt optimizer that proposes better prompt versions from failing dataset items.
+
+Everything stays local to `@terreno/ai`, with screens in `admin-frontend`.
+
+Out of scope: conversation-level evaluators, eval backfill and budgets, email alerts, custom dashboards, and other optimizer algorithms.
+
+- **Implementation plan:** [ai-observability-production-loop.md](https://github.com/TerrenoLabs/terreno/blob/master/docs/implementationPlans/ai-observability-production-loop.md)
+- **Tasks:** [ai-observability-production-loop.md](https://github.com/TerrenoLabs/terreno/blob/master/docs/tasks/ai-observability-production-loop.md)
+- **Depends on:** `ai-observability` (phases 1–4)
+
+---
+
 ## better-auth-strict-oauth-provider
 
 **Title:** `Better Auth sync omits unset oauthProvider`
@@ -1401,6 +1429,28 @@ Open for discussion: how this relates to the support agent's `support/kb/` docs
 
 ---
 
+## remove-rtk
+
+**Title:** `Remove @terreno/rtk and Redux in Terreno 58`
+
+**Labels:** `area:syncdb`, `type:feature`  
+**Project fields:** Area=`syncdb`, Target=`58`, Impact=`Breaking`, IP=`remove-rtk`, Status=`Planned`
+
+Delete the `@terreno/rtk` package and every Redux dependency from the monorepo. After this change `@terreno/syncdb` is the only frontend data and session layer:
+
+- Synced collections use `useQuery` / `useMutate` (unchanged).
+- Every other HTTP call (`/auth/me`, modelRouter actions, AI, consents, announcements,
+  version check) goes through a new syncdb request client with generated, typed hooks.
+- Session state comes from `@terreno/syncdb/auth` (Better Auth, plus a ported JWT adapter).
+- Feature flags and server pushes ride syncdb's existing socket.
+- GptHistory and McpServiceToken become synced collections.
+- All unit and e2e tests exercise these syncdb paths. No test imports RTK or Redux.
+
+- **Implementation plan:** [remove-rtk.md](https://github.com/TerrenoLabs/terreno/blob/master/docs/implementationPlans/remove-rtk.md)
+- **Tasks:** [remove-rtk.md](https://github.com/TerrenoLabs/terreno/blob/master/docs/tasks/remove-rtk.md)
+
+---
+
 # Shipped, umbrella, and declined IPs
 
 Tracking issues created for IPs that previously lacked a `**Roadmap issue:**` header.

package/src/docs/versioned/next/how-to/README.md

@@ -60,6 +60,7 @@ Problem-oriented, practical steps. Use these when you know what you want to do.
 - [Compose charts in a dashboard grid](charts-and-dashboards.md) — `LineChart` / `BarChart` / `AreaChart` / `DonutChart` inside `Card`s in `DashboardGrid` (ops-dashboard scorecards and spans: [dashboard-chart-parity](../implementationPlans/dashboard-chart-parity.md))
 - [Compare rendered chart snapshots](compare-chart-rendered-snapshots.md) — PNG goldens for the chart visual gallery, not JSON snapshots
 - [Upgrade from @terreno/rtk to @terreno/syncdb (Terreno 58)](migrate-rtk-to-syncdb.md) — Remove `@terreno/rtk` and Redux; every removed export mapped to its replacement
+- [Upgrade to the server-rendered admin SPA](upgrade-admin-spa-ssr.md) — New `dist/client` + `dist/server` layout, the baked `/console` mount, `internalOrigin`, and proxy cache rules
 - [Use syncdb correctly](use-syncdb-correctly.md) — Serial snapshot startup, outbox-halting 403s, phantom creates, conflicts, and other library behavior apps must handle
 - [Generate a REST CLI from OpenAPI](generate-a-rest-cli.md) — Scaffold `terreno generate rest-cli` for an app API
 - [Generate the client SDK](generate-the-client-sdk.md) — `terreno-syncdb-codegen` writes collection hooks and typed request hooks from OpenAPI

package/src/docs/versioned/next/how-to/ai-feature-development.md

@@ -1,6 +1,6 @@
 # Develop an AI feature
 
-Work this loop in **Terreno admin** when observability primaries are `local`. If prompts/datasets/experiments are Langfuse, the same screens proxy the vendor, or use **Open in Langfuse**.
+Work this loop in **Terreno admin**. Observability primaries are always `local`.
 
 Architecture: [AI observability](../explanation/ai-observability.md). Routes and models: [AI reference](../reference/ai.md). Register plugins: [Observe LLM calls](./observe-llm-calls.md).
 
@@ -43,9 +43,9 @@ Skip unlabeled items in experiments unless you only need generation smoke (still
 
 ## 3. Create the prompt
 
-1. Open **Admin → AI Observability → Prompts**. The folder rail shows counts; search filters name and folder. If your role lacks `aiPrompt:create`, **Create prompt** is hidden (API writes still 403).
-2. **Create prompt** (folder, name, system, user template with `{{variables}}`). That write is immutable **v1**. Open the row to the prompt hub (**Overview**, **Versions**, **Traces**, **Experiments**).
-3. On **Versions**, edit system/template/variables/temperature when `aiPrompt:update` is allowed, then **Save as vN+1**. There is no in-place save. **Playground → Run once** (when `aiPrompt:playground` is allowed) compiles and calls the model; it does **not** create a version. **Traces** and **Experiments** show recent items for this prompt from prompt detail relationships.
+1. Open **Admin → AI Observability → Prompts**. The folder filter shows counts; search filters name and folder. If your role lacks `aiPrompt:create`, the header **New prompt** is hidden (API writes still 403).
+2. **New prompt** (folder, name, system, user template with `{{variables}}`). That write is immutable **v1**. Open the row to the prompt hub (**Versions**, **Traces**, **Experiments**).
+3. **Versions** shows the description, folder, tags, and the editor together. Edit system/template/variables/temperature when `aiPrompt:update` is allowed, then **Save as vN+1**. There is no in-place save. System and template fields explain what each one is for. **Playground → Run once** (when `aiPrompt:playground` is allowed) compiles and calls the model; it does **not** create a version. **Traces** and **Experiments** show recent items for this prompt from prompt detail relationships.
 4. Do not paste that string into app routes. Apps call `AIService` with `promptName` + `promptLabel` (step 6).
 
 Read-only auditors with `aiPrompt:list` and `read` can review overview metadata and relationship tabs but do not see create, save, promote, or playground controls. Operators with the full `aiPrompt` action set see those controls. UI gating follows `permissions` on `GET /ai/observability/status`; assign roles in **Admin → Roles** per [API reference — AI observability RBAC](../reference/api.md#ai-observability-rbac).
@@ -126,15 +126,16 @@ Do this in parallel with steps 1–6 once the feature is reachable.
 4. In **Admin → AI Observability → Traces**, filter by time, prompt dropdown, status, user, session, score presence, or data sensitivity. Score and sensitivity dropdowns include both positive and negative states. Rows show a status dot, `sensitive` badge, error line, `N prompts`, tokens, cost, and latency. Open a row for the span tree (kind, indent, duration bar). Sensitive I/O disclosures start **collapsed**.
 5. Select one or more traces. The bulk bar warns when any selected row is `sensitive`. Choose **Send to human review**, read the queue explanation, and pick a human evaluator. The evaluator's dimensions become the score fields and its instructions guide the reviewer. Confirming posts `POST /ai/observability/traces/review` with `reason: "manual"`. Automatic (`llm-judge` / `json-assert`) evaluators cannot enter this queue.
 6. Open **Review queue**. Tabs show Pending / In progress / Done / Skipped counts, and pending
-   items are oldest-first. Select **Start reviewing — oldest first**.
+   items are oldest-first. Select the header **Review oldest pending**.
 7. On each item, compare **What the AI was given** with **What the AI wrote**. Long fields begin
    collapsed with a word count; structured outputs retain field labels and reviewer notes. Use
    **Raw JSON** only when the field view is insufficient.
-8. Score every required evaluator dimension: numeric dimensions use the declared range slider,
-   boolean dimensions use Pass / Fail, and categorical dimensions use pills. Add an optional
-   comment, then **Submit & next**. **Skip** records no score. For the second phase-1 intake path,
-   use **Assign to me** to manually claim an item and move it to In progress. Continue until the
-   toast says **Queue clear**.
+8. Score every required evaluator dimension on the same screen. The segmented **Yes** / **No**
+   control, the numeric slider, and categorical pills sit in a fixed column on the left of each
+   question. On web, `Y` / `N` answer the highlighted question as soon as the item opens, `Enter`
+   submits, and `S` skips. Add an optional note, then **Submit and open next**. **Skip** records
+   no score. For the second phase-1 intake path, use **Assign to me** to manually claim an item
+   and move it to In progress. Continue until the toast says **Queue clear**.
 
 Feedback is a score (`source: "user-feedback"`) fanned out to every ScoreSink. Non-owners who are not admin get 403.
 

package/src/docs/versioned/next/how-to/build-admin-screens.md

@@ -45,6 +45,9 @@ const syncConflicts = useConflicts();
 </AdminProvider>
 ```
 
+The nested `app/admin/[model]/_layout.tsx` stack sets `headerShown: false` too. Without that,
+Expo titles the screen `index` and draws a second back arrow above `AdminScreenPage`.
+
 Render this tree inside `SyncDbProvider client={adminSyncDb}`. If product screens
 also sync the same collection, make `adminSyncDb` a dedicated client with a unique
 store name and `windowCollections` set; do not mix admin rows into the owner/tenant

package/src/docs/versioned/next/how-to/expose-mcp-tools.md

@@ -72,10 +72,10 @@ const tools = getMCPTools(req.user);
 ```
 
 To list the signed-in user's tools in an app, use `useMCPTools` from
-`@terreno/syncdb/app` with the same auth provider you pass to `createSyncDb`:
+`@terreno/syncdb/mcp` with the same auth provider you pass to `createSyncDb`:
 
 ```typescript
-import {useMCPTools} from "@terreno/syncdb/app";
+import {useMCPTools} from "@terreno/syncdb/mcp";
 import {authProvider} from "@/store/syncdb";
 
 const {tools, isLoading, error, refetch} = useMCPTools({authProvider});

package/src/docs/versioned/next/how-to/migrate-rtk-to-syncdb.md

@@ -46,7 +46,14 @@ bunx expo install expo-sqlite expo-secure-store @react-native-async-storage/asyn
 
 `@terreno/rtk` depended on `expo-constants` and `expo-network`, so apps got them transitively.
 `@terreno/syncdb` lists them only as optional peers, and `baseUrl` imports `expo-constants`, so
-install both explicitly.
+install both explicitly. `@better-auth/expo`'s client also imports `expo-network` when the auth
+client is created. A missing `expo-network` doesn't show up in development, because the old copy
+stays in `node_modules` after `@terreno/rtk` is removed, but a clean install (an EAS build) leaves
+it out. The release build then crashes on launch with `Unhandled JS Exception: Error: Cannot find
+module`, thrown from `createAuthClient`, and expo-updates aborts the app. The `.catch` in that
+import doesn't help, because Metro replaces an unresolvable import with a stub that throws
+synchronously. To check an upgraded app, run `rm -rf node_modules && bun install --frozen-lockfile`
+and confirm both packages are installed.
 
 Keep or add the other optional peers of `@terreno/syncdb` for the features the app uses:
 
@@ -391,7 +398,7 @@ Every export of `@terreno/rtk` 57.x and its 58 replacement. Paths: `api` =
 | `installTerrenoDevConsoleLogger` | `installDevConsoleLogger` (app) |
 | `resetTerrenoDevConsoleLoggerForTests` | Removed from the public API |
 | `registerTerrenoDevStore` | Removed. `createSyncDb({debug})` registers the client for terreno-mcp-local (`get_syncdb_state`, `syncdb_snapshot`, `syncdb_action`). |
-| `useMCPTools`, `MCPToolInfo`, `UseMCPToolsOptions`, `UseMCPToolsResult` | Same names (app); pass `authProvider` |
+| `useMCPTools`, `MCPToolInfo`, `UseMCPToolsOptions`, `UseMCPToolsResult` | Same names, from `@terreno/syncdb/mcp`; pass `authProvider` |
 | `useTerrenoChat`, `UseTerrenoChatOptions` | Removed (no consumers). Use `useChat` from `@ai-sdk/react` with a bearer header from `authProvider.getToken()`, or `GPTChat` from `@terreno/ui`. |
 | `@terreno/rtk/buildNumber`: `coerceBuildNumber`, `resolveBuildNumber` | `@terreno/syncdb/buildNumber` (node-only) |
 

package/src/docs/versioned/next/how-to/observe-llm-calls.md

@@ -12,7 +12,7 @@ const priceMap = parseObservabilityPriceMap(process.env.AI_OBS_PRICE_MAP_JSON);
 server.register(new ObservabilityApp({
   accessControl: access,
   aiService,
-  plugins: [localPlugin, langfuseAdapter, otelSink].filter(Boolean),
+  plugins: [localPlugin, otelSink].filter(Boolean),
   control: {
     datasets: process.env.AI_OBS_DATASETS_PRIMARY ?? "local",
     experiments: process.env.AI_OBS_EXPERIMENTS_PRIMARY ?? "local",
@@ -36,9 +36,7 @@ Validate the parsed object before registration: each model needs non-negative nu
 `example-backend/src/utils/observabilityConfig.ts`; malformed values fail startup with the
 variable name.
 
-Keep existing `LangfuseApp` if you already use Langfuse keys. The Langfuse **adapter** must reuse `getLangfuseClient()` and must not start a second OpenTelemetry SDK.
-
-Boot **fails** if `experiments.primary !== datasets.primary`, if `reviewQueue` is `langfuse`, or if a primary’s plugin is missing.
+`local` is the only control primary. Boot **fails** if `experiments.primary !== datasets.primary`, if `reviewQueue` is not `local`, or if a primary’s plugin is missing.
 
 Pass the same `access` object you use for `TerrenoApp` and `AdminApp` so
 `GET /ai/observability/status` permissions match `/admin/config` screen filtering and every
@@ -63,12 +61,11 @@ Terreno does not ship domain-specific folder names or an in-app approval workflo
 
 | Variable | Role |
 | --- | --- |
-| `AI_OBS_PROMPTS_PRIMARY` | `local` or `langfuse` (default `local`) |
+| `AI_OBS_PROMPTS_PRIMARY` | default `local` (the only primary) |
 | `AI_OBS_DATASETS_PRIMARY` | default `local` |
 | `AI_OBS_EXPERIMENTS_PRIMARY` | must equal datasets |
 | `AI_OBS_SAMPLE_RATE` | `0`–`1`, default `0` (live evals off) |
 | `AI_OBS_PRICE_MAP_JSON` | `{ "gemini-2.5-flash": { "inputPerMTok": 0.1, "outputPerMTok": 0.4 } }` |
-| `LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY` | existing Langfuse plugin |
 | `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | optional OTLP trace sink |
 
 ## App generate calls
@@ -90,10 +87,9 @@ Opt out of tracing with `skipTrace: true`. Sink failures are logged and never fa
 ## Admin
 
 Phase 1 admin ships **Prompts**, **Traces**, and **Review**. Datasets, Evaluators, and
-Experiments screens arrive in phase 2. **Open in Langfuse** appears in phase 3 when the
-Langfuse plugin is registered. Review is hidden when the local plugin is off.
+Experiments screens arrive in phase 2. Review is hidden when the local plugin is off.
 
-On **Traces**, **Run multi-stage trace test** (local sink only) runs two schema-backed
+On **Traces**, the header **Run a test trace** action (local sink only) runs two schema-backed
 `generateJsonObject` calls, a deterministic tool stage, and a final schema-backed combine
 call, then opens the nested trace. Each LLM span input includes `outputSchema`. It uses the
 server `aiService` when one is configured; otherwise it sends the admin's saved key as
@@ -112,6 +108,37 @@ that key exists.
    evaluator. On a database that already booted, run Admin → Scripts →
    `seedChatSafetyDataset` as a wet run, or
    `cd example-backend && bun run script seedChatSafetyDataset --wet`.
+
+   The same seed also fills every observability screen, so you do not have to build data
+   by hand:
+
+   | Screen | Seeded data |
+   | --- | --- |
+   | Prompts | `example-summarize` v3 (staging, output capped at 120 characters); `joke-generator` and `chat-assistant` v1 (production) and v2 (staging); judge prompts `eval-judge-correctness` and `eval-judge-helpfulness`; `eval-judge-summary-questions` v1 (short paraphrased wording) and v2 (careful wording, production) |
+   | Evaluators | `correctness-human`, `helpfulness-human`, `schema-assert`, the `correctness` and `helpfulness` LLM judges, and the review-queue question sets `output-quality-questions` and `helpfulness-questions` (three yes/no questions each) |
+   | Datasets | `example-summarize-gold` (8 proofread rows, 2 unproofread), `joke-generator-topics` (6), `chat-assistant-gold` (6), and `summary-review-set` (the 10 newest seeded summaries, every item answered by humans with `summary-review-questions`) |
+   | Experiments | `Summarize v1 vs v2 (gold)`: v1 fails correctness, v2 passes. `Summarize v2 vs v3 (length cap)`: v3 fails `schema-assert` on 2 rows. `Jokes v1 vs v2 (one-liners)` and `Chat assistant v1 vs v2 (brevity)`: v1 fails helpfulness, v2 passes. `Summary judge v1 vs v2 (calibration)`: a judge calibration experiment on `summary-review-set`; v1 agrees with humans 77.5% (fails the 90% gate), v2 agrees 95% (passes) |
+   | Traces | 18 Todos → Summarize traces over the past week (2 errors, 1 sensitive, 2 flagged for the dataset), 6 `joke-generator` traces, and 6 `chat-assistant` turns (4 with a tool span); user-feedback, LLM-judge, and code scores |
+   | Review | 34 items: 11 pending, 1 in progress, 21 done, 1 skipped, including the batches `Recent summaries — batch 1` (8 of 10 answered) and `summary-review-set — human answers` (10 of 10) |
+
+   Seeded experiments run through the real experiment runner with a deterministic
+   `seed-fixture` client, so they need no API key. Seeded traces use session IDs that
+   start with `seed-obs-`. Production stays on v1 for every product prompt, so promoting v2 from
+   an experiment changes what the app sends.
+
+   **New experiment** (pipeline **Compare prompt versions**) keeps **Continue** disabled and
+   shows "Select a dataset." until you choose a dataset; the field reads "Choose a dataset"
+   until then. **Experiment results** shows one verdict banner, then one **Gates** table: a
+   row per gate (evaluator and dimension in plain words), **Needs** with the threshold as a
+   symbol (for example "≥ 0.8"), and one column per version with the measured value and Pass
+   or Fail. The last **Result** row says whether each version passes (for example "Fails 1 of
+   3 gates"). When no version passes, the screen explains why, for example "v1 fails 1 gate:
+   correctness: correct ≥ 0.8 (got 0.75) — can't promote v1.", and the header **Promote vN to
+   production** stays disabled.
+
+   On a database that already booted, run
+   Admin → Scripts → `seedAiObservabilityDemo`, or
+   `cd example-backend && bun run script seedAiObservabilityDemo --wet`.
 3. Start `bun run backend:dev` and `bun run frontend:web`, then sign in as the seeded
    admin.
 4. Open **AI Observability → Prompts** to inspect or save a new immutable version. Move
@@ -137,6 +164,13 @@ that key exists.
 6. Open its `example-summarize` trace, send it to Review, score **correct**, and submit
    until the queue reports **Queue clear**.
 
+The example chat reads two more registry prompts. `addGptRoutes` takes
+`systemPromptName: "chat-assistant"`, so each chat turn prepends that prompt's `production`
+version to the system prompt and its trace records the version. The chat's `generate_joke`
+tool calls `generateText` with `promptName: "joke-generator"` and falls back to a built-in
+prompt when the registry has no `production` version. Move either label in **Prompts** and
+the next chat message uses the new version.
+
 The example app also calls this route from the UI: **Todos → Summarize** sends the user's todo
 list, so each press produces a trace with the production prompt reference, `userId`, and a
 per-app-run `x-ai-session-id`. Without server credentials the screen forwards the Gemini key
@@ -145,7 +179,7 @@ saved in **Profile** as `x-ai-api-key`; otherwise the route answers **503**.
 The route sends `text` as the user prompt. The labelled registry version supplies the
 system prompt and the route uses `TemperaturePresets.LOW`.
 
-The example registers the local plugin even when no Langfuse keys exist. Set
+The example always registers the local plugin. Set
 `AI_OBS_PRICE_MAP_JSON` to calculate USD cost; unlisted models show tokens with no cost,
 never `$0`.
 
@@ -182,9 +216,66 @@ curl -X POST "$API/ai/observability/prompts/example-summarize/labels" \
 The example backend registers `requestAiServiceFactory` but no server `aiService`, so status reports
 `playgroundAi.source: "request-key"`. In **AI Observability → Prompts → Playground**, save a Gemini
 API key on **Profile** first; the admin UI blocks **Run once** with that hint until the key loads
-from storage and is sent as `x-ai-api-key`. **Traces → Run multi-stage trace test** uses the same
+from storage and is sent as `x-ai-api-key`. **Traces → Run a test trace** uses the same
 saved key and is blocked the same way.
 
+## Share instructions across prompts
+
+Put the instructions that a set of prompts repeats in one prompt, then reference it from each
+prompt with `{{> name}}`.
+
+1. Create the shared prompt and pin `production`:
+
+   ```bash
+   curl -X POST "$API/ai/observability/prompts" \
+     -H "Authorization: Bearer $ADMIN_TOKEN" -H "Content-Type: application/json" \
+     -d '{"folder":"shared","name":"base-rules","template":"Answer in plain language for {{audience}}.","type":"text"}'
+   curl -X POST "$API/ai/observability/prompts/base-rules/labels" \
+     -H "Authorization: Bearer $ADMIN_TOKEN" -H "Content-Type: application/json" \
+     -d '{"label":"production","version":1}'
+   ```
+
+2. Reference it from each prompt's `system` or `template`. In the prompt editor, open the version and choose **Insert prompt reference**. That opens a modal where you pick the shared prompt. That appends:
+
+   ```text
+   {{> base-rules}}
+   Triage the intake form.
+   ```
+
+   - `{{> base-rules}}` follows the `production` label.
+   - `{{> base-rules label=staging}}` follows another label.
+   - `{{> base-rules version=3}}` pins one version. Use `label=` or `version=`, not both.
+   - A shared prompt can reference another one, up to 5 levels deep.
+
+3. Promote a new `base-rules` version. Every prompt that references it by label uses the new text
+   at once, with no new version of its own.
+
+Every reference must resolve when you save, so a saved prompt never fails at call time. Creating a
+prompt, saving a version, and moving a label return 400:
+
+- `Invalid prompt reference` when a tag is malformed, names an unknown prompt or version, or names
+  a label the shared prompt does not have yet. Promote the shared prompt before you reference it.
+- `Prompt reference cycle` when the save or label move would make prompts include each other. A
+  new version counts as `latest` for this check.
+- `Prompt references nested too deeply` when the change would nest references more than 5 levels,
+  in the saved prompt or in any prompt that includes it.
+
+The prompt editor keeps the tag; `PromptRegistry.get`, the playground, experiments, and judges use
+the resolved text. `{{variables}}` in a shared prompt are filled like the prompt's own.
+
+The trace stays named after the prompt you called. Its `prompts` list holds that prompt first,
+then one entry per shared prompt with `includedBy` set to the prompt whose tag pulled it in:
+
+```json
+[
+  {"label": "production", "name": "triage-intake", "version": 3},
+  {"includedBy": "triage-intake", "label": "production", "name": "base-rules", "version": 1}
+]
+```
+
+So `?prompt=base-rules` on the trace list finds every call that used the shared text, and
+`?prompt=triage-intake` still finds only that prompt's calls.
+
 ## Review a trace
 
 1. Use the seeded human `correctness-human` evaluator. If you skipped `bun run backend:seed`,
@@ -195,3 +286,60 @@ saved key and is blocked the same way.
 4. Open an item: `GET /ai/observability/review/:id` returns evaluator dimensions plus `given` / `wrote` panels (variable labels when present, otherwise raw keys).
 5. `POST /ai/observability/review/:id` with `action: "submit"` writes dimension-keyed scores through every ScoreSink and marks the item `done` (it leaves the pending list). `skip` and `assign` move status to `skipped` / `in_progress`.
 
+The seeded queue uses `output-quality-questions` and `helpfulness-questions`: three required yes/no questions on one screen. Re-running `bun run backend:seed` updates those question sets in place and points any queue item still using `correctness-human` or `helpfulness-human` at the matching three-question set. On the item screen, **What the AI was given** and **What the AI wrote** are headings above equal columns. Each column's card holds the field labels (for example source text) and the values. The output card uses the `ai` surface. Segmented Yes / No (and a numeric or categorical control) sit in a fixed left column. On web, `Y` / `N` answer the highlighted question as soon as the item opens, including when a Yes/No segment is focused, `Enter` submits, and `S` skips.
+
+## Check that an LLM judge agrees with people
+
+Use a review batch: a fixed set of traces (one input and one output each) that reviewers answer
+the same list of questions for. Then run LLM judges over the same traces and compare.
+
+1. Create a **question set**: a `human` evaluator with one `boolean` dimension per question. Put
+   the reviewer-facing wording in each dimension's `question`, for example
+   `{key: "covers_every_item", dataType: "boolean", question: "Does it mention every todo?"}`.
+   The seed installs `summary-review-questions` with four questions.
+2. Open **AI Observability → Review batches** (or **Review batches** in the Review queue header),
+   pick the question set and (optionally) a prompt, and press **Save batch** on the New batch
+   card. That grabs the 10 newest successful traces. API: `POST /ai/observability/review/batches` with
+   `{evaluatorId, promptName?, limit?: 10, name?}`. It takes the newest successful traces with an
+   output that are not already in a batch for that question set, so pressing it again gives the
+   next 10. Pass `traceIds` instead to pick traces by hand.
+3. Reviewers press **Answer N remaining**. Each item shows **What the AI was given** and **What
+   the AI wrote** as headings above equal columns, and every question as a segmented **Yes** /
+   **No** in a fixed left column. On web, `Y` / `N` answer the highlighted question as soon as the
+   item opens, including when a Yes/No segment is focused, `Enter` submits and
+   opens the next item in the batch (**Submit and open next**), and `S` skips. Items someone
+   assigned to themselves (in progress) stay in the batch's next/previous walk until they are
+   submitted or skipped.
+4. Create an `llm-judge` evaluator whose dimension keys match the question keys and whose judge
+   prompt returns those keys. On the batch, choose the judge and press **Run on N items**
+   (`POST /ai/observability/review/batches/:id/judgeRuns` with `{evaluatorId}`). The judge runs
+   on every batch trace through `ObservabilityApp.aiService` (or the caller's `x-ai-api-key`).
+5. Read **Which judge matches people?**: overall and per-question agreement for each judge,
+   counted only on items people answered and questions the judge also answers. 90% or more reads
+   **Matches**, 75–89% **Close**, and under 75% **Off**. Click a judge to list where it disagrees, then open the item to see why.
+   Re-running a judge after editing its prompt replaces that judge's previous result.
+
+The seed adds the batch `Recent summaries — batch 1` (8 of 10 answered) with two judges run:
+`summary-questions-judge` agrees 97% and `summary-questions-judge-quick` agrees 78%.
+
+## Calibrate a judge prompt against a human-judged dataset
+
+A batch answers "does this judge match people on these traces?". To improve a judge prompt
+version by version, judge a dataset once and reuse those answers:
+
+1. Build a dataset of traces (**Add to dataset** on traces, or
+   `POST /ai/observability/traces/add-to-dataset`).
+2. Create a review batch from it: `POST /ai/observability/review/batches` with
+   `{evaluatorId: <question set>, datasetId}`. Only items without a current answer are queued.
+   Every submitted answer is saved on its dataset item, and the dataset's `judgments` show
+   answered, current, and stale counts per question set.
+3. Save 2–3 versions of the judge prompt, then create a judge experiment:
+   `POST /ai/observability/experiments` with
+   `{kind: "judge", name, datasetId, questionSetId, judgeEvaluatorId, versions: [1, 2], targetAgreement: 0.9}`.
+   Each version answers every human-answered item; `results.agreement` ranks the versions and
+   each version gets an `agreement` gate.
+4. Promote the version that passes (`POST /ai/observability/experiments/:id/promote`); the
+   judge evaluator then shows its `calibration`. Use that judge in prompt experiments.
+
+Rewording a question makes earlier answers stale: re-run step 2 to re-queue those items.
+

package/src/docs/versioned/next/how-to/run-tests-locally.md

@@ -229,6 +229,15 @@ For `test:integration`, start example-backend with `ADMIN_SPA_ENABLED=true`,
 `AUTH_PROVIDER=better-auth`, and, for the MCP OAuth consent spec, `MCP_OAUTH_ENABLED=true` and
 `PUBLIC_API_URL=http://localhost:4000`. Without the last two the consent spec is skipped.
 
+Job `admin-spa-image-smoke` builds `example-backend/Dockerfile` and checks the image's
+server-rendered admin SPA. Locally, with Docker, or against any running backend:
+
+```bash
+docker build --file example-backend/Dockerfile --tag terreno-example-backend:admin-spa-smoke .
+# run it with ADMIN_SPA_ENABLED=true and a Mongo replica set (see the job), then:
+ADMIN_SPA_SMOKE_BRAND="Terreno Example" bash scripts/admin-spa-image-smoke.sh http://localhost:4000
+```
+
 ## Maestro web e2e
 
 Install CLI if missing: `curl -fsSL "https://get.maestro.mobile.dev" | bash`, then put