@terreno/mcp 57.20.0 → 57.22.0

package/src/docs/versioned/next/how-to/install-agent-skills.md

@@ -19,12 +19,12 @@ takes a plugin skill's command from the frontmatter `name`.
 
 | Host | Plugin | Invoke Grow | Scan plugin | Invoke Aim |
 | --- | --- | --- | --- | --- |
-| Cursor | `terreno-planning` | `/terreno-1-grow` | `terreno-scan` | `/terreno-scan-1-aim` |
-| Codex | `terreno-planning` | `$terreno-1-grow` | `terreno-scan` | `$terreno-scan-1-aim` |
-| Claude Code | `terreno` | `/terreno:1-grow` | `terreno-scan` | `/terreno-scan:1-aim` |
+| Cursor | `terreno-planning` | `/terreno-1-grow` | `terreno-loop` | `/terreno-loop-1-aim` |
+| Codex | `terreno-planning` | `$terreno-1-grow` | `terreno-loop` | `$terreno-loop-1-aim` |
+| Claude Code | `terreno` | `/terreno:1-grow` | `terreno-loop` | `/terreno-loop:1-aim` |
 
-`terreno-scan` depends on the lifecycle plugin: Plot hands each slice to Grow. Its
-resident loop (`/terreno-scan:loop`, `/terreno-scan-loop`, `$terreno-scan-loop`) keeps a
+`terreno-loop` depends on the lifecycle plugin: Plot hands each slice to Grow. Its
+resident loop (`/terreno-loop:run`, `/terreno-loop-run`, `$terreno-loop-run`) keeps a
 goal running and heartbeats over the PRs it opens. See the
 [scan plugin reference](../reference/scan-plugin.md).
 
@@ -37,7 +37,7 @@ Install `terreno-planning` from [`.cursor-plugin/marketplace.json`](https://gith
 ```text
 codex plugin marketplace add TerrenoLabs/terreno
 codex plugin install terreno-planning --source terreno-plugins
-codex plugin install terreno-scan --source terreno-plugins
+codex plugin install terreno-loop --source terreno-plugins
 $terreno-1-grow
 ```
 
@@ -51,7 +51,7 @@ Codex installs the canonical plugin at
 ```text
 /plugin marketplace add TerrenoLabs/terreno
 /plugin install terreno@terreno-plugins
-/plugin install terreno-scan@terreno-plugins
+/plugin install terreno-loop@terreno-plugins
 /terreno:1-grow
 ```
 
@@ -59,7 +59,29 @@ Marketplace: [`.claude-plugin/marketplace.json`](https://github.com/TerrenoLabs/
 Claude Code stages, app skills, and agents come from the generated copy at
 [`plugins/terreno-claude/`](https://github.com/TerrenoLabs/terreno/tree/master/plugins/terreno-claude);
 scan stages come from
-[`plugins/terreno-scan-claude/`](https://github.com/TerrenoLabs/terreno/tree/master/plugins/terreno-scan-claude).
+[`plugins/terreno-loop-claude/`](https://github.com/TerrenoLabs/terreno/tree/master/plugins/terreno-loop-claude).
+
+#### Pick a channel
+
+Register one channel. Each one provides plugins named `terreno` and `terreno-loop`, so
+two channels give duplicate `/terreno:*` skills.
+
+| Channel | Tracks | Add the marketplace | Install |
+| --- | --- | --- | --- |
+| Master | Every merge to `master` | `/plugin marketplace add TerrenoLabs/terreno` | `/plugin install terreno@terreno-plugins` |
+| Pinned release | One release tag, frozen | `/plugin marketplace add TerrenoLabs/terreno#<version>` | `/plugin install terreno@terreno-plugins` |
+| Latest release | The newest stable release, from npm | `/plugin marketplace add https://raw.githubusercontent.com/TerrenoLabs/terreno/master/plugins/claude-release/.claude-plugin/marketplace.json` | `/plugin install terreno@terreno-releases` |
+
+The release channel installs `@terreno/agent-plugin` and `@terreno/agent-plugin-loop`
+from npm. Both are published in lockstep with every other `@terreno/*` package, and
+their plugin version is the Terreno release. To move a pinned install to another release, remove
+the marketplace and add it again at the new tag. Only tags from 57.21.0 on carry
+distinct plugin versions. Older tags all report `2.12.0`, so switching between two of
+them can be skipped as already up to date.
+
+Claude Code does not auto-update third-party marketplaces by default. Run
+`/plugin update terreno@<marketplace>`, or turn on auto-update for the marketplace in
+`/plugin` → **Marketplaces**.
 
 ## What you get
 
@@ -67,10 +89,10 @@ scan stages come from
 | --- | --- |
 | Lifecycle | `terreno-1-grow` … `terreno-5-taste`, `terreno-pick-roast-loop`, `terreno-planning-loop`, `terreno-taste-sweep` |
 | Terreno apps | backend, UI, admin interfaces, data, schema, SDK, prompts, upgrades, deployment |
-| Testing | `terreno-test-setup` (bootstrap backend, frontend, mobile), `terreno-backend-tests`, `terreno-frontend-tests`, `terreno-mobile-tests`, `backend-test-env` |
-| Docs | `update-docs`, `update-agent-docs`, architecture skills |
+| Testing | `terreno-test-setup` (bootstrap backend, frontend, mobile), `terreno-backend-tests`, `terreno-frontend-tests`, `terreno-mobile-tests`, `backend-test-env`, `backfill-regression` (propose manual regression cases for approval) |
+| Docs | `update-docs`, `backfill-user-guide` (draft end-user guide pages for approval), `update-agent-docs`, architecture skills |
 | GitHub | issues, review, UI verification, PR screenshots (`attach-pr-media`), release, dependency updates |
-| Code scans | `terreno-scan-1-aim` … `terreno-scan-5-track`, `terreno-scan-campaign`, `terreno-scan-loop` |
+| Code scans | `terreno-loop-1-aim` … `terreno-loop-5-track`, `terreno-loop-campaign`, `terreno-loop-run` |
 | Expo and native | `track-upstream-expo`, `upgrading-expo`, `store-assets`, deployment / EAS skills |
 | Plugin agents | `pre-commit`, `ui-verifier` |
 
@@ -84,12 +106,12 @@ Canonical sources:
 
 1. `plugins/terreno-planning/skills/` — lifecycle and reusable Terreno app workflows
 2. `plugins/terreno-planning/agents/` — reusable verification agents
-3. `plugins/terreno-scan/skills/` — scan stages and the campaign loop
+3. `plugins/terreno-loop/skills/` — scan stages and the campaign loop
 4. `.rulesync/skills/` — repository-only and optional Expo skills (`bun run rules` generates agent copies)
 5. `<package>/.ai/skills/` — package/MCP-specific copies; not installable overlays
 
 `plugins/terreno-claude/` is generated from sources 1 and 2, and
-`plugins/terreno-scan-claude/` from source 3, both with shortened skill names.
+`plugins/terreno-loop-claude/` from source 3, both with shortened skill names.
 Codex uses the canonical plugin plus committed `.codex-plugin/plugin.json` and
 `.agents/plugins/marketplace.json` — do not generate a third plugin tree.
 

package/src/docs/versioned/next/how-to/manage-mcp-in-admin.md

@@ -0,0 +1,79 @@
+# Manage MCP from the admin panel
+
+Add admin screens for an app's MCP server: service tokens, OAuth clients, OAuth connections, a request log, and a tool registry with on/off switches. Add a one-click "sign out of MCP everywhere" script for a user.
+
+## Prerequisites
+
+- A `TerrenoApp` backend with at least one MCP tool ([Expose MCP tools](expose-mcp-tools.md))
+- `AdminApp` from `@terreno/admin-backend` and an admin UI (`@terreno/admin-spa` or `@terreno/admin-frontend`)
+- Service tokens or MCP OAuth turned on if you want those screens to show data ([service tokens](connect-mcp-service-token.md), [MCP OAuth](enable-mcp-oauth.md))
+
+## 1. Turn on the request log and tool registry
+
+Both are off by default.
+
+```typescript
+new TerrenoApp({
+  userModel: User,
+  mcpRequestLog: {enabled: true, retentionDays: 30},
+  mcpToolSettings: true,
+  // mcpServiceTokens, mcpOAuth, ...
+});
+```
+
+- The request log stores one row per `tools/call`: tool, user, credential type, service token or OAuth client, duration, outcome, and the error class when the tool threw. It never stores tool arguments, results, or error messages, so it is safe for apps that handle personal data. Rows are deleted after `retentionDays`.
+- The tool registry records every tool at startup. An admin can switch a tool off; it then disappears from `tools/list` and `tools/call` rejects it, for every user and client. The process that saved the switch applies it on the next request; other server instances apply it within 5 seconds. If the settings read fails, the server keeps the last switches it read, and refuses MCP requests when it has none.
+
+## 2. Register the screens and script
+
+```typescript
+import {AdminApp, mcpAdminModels, mcpAdminScripts} from "@terreno/admin-backend";
+
+new AdminApp({
+  models: [...appAdminModels, ...mcpAdminModels({requestLog: true, toolSettings: true})],
+  scripts: [...appScripts, ...mcpAdminScripts()],
+});
+```
+
+Set `requestLog` and `toolSettings` to match the `TerrenoApp` options from step 1. Both default to `false`, like the server options: a request log screen with logging off stays empty, and tool switches with `mcpToolSettings` off do nothing.
+
+With RBAC (`AdminApp({accessControl})`), add the statements and grant them to the roles that run MCP:
+
+```typescript
+import {mcpAdminStatements} from "@terreno/admin-backend";
+
+export const appStatements = {...terrenoStatements, ...mcpAdminStatements, /* ... */};
+```
+
+## 3. Use the screens
+
+The admin sidebar shows an **MCP** group:
+
+| Screen | Use it to |
+| --- | --- |
+| MCP service tokens | See each user's `mcp_` keys and when they were last used. Delete revokes a key. |
+| MCP OAuth clients | See registered connectors (Claude.ai, ChatGPT, headless clients) and their redirect URLs. Delete removes the client and revokes everything issued to it. |
+| MCP OAuth connections | See who approved which client and when. Each row is one token pair; rows with the same `familyId` are one connection. Delete revokes that connection. |
+| MCP request log | See recent tool calls by tool, user, client, outcome, and duration. Filter by user or client to see which clients a user has connected recently. |
+| MCP tools | Switch a tool off or on. Only `enabled` can be changed. `registered: false` means the running server no longer has that tool. |
+
+MCP over HTTP is stateless, so there is no list of open connections. "Who is connected" means recent activity in the request log plus active rows in the token and connection screens.
+
+## 4. Sign a user out of MCP everywhere
+
+Run the `mcp-sign-out-everywhere` script from **Scripts** with `user` set to the user's id or email. Run it as a dry run first: it reports how many OAuth connections and service tokens it would revoke. The wet run revokes them all; the user's clients must reconnect.
+
+To end every MCP credential and every app session at once, reset the user's password or revoke their sessions instead. That bumps `tokenEpoch`, which ends OAuth grants and service tokens on their next use.
+
+## Check it
+
+1. Call a tool with a service token, then open **MCP request log**: a row shows the tool, `outcome: ok`, and the token.
+2. Switch that tool off in **MCP tools** and run `tools/list` again: the tool is gone.
+3. Delete the token's row in **MCP service tokens** and call the tool again: the token no longer authenticates (401 with MCP OAuth on, otherwise the tool refuses an anonymous caller).
+
+## Related
+
+- [Expose an MCP server safely](expose-mcp-server-safely.md)
+- [Enable MCP OAuth](enable-mcp-oauth.md)
+- [admin-backend reference: MCP screens](../reference/admin-backend.md#mcp-screens)
+- [api reference: MCP request log and tool settings](../reference/api.md#mcp-request-log-and-tool-settings)

package/src/docs/versioned/next/how-to/observe-llm-calls.md

@@ -12,7 +12,7 @@ const priceMap = parseObservabilityPriceMap(process.env.AI_OBS_PRICE_MAP_JSON);
 server.register(new ObservabilityApp({
   accessControl: access,
   aiService,
-  plugins: [localPlugin, langfuseAdapter, otelSink].filter(Boolean),
+  plugins: [localPlugin, otelSink].filter(Boolean),
   control: {
     datasets: process.env.AI_OBS_DATASETS_PRIMARY ?? "local",
     experiments: process.env.AI_OBS_EXPERIMENTS_PRIMARY ?? "local",
@@ -36,9 +36,7 @@ Validate the parsed object before registration: each model needs non-negative nu
 `example-backend/src/utils/observabilityConfig.ts`; malformed values fail startup with the
 variable name.
 
-Keep existing `LangfuseApp` if you already use Langfuse keys. The Langfuse **adapter** must reuse `getLangfuseClient()` and must not start a second OpenTelemetry SDK.
-
-Boot **fails** if `experiments.primary !== datasets.primary`, if `reviewQueue` is `langfuse`, or if a primary’s plugin is missing.
+`local` is the only control primary. Boot **fails** if `experiments.primary !== datasets.primary`, if `reviewQueue` is not `local`, or if a primary’s plugin is missing.
 
 Pass the same `access` object you use for `TerrenoApp` and `AdminApp` so
 `GET /ai/observability/status` permissions match `/admin/config` screen filtering and every
@@ -63,12 +61,11 @@ Terreno does not ship domain-specific folder names or an in-app approval workflo
 
 | Variable | Role |
 | --- | --- |
-| `AI_OBS_PROMPTS_PRIMARY` | `local` or `langfuse` (default `local`) |
+| `AI_OBS_PROMPTS_PRIMARY` | default `local` (the only primary) |
 | `AI_OBS_DATASETS_PRIMARY` | default `local` |
 | `AI_OBS_EXPERIMENTS_PRIMARY` | must equal datasets |
 | `AI_OBS_SAMPLE_RATE` | `0`–`1`, default `0` (live evals off) |
 | `AI_OBS_PRICE_MAP_JSON` | `{ "gemini-2.5-flash": { "inputPerMTok": 0.1, "outputPerMTok": 0.4 } }` |
-| `LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY` | existing Langfuse plugin |
 | `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | optional OTLP trace sink |
 
 ## App generate calls
@@ -90,10 +87,9 @@ Opt out of tracing with `skipTrace: true`. Sink failures are logged and never fa
 ## Admin
 
 Phase 1 admin ships **Prompts**, **Traces**, and **Review**. Datasets, Evaluators, and
-Experiments screens arrive in phase 2. **Open in Langfuse** appears in phase 3 when the
-Langfuse plugin is registered. Review is hidden when the local plugin is off.
+Experiments screens arrive in phase 2. Review is hidden when the local plugin is off.
 
-On **Traces**, **Run multi-stage trace test** (local sink only) runs two schema-backed
+On **Traces**, the header **Run a test trace** action (local sink only) runs two schema-backed
 `generateJsonObject` calls, a deterministic tool stage, and a final schema-backed combine
 call, then opens the nested trace. Each LLM span input includes `outputSchema`. It uses the
 server `aiService` when one is configured; otherwise it sends the admin's saved key as
@@ -112,6 +108,37 @@ that key exists.
    evaluator. On a database that already booted, run Admin → Scripts →
    `seedChatSafetyDataset` as a wet run, or
    `cd example-backend && bun run script seedChatSafetyDataset --wet`.
+
+   The same seed also fills every observability screen, so you do not have to build data
+   by hand:
+
+   | Screen | Seeded data |
+   | --- | --- |
+   | Prompts | `example-summarize` v3 (staging, output capped at 120 characters); `joke-generator` and `chat-assistant` v1 (production) and v2 (staging); judge prompts `eval-judge-correctness` and `eval-judge-helpfulness`; `eval-judge-summary-questions` v1 (short paraphrased wording) and v2 (careful wording, production) |
+   | Evaluators | `correctness-human`, `helpfulness-human`, `schema-assert`, the `correctness` and `helpfulness` LLM judges, and the review-queue question sets `output-quality-questions` and `helpfulness-questions` (three yes/no questions each) |
+   | Datasets | `example-summarize-gold` (8 proofread rows, 2 unproofread), `joke-generator-topics` (6), `chat-assistant-gold` (6), and `summary-review-set` (the 10 newest seeded summaries, every item answered by humans with `summary-review-questions`) |
+   | Experiments | `Summarize v1 vs v2 (gold)`: v1 fails correctness, v2 passes. `Summarize v2 vs v3 (length cap)`: v3 fails `schema-assert` on 2 rows. `Jokes v1 vs v2 (one-liners)` and `Chat assistant v1 vs v2 (brevity)`: v1 fails helpfulness, v2 passes. `Summary judge v1 vs v2 (calibration)`: a judge calibration experiment on `summary-review-set`; v1 agrees with humans 77.5% (fails the 90% gate), v2 agrees 95% (passes) |
+   | Traces | 18 Todos → Summarize traces over the past week (2 errors, 1 sensitive, 2 flagged for the dataset), 6 `joke-generator` traces, and 6 `chat-assistant` turns (4 with a tool span); user-feedback, LLM-judge, and code scores |
+   | Review | 34 items: 11 pending, 1 in progress, 21 done, 1 skipped, including the batches `Recent summaries — batch 1` (8 of 10 answered) and `summary-review-set — human answers` (10 of 10) |
+
+   Seeded experiments run through the real experiment runner with a deterministic
+   `seed-fixture` client, so they need no API key. Seeded traces use session IDs that
+   start with `seed-obs-`. Production stays on v1 for every product prompt, so promoting v2 from
+   an experiment changes what the app sends.
+
+   **New experiment** (pipeline **Compare prompt versions**) keeps **Continue** disabled and
+   shows "Select a dataset." until you choose a dataset; the field reads "Choose a dataset"
+   until then. **Experiment results** shows one verdict banner, then one **Gates** table: a
+   row per gate (evaluator and dimension in plain words), **Needs** with the threshold as a
+   symbol (for example "≥ 0.8"), and one column per version with the measured value and Pass
+   or Fail. The last **Result** row says whether each version passes (for example "Fails 1 of
+   3 gates"). When no version passes, the screen explains why, for example "v1 fails 1 gate:
+   correctness: correct ≥ 0.8 (got 0.75) — can't promote v1.", and the header **Promote vN to
+   production** stays disabled.
+
+   On a database that already booted, run
+   Admin → Scripts → `seedAiObservabilityDemo`, or
+   `cd example-backend && bun run script seedAiObservabilityDemo --wet`.
 3. Start `bun run backend:dev` and `bun run frontend:web`, then sign in as the seeded
    admin.
 4. Open **AI Observability → Prompts** to inspect or save a new immutable version. Move
@@ -137,6 +164,13 @@ that key exists.
 6. Open its `example-summarize` trace, send it to Review, score **correct**, and submit
    until the queue reports **Queue clear**.
 
+The example chat reads two more registry prompts. `addGptRoutes` takes
+`systemPromptName: "chat-assistant"`, so each chat turn prepends that prompt's `production`
+version to the system prompt and its trace records the version. The chat's `generate_joke`
+tool calls `generateText` with `promptName: "joke-generator"` and falls back to a built-in
+prompt when the registry has no `production` version. Move either label in **Prompts** and
+the next chat message uses the new version.
+
 The example app also calls this route from the UI: **Todos → Summarize** sends the user's todo
 list, so each press produces a trace with the production prompt reference, `userId`, and a
 per-app-run `x-ai-session-id`. Without server credentials the screen forwards the Gemini key
@@ -145,7 +179,7 @@ saved in **Profile** as `x-ai-api-key`; otherwise the route answers **503**.
 The route sends `text` as the user prompt. The labelled registry version supplies the
 system prompt and the route uses `TemperaturePresets.LOW`.
 
-The example registers the local plugin even when no Langfuse keys exist. Set
+The example always registers the local plugin. Set
 `AI_OBS_PRICE_MAP_JSON` to calculate USD cost; unlisted models show tokens with no cost,
 never `$0`.
 
@@ -182,9 +216,66 @@ curl -X POST "$API/ai/observability/prompts/example-summarize/labels" \
 The example backend registers `requestAiServiceFactory` but no server `aiService`, so status reports
 `playgroundAi.source: "request-key"`. In **AI Observability → Prompts → Playground**, save a Gemini
 API key on **Profile** first; the admin UI blocks **Run once** with that hint until the key loads
-from storage and is sent as `x-ai-api-key`. **Traces → Run multi-stage trace test** uses the same
+from storage and is sent as `x-ai-api-key`. **Traces → Run a test trace** uses the same
 saved key and is blocked the same way.
 
+## Share instructions across prompts
+
+Put the instructions that a set of prompts repeats in one prompt, then reference it from each
+prompt with `{{> name}}`.
+
+1. Create the shared prompt and pin `production`:
+
+   ```bash
+   curl -X POST "$API/ai/observability/prompts" \
+     -H "Authorization: Bearer $ADMIN_TOKEN" -H "Content-Type: application/json" \
+     -d '{"folder":"shared","name":"base-rules","template":"Answer in plain language for {{audience}}.","type":"text"}'
+   curl -X POST "$API/ai/observability/prompts/base-rules/labels" \
+     -H "Authorization: Bearer $ADMIN_TOKEN" -H "Content-Type: application/json" \
+     -d '{"label":"production","version":1}'
+   ```
+
+2. Reference it from each prompt's `system` or `template`. In the prompt editor, open the version and choose **Insert prompt reference**. That opens a modal where you pick the shared prompt. That appends:
+
+   ```text
+   {{> base-rules}}
+   Triage the intake form.
+   ```
+
+   - `{{> base-rules}}` follows the `production` label.
+   - `{{> base-rules label=staging}}` follows another label.
+   - `{{> base-rules version=3}}` pins one version. Use `label=` or `version=`, not both.
+   - A shared prompt can reference another one, up to 5 levels deep.
+
+3. Promote a new `base-rules` version. Every prompt that references it by label uses the new text
+   at once, with no new version of its own.
+
+Every reference must resolve when you save, so a saved prompt never fails at call time. Creating a
+prompt, saving a version, and moving a label return 400:
+
+- `Invalid prompt reference` when a tag is malformed, names an unknown prompt or version, or names
+  a label the shared prompt does not have yet. Promote the shared prompt before you reference it.
+- `Prompt reference cycle` when the save or label move would make prompts include each other. A
+  new version counts as `latest` for this check.
+- `Prompt references nested too deeply` when the change would nest references more than 5 levels,
+  in the saved prompt or in any prompt that includes it.
+
+The prompt editor keeps the tag; `PromptRegistry.get`, the playground, experiments, and judges use
+the resolved text. `{{variables}}` in a shared prompt are filled like the prompt's own.
+
+The trace stays named after the prompt you called. Its `prompts` list holds that prompt first,
+then one entry per shared prompt with `includedBy` set to the prompt whose tag pulled it in:
+
+```json
+[
+  {"label": "production", "name": "triage-intake", "version": 3},
+  {"includedBy": "triage-intake", "label": "production", "name": "base-rules", "version": 1}
+]
+```
+
+So `?prompt=base-rules` on the trace list finds every call that used the shared text, and
+`?prompt=triage-intake` still finds only that prompt's calls.
+
 ## Review a trace
 
 1. Use the seeded human `correctness-human` evaluator. If you skipped `bun run backend:seed`,
@@ -195,3 +286,60 @@ saved key and is blocked the same way.
 4. Open an item: `GET /ai/observability/review/:id` returns evaluator dimensions plus `given` / `wrote` panels (variable labels when present, otherwise raw keys).
 5. `POST /ai/observability/review/:id` with `action: "submit"` writes dimension-keyed scores through every ScoreSink and marks the item `done` (it leaves the pending list). `skip` and `assign` move status to `skipped` / `in_progress`.
 
+The seeded queue uses `output-quality-questions` and `helpfulness-questions`: three required yes/no questions on one screen. Re-running `bun run backend:seed` updates those question sets in place and points any queue item still using `correctness-human` or `helpfulness-human` at the matching three-question set. On the item screen, **What the AI was given** and **What the AI wrote** are headings above equal columns. Each column's card holds the field labels (for example source text) and the values. The output card uses the `ai` surface. Segmented Yes / No (and a numeric or categorical control) sit in a fixed left column. On web, `Y` / `N` answer the highlighted question as soon as the item opens, including when a Yes/No segment is focused, `Enter` submits, and `S` skips.
+
+## Check that an LLM judge agrees with people
+
+Use a review batch: a fixed set of traces (one input and one output each) that reviewers answer
+the same list of questions for. Then run LLM judges over the same traces and compare.
+
+1. Create a **question set**: a `human` evaluator with one `boolean` dimension per question. Put
+   the reviewer-facing wording in each dimension's `question`, for example
+   `{key: "covers_every_item", dataType: "boolean", question: "Does it mention every todo?"}`.
+   The seed installs `summary-review-questions` with four questions.
+2. Open **AI Observability → Review batches** (or **Review batches** in the Review queue header),
+   pick the question set and (optionally) a prompt, and press **Save batch** on the New batch
+   card. That grabs the 10 newest successful traces. API: `POST /ai/observability/review/batches` with
+   `{evaluatorId, promptName?, limit?: 10, name?}`. It takes the newest successful traces with an
+   output that are not already in a batch for that question set, so pressing it again gives the
+   next 10. Pass `traceIds` instead to pick traces by hand.
+3. Reviewers press **Answer N remaining**. Each item shows **What the AI was given** and **What
+   the AI wrote** as headings above equal columns, and every question as a segmented **Yes** /
+   **No** in a fixed left column. On web, `Y` / `N` answer the highlighted question as soon as the
+   item opens, including when a Yes/No segment is focused, `Enter` submits and
+   opens the next item in the batch (**Submit and open next**), and `S` skips. Items someone
+   assigned to themselves (in progress) stay in the batch's next/previous walk until they are
+   submitted or skipped.
+4. Create an `llm-judge` evaluator whose dimension keys match the question keys and whose judge
+   prompt returns those keys. On the batch, choose the judge and press **Run on N items**
+   (`POST /ai/observability/review/batches/:id/judgeRuns` with `{evaluatorId}`). The judge runs
+   on every batch trace through `ObservabilityApp.aiService` (or the caller's `x-ai-api-key`).
+5. Read **Which judge matches people?**: overall and per-question agreement for each judge,
+   counted only on items people answered and questions the judge also answers. 90% or more reads
+   **Matches**, 75–89% **Close**, and under 75% **Off**. Click a judge to list where it disagrees, then open the item to see why.
+   Re-running a judge after editing its prompt replaces that judge's previous result.
+
+The seed adds the batch `Recent summaries — batch 1` (8 of 10 answered) with two judges run:
+`summary-questions-judge` agrees 97% and `summary-questions-judge-quick` agrees 78%.
+
+## Calibrate a judge prompt against a human-judged dataset
+
+A batch answers "does this judge match people on these traces?". To improve a judge prompt
+version by version, judge a dataset once and reuse those answers:
+
+1. Build a dataset of traces (**Add to dataset** on traces, or
+   `POST /ai/observability/traces/add-to-dataset`).
+2. Create a review batch from it: `POST /ai/observability/review/batches` with
+   `{evaluatorId: <question set>, datasetId}`. Only items without a current answer are queued.
+   Every submitted answer is saved on its dataset item, and the dataset's `judgments` show
+   answered, current, and stale counts per question set.
+3. Save 2–3 versions of the judge prompt, then create a judge experiment:
+   `POST /ai/observability/experiments` with
+   `{kind: "judge", name, datasetId, questionSetId, judgeEvaluatorId, versions: [1, 2], targetAgreement: 0.9}`.
+   Each version answers every human-answered item; `results.agreement` ranks the versions and
+   each version gets an `agreement` gate.
+4. Promote the version that passes (`POST /ai/observability/experiments/:id/promote`); the
+   judge evaluator then shows its `calibration`. Use that judge in prompt experiments.
+
+Rewording a question makes earlier answers stale: re-run step 2 to re-queue those items.
+

package/src/docs/versioned/next/how-to/run-tests-locally.md

@@ -11,7 +11,7 @@ section).
 ## Do now — prepush mirrors CI
 
 ```bash
-bun run prepush            # lint, compile, then every CI job this branch triggers
+bun run prepush            # compile, lint + typecheck, then every CI job this branch triggers
 bun run prepush --dry-run  # print the plan only
 bun run prepush --all      # every check, regardless of changed files
 ```
@@ -29,6 +29,20 @@ are listed at the end with the command to run by hand. A test in
 `scripts/ci/prepush/plan.test.ts` fails when the mapping gains a parameter that
 `prepush` neither runs nor lists.
 
+## Typecheck
+
+Every package has a `typecheck` script that runs `tsc` with its
+`tsconfig.typecheck.json`. That config extends the build `tsconfig.json`, emits
+nothing, and includes the test files the build excludes (`*.test.ts(x)`,
+`*.isolated.ts(x)`, `src/tests/**`). Root `bun run lint` ends with `bun run typecheck`.
+CircleCI runs it in the `typecheck` job beside `repo-policies`.
+
+```bash
+bun run compile                # typecheck reads workspace packages' dist types
+bun run typecheck              # every package
+cd ui && bun run typecheck     # one package
+```
+
 ## Package tests
 
 From the repo root after `bun bootstrap`:
@@ -58,6 +72,7 @@ bun test --only-failures <path>
 | `create-terreno-app-ci` | `cd create-terreno-app && bun run test:ci` |
 | `packages-ci` (`admin-backend`, `admin-frontend`, `api-health`, `feature-flags`, `test`) | `cd <package> && bun run test:ci` |
 | `example-backend-ci` | `cd example-backend && bun run test:ci` |
+| `smoke` | `bun run smoke` (see [Package smokes](#package-smokes-smoke)) |
 | `example-frontend-ci` | `bun run frontend:test` |
 | `ui-demo-ci` | `cd demo && bun run test:ci` then `bun run check:demo-coverage` |
 | `new-file-coverage` | `bun run check:new-file-coverage --base=origin/master --threshold=90` |
@@ -65,6 +80,35 @@ bun test --only-failures <path>
 
 Coverage gate for a published package: `cd <package> && bun run test:coverage`.
 
+## Package smokes (`smoke`)
+
+The CircleCI `smoke` job runs on any package change. It compiles the workspace, builds
+the admin-spa, demo, and example-frontend smoke web exports, installs Playwright Chromium, then runs every
+workspace `smoke` script against the job's replica-set Mongo sidecar
+(`TERRENO_TEST_MONGODB_URI`). `prepush` lists it with the command below instead of
+running it.
+
+`bun run smoke:local` runs every step below in order. Run them yourself to reuse builds:
+
+```bash
+bun run compile
+(cd admin-spa && bun run build:web) && (cd demo && bun run export)
+(cd example-frontend && bun run export:smoke)
+bunx playwright install --with-deps chromium   # once
+bun run smoke              # every package; prints each one's time, flags any over 30 s
+bun run smoke api rtk      # only these workspaces
+```
+
+A package over 30 s is flagged, not failed: the CircleCI `smoke` job's table judges the budget
+([smoke conventions](../testing/e2e.md#smoke-conventions)). If a browser smoke fails with
+`Executable doesn't exist`, the preinstalled Playwright browsers do not match the locked
+version; install Chromium with `PLAYWRIGHT_BROWSERS_PATH=$HOME/.cache/ms-playwright` set and run
+`bun run smoke` with the same variable.
+
+Without `TERRENO_TEST_MONGODB_URI`, a smoke that needs Mongo starts its own in-memory
+replica set. To reuse a local replica-set `mongod`, export
+`TERRENO_TEST_MONGODB_URI="mongodb://127.0.0.1:27017/?replicaSet=rs0"` first.
+
 ## Playwright (example-frontend e2e)
 
 1. Install Chromium once: `cd example-frontend && bunx playwright install --with-deps chromium`
@@ -182,6 +226,10 @@ bun run test:e2e
 bun run test:integration    # needs Mongo + example-backend; job `admin-spa-integration`
 ```
 
+For `test:integration`, start example-backend with `ADMIN_SPA_ENABLED=true`,
+`AUTH_PROVIDER=better-auth`, and, for the MCP OAuth consent spec, `MCP_OAUTH_ENABLED=true` and
+`PUBLIC_API_URL=http://localhost:4000`. Without the last two the consent spec is skipped.
+
 ## Maestro web e2e
 
 Install CLI if missing: `curl -fsSL "https://get.maestro.mobile.dev" | bash`, then put

package/src/docs/versioned/next/how-to/upgrade-terreno.md

@@ -48,7 +48,17 @@ From `publish-on-tag.yml`: **backend** `api`, `test`, `admin-backend`, `blocks`,
 
 If the range includes the syncdb release (`56.0.0-beta.2` and later), treat collection CRUD as a **platform migration**. Follow [migrate-rtk-to-syncdb.md](migrate-rtk-to-syncdb.md) in a separate change from the routine lockstep bump. Keep `@terreno/rtk` for the OpenAPI SDK, Better Auth, feature flags, and sockets until that migration is done.
 
-## 5. Manual steps
+## 5. Langfuse is removed from `@terreno/ai`
+
+If the range includes the release that removes Langfuse, move off it before or with the bump:
+
+1. Register `ObservabilityApp` with `createLocalObservabilityPlugin()` for prompts, traces, and review. See [observe-llm-calls.md](observe-llm-calls.md).
+2. Recreate the prompts you kept in Langfuse in the built-in prompt registry.
+3. Replace `addGptRoutes({langfuseSystemPromptName})` with `systemPromptName` (and optionally `systemPromptLabel`).
+4. Remove imports of the deleted exports (`LangfuseApp`, `getPrompt`, `compilePrompt`, `initTracing`, and the rest listed in the release notes) and of the `@terreno/ai/langfuseApp` and `@terreno/ai/langfuseClient` subpaths. `ControlPrimary` is now only `"local"`.
+5. Drop the `LANGFUSE_*` environment variables. Deploy a backend revision that no longer mounts the Langfuse secrets **before** deleting those secrets (for example with `terraform apply`), or the running revision fails to start.
+
+## 6. Manual steps
 
 1. Branch from a clean tree. Confirm tests pass on HEAD so later failures are from the upgrade.
 2. Print `fromVersion`, `toVersion`, notes, affected packages, and this order. **Stop until the human confirms.**
@@ -61,13 +71,13 @@ If the range includes the syncdb release (`56.0.0-beta.2` and later), treat coll
 
 Stop at the first failed compile or test. Do not continue. Compile is a typecheck (`tsc --noEmit`); a bundler that ignores types is not enough.
 
-## 6. Verification
+## 7. Verification
 
 - Every `@terreno/*` version in the lockfile is `toVersion`.
 - `terreno_get_upgrade_guide` verification sections for notes in the range all pass.
 - App boots; auth and one write path work.
 
-## 7. Rollback
+## 8. Rollback
 
 On the upgrade branch, with no extra uncommitted work you need:
 

package/src/docs/versioned/next/implementationPlans/IP_TEMPLATE.md

@@ -47,6 +47,42 @@ the roadmap entry is scaffolded with on merge; omit it to have them inferred.
 
 Link to `docs/tasks/<slug>.md`.
 
+## E2E tests
+
+Every planned spec or flow, written by the task that delivers its behaviour. Follow
+[`docs/testing/e2e.md`](../testing/e2e.md) for paths and layers. Write
+`None — <reason>` when the plan adds none; never delete the section.
+
+| ID | Spec or flow | Layer | Proves | Written in |
+| --- | --- | --- | --- | --- |
+| E1 | `<package>/e2e/<name>.spec.ts` | browser \| native flow \| http \| process | `<acceptance criterion, observable>` | T1 |
+
+## Regression cases
+
+None — no hard-to-automate change
+
+Most plans keep the line above; never delete the section. Replace it only when the plan
+changes something a developer or QA must check by hand (push notifications and other
+hardware or OS behavior, a full cross-device flow, feel) that no existing case covers. See
+"Keep regression minimal" in the planning plugin's testing plan. Cases use the suite format
+of `docs/regression/` with provisional IDs (`<AREA>-new-<n>`); Brew assigns real IDs:
+
+Destination: `<AREA>` → `docs/regression/<area>.md`
+
+| ID | Case | Steps | Expected | Platforms | Automated by | Added |
+| --- | --- | --- | --- | --- | --- | --- |
+| `<AREA>-new-1` | `<case>` | `<starting state, then actions>` | `<what a person observes>` | `<platforms>` | manual \| E1 | `<slug>` |
+
+## User guide
+
+End-user guide pages this plan adds, updates, or retires, in the format of
+[`docs/user-guide/`](../user-guide/README.md). Write `None — <reason>` when the plan
+changes nothing a user sees; never delete the section.
+
+| ID | Page | Change | Audience | Written in |
+| --- | --- | --- | --- | --- |
+| G1 | `docs/user-guide/<area>/<feature>.md` | new \| update \| retire | user \| admin \| staff | T1 |
+
 ## Acceptance Criteria
 
 - [ ]

package/src/docs/versioned/next/implementationPlans/README.md

@@ -15,6 +15,29 @@ for *the concrete steps*. External trackers (a GitHub roadmap issue, a Linear is
 discussion thread) link **to** these files — they never replace them. See
 [`docs/tasks/README.md`](../tasks/README.md) for the task-file shape.
 
+## E2E tests and regression cases
+
+Every IP carries two testing sections from [`IP_TEMPLATE.md`](IP_TEMPLATE.md), placed
+before `## Acceptance Criteria`:
+
+- **`## E2E tests`** lists each planned spec or flow: its path, layer, the criterion it
+  proves, and the task that writes it. Paths and layers follow the
+  [e2e guide](../testing/e2e.md).
+- **`## Regression cases`** lists the manual cases the feature adds or changes, with
+  provisional IDs (`AUTH-new-1`). Brew writes them into the area's suite under
+  `docs/regression/` and assigns the real IDs.
+
+Either section may be `None — <reason>`; neither may be missing. Terreno's paths are
+declared in the `## Testing conventions` table in `.rulesync/rules/00-root.md` (which
+generates `CLAUDE.md`) and in the hand-kept `AGENTS.md`. The full
+contract is the planning plugin's
+[`testing-plan.md`](../../plugins/terreno-planning/references/testing-plan.md).
+
+Every IP also carries a **`## User guide`** section: the end-user guide pages under
+[`docs/user-guide/`](../user-guide/README.md) the plan adds, updates, or retires, or
+`None — <reason>`. The task that ships the feature writes its page. The contract is
+[`user-guide.md`](../../plugins/terreno-planning/references/user-guide.md).
+
 ## Status lives on each IP
 
 There is no shared `PLAN_INDEX.md`. That file caused merge conflicts the same way a