@terreno/mcp 58.0.0-beta.4 → 58.0.1

package/src/docs/versioned/58.0.0/regression/README.md

@@ -0,0 +1,87 @@
+# Regression suites
+
+Run `bun run check:regression-suites` after you edit any file here. CI runs the same check in
+the `repo-policies` job.
+
+These files hold a minimal manual pass a developer or QA walks before a release: a short check
+that the app works as intended, plus the behavior no unit test, smoke, or e2e spec can reach,
+such as push notifications, real email delivery, other hardware or operating-system behavior,
+full flows across devices, and how the app feels. Keep them small. Most features add no case
+here, and anything an e2e spec can drive and assert belongs in an e2e spec instead. Each area
+has one living file. Implementation plans
+propose cases under `## Regression cases`, and Brew writes them here with real IDs before it
+opens the pull request. See [Where regression suites live](../testing/e2e.md#where-regression-suites-live)
+for how this tier fits with the automated ones.
+
+## Areas
+
+| File | Area | Covers |
+| --- | --- | --- |
+| [planning.md](planning.md) | `PLAN` | Retired: planning-stage checks, kept so their IDs are never reused |
+| [auth.md](auth.md) | `AUTH` | Email delivery for password reset and verification, and native session storage |
+| [admin.md](admin.md) | `ADMIN` | Admin edits and announcements seen live on a phone |
+| [syncdb.md](syncdb.md) | `SYNC` | Offline writes, persistence across an app kill, and account switches on a device |
+| [ui-native.md](ui-native.md) | `NATIVE` | Notification permission prompts, finger-drawn signatures, and keyboard behavior on iOS and Android |
+
+## Suite format
+
+Every file follows the same shape:
+
+```markdown
+# Auth regression suite
+
+Area: AUTH
+Owner: <team or person>
+Last full run: 2026-01-15 (<who>, <short sha>)
+
+| ID | Case | Steps | Expected | Platforms | Automated by | Added |
+| --- | --- | --- | --- | --- | --- | --- |
+| AUTH-001 | ... | ... | ... | ... | manual | <IP slug> |
+```
+
+- **Title** is the first line, a `#` heading.
+- **`Area:`** declares the ID prefix: uppercase letters and digits. Find a suite by this line,
+  not by its file name. No two files declare the same area.
+- **`Owner:`** names the team or person who keeps the file current.
+- **`Last full run:`** is `never`, or the date, who walked it, and the short commit SHA. Change
+  it only after a person walks every case in the file.
+- **IDs** are `<AREA>-NNN`, zero-padded to three digits, and never reused. The next free ID is
+  one past the highest the file has ever held. A retired case keeps its row with `Retired` in
+  `Expected`. Provisional IP IDs such as `AUTH-new-1` never appear here.
+- **Steps** name a concrete starting state (who is signed in, what data exists, which build or
+  device), then an action a person can follow without guessing.
+- **Expected** is observable: what is on screen, in the inbox, or in the response. "It works"
+  or a restatement of the action does not count.
+- **Automated by** is `manual` or the path of the spec that covers the case. It never names an
+  IP-local `E<n>` row.
+- **Added** is the slug of the implementation plan that added the case.
+
+Escape a literal pipe inside a cell as `\|`.
+
+## What the check enforces
+
+`scripts/regression/checkSuites.ts` reads every `.md` file here except this README and fails
+when a file:
+
+- lacks the title, `Area:`, `Owner:`, or `Last full run:` line, or one of them is malformed
+- has a table whose columns differ from the seven above, or more than one table
+- has a row with the wrong number of cells, an ID that is not `<AREA>-NNN`, a `-new-` ID, or a
+  duplicate ID
+- has an empty cell, including Steps and Expected
+- names an `E<n>` row in Automated by
+
+It also fails when two files declare the same area. It does not judge whether a step meets the
+bar above; reviewers do.
+
+## Add a case
+
+1. Open the area's file, or create `<area>.md` with `Last full run: never` and an `Owner:`.
+2. Add a row with the next free ID.
+3. Run `bun run check:regression-suites`.
+
+## Backfill the suites
+
+To find cases an older feature should have had, run the `backfill-regression` skill. It
+surveys the app for behavior automation cannot reach, proposes cases one area at a time, and
+writes only the sets you approve. It records denied cases in
+`.terreno/regression-backfill.json` (not committed) so it never proposes them again.

package/src/docs/versioned/58.0.0/regression/admin.md

@@ -0,0 +1,13 @@
+# Admin regression suite
+
+Area: ADMIN
+Owner: Josh Gachnang
+Last full run: never
+
+Walk these cases with an admin account on web and a second, non-admin account signed in on a
+phone. Both run against the same deployed example-backend.
+
+| ID | Case | Steps | Expected | Platforms | Automated by | Added |
+| --- | --- | --- | --- | --- | --- | --- |
+| ADMIN-001 | An admin edit reaches the owner's phone live | A non-admin user is signed in on a phone with the Todos tab open and one todo titled "Buy milk". On web, an admin opens the Admin Panel, opens that todo, changes its title to "Buy oat milk", and saves | Within 10 seconds, without pulling to refresh or reopening the app, the phone's Todos list shows "Buy oat milk" | iOS, Android | manual | e2e-and-regression-testing |
+| ADMIN-002 | A published announcement shows on a phone | A non-admin user is signed in on a phone. On web, an admin creates and publishes an announcement from the Admin Panel, then the user force quits and reopens the app | The announcement shows once on the phone with the admin's title and body, and does not show again after it is dismissed and the app is reopened | iOS, Android | manual | e2e-and-regression-testing |

package/src/docs/versioned/58.0.0/regression/auth.md

@@ -0,0 +1,15 @@
+# Auth regression suite
+
+Area: AUTH
+Owner: Josh Gachnang
+Last full run: never
+
+Walk these cases against example-frontend and example-backend on a deployed environment whose
+backend sends real email. Use an inbox you can open, not a seeded `@example.com` address.
+
+| ID | Case | Steps | Expected | Platforms | Automated by | Added |
+| --- | --- | --- | --- | --- | --- | --- |
+| AUTH-001 | Password reset email reaches a real inbox | Signed out, with an account whose email is a real inbox you can open: on the login screen tap the forgot-password link, enter that email, tap "Send reset link", open the email, follow its link, enter a new password, tap "Save new password", then sign in with the new password | One reset email arrives within 2 minutes from the configured sender. Its link opens the "Reset password" screen in the app. Sign-in with the new password lands on Todos, and sign-in with the old password shows an error | Web, iOS, Android | manual | e2e-and-regression-testing |
+| AUTH-002 | Verification email confirms the account | Signed in as a new account with a real inbox that has not verified its email: open Profile, tap "Resend" on the verify-email banner, open the email, follow its link, tap "Confirm email", then reopen Profile | One verification email arrives within 2 minutes. After confirming, Profile no longer shows the "Verify your email" banner | Web, iOS, Android | manual | e2e-and-regression-testing |
+| AUTH-003 | Session survives a force quit on a device | Signed in on a release build on a phone, with at least one todo: force quit the app from the app switcher, wait 10 seconds, then open it again | The app opens on Todos with the same todos and no login screen, because the session is read back from the device's secure storage | iOS, Android | manual | e2e-and-regression-testing |
+| AUTH-004 | Logout clears the stored session on a device | Signed in on a release build on a phone: open Profile, tap "Logout", force quit the app, then open it again | The app opens on the login screen, and no data from the previous account shows before sign-in | iOS, Android | manual | e2e-and-regression-testing |

package/src/docs/versioned/58.0.0/regression/planning.md

@@ -0,0 +1,16 @@
+# Planning regression suite
+
+Area: PLAN
+Owner: Josh Gachnang
+Last full run: never
+
+Every case here is retired: they checked agent behavior, not the app, which the minimal
+regression rule leaves out. The rows stay so their IDs are never reused.
+
+| ID | Case | Steps | Expected | Platforms | Automated by | Added |
+| --- | --- | --- | --- | --- | --- | --- |
+| PLAN-001 | Grow plans e2e and regression | In a fresh session in a repo that has `docs/testing/e2e.md`, run Grow on "add a due date to todos" and approve it | Retired: an agent-behavior check, not an app check | Claude Code, Cursor, Codex | manual | e2e-and-regression-testing |
+| PLAN-002 | Grow without a guide | Repeat PLAN-001 in a repo with no `docs/testing/e2e.md` and no Testing conventions table | Retired: an agent-behavior check, not an app check | Claude Code | manual | e2e-and-regression-testing |
+| PLAN-003 | Brew writes the suite | Take an approved IP with one regression case through Brew | Retired: an agent-behavior check, not an app check | Claude Code | manual | e2e-and-regression-testing |
+| PLAN-004 | Pick writes, Roast enforces a planned spec | Run Pick on a PLAN-001 task that owns an `## E2E tests` row. Then delete the spec file and run Roast | Retired: an agent-behavior check, not an app check | Claude Code | manual | e2e-and-regression-testing |
+| PLAN-005 | Taste renumbers a collision | On a branch that added `AUTH-00N`, merge a master commit that also added `AUTH-00N`, then run Taste | Retired: an agent-behavior check, not an app check | Claude Code | manual | e2e-and-regression-testing |

package/src/docs/versioned/58.0.0/regression/syncdb.md

@@ -0,0 +1,15 @@
+# SyncDB regression suite
+
+Area: SYNC
+Owner: Josh Gachnang
+Last full run: never
+
+Walk these cases on a release build on a physical phone, against a deployed example-backend.
+Turn the network off with airplane mode, not by stopping the backend, so the operating system
+reports the device as offline.
+
+| ID | Case | Steps | Expected | Platforms | Automated by | Added |
+| --- | --- | --- | --- | --- | --- | --- |
+| SYNC-001 | An offline edit replays after airplane mode | Signed in on a phone with the Todos tab open and one todo, and the same account open on web: turn on airplane mode, mark the todo complete, wait 30 seconds, then turn airplane mode off | The phone shows the todo complete at once while offline. Within 10 seconds of reconnecting, web shows it complete without a reload, and the phone shows no conflict | iOS, Android | manual | e2e-and-regression-testing |
+| SYNC-002 | An offline create survives a force quit | Signed in on a phone in airplane mode: create a todo titled "Offline note", force quit the app, reopen it still in airplane mode, then turn airplane mode off | After reopening offline the list still shows "Offline note". After reconnecting it appears on web for the same account within 10 seconds, exactly once | iOS, Android | manual | e2e-and-regression-testing |
+| SYNC-003 | Switching accounts on one phone leaks no rows | Signed in on a phone as user A with at least two todos: open Profile, tap "Logout", then sign in as user B, who has no todos | User B's Todos tab is empty, both before and after the first sync completes. None of user A's todos appear at any point | iOS, Android | manual | e2e-and-regression-testing |

package/src/docs/versioned/58.0.0/regression/ui-native.md

@@ -0,0 +1,16 @@
+# Native UI regression suite
+
+Area: NATIVE
+Owner: Josh Gachnang
+Last full run: never
+
+Walk these cases on a fresh install on a physical phone. Delete the app first so the operating
+system asks for each permission again. NATIVE-001 and NATIVE-002 need a development build,
+because Profile shows "Send test push" only in development builds; the others run on either.
+
+| ID | Case | Steps | Expected | Platforms | Automated by | Added |
+| --- | --- | --- | --- | --- | --- | --- |
+| NATIVE-001 | Allowing notifications enables push | On a fresh install of a development build, sign in, tap Allow on the system notification prompt, open Profile, tap "Send test push", then lock the phone | The system prompt appears once, right after sign-in. A test notification shows on the lock screen within 1 minute | iOS, Android | manual | e2e-and-regression-testing |
+| NATIVE-002 | Denying notifications registers no device | On a fresh install of a development build, sign in as an account with no other signed-in devices, tap Don't Allow on the system notification prompt, then open Profile and tap "Send test push" | The app keeps working on Todos. Profile shows "No registered push tokens for this account." under the push button, and the prompt does not reappear on the next launch | iOS, Android | manual | e2e-and-regression-testing |
+| NATIVE-003 | A consent signature can be drawn with a finger | On a fresh install, sign in as an account that has never accepted the consent forms. On each consent screen that asks for a signature, draw a signature with one finger, then accept every form | The stroke follows the finger and the screen does not scroll while drawing. After the last form the app opens on Todos, and after a force quit and reopen no consent screen shows again | iOS, Android | manual | e2e-and-regression-testing |
+| NATIVE-004 | The keyboard does not cover a focused field | Signed in on a phone in portrait: open Profile, scroll to the "API Key" field in the Gemini API Key card, and tap it | The keyboard opens and the focused field stays visible above it, with its "API Key" label on screen | iOS, Android | manual | e2e-and-regression-testing |

package/src/docs/versioned/58.0.0/tasks/ai-dev-loop-boost.md

@@ -0,0 +1,132 @@
+# Task List: The AI Development Loop (Boost)
+
+See: [`docs/implementationPlans/ai-dev-loop-boost.md`](../implementationPlans/ai-dev-loop-boost.md)
+
+**RTK deprecation flag:** **Partial.** Tasks marked `[RTK]` rename `get_rtk_state` and relocate `installTerrenoDevConsoleLogger`; they require PR #869 merged in addition to PR #802. Unmarked tasks require only #802.
+
+## Instructions for the implementing agent
+
+- **Phase 1 gates everything.** PR #802 was in draft when this IP was written. Verify every tool name, parameter, and environment variable against the merged source in `mcp-server/src/local/` and `mcp-server/src/tools.ts` before documenting anything.
+- Every tool output shown in documentation must be real output from a real run, not a plausible-looking example.
+- Do not add new MCP tools. This IP documents and renames; it does not expand the surface.
+- Run `bun run mcp:build`, `bun run lint`, and `bun run compile` before each commit. Run `bun run rules:check` after touching `.rulesync/`.
+
+## Phase 1: Reconcile with merged #802
+
+- [x] **Task 1.1**: Inventory the actual MCP surface
+  - Description: Read `mcp-server/src/tools.ts`, `mcp-server/src/prompts.ts`, `mcp-server/src/resources.ts`, `mcp-server/src/local/localTools.ts`, and every file under `mcp-server/src/local/tools/`. Produce a table of every tool with its exact name, input schema, and which server hosts it. Separately list every environment variable the servers read (`TERRENO_MCP_EVAL`, `TERRENO_METRO_URL`, `TERRENO_PROJECT_ROOT`, `TERRENO_LOG_FILE`, `TERRENO_BROWSER_LOGS`, and any others found). Compare against the tool table in the IP and correct the IP where it is wrong. Put the deltas in the PR body.
+  - Files: `docs/implementationPlans/ai-dev-loop-boost.md`
+  - Depends on: PR #802 merged
+  - Acceptance: every tool in the corrected table is greppable in the source; every environment variable cites its reading file; the PR body lists what differed from the pre-merge IP.
+
+- [x] **Task 1.2**: Verify the log pipeline end to end
+  - Description: Start the example backend and frontend, then exercise each `read_logs` source and record real output: `backend` (winston JSONL file transport), `app` (console output over Metro CDP), `metro` (bundler events), and `browser` (POST fallback). Deliberately break something to produce an error and confirm `last_error` returns it with a stack. Note which sources required extra setup and what happens when Metro is not running. Save the real outputs for use in the documentation tasks.
+  - Files: none (findings recorded for later tasks)
+  - Depends on: Task 1.1
+  - Acceptance: all four sources produced output, or any that did not is documented with the reason; `last_error` output captured verbatim including a stack trace.
+
+- [x] **Task 1.3**: Reproduce the Hermes CDP contention symptom
+  - Description: With `read_logs` connected and streaming `app` source entries, open React Native DevTools (press `j` in Metro) and observe what happens to the log tool. Record the exact status message or failure the tool reports. Then close DevTools and confirm whether the tool recovers automatically. This becomes the documented symptom and workaround.
+  - Files: none (findings recorded for later tasks)
+  - Depends on: Task 1.2
+  - Acceptance: the symptom is recorded verbatim; recovery behavior (automatic or requiring a retry) is confirmed by observation.
+
+## Phase 2: Reference completeness
+
+- [x] **Task 2.1**: Update `docs/reference/mcp-server.md`
+  - Description: Rewrite the tool sections to match the Task 1.1 inventory exactly. Split clearly into "Hosted server (`terreno-mcp`)" and "Local server (`terreno-mcp-local`)" sections. For each tool: name, purpose, input schema table, example call, and example output. Add an "Environment variables" section listing every variable from Task 1.1 with its effect and default. Add a "What the local server can and cannot see" section — it reads `package.json`/lockfiles, connects to Mongo via the consumer's `.env`, tails log files, and attaches over CDP; it never imports consumer code. Document the Hermes limitation from Task 1.3. Update the hosted URL per IP question AI6 if the domain decision has been made.
+  - Files: `docs/reference/mcp-server.md`
+  - Depends on: Task 1.1, Task 1.3
+  - Acceptance: tool list matches the inventory one-to-one; every environment variable documented; the can/cannot-see section is present; the Hermes symptom is quoted from Task 1.3.
+
+- [x] **Task 2.2**: Sync `mcp-server/README.md`
+  - Description: Bring the package README's tool list in line with the reference page (it is known to omit `terreno_search_docs`, `terreno_get_component_docs`, and `terreno_get_upgrade_guide`). Keep the README short — the six-section template from [`docs-reference-coverage`](../implementationPlans/docs-reference-coverage.md) — and link the reference page for full detail.
+  - Files: `mcp-server/README.md`
+  - Depends on: Task 2.1
+  - Acceptance: the README and reference page list identical tool sets; the README follows the standard template; all links absolute.
+
+- [x] **Task 2.3**: Document the browser-log route
+  - Description: Add `POST /__terreno/browser-logs` to `docs/reference/api.md` (or the appropriate reference page) documenting: that it is registered only when `NODE_ENV !== "production"`, that it 404s in production, the `TERRENO_BROWSER_LOGS=false` override, the JSONL file it appends to, and the payload shape. Verify all of this against `api/src/browserLogsRoute.ts` in the merged tree.
+  - Files: `docs/reference/api.md`, `docs/reference/environment-variables.md`
+  - Depends on: Task 1.1
+  - Acceptance: every behavior claimed matches `api/src/browserLogsRoute.ts`; the production 404 behavior is stated; the env var appears in the environment reference.
+
+## Phase 3: The loop explainer
+
+- [x] **Task 3.1**: Write `docs/explanation/ai-development-loop.md`
+  - Description: Explainer covering the five stages (Ask, Generate, Run, Observe, Fix) with the mermaid diagram from the IP. For each stage, name the tools involved and what they contribute. Then a "Tool layer versus process layer" section using the table from the IP, stating that this page covers the tool layer and that the `/terreno-*` SDLC pipeline is the process layer, with a link to `docs/explanation/agentic-sdlc.md` — do not duplicate that page's content. Then a "What makes this different" section built on one narrow, defensible claim: the agent observes what the app actually did rather than a description of it. Then a "What the agent cannot see" section — production logs, native crash reports outside the dev loop, anything requiring a physical device not attached to a dev server, and the Hermes single-connection limitation. Then a "How this composes" section covering `expo-mcp` for simulator interaction, Playwright MCP for web, and Maestro for durable E2E flows, with the division of labor. No commands — this is conceptual.
+  - Files: `docs/explanation/ai-development-loop.md` (new), `docs/explanation/README.md`
+  - Depends on: Task 2.1
+  - Acceptance: all five stages present with real tool names from the inventory; the tool/process table is present and links the companion page; the cannot-see section has at least four honest limitations; no shell commands; listed in the explanation index.
+
+- [x] **Task 3.2**: Write `docs/explanation/agent-guidelines.md`
+  - Description: Explainer on how conventions reach an agent: the hosted server's docs resources and search, per-package `.ai/guidelines/` shipped in npm tarballs (verify whether #802 Phase 2 shipped this before documenting it — if not, describe only what exists and note the rest as planned), `terreno_bootstrap_ai_rules` for generating editor-specific rule files, and the `.rulesync/` source-of-truth pattern with its generated targets. Add a section for consumers on extending the generated rules with their own project conventions without losing them on regeneration.
+  - Files: `docs/explanation/agent-guidelines.md` (new), `docs/explanation/README.md`
+  - Depends on: Task 1.1
+  - Acceptance: every mechanism described is verified present in the merged tree, with anything unshipped clearly labeled as planned; the consumer-extension section gives a concrete pattern.
+
+## Phase 4: Setup and debug how-tos
+
+- [x] **Task 4.1**: Write `docs/how-to/set-up-terreno-mcp.md`
+  - Description: Per-editor setup for both Terreno servers. Cover at minimum Cursor (`.cursor/mcp.json`) and Claude Code, with the exact JSON for each including the hosted HTTP entry and the `bunx terreno-mcp-local` stdio entry. Then: what `terreno-mcp-local` needs to work (project root discovery and the `TERRENO_PROJECT_ROOT` override, a reachable `MONGO_URI`, the backend log file, Metro running for app logs); an optional section on adding `expo-mcp` and Playwright MCP with the division of labor from Task 3.1; and a verification section — ask the agent to call `application_info` and confirm it reports the project's `@terreno/*` versions. Compare the JSON against what `terreno_bootstrap_app` actually generates and make them identical.
+  - Files: `docs/how-to/set-up-terreno-mcp.md` (new), `docs/how-to/README.md`
+  - Depends on: Task 2.1
+  - Acceptance: configuration for at least two editors, each verified by actually loading it; the JSON matches bootstrap output byte-for-byte in structure; the `application_info` verification step works.
+
+- [x] **Task 4.2**: Write `docs/how-to/debug-with-mcp.md`
+  - Description: Task-focused guide using the real failure and real outputs captured in Task 1.2. Structure: the symptom; `last_error` to locate it; `read_logs` with source filtering to see the surrounding context; `database_query` to check the data; `get_client_state` to check the client; `navigate` plus a screenshot from the composed server to confirm the fix. Include a "which tool for which symptom" table (app crashes on a screen; API returns 403; data looks wrong; screen renders empty; bundler fails). Use verbatim tool output.
+  - Files: `docs/how-to/debug-with-mcp.md` (new), `docs/how-to/README.md`
+  - Depends on: Task 1.2, Task 4.1
+  - Acceptance: all tool outputs are verbatim from Task 1.2; the symptom-to-tool table has at least five rows; the walkthrough ends with a verified fix.
+
+- [x] **Task 4.3**: Add discovery paths for the local server
+  - Description: Nothing currently tells a developer that `terreno-mcp-local` exists. Add links to `docs/how-to/set-up-terreno-mcp.md` from: the README's MCP section, `docs/README.md` quick links, `docs/tutorials/README.md`, and the `terreno://docs/overview` MCP resource. Also confirm `terreno_bootstrap_app` output includes a comment or README line pointing at the setup guide.
+  - Files: `README.md`, `docs/README.md`, `docs/tutorials/README.md`, `mcp-server/src/docs/resources/overview.md`, `mcp-server/src/docs/templates/bootstrap/**`
+  - Depends on: Task 4.1
+  - Acceptance: at least four discovery paths link the setup guide; bootstrap output references it; `bun run mcp:build` succeeds.
+
+## Phase 5: Gap fixes
+
+- [x] **Task 5.1**: `[RTK]` Rename `get_rtk_state` to `get_client_state`
+  - Description: Per IP question AI1, rename the tool to `get_client_state` and extend it to detect either store type, returning a field identifying which layer it found (RTK Query cache versus syncdb store). Keep `get_rtk_state` registered as an alias whose description marks it deprecated and names the replacement. Update the tool description to explain what it returns for each layer. Add tests covering: syncdb store present, RTK store present, neither present, and the alias resolving to the same handler.
+  - Files: `mcp-server/src/local/tools/runtime.ts`, `mcp-server/src/local/localTools.ts`, tests under `mcp-server/src/local/`
+  - Depends on: Task 1.1, PR #869 merged
+  - Acceptance: both names work; the response identifies the detected layer; the alias description marks it deprecated; all four test cases pass; `bun run mcp:build` succeeds.
+
+- [x] **Task 5.2**: `[RTK]` Relocate the dev console logger
+  - Description: Per IP question AI2, move `installTerrenoDevConsoleLogger` from `rtk/src/devConsoleLogger.ts` to the syncdb package, keeping a re-export from `@terreno/rtk` for the deprecation window so existing consumers are unaffected. Move its tests with it. Update the bootstrap `_layout` template and `example-frontend/app/_layout.tsx` to import from the new location. Add the re-export to the RTK deprecation notes.
+  - Files: `syncdb/src/devConsoleLogger.ts` (moved), `rtk/src/devConsoleLogger.ts` (re-export), `rtk/src/index.ts`, `syncdb/src/index.ts`, `mcp-server/src/docs/templates/bootstrap/frontend/app/_layout.template.tsx`, `example-frontend/app/_layout.tsx`, tests
+  - Depends on: Task 5.1
+  - Acceptance: both import paths work; tests pass from the new location; `bun run compile` succeeds across the workspace; the example app still ships browser logs to the backend route (verify by producing a `console.error` and reading it back with `read_logs`).
+
+- [x] **Task 5.3**: Add rate limiting to the hosted MCP server
+  - Description: Per IP question AI5 — a launch prerequisite. Add per-IP rate limiting to the hosted server's JSON-RPC endpoint. Choose limits appropriate to the workload (search and codegen are cheap per call; the index is built once at startup) and make them configurable by environment variable with a sensible default. Return a clear JSON-RPC error when exceeded, naming the limit and the retry window. Add tests for under-limit, at-limit, and over-limit behavior. Document the limit in `docs/reference/mcp-server.md`.
+  - Files: `mcp-server/src/index.ts`, tests under `mcp-server/src/`, `docs/reference/mcp-server.md`, `docs/reference/environment-variables.md`
+  - Depends on: Task 2.1
+  - Acceptance: exceeding the limit returns a structured error naming the limit and retry window; the limit is configurable by environment variable; three test cases pass; the limit is documented.
+
+- [x] **Task 5.4**: Handle the MCP domain change
+  - Description: If program question P4 / IP question AI6 was answered in favor of moving off `mcp.terreno.flourish.health`, update every occurrence of the URL: `README.md`, `docs/reference/mcp-server.md`, `mcp-server/README.md`, `mcp-server/src/**` defaults, bootstrap templates, and `infra/flourish/` deployment config. Keep the old hostname serving with a redirect or CNAME for at least one release and document the cutover. If the decision was to keep the current domain, make no change and record that.
+  - Files: various (grep `rg -n "mcp\.terreno"`)
+  - Depends on: Task 2.1
+  - Acceptance: `rg -n "mcp\.terreno\.flourish\.health"` returns only intentional legacy references; a bootstrapped app's `mcp.json` uses the new URL; the cutover plan is documented; or the no-change decision is recorded in the PR body.
+
+- [x] **Task 5.5**: Let `terreno-mcp-local` attach to Metro's inspector proxy
+  - Description: Found by Task 1.3. `@react-native/dev-middleware` 0.86 (Expo 57) rejects debugger WebSocket upgrades that carry no `Origin` header (`InspectorProxy.js` `verifyClient`), and neither Bun nor Node sends one, so the `app` log source and every CDP-backed tool fail with `CDP websocket error: ... Expected 101 status code`. Send an `Origin` matching the Metro base URL's loopback origin on the CDP socket, and surface Metro's close code and reason (for example `[NEW_DEBUGGER_OPENED]`) in the status hint instead of the bare `CDP disconnected`. Add tests at the session seam.
+  - Files: `mcp-server/src/local/metro/metroDevSession.ts`, its tests
+  - Depends on: Task 1.3
+  - Acceptance: against a real Metro, `read_logs` with source `app` connects to a debuggable target without a preload shim; when another debugger takes the session, the status names Metro's close reason; tests cover the header and the close-reason hint; `bun run mcp:build` succeeds.
+
+- [x] **Task 5.6**: Make the local database tools load Mongoose under Bun
+  - Description: Found by Task 2.1. `database_schema` and `database_query` call `await import("mongoose")` and read `.connection` on the namespace object; under Bun with Mongoose 9 that property is undefined, so both tools fail with `undefined is not an object (evaluating 'mongoose.connection.readyState')` whenever a Mongo URI is found. Use the module's default export, and add a test that exercises the real import path against the in-memory Mongo used by the package tests.
+  - Files: `mcp-server/src/local/tools/databaseSchema.ts`, `mcp-server/src/local/tools/databaseQuery.ts`, their tests
+  - Depends on: Task 2.1
+  - Acceptance: against the running example database, `database_schema` and `database_query` return real results through `terreno-mcp-local`; tests fail before the fix and pass after; `bun run mcp:build` succeeds.
+
+## Phase 6: Verification
+
+- [x] **Task 6.1**: Run the loop end to end and capture evidence
+  - Description: In a fresh agent session with both MCP servers configured per `docs/how-to/set-up-terreno-mcp.md`, complete a full loop: ask for a small feature, let the agent generate it, run the app, introduce a bug, use `last_error` and `read_logs` to find it, fix it, and verify with `navigate` plus a screenshot. Capture the transcript highlights and screenshots to `/opt/cursor/artifacts/`. Fix any documentation gap the run exposed.
+  - Files: any documentation needing correction
+  - Depends on: Task 4.2, Task 5.1, Task 5.3
+  - Acceptance: the full loop completed in one session using only the public docs; artifacts captured showing the `last_error` output and the verified fix; every gap fixed.

package/src/docs/versioned/58.0.0/tasks/ai-observability-production-loop.md

@@ -0,0 +1,204 @@
+# Task List: AI observability production loop
+
+**Status:** approved 2026-10-18. Start with T1 (or T2, which is independent).
+**IP:** [`docs/implementationPlans/ai-observability-production-loop.md`](../implementationPlans/ai-observability-production-loop.md). The contracts, acceptance criteria (AC1–AC21; AC9 was dropped with T10), and open questions live there. T10 moved to Expansions (X9) during cut, so the IDs skip it.
+**Supporting skills:** `terreno-backend-api`, `mongoose-schema-safety`, `ai-prompt-governance`, `backend-test-env`, `building-admin-interfaces`, `terreno-ui`, `dataviz`, `update-docs`, `verify-ui-changes`, `accessibility-testing`.
+
+Shared files force three write chains:
+
+- **Backend chain:** T1 → T3 → T4 → T5 → T6 → T7 → T8 → T9 → T11 → T12 → T13 → T14 → T15.
+  - Shared files: `ai/src/observability/observabilityApp.ts` (route mounting and options),
+    `ai/src/observability/local/localPlugin.ts` (model registration), `ai/src/index.ts`,
+    `example-backend/src/__snapshots__/openapi.test.ts.snap`, and `docs/reference/ai.md`.
+  - Each backend task updates the reference rows for what it ships.
+- **UI chain:** T19 → T21 → T22 → T23 → T24.
+  - Shared files: `ai/src/observability/adminScreens.ts`, `shell/AiObservabilityScreenWidgets.tsx`,
+    `shell/aiObservabilityNav.ts`, and `isolated/aiObservabilityScreens.isolated.tsx`.
+  - Each UI task writes the operator how-to page for its feature.
+- **Jobs chain:** T16 → T17. Shared files: `ai/src/observability/jobsRunners.ts`, `example-backend/src/server.ts`, and `docs/how-to/background-jobs.md`. They plug into runner options that T1, T8, and T9 define, so they never edit `observabilityApp.ts`.
+- **Example server and jobs:** `example-backend/src/server.ts` is written by T1, T16, T17, and T25, in that order; the example jobs files by T16 and T17.
+
+Parallel work: T2 (api RBAC), T16 (after T1), T18 (CLI, after T13), T19 (after T5), and T20 (after T6).
+UI tasks prove themselves with data they create through their own API; T25 seeds the demo last and re-captures every screen.
+Every UI task runs `verify-ui-changes` against the seeded example and `accessibility-testing`
+before Roast. Every user-visible task adds a `changes/unreleased/*.md` note.
+
+Paths below are relative to the repo root. `obs/` means `ai/src/observability/`; `widgets/` means
+`admin-frontend/src/widgets/aiObservability/`.
+
+### Phase 1: Tracer
+
+- [ ] **T1** — Live evaluation scores a sampled production trace
+  - Depends on: none
+  - Files: `ai/src/service/aiService.ts`, `obs/observabilityAppRegistry.ts`, `obs/observabilityApp.ts`, `obs/liveEval.ts` (new), `obs/liveEval.test.ts` (new), `obs/evaluate.ts`, `obs/evaluate.test.ts`, `ai/src/service/prompts.ts` (`LIVE_JUDGE_*`), `obs/local/models/obsScore.ts` (`origin`), `obs/local/models/obsEvalRun.ts` (new), `obs/local/localPlugin.ts`, `obs/types.ts`, `ai/src/types/observability.ts`, `ai/src/index.ts`, `example-backend/src/server.ts` (`sampleRate` from `AI_OBS_SAMPLE_RATE`), `docs/reference/ai.md`, `docs/explanation/ai-observability.md`.
+  - Delivers:
+    - The trace export seam: `logRequestAndTrace` calls `app.exportTrace`, and `exportTrace` calls `liveEval.schedule` after the local sink persists.
+    - The `liveEvalRunner` option (default in-process) that T16 plugs into.
+    - `ObsEvalRun` with a unique `{traceId, evaluatorId}` index, a 10-minute `pending` lease that can be reclaimed, and judge tokens and cost. Live scores are upserted on `{traceId, evaluatorId, name, origin}` (partial unique index).
+    - The scheduler: evaluator selection with a 30 s cache, the effective rate, the deterministic roll, the error, no-output, and sensitive skips, and the in-process runner (concurrency 2, cap 500).
+    - Trace-worded judge prompt via `subject: "trace"`.
+    - Scores fan out through `scoreSinks` with `origin: "live"`, and each run writes an `ObsEvalRun`.
+    - With no server `aiService`, live eval logs once and stays off.
+  - Acceptance: AC1, AC2. The new tests fail on the base branch.
+
+### Phase 2: Access and threads foundation
+
+- [ ] **T2** — RBAC resources for alerts, test suites, and the optimizer
+  - Depends on: none
+  - Files: `api/src/rbac/statements.ts`, `api/src/rbac/roleModel.ts`, `api/src/rbac/*.test.ts` (catalog and seed tests), `obs/permissions.ts`, `example-backend/src/rbacRoles.ts`, `example-backend/src/rbacRoles.test.ts`, `docs/reference/api.md` (AI observability RBAC).
+  - Delivers: the `aiAlert` and `aiTestSuite` resources and `aiPrompt.optimize`. The seeded admin, auditor, and superadmin grants are additive on re-seed, and the example viewer and operator roles get the matching grants.
+  - Acceptance: AC16.
+
+- [ ] **T3** — Traces record named groups
+  - Depends on: T1
+  - Files: `ai/src/service/aiService.ts` (`groups` option), `ai/src/service/chatTurn.ts` (`conversation` group), `obs/types.ts` (`TraceRecord.groups`), `obs/traceGroups.ts` (new: validation), `obs/traceGroups.test.ts` (new), `obs/local/models/obsTrace.ts` (field + index), `obs/local/traceStore.ts` (persist, list filter, detail map), `obs/routes/traces.ts`, `obs/routes/traces.test.ts`, `ai/src/types/index.ts` (generate option type), `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`.
+  - Delivers: the Groups contract rows for shape, validation, `session`, chat, the index, and the trace-list filter.
+  - Acceptance: AC4.
+
+### Phase 3: Online evaluation and threads API
+
+- [ ] **T4** — Live sampling filters by prompt and lets sensitive traces opt in
+  - Depends on: T3
+  - Files: `obs/local/models/obsEvaluator.ts` (`livePromptNames`, `includeSensitive`), `obs/local/evaluatorStore.ts`, `obs/routes/evaluators.ts`, `obs/routes/evaluators.test.ts`, `obs/liveEval.ts`, `obs/liveEval.test.ts`, `ai/src/types/observability.ts`, `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`.
+  - Delivers: the required prompt list when a write sets the rate above 0 (400 otherwise), production-label prompt matching, and the `includeSensitive` opt-in (D2). Stored evaluators with a rate but no prompt list are not sampled and stay editable.
+  - Acceptance: AC3, and the sensitive half of AC2.
+
+- [ ] **T5** — Groups list and timeline API
+  - Depends on: T4
+  - Files: `obs/local/groupStore.ts` (new), `obs/routes/groups.ts` (new), `obs/routes/groups.test.ts` (new), `obs/observabilityApp.ts` (mount), `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`, `docs/explanation/model-router-actions.md` (observability plugin routes join the documented exceptions).
+  - Delivers: `GET /groups/kinds`, `GET /groups`, and `GET /groups/:kind/:id`, with rollups and an ordered timeline; `session` reads `sessionId`.
+  - Acceptance: AC5.
+
+- [ ] **T6** — Evaluator live stats and sampling ceiling in status
+  - Depends on: T5
+  - Files: `obs/local/evaluatorStore.ts` (live stats from `ObsEvalRun` and live scores), `obs/routes/evaluators.ts`, `obs/routes/evaluators.test.ts`, `obs/status.ts`, `obs/routes/status.ts`, `ai/src/types/observability.ts`, `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`.
+  - Delivers: evaluator detail `live` and status `liveEval: {available, ceilingPercent, misconfiguredEvaluatorIds}`.
+  - Acceptance: a route test seeds `ObsEvalRun` rows and live scores and reads back the counts and means; a status test covers both `available` values and lists an evaluator with a rate but no prompt list (AC3).
+
+### Phase 4: Monitoring and alerts API
+
+- [ ] **T7** — Metrics API returns time series and breakdowns
+  - Depends on: T6
+  - Files: `obs/local/metricsStore.ts` (new, window functions reused by T9), `obs/local/metricsStore.test.ts` (new), `obs/routes/metrics.ts` (new), `obs/routes/metrics.test.ts` (new), `obs/observabilityApp.ts`, `ai/src/types/observability.ts` (`MetricsResponse`), `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`.
+  - Delivers: the Metrics contract, including the range limits, `null` cost for unpriced buckets, `costPartial`, and `liveEvalCostUsd` from `ObsEvalRun`.
+  - Acceptance: AC6.
+
+- [ ] **T8** — Webhook destinations with signed, metadata-only delivery
+  - Depends on: T7, T2
+  - Files: `obs/local/models/obsWebhook.ts` (new), `obs/local/models/obsWebhookDelivery.ts` (new), `obs/webhooks.ts` (new: payload builder, signer, Slack renderer, URL guard, in-process retry), `obs/webhooks.test.ts` (new), `obs/routes/webhooks.ts` (new: CRUD, rotate secret, `POST /:id/test`, deliveries list), `obs/routes/webhooks.test.ts` (new), `obs/local/localPlugin.ts`, `obs/observabilityApp.ts` (`publicAdminUrl` and `webhookDeliveryRunner` options, mount), `obs/types.ts`, `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`.
+  - Delivers: the Webhooks contract and D1. The payload builder accepts only `ObsWebhookPayload`, the secret is returned once, `url` is never returned after create (`urlHost` only), delivery connects to the checked IP through an undici `Agent` lookup, and redirects are not followed.
+  - Acceptance: AC7. A test seeds a trace whose input and output hold a marker string, fires a delivery, and asserts the body never contains the marker.
+
+- [ ] **T9** — Alert rules evaluate and notify
+  - Depends on: T8
+  - Files: `obs/local/models/obsAlertRule.ts` (new), `obs/alertRules.ts` (new: evaluate, state machine, lease, interval scheduler), `obs/alertRules.test.ts` (new, fake clock), `obs/routes/alertRules.ts` (new: CRUD + `POST /:id/evaluate` dry run), `obs/routes/alertRules.test.ts` (new), `obs/local/localPlugin.ts`, `obs/observabilityApp.ts` (`alerts: {interval, scheduler}` options, mount), `obs/types.ts`, `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`.
+  - Delivers: every alert metric in the IP, the `minSamples` guard, `alert.firing` and `alert.resolved` once each (Q7), a single evaluator per tick via the lease, and the per-metric `traceIds` selection plus `adminUrl` in the payload.
+  - Acceptance: AC8.
+
+### Phase 5: CI test suites
+
+- [ ] **T11** — Test suites and suite runs
+  - Depends on: T9, T2
+  - Files: `obs/local/models/obsTestSuite.ts` (new), `obs/local/models/obsExperiment.ts` (`suiteId`, `kind`), `obs/local/experimentRunner.ts` (one-version suite runs, candidate-only gates), `obs/local/experimentRunner.test.ts`, `obs/local/testSuiteStore.ts` (new), `obs/routes/testSuites.ts` (new), `obs/routes/testSuites.test.ts` (new), `obs/routes/experiments.ts` (`?kind` filter), `obs/routes/experiments.test.ts`, `obs/local/localPlugin.ts`, `obs/observabilityApp.ts`, `ai/src/types/observability.ts`, `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`.
+  - Delivers: suite CRUD with a required stored `target` and `maxCostUsd`; runs that use the stored target or one `{version}` / `{label}` override (Q3); a kind-aware `versions` validator; and the run status shape with `estimatedCostUsd`. The experiments list excludes `kind: "suite"` unless `?kind=suite`.
+  - Acceptance: AC10 for version and label runs, the route half of AC21 (403 without `aiTestSuite:run`), plus a test that a normal experiment still needs 2–3 versions.
+
+- [ ] **T12** — Suite runs accept an inline candidate prompt
+  - Depends on: T11
+  - Files: `obs/local/experimentRunner.ts` (`"candidate"` version key with an inline body), `obs/local/experimentRunner.test.ts`, `obs/routes/testSuites.ts`, `obs/routes/testSuites.test.ts`, `obs/local/models/obsExperiment.ts` (stored candidate), `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`.
+  - Delivers: `{candidate: PromptVersionFields}` stored as version `0` with `candidateBody`, validated like `createVersion`, with a 400 when its variable set differs from the base prompt's production version. No prompt version is created. `mongoose-schema-safety` risk matrix recorded in the PR.
+  - Acceptance: AC10 (inline half).
+
+- [ ] **T13** — Observability service tokens for CI
+  - Depends on: T12
+  - Files: `obs/local/models/obsServiceToken.ts` (new), `obs/serviceTokens.ts` (new: mint, verify, revoke), `obs/serviceTokens.test.ts` (new), `obs/routes/observabilityRouteAccess.ts` (accepts `tobs_` only where a route opts in with `serviceToken: true`), `obs/routes/serviceTokens.ts` (new), `obs/routes/testSuites.ts` (opt in on list, read, run, and run status; rate limit), `obs/local/localPlugin.ts`, `obs/observabilityApp.ts`, `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`.
+  - Delivers: the token contract (D5): hash only, shown once, a required expiry of at most 365 days, at most 10 active per user, scope checked before the legacy-admin shortcut, 401 everywhere else and for a disabled user, 10 runs per hour per token, and audit entries.
+  - Acceptance: AC11.
+
+### Phase 6: Prompt optimizer
+
+- [ ] **T14** — Prompt optimizer engine
+  - Depends on: T13
+  - Files: `obs/local/experimentRunner.ts` (extract `scoreVersionOnItems`), `obs/local/experimentRunner.test.ts`, `obs/local/promptOptimizer.ts` (new), `obs/local/promptOptimizer.test.ts` (new, deterministic fixture model), `obs/local/models/obsOptimization.ts` (new), `obs/local/localPlugin.ts`, `ai/src/service/prompts.ts` (`PROMPT_OPTIMIZER_*`), `ai/src/types/observability.ts`, `docs/reference/ai.md`.
+  - Delivers: the reflective-rewrite loop (Q5), the train/validation split, variable and schema guards, the PHI guards (D6: only person-written or annotator-signed items; rejection of 8-word verbatim runs and of new item tokens or digit runs), estimated-cost tracking from returned token usage, budget stops, and early stop. Existing experiment tests stay green after the extraction.
+  - Acceptance: AC13.
+
+- [ ] **T15** — Optimizer API
+  - Depends on: T14, T2
+  - Files: `obs/routes/optimizations.ts` (new), `obs/routes/optimizations.test.ts` (new), `obs/observabilityApp.ts`, `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`.
+  - Delivers: estimate, create, list, detail, cancel, and `saveVersion` (Q4: only `latest` moves; requires `reviewedForPersonalData: true`, recorded in the audit entry).
+  - Acceptance: AC14.
+
+### Phase 7: Durable jobs (parallel)
+
+- [ ] **T16** — Live evals run as durable jobs
+  - Depends on: T1
+  - Files: `obs/jobsRunners.ts` (new subpath), `obs/jobsRunners.test.ts` (new), `ai/package.json` (`exports["./observability/jobsRunners"]`), `ai/src/harness/runners/jobsRunner.test.ts` (extend the import guard to the new subpath's reachability from `ai/src/index.ts`), `example-backend/src/server.ts` (pass the jobs runner through the T1 `liveEvalRunner` option when JobsApp is registered), `example-backend/src/utils/observabilityConfig.ts` (add a shared `createExampleObservabilityApp()` that the server and the worker both call), `example-backend/src/jobs/createExampleJobsApp.ts`, `example-backend/src/jobs/defineExampleJobs.ts`, `example-backend/src/jobs/bootstrapJobsWorker.ts` (the standalone worker builds the same observability context, so job handlers have the `aiService` and score sinks), `docs/how-to/background-jobs.md`.
+  - Delivers: the `ai-observability/live-eval` job with 3 attempts, relying on the T1 `ObsEvalRun` claim for idempotent retries, and the in-process fallback when no JobsApp is registered.
+  - Acceptance: AC15 (live-eval half).
+
+- [ ] **T17** — Alert ticks and webhook deliveries run as jobs
+  - Depends on: T9, T16
+  - Files: `obs/jobsRunners.ts`, `obs/jobsRunners.test.ts`, `example-backend/src/server.ts` (pass the jobs scheduler and delivery runner through the T8/T9 options), `example-backend/src/utils/observabilityConfig.ts`, `example-backend/src/jobs/defineExampleJobs.ts`, `docs/how-to/background-jobs.md`.
+  - Delivers: the `ai-observability/alerts-tick` cron (every minute) and the `ai-observability/webhook-delivery` job, which keeps the 0 / 10 s / 60 s schedule. When jobs are used, the interval scheduler stays off.
+  - Acceptance: AC15 (alerts half).
+
+### Phase 8: CLI (parallel)
+
+- [ ] **T18** — `terreno ai test` CLI command
+  - Depends on: T13
+  - Files: `cli/src/commands/ai.ts` (new), `cli/src/commands/ai.test.ts` (new, fake `io.fetch`), `cli/src/runCli.ts`, `cli/src/help.ts`, `docs/reference/cli.md`, `docs/how-to/run-ai-eval-suites-in-ci.md` (new: GitHub Actions and CircleCI snippets, each with the required-secret validation step).
+  - Delivers: the CLI contract in the IP, including the gate table output, exactly one target flag (Q3), and exit codes.
+  - Acceptance: AC12.
+
+### Phase 9: Admin UI
+
+- [ ] **T19** — Threads screens
+  - Depends on: T5
+  - Files: `widgets/threads/*` (new: list and detail screens and views, `useAiObservabilityThreadsApi.ts` via `adminRequest`, tests), `widgets/traces/AiTraceDetailView.tsx` (group chips), `widgets/shell/AiObservabilityScreenWidgets.tsx`, `widgets/shell/aiObservabilityNav.ts`, `admin-frontend/src/isolated/aiObservabilityScreens.isolated.tsx`, `obs/adminScreens.ts`, `obs/adminScreens.test.ts`, `docs/how-to/group-ai-traces.md` (new).
+  - Delivers: the Threads row of the IP's Admin UI table.
+  - Acceptance: AC17, AC18.
+
+- [ ] **T20** — Evaluator live controls and stats
+  - Depends on: T6
+  - Files: `widgets/evaluators/AiEvaluatorPanels.tsx`, `widgets/evaluators/AiEvaluatorNewScreen.tsx`, `widgets/evaluators/evaluatorTypes.ts`, `widgets/evaluators/evaluatorTypes.test.ts`, `widgets/evaluators/useAiObservabilityEvaluatorsApi.ts` (additive response types only), `widgets/shell/AiObservabilityStatusChip.tsx` (additive `liveEval` status type), `admin-frontend/src/isolated/observabilityStatusFixtures.isolated.ts`, `admin-frontend/src/isolated/aiEvaluatorLive.isolated.tsx` (new), `docs/how-to/evaluate-production-traces.md` (new).
+  - Delivers: the Evaluator row of the Admin UI table. Save is disabled while the rate is above 0 and no prompt is chosen; a warning shows when sensitive is on, and when status lists the evaluator as misconfigured.
+  - Acceptance: AC17, AC18. With `GEMINI_API_KEY` set and `AI_OBS_SAMPLE_RATE=1`, a Summarize press in the example app shows a new live score on the evaluator screen (screenshot).
+
+- [ ] **T21** — Monitoring screen
+  - Depends on: T7, T19
+  - Files: `widgets/monitoring/*` (new), `widgets/shell/AiObservabilityScreenWidgets.tsx`, `widgets/shell/aiObservabilityNav.ts`, `admin-frontend/src/isolated/aiObservabilityScreens.isolated.tsx`, `obs/adminScreens.ts`, `obs/adminScreens.test.ts`, `docs/how-to/monitor-ai-quality.md` (new).
+  - Delivers: the Monitoring row of the Admin UI table, with charts per the `dataviz` skill.
+  - Acceptance: AC17, AC18.
+
+- [ ] **T22** — Alerts and webhooks screens
+  - Depends on: T9, T21
+  - Files: `widgets/alerts/*` (new), `widgets/shell/AiObservabilityScreenWidgets.tsx`, `widgets/shell/aiObservabilityNav.ts`, `admin-frontend/src/isolated/aiObservabilityScreens.isolated.tsx`, `obs/adminScreens.ts`, `obs/adminScreens.test.ts`, `docs/how-to/monitor-ai-quality.md`.
+  - Delivers: the Alerts and Webhooks row of the Admin UI table.
+  - Acceptance: AC17, AC18. A screenshot shows the secret-once modal and a delivery log entry from **Send test** against a local receiver.
+
+- [ ] **T23** — Test suites screens, admin runs, and CI tokens
+  - Depends on: T13, T18, T22
+  - Files: `widgets/suites/*` (new), `widgets/shell/AiObservabilityScreenWidgets.tsx`, `widgets/shell/aiObservabilityNav.ts`, `admin-frontend/src/isolated/aiObservabilityScreens.isolated.tsx`, `obs/adminScreens.ts`, `obs/adminScreens.test.ts`, `docs/how-to/run-ai-eval-suites-in-ci.md`.
+  - Delivers: the Test suites row of the Admin UI table, including the stored-target picker in the editor and the **Run** button with an optional version/label override (D4).
+  - Acceptance: AC17, AC18, AC21. The walkthrough starts a run from admin, then mints a token in admin and runs `terreno ai test` against the local backend with it.
+
+- [ ] **T24** — Optimizer screens
+  - Depends on: T15, T23
+  - Files: `widgets/optimizer/*` (new), `widgets/prompts/*` (the **Optimize** button on the prompt hub), `widgets/shell/AiObservabilityScreenWidgets.tsx`, `widgets/shell/aiObservabilityNav.ts`, `admin-frontend/src/isolated/aiObservabilityScreens.isolated.tsx`, `obs/adminScreens.ts`, `obs/adminScreens.test.ts`, `docs/how-to/optimize-a-prompt.md` (new).
+  - Delivers: the Optimizer row of the Admin UI table, including rejected candidates with their reason and the personal-data review tick before **Save as vN+1**.
+  - Acceptance: AC17, AC18.
+
+### Phase 10: Demo and SOP
+
+- [ ] **T25** — Seeded demo data, SOP docs, and full walkthrough
+  - Depends on: T15, T17, T20, T24
+  - Files: `example-backend/src/scripts/seedAiObservabilityDemo.ts`, `example-backend/src/scripts/seedAiObservabilityDemo.test.ts`, `example-backend/src/server.ts` (`publicAdminUrl` from `PUBLIC_APP_URL`), `docs/how-to/observe-llm-calls.md` (env `AI_OBS_SAMPLE_RATE` meaning, seed table), `docs/how-to/ai-feature-development.md` (SOP steps 5–8 use suites, live eval, alerts, and the optimizer), `docs/explanation/ai-observability.md` (production loop section), `docs/implementationPlans/ai-observability.md` (pointer to this plan).
+  - Delivers:
+    - `conversation` groups on seeded chat traces.
+    - Live scores and `ObsEvalRun` rows over the last 7 days, with a dip on one prompt.
+    - One firing alert rule and a disabled demo webhook.
+    - The `summarize-gold` suite with one passed and one failed run.
+    - One finished optimization run from the deterministic `seed-fixture` client.
+    - The seed stays idempotent.
+  - Acceptance: AC19, plus a `verify-ui-changes` pass that re-captures Threads, Evaluator live panel, Monitoring, Alerts, Webhooks, Test suites, and Optimizer on the seeded demo (AC18).