@terreno/mcp 58.0.0-beta.2 → 58.0.0-beta.4

package/src/docs/versioned/next/tasks/ai-observability-production-loop.md

@@ -0,0 +1,204 @@
+# Task List: AI observability production loop
+
+**Status:** approved 2026-10-18. Start with T1 (or T2, which is independent).
+**IP:** [`docs/implementationPlans/ai-observability-production-loop.md`](../implementationPlans/ai-observability-production-loop.md). The contracts, acceptance criteria (AC1–AC21; AC9 was dropped with T10), and open questions live there. T10 moved to Expansions (X9) during cut, so the IDs skip it.
+**Supporting skills:** `terreno-backend-api`, `mongoose-schema-safety`, `ai-prompt-governance`, `backend-test-env`, `building-admin-interfaces`, `terreno-ui`, `dataviz`, `update-docs`, `verify-ui-changes`, `accessibility-testing`.
+
+Shared files force three write chains:
+
+- **Backend chain:** T1 → T3 → T4 → T5 → T6 → T7 → T8 → T9 → T11 → T12 → T13 → T14 → T15.
+  - Shared files: `ai/src/observability/observabilityApp.ts` (route mounting and options),
+    `ai/src/observability/local/localPlugin.ts` (model registration), `ai/src/index.ts`,
+    `example-backend/src/__snapshots__/openapi.test.ts.snap`, and `docs/reference/ai.md`.
+  - Each backend task updates the reference rows for what it ships.
+- **UI chain:** T19 → T21 → T22 → T23 → T24.
+  - Shared files: `ai/src/observability/adminScreens.ts`, `shell/AiObservabilityScreenWidgets.tsx`,
+    `shell/aiObservabilityNav.ts`, and `isolated/aiObservabilityScreens.isolated.tsx`.
+  - Each UI task writes the operator how-to page for its feature.
+- **Jobs chain:** T16 → T17. Shared files: `ai/src/observability/jobsRunners.ts`, `example-backend/src/server.ts`, and `docs/how-to/background-jobs.md`. They plug into runner options that T1, T8, and T9 define, so they never edit `observabilityApp.ts`.
+- **Example server and jobs:** `example-backend/src/server.ts` is written by T1, T16, T17, and T25, in that order; the example jobs files by T16 and T17.
+
+Parallel work: T2 (api RBAC), T16 (after T1), T18 (CLI, after T13), T19 (after T5), and T20 (after T6).
+UI tasks prove themselves with data they create through their own API; T25 seeds the demo last and re-captures every screen.
+Every UI task runs `verify-ui-changes` against the seeded example and `accessibility-testing`
+before Roast. Every user-visible task adds a `changes/unreleased/*.md` note.
+
+Paths below are relative to the repo root. `obs/` means `ai/src/observability/`; `widgets/` means
+`admin-frontend/src/widgets/aiObservability/`.
+
+### Phase 1: Tracer
+
+- [ ] **T1** — Live evaluation scores a sampled production trace
+  - Depends on: none
+  - Files: `ai/src/service/aiService.ts`, `obs/observabilityAppRegistry.ts`, `obs/observabilityApp.ts`, `obs/liveEval.ts` (new), `obs/liveEval.test.ts` (new), `obs/evaluate.ts`, `obs/evaluate.test.ts`, `ai/src/service/prompts.ts` (`LIVE_JUDGE_*`), `obs/local/models/obsScore.ts` (`origin`), `obs/local/models/obsEvalRun.ts` (new), `obs/local/localPlugin.ts`, `obs/types.ts`, `ai/src/types/observability.ts`, `ai/src/index.ts`, `example-backend/src/server.ts` (`sampleRate` from `AI_OBS_SAMPLE_RATE`), `docs/reference/ai.md`, `docs/explanation/ai-observability.md`.
+  - Delivers:
+    - The trace export seam: `logRequestAndTrace` calls `app.exportTrace`, and `exportTrace` calls `liveEval.schedule` after the local sink persists.
+    - The `liveEvalRunner` option (default in-process) that T16 plugs into.
+    - `ObsEvalRun` with a unique `{traceId, evaluatorId}` index, a 10-minute `pending` lease that can be reclaimed, and judge tokens and cost. Live scores are upserted on `{traceId, evaluatorId, name, origin}` (partial unique index).
+    - The scheduler: evaluator selection with a 30 s cache, the effective rate, the deterministic roll, the error, no-output, and sensitive skips, and the in-process runner (concurrency 2, cap 500).
+    - Trace-worded judge prompt via `subject: "trace"`.
+    - Scores fan out through `scoreSinks` with `origin: "live"`, and each run writes an `ObsEvalRun`.
+    - With no server `aiService`, live eval logs once and stays off.
+  - Acceptance: AC1, AC2. The new tests fail on the base branch.
+
+### Phase 2: Access and threads foundation
+
+- [ ] **T2** — RBAC resources for alerts, test suites, and the optimizer
+  - Depends on: none
+  - Files: `api/src/rbac/statements.ts`, `api/src/rbac/roleModel.ts`, `api/src/rbac/*.test.ts` (catalog and seed tests), `obs/permissions.ts`, `example-backend/src/rbacRoles.ts`, `example-backend/src/rbacRoles.test.ts`, `docs/reference/api.md` (AI observability RBAC).
+  - Delivers: the `aiAlert` and `aiTestSuite` resources and `aiPrompt.optimize`. The seeded admin, auditor, and superadmin grants are additive on re-seed, and the example viewer and operator roles get the matching grants.
+  - Acceptance: AC16.
+
+- [ ] **T3** — Traces record named groups
+  - Depends on: T1
+  - Files: `ai/src/service/aiService.ts` (`groups` option), `ai/src/service/chatTurn.ts` (`conversation` group), `obs/types.ts` (`TraceRecord.groups`), `obs/traceGroups.ts` (new: validation), `obs/traceGroups.test.ts` (new), `obs/local/models/obsTrace.ts` (field + index), `obs/local/traceStore.ts` (persist, list filter, detail map), `obs/routes/traces.ts`, `obs/routes/traces.test.ts`, `ai/src/types/index.ts` (generate option type), `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`.
+  - Delivers: the Groups contract rows for shape, validation, `session`, chat, the index, and the trace-list filter.
+  - Acceptance: AC4.
+
+### Phase 3: Online evaluation and threads API
+
+- [ ] **T4** — Live sampling filters by prompt and lets sensitive traces opt in
+  - Depends on: T3
+  - Files: `obs/local/models/obsEvaluator.ts` (`livePromptNames`, `includeSensitive`), `obs/local/evaluatorStore.ts`, `obs/routes/evaluators.ts`, `obs/routes/evaluators.test.ts`, `obs/liveEval.ts`, `obs/liveEval.test.ts`, `ai/src/types/observability.ts`, `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`.
+  - Delivers: the required prompt list when a write sets the rate above 0 (400 otherwise), production-label prompt matching, and the `includeSensitive` opt-in (D2). Stored evaluators with a rate but no prompt list are not sampled and stay editable.
+  - Acceptance: AC3, and the sensitive half of AC2.
+
+- [ ] **T5** — Groups list and timeline API
+  - Depends on: T4
+  - Files: `obs/local/groupStore.ts` (new), `obs/routes/groups.ts` (new), `obs/routes/groups.test.ts` (new), `obs/observabilityApp.ts` (mount), `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`, `docs/explanation/model-router-actions.md` (observability plugin routes join the documented exceptions).
+  - Delivers: `GET /groups/kinds`, `GET /groups`, and `GET /groups/:kind/:id`, with rollups and an ordered timeline; `session` reads `sessionId`.
+  - Acceptance: AC5.
+
+- [ ] **T6** — Evaluator live stats and sampling ceiling in status
+  - Depends on: T5
+  - Files: `obs/local/evaluatorStore.ts` (live stats from `ObsEvalRun` and live scores), `obs/routes/evaluators.ts`, `obs/routes/evaluators.test.ts`, `obs/status.ts`, `obs/routes/status.ts`, `ai/src/types/observability.ts`, `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`.
+  - Delivers: evaluator detail `live` and status `liveEval: {available, ceilingPercent, misconfiguredEvaluatorIds}`.
+  - Acceptance: a route test seeds `ObsEvalRun` rows and live scores and reads back the counts and means; a status test covers both `available` values and lists an evaluator with a rate but no prompt list (AC3).
+
+### Phase 4: Monitoring and alerts API
+
+- [ ] **T7** — Metrics API returns time series and breakdowns
+  - Depends on: T6
+  - Files: `obs/local/metricsStore.ts` (new, window functions reused by T9), `obs/local/metricsStore.test.ts` (new), `obs/routes/metrics.ts` (new), `obs/routes/metrics.test.ts` (new), `obs/observabilityApp.ts`, `ai/src/types/observability.ts` (`MetricsResponse`), `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`.
+  - Delivers: the Metrics contract, including the range limits, `null` cost for unpriced buckets, `costPartial`, and `liveEvalCostUsd` from `ObsEvalRun`.
+  - Acceptance: AC6.
+
+- [ ] **T8** — Webhook destinations with signed, metadata-only delivery
+  - Depends on: T7, T2
+  - Files: `obs/local/models/obsWebhook.ts` (new), `obs/local/models/obsWebhookDelivery.ts` (new), `obs/webhooks.ts` (new: payload builder, signer, Slack renderer, URL guard, in-process retry), `obs/webhooks.test.ts` (new), `obs/routes/webhooks.ts` (new: CRUD, rotate secret, `POST /:id/test`, deliveries list), `obs/routes/webhooks.test.ts` (new), `obs/local/localPlugin.ts`, `obs/observabilityApp.ts` (`publicAdminUrl` and `webhookDeliveryRunner` options, mount), `obs/types.ts`, `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`.
+  - Delivers: the Webhooks contract and D1. The payload builder accepts only `ObsWebhookPayload`, the secret is returned once, `url` is never returned after create (`urlHost` only), delivery connects to the checked IP through an undici `Agent` lookup, and redirects are not followed.
+  - Acceptance: AC7. A test seeds a trace whose input and output hold a marker string, fires a delivery, and asserts the body never contains the marker.
+
+- [ ] **T9** — Alert rules evaluate and notify
+  - Depends on: T8
+  - Files: `obs/local/models/obsAlertRule.ts` (new), `obs/alertRules.ts` (new: evaluate, state machine, lease, interval scheduler), `obs/alertRules.test.ts` (new, fake clock), `obs/routes/alertRules.ts` (new: CRUD + `POST /:id/evaluate` dry run), `obs/routes/alertRules.test.ts` (new), `obs/local/localPlugin.ts`, `obs/observabilityApp.ts` (`alerts: {interval, scheduler}` options, mount), `obs/types.ts`, `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`.
+  - Delivers: every alert metric in the IP, the `minSamples` guard, `alert.firing` and `alert.resolved` once each (Q7), a single evaluator per tick via the lease, and the per-metric `traceIds` selection plus `adminUrl` in the payload.
+  - Acceptance: AC8.
+
+### Phase 5: CI test suites
+
+- [ ] **T11** — Test suites and suite runs
+  - Depends on: T9, T2
+  - Files: `obs/local/models/obsTestSuite.ts` (new), `obs/local/models/obsExperiment.ts` (`suiteId`, `kind`), `obs/local/experimentRunner.ts` (one-version suite runs, candidate-only gates), `obs/local/experimentRunner.test.ts`, `obs/local/testSuiteStore.ts` (new), `obs/routes/testSuites.ts` (new), `obs/routes/testSuites.test.ts` (new), `obs/routes/experiments.ts` (`?kind` filter), `obs/routes/experiments.test.ts`, `obs/local/localPlugin.ts`, `obs/observabilityApp.ts`, `ai/src/types/observability.ts`, `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`.
+  - Delivers: suite CRUD with a required stored `target` and `maxCostUsd`; runs that use the stored target or one `{version}` / `{label}` override (Q3); a kind-aware `versions` validator; and the run status shape with `estimatedCostUsd`. The experiments list excludes `kind: "suite"` unless `?kind=suite`.
+  - Acceptance: AC10 for version and label runs, the route half of AC21 (403 without `aiTestSuite:run`), plus a test that a normal experiment still needs 2–3 versions.
+
+- [ ] **T12** — Suite runs accept an inline candidate prompt
+  - Depends on: T11
+  - Files: `obs/local/experimentRunner.ts` (`"candidate"` version key with an inline body), `obs/local/experimentRunner.test.ts`, `obs/routes/testSuites.ts`, `obs/routes/testSuites.test.ts`, `obs/local/models/obsExperiment.ts` (stored candidate), `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`.
+  - Delivers: `{candidate: PromptVersionFields}` stored as version `0` with `candidateBody`, validated like `createVersion`, with a 400 when its variable set differs from the base prompt's production version. No prompt version is created. `mongoose-schema-safety` risk matrix recorded in the PR.
+  - Acceptance: AC10 (inline half).
+
+- [ ] **T13** — Observability service tokens for CI
+  - Depends on: T12
+  - Files: `obs/local/models/obsServiceToken.ts` (new), `obs/serviceTokens.ts` (new: mint, verify, revoke), `obs/serviceTokens.test.ts` (new), `obs/routes/observabilityRouteAccess.ts` (accepts `tobs_` only where a route opts in with `serviceToken: true`), `obs/routes/serviceTokens.ts` (new), `obs/routes/testSuites.ts` (opt in on list, read, run, and run status; rate limit), `obs/local/localPlugin.ts`, `obs/observabilityApp.ts`, `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`.
+  - Delivers: the token contract (D5): hash only, shown once, a required expiry of at most 365 days, at most 10 active per user, scope checked before the legacy-admin shortcut, 401 everywhere else and for a disabled user, 10 runs per hour per token, and audit entries.
+  - Acceptance: AC11.
+
+### Phase 6: Prompt optimizer
+
+- [ ] **T14** — Prompt optimizer engine
+  - Depends on: T13
+  - Files: `obs/local/experimentRunner.ts` (extract `scoreVersionOnItems`), `obs/local/experimentRunner.test.ts`, `obs/local/promptOptimizer.ts` (new), `obs/local/promptOptimizer.test.ts` (new, deterministic fixture model), `obs/local/models/obsOptimization.ts` (new), `obs/local/localPlugin.ts`, `ai/src/service/prompts.ts` (`PROMPT_OPTIMIZER_*`), `ai/src/types/observability.ts`, `docs/reference/ai.md`.
+  - Delivers: the reflective-rewrite loop (Q5), the train/validation split, variable and schema guards, the PHI guards (D6: only person-written or annotator-signed items; rejection of 8-word verbatim runs and of new item tokens or digit runs), estimated-cost tracking from returned token usage, budget stops, and early stop. Existing experiment tests stay green after the extraction.
+  - Acceptance: AC13.
+
+- [ ] **T15** — Optimizer API
+  - Depends on: T14, T2
+  - Files: `obs/routes/optimizations.ts` (new), `obs/routes/optimizations.test.ts` (new), `obs/observabilityApp.ts`, `example-backend/src/__snapshots__/openapi.test.ts.snap`, `docs/reference/ai.md`.
+  - Delivers: estimate, create, list, detail, cancel, and `saveVersion` (Q4: only `latest` moves; requires `reviewedForPersonalData: true`, recorded in the audit entry).
+  - Acceptance: AC14.
+
+### Phase 7: Durable jobs (parallel)
+
+- [ ] **T16** — Live evals run as durable jobs
+  - Depends on: T1
+  - Files: `obs/jobsRunners.ts` (new subpath), `obs/jobsRunners.test.ts` (new), `ai/package.json` (`exports["./observability/jobsRunners"]`), `ai/src/harness/runners/jobsRunner.test.ts` (extend the import guard to the new subpath's reachability from `ai/src/index.ts`), `example-backend/src/server.ts` (pass the jobs runner through the T1 `liveEvalRunner` option when JobsApp is registered), `example-backend/src/utils/observabilityConfig.ts` (add a shared `createExampleObservabilityApp()` that the server and the worker both call), `example-backend/src/jobs/createExampleJobsApp.ts`, `example-backend/src/jobs/defineExampleJobs.ts`, `example-backend/src/jobs/bootstrapJobsWorker.ts` (the standalone worker builds the same observability context, so job handlers have the `aiService` and score sinks), `docs/how-to/background-jobs.md`.
+  - Delivers: the `ai-observability/live-eval` job with 3 attempts, relying on the T1 `ObsEvalRun` claim for idempotent retries, and the in-process fallback when no JobsApp is registered.
+  - Acceptance: AC15 (live-eval half).
+
+- [ ] **T17** — Alert ticks and webhook deliveries run as jobs
+  - Depends on: T9, T16
+  - Files: `obs/jobsRunners.ts`, `obs/jobsRunners.test.ts`, `example-backend/src/server.ts` (pass the jobs scheduler and delivery runner through the T8/T9 options), `example-backend/src/utils/observabilityConfig.ts`, `example-backend/src/jobs/defineExampleJobs.ts`, `docs/how-to/background-jobs.md`.
+  - Delivers: the `ai-observability/alerts-tick` cron (every minute) and the `ai-observability/webhook-delivery` job, which keeps the 0 / 10 s / 60 s schedule. When jobs are used, the interval scheduler stays off.
+  - Acceptance: AC15 (alerts half).
+
+### Phase 8: CLI (parallel)
+
+- [ ] **T18** — `terreno ai test` CLI command
+  - Depends on: T13
+  - Files: `cli/src/commands/ai.ts` (new), `cli/src/commands/ai.test.ts` (new, fake `io.fetch`), `cli/src/runCli.ts`, `cli/src/help.ts`, `docs/reference/cli.md`, `docs/how-to/run-ai-eval-suites-in-ci.md` (new: GitHub Actions and CircleCI snippets, each with the required-secret validation step).
+  - Delivers: the CLI contract in the IP, including the gate table output, exactly one target flag (Q3), and exit codes.
+  - Acceptance: AC12.
+
+### Phase 9: Admin UI
+
+- [ ] **T19** — Threads screens
+  - Depends on: T5
+  - Files: `widgets/threads/*` (new: list and detail screens and views, `useAiObservabilityThreadsApi.ts` via `adminRequest`, tests), `widgets/traces/AiTraceDetailView.tsx` (group chips), `widgets/shell/AiObservabilityScreenWidgets.tsx`, `widgets/shell/aiObservabilityNav.ts`, `admin-frontend/src/isolated/aiObservabilityScreens.isolated.tsx`, `obs/adminScreens.ts`, `obs/adminScreens.test.ts`, `docs/how-to/group-ai-traces.md` (new).
+  - Delivers: the Threads row of the IP's Admin UI table.
+  - Acceptance: AC17, AC18.
+
+- [ ] **T20** — Evaluator live controls and stats
+  - Depends on: T6
+  - Files: `widgets/evaluators/AiEvaluatorPanels.tsx`, `widgets/evaluators/AiEvaluatorNewScreen.tsx`, `widgets/evaluators/evaluatorTypes.ts`, `widgets/evaluators/evaluatorTypes.test.ts`, `widgets/evaluators/useAiObservabilityEvaluatorsApi.ts` (additive response types only), `widgets/shell/AiObservabilityStatusChip.tsx` (additive `liveEval` status type), `admin-frontend/src/isolated/observabilityStatusFixtures.isolated.ts`, `admin-frontend/src/isolated/aiEvaluatorLive.isolated.tsx` (new), `docs/how-to/evaluate-production-traces.md` (new).
+  - Delivers: the Evaluator row of the Admin UI table. Save is disabled while the rate is above 0 and no prompt is chosen; a warning shows when sensitive is on, and when status lists the evaluator as misconfigured.
+  - Acceptance: AC17, AC18. With `GEMINI_API_KEY` set and `AI_OBS_SAMPLE_RATE=1`, a Summarize press in the example app shows a new live score on the evaluator screen (screenshot).
+
+- [ ] **T21** — Monitoring screen
+  - Depends on: T7, T19
+  - Files: `widgets/monitoring/*` (new), `widgets/shell/AiObservabilityScreenWidgets.tsx`, `widgets/shell/aiObservabilityNav.ts`, `admin-frontend/src/isolated/aiObservabilityScreens.isolated.tsx`, `obs/adminScreens.ts`, `obs/adminScreens.test.ts`, `docs/how-to/monitor-ai-quality.md` (new).
+  - Delivers: the Monitoring row of the Admin UI table, with charts per the `dataviz` skill.
+  - Acceptance: AC17, AC18.
+
+- [ ] **T22** — Alerts and webhooks screens
+  - Depends on: T9, T21
+  - Files: `widgets/alerts/*` (new), `widgets/shell/AiObservabilityScreenWidgets.tsx`, `widgets/shell/aiObservabilityNav.ts`, `admin-frontend/src/isolated/aiObservabilityScreens.isolated.tsx`, `obs/adminScreens.ts`, `obs/adminScreens.test.ts`, `docs/how-to/monitor-ai-quality.md`.
+  - Delivers: the Alerts and Webhooks row of the Admin UI table.
+  - Acceptance: AC17, AC18. A screenshot shows the secret-once modal and a delivery log entry from **Send test** against a local receiver.
+
+- [ ] **T23** — Test suites screens, admin runs, and CI tokens
+  - Depends on: T13, T18, T22
+  - Files: `widgets/suites/*` (new), `widgets/shell/AiObservabilityScreenWidgets.tsx`, `widgets/shell/aiObservabilityNav.ts`, `admin-frontend/src/isolated/aiObservabilityScreens.isolated.tsx`, `obs/adminScreens.ts`, `obs/adminScreens.test.ts`, `docs/how-to/run-ai-eval-suites-in-ci.md`.
+  - Delivers: the Test suites row of the Admin UI table, including the stored-target picker in the editor and the **Run** button with an optional version/label override (D4).
+  - Acceptance: AC17, AC18, AC21. The walkthrough starts a run from admin, then mints a token in admin and runs `terreno ai test` against the local backend with it.
+
+- [ ] **T24** — Optimizer screens
+  - Depends on: T15, T23
+  - Files: `widgets/optimizer/*` (new), `widgets/prompts/*` (the **Optimize** button on the prompt hub), `widgets/shell/AiObservabilityScreenWidgets.tsx`, `widgets/shell/aiObservabilityNav.ts`, `admin-frontend/src/isolated/aiObservabilityScreens.isolated.tsx`, `obs/adminScreens.ts`, `obs/adminScreens.test.ts`, `docs/how-to/optimize-a-prompt.md` (new).
+  - Delivers: the Optimizer row of the Admin UI table, including rejected candidates with their reason and the personal-data review tick before **Save as vN+1**.
+  - Acceptance: AC17, AC18.
+
+### Phase 10: Demo and SOP
+
+- [ ] **T25** — Seeded demo data, SOP docs, and full walkthrough
+  - Depends on: T15, T17, T20, T24
+  - Files: `example-backend/src/scripts/seedAiObservabilityDemo.ts`, `example-backend/src/scripts/seedAiObservabilityDemo.test.ts`, `example-backend/src/server.ts` (`publicAdminUrl` from `PUBLIC_APP_URL`), `docs/how-to/observe-llm-calls.md` (env `AI_OBS_SAMPLE_RATE` meaning, seed table), `docs/how-to/ai-feature-development.md` (SOP steps 5–8 use suites, live eval, alerts, and the optimizer), `docs/explanation/ai-observability.md` (production loop section), `docs/implementationPlans/ai-observability.md` (pointer to this plan).
+  - Delivers:
+    - `conversation` groups on seeded chat traces.
+    - Live scores and `ObsEvalRun` rows over the last 7 days, with a dip on one prompt.
+    - One firing alert rule and a disabled demo webhook.
+    - The `summarize-gold` suite with one passed and one failed run.
+    - One finished optimization run from the deterministic `seed-fixture` client.
+    - The seed stays idempotent.
+  - Acceptance: AC19, plus a `verify-ui-changes` pass that re-captures Threads, Evaluator live panel, Monitoring, Alerts, Webhooks, Test suites, and Optimizer on the seeded demo (AC18).

package/src/docs/versioned/next/tasks/ai-observability.md

@@ -18,7 +18,7 @@ Ships design screens: Prompts, Prompt editor (+ Playground), Traces, Trace detai
   - Files: `ai/src/observability/types.ts`, `ai/src/observability/observabilityApp.ts`, `ai/src/observability/observabilityApp.test.ts`, `ai/src/index.ts`
   - Blocked by: none
   - Docs: `docs/explanation/ai-observability.md` (two planes + config errors)
-  - Acceptance: unit tests for local-only valid config, illegal mix, langfuse review queue, missing plugin for a primary
+  - Acceptance: unit tests for local-only valid config, illegal mix, non-local review queue, missing plugin for a primary
 
 - [x] **Task 1.2**: In-memory sinks + `AIService` emit + production prompt resolve
   - Delivers: `promptName` + `promptLabel` resolves via `PromptRegistry` **before the model call**, even when `skipTrace` is true or no trace sink is registered; missing registry/label/version returns 400 without a model call; resolved-version `sensitive` applies unless the generate option overrides it; every `generate*` exports a trace unless `skipTrace`; sink failures logged not thrown; `AIRequest` still written; `userId` / `sessionId` / `sensitive` / `promptRef` / `priceMap` on options; `costUsd` omitted for unpriced models
@@ -171,7 +171,7 @@ Ships design screens: Evaluators, Evaluator detail, New evaluator, Datasets, Dat
   - Acceptance: widget tests for each tab and the empty dataset; `verify-ui-changes` bulk-adds traces and sees them under Needs review
 
 - [ ] **Task 2.9**: Experiment screens (list, wizard, results)
-  - Delivers: list with status badge, running progress bar and cost, plus the BackgroundTask/Langfuse footnote; 4-step wizard with a completed-step rail, dataset picker showing counts and schema match, version checkboxes tagged latest/production/superseded, evaluator checkboxes with per-item cost, and a Review & run step showing the estimate; results with gate tiles, "N gates failing", the **Promote blocked** state naming the failing gate, the outliers list with reason chips, the side-by-side table with failed rows floated and highlighted, and the promote confirm modal
+  - Delivers: list with status badge, running progress bar and cost, plus the BackgroundTask footnote; 4-step wizard with a completed-step rail, dataset picker showing counts and schema match, version checkboxes tagged latest/production/superseded, evaluator checkboxes with per-item cost, and a Review & run step showing the estimate; results with gate tiles, "N gates failing", the **Promote blocked** state naming the failing gate, the outliers list with reason chips, the side-by-side table with failed rows floated and highlighted, and the promote confirm modal
   - Files: `admin-frontend/src/widgets/aiObservability/experiments/*.tsx`
   - Blocked by: 2.5, 2.8, 1.9
   - Docs: how-to steps 5–6
@@ -188,7 +188,7 @@ Ships design screens: Evaluators, Evaluator detail, New evaluator, Datasets, Dat
 
 # Phase 3 — Analytics and pluggable backends
 
-Ships design screens: Sessions, Session timeline, Users, User AI profile, Costs, plus sink health and Open in Langfuse.
+Ships design screens: Sessions, Session timeline, Users, User AI profile, Costs, plus sink health.
 
 - [ ] **Task 3.1**: Session, user, and cost aggregates + screens
   - Delivers: `GET /sessions`, `/sessions/:id`, `/users`, `/users/:id`, `/costs`; Sessions list with the Traces/Sessions toggle and the timeline detail; Users list subtitled "AI cost and usage per user — not the user CRUD table" and the profile with masked email, **no prompt content**, and 7d/30d/all-time rollups; Costs with the KPI row, by-model table showing **tokens only** for unpriced models plus the "no $0 lie" note, and the by-prompt table
@@ -197,19 +197,9 @@ Ships design screens: Sessions, Session timeline, Users, User AI profile, Costs,
   - Docs: `docs/reference/ai.md` routes; `docs/reference/admin-frontend.md`
   - Acceptance: an unpriced model renders `tokens only` and never `$0`; no prompt or output text appears in any users response; widget tests for each screen
 
-- [ ] **Task 3.2**: Langfuse `TraceSink` + `ScoreSink`
-  - Delivers: adapters wrapping the existing `getLangfuseClient()`; **no second OTel SDK**; best-effort with logged failures
-  - Files: `ai/src/observability/langfuse/traceAdapter.ts`, tests with a mocked client
-  - Blocked by: 1.2
-  - Docs: explanation — `LangfuseApp` vs `ObservabilityApp`
-  - Acceptance: one generate dual-writes; a throwing Langfuse mock leaves the local row intact and the caller unaffected
+- [x] ~~**Task 3.2**: Langfuse `TraceSink` + `ScoreSink`~~ — dropped: Langfuse support was removed from `@terreno/ai`.
 
-- [ ] **Task 3.3**: Langfuse `PromptRegistry` / `DatasetStore` / `ExperimentRunner` + primaries
-  - Delivers: `AI_OBS_*_PRIMARY` env wiring; Terreno HTTP facade delegating to the existing Langfuse admin routes; **Open in Langfuse** deep-link bases on `GET /status`
-  - Files: `ai/src/observability/langfuse/*.ts`, tests
-  - Blocked by: 3.2, 1.5, 2.5
-  - Docs: how-to — switching primaries by env
-  - Acceptance: prompts primary `langfuse` reads through the client; experiments primary `langfuse` starts no `BackgroundTask`; the illegal mixes still fail boot
+- [x] ~~**Task 3.3**: Langfuse `PromptRegistry` / `DatasetStore` / `ExperimentRunner` + primaries~~ — dropped: Langfuse support was removed from `@terreno/ai`.
 
 - [ ] **Task 3.4**: `OtelTraceSink` (OpenInference)
   - Delivers: OTLP exporter behind `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`; `openinference.span.kind` set from the span kind; model and token attributes
@@ -219,11 +209,11 @@ Ships design screens: Sessions, Session timeline, Users, User AI profile, Costs,
   - Acceptance: the mock exporter receives an `LLM` span with model and token attributes
 
 - [ ] **Task 3.5**: Sink health chip, example wiring, and phase-3 docs
-  - Delivers: last-error-per-sink on `GET /status`; the chip renders Langfuse health and active primaries and shows **Open in Langfuse** only when that plugin is registered; example-backend adds the Langfuse and OTel sinks conditionally; docs and the `terreno-langfuse-integration.md` pointer finalized
+  - Delivers: last-error-per-sink on `GET /status`; the chip renders sink health and active primaries; example-backend adds the OTel sink conditionally; docs finalized
   - Files: `ai/src/observability/routes/status.ts`, `admin-frontend/.../shell/*.tsx`, `example-backend/src/server.ts`, docs
-  - Blocked by: 3.1, 3.3, 3.4
+  - Blocked by: 3.1, 3.4
   - Docs: (this task)
-  - Acceptance: chip states covered by widget tests; a Langfuse-less boot shows no deep link; `verify-ui-changes` evidence attached
+  - Acceptance: chip states covered by widget tests; a boot without sinks shows a healthy local-only chip; `verify-ui-changes` evidence attached
 
 ---
 

package/src/docs/versioned/next/tasks/docs-reference-coverage.md

@@ -15,7 +15,7 @@ See: [`docs/implementationPlans/docs-reference-coverage.md`](../implementationPl
 ## Phase 1: Missing reference pages
 
 - [x] **Task 1.1**: Write `docs/reference/ai.md`
-  - Description: Convert `.cursor/rules/ai/00-ai.mdc` into a public reference page. Sections, following `docs/reference/api.md`'s shape: Install; Commands; Architecture (file structure); `AIService` setup and full method table; structured-output helpers (`generateJsonValue`, `generateJsonObject`, `generateJsonArray`) including the JSON-normalization behavior and error logging; temperature presets; the `AIRequest` and `GptHistory` models with field tables; route registrars (`addGptRoutes`, `addGptHistoryRoutes`, `addAiRequestsExplorerRoutes`) with endpoints, methods, and permissions; Langfuse integration; integration example with `setupServer`/`TerrenoApp`; environment variables; conventions; testing (including the mock-model pattern). Verify every symbol against `ai/src/index.ts`. Remove the internal-only conventions that make no sense to a consumer (for example instructions about not mocking `@terreno/api` belong in agent rules, not reference docs — use judgment).
+  - Description: Convert `.cursor/rules/ai/00-ai.mdc` into a public reference page. Sections, following `docs/reference/api.md`'s shape: Install; Commands; Architecture (file structure); `AIService` setup and full method table; structured-output helpers (`generateJsonValue`, `generateJsonObject`, `generateJsonArray`) including the JSON-normalization behavior and error logging; temperature presets; the `AIRequest` and `GptHistory` models with field tables; route registrars (`addGptRoutes`, `addGptHistoryRoutes`, `addAiRequestsExplorerRoutes`) with endpoints, methods, and permissions; integration example with `setupServer`/`TerrenoApp`; environment variables; conventions; testing (including the mock-model pattern). Verify every symbol against `ai/src/index.ts`. Remove the internal-only conventions that make no sense to a consumer (for example instructions about not mocking `@terreno/api` belong in agent rules, not reference docs — use judgment).
   - Files: `docs/reference/ai.md` (new)
   - Depends on: none
   - Acceptance: every documented symbol appears in `ai/src/index.ts`; the method table lists all `AIService` methods; all three route registrars documented with their permissions; no `.claude/` or `.cursor/` path referenced.

package/src/docs/versioned/next/testing/e2e/admin-spa.md

@@ -10,14 +10,32 @@ bun run compile && bun run --cwd admin-spa build:web && bun run --cwd admin-spa
 
 | Tier | What it covers | Command |
 | --- | --- | --- |
-| Unit and integration | `AdminSpaServeApp` routing, index rewriting, the base-path global, app-config merging, the dev proxy, and the SPA's store and components (`src/**/*.test.ts(x)`, `store/`, `components/`) | `bun run --cwd admin-spa test:ci` |
-| Smoke | The compiled serve plugin serving the built SPA from a real server in a child process | `bun run --cwd admin-spa smoke` |
-| E2E | `test:e2e`: Chromium against the built SPA on `e2e/serveTestApp.ts`, a backend-free server with mocked Better Auth and `/admin/config`. It covers the login redirect, an admin signing in to AdminHome, and a wrong password showing an error (`e2e/login.spec.ts`) | `bun run --cwd admin-spa test:e2e` |
-| E2E (integration) | `test:integration`: Chromium against example-backend serving the SPA at `/console` with Better Auth and the admin API. It covers sign-in and admin CRUD (`e2e/integration/*.spec.ts`). It needs a running backend with `ADMIN_SPA_ENABLED=true` and a seeded admin; see `playwright.integration.config.ts` | `bun run --cwd admin-spa test:integration` |
+| Unit and integration | `AdminSpaServeApp` routing, the server render and its 503 without a server build, the basePath/baseUrl check, the access gate, cache headers, `loadSsrPayload`, the route loaders, app-config merging, the dev proxy, and the SPA's store and components (`src/**/*.test.ts(x)`, `store/`, `components/`, `server/`) | `bun run --cwd admin-spa test:ci` |
+| Smoke | The compiled serve plugin server-rendering the built export from a real server in a child process | `bun run --cwd admin-spa smoke` |
+| E2E | `test:e2e`: Chromium against the built export on `e2e/serveTestApp.ts`, a backend-free server with mocked Better Auth and `/admin/config` | `bun run --cwd admin-spa test:e2e` |
+| E2E (integration) | `test:integration`: Chromium against example-backend serving the SPA at `/console` with Better Auth and the admin API. It needs a running backend with `ADMIN_SPA_ENABLED=true` and a seeded admin; see `playwright.integration.config.ts` | `bun run --cwd admin-spa test:integration` |
+| Image smoke | The built `example-backend/Dockerfile` image server-rendering `/console/login` and the root loader | `scripts/admin-spa-image-smoke.sh <base-url>` |
 | Manual regression | Cases in `docs/regression/` | none |
 
-CircleCI runs `test:e2e` in `admin-spa-ci` and `test:integration` in `admin-spa-integration`.
-The smoke runs only in the `smoke` job.
+### Specs
+
+| Spec | Layer | Covers |
+| --- | --- | --- |
+| `e2e/ssr.spec.ts` | `browser` | The login page holds the brand and form with JavaScript off; an anonymous protected route gets a server 302; login hydrates without a warning, by direct visit or client navigation |
+| `e2e/login.spec.ts` | `browser` | The anonymous redirect to `/login` with `returnTo` (unsafe values ignored), an admin signing in to AdminHome, a wrong password showing an error |
+| `e2e/shell.ssr.spec.ts` | `browser` | The admin shell is in the first HTML with no spinner; hydration does not request the admin config again; no loader data or shell payload stays in the page |
+| `e2e/integration/adminSpa.ssr.list.integration.spec.ts` | `browser` | `/console/User` shows seeded rows with JavaScript off and hydrates without `GET /admin/users` |
+| `e2e/integration/adminSpa.ssr.detail.integration.spec.ts` | `browser` | `/console/User/<id>` (also fully percent-encoded) shows the form's values with JavaScript off and hydrates without a read |
+| `e2e/integration/adminSpa.ssr.windowed.integration.spec.ts` | `browser` | `/console/Todo` shows seeded rows; syncdb starts and keeps them; a change made right after the render shows without a refresh |
+| `e2e/integration/adminSpa.ssr.rpc.integration.spec.ts` | `browser` | `/console/configuration` and the home widgets show their data with JavaScript off |
+| `e2e/integration/adminSpa.ssr.jobs.integration.spec.ts` | `browser` | `/console/jobs` shows dashboard data, not a spinner, with JavaScript off |
+| `e2e/integration/adminSpa.ssr.comms.integration.spec.ts` | `browser` | `/console/comms` shows seeded message rows with JavaScript off |
+| `e2e/integration/adminSpa.ssr.orgs.integration.spec.ts` | `browser` | `/console/orgs` and an organization's members page show their data with JavaScript off |
+
+The other `e2e/integration/` specs cover sign-in, admin CRUD and the MCP OAuth consent page.
+
+CircleCI runs `test:e2e` in `admin-spa-ci`, `test:integration` in `admin-spa-integration`, the
+smoke in the `smoke` job, and the image smoke in `admin-spa-image-smoke`.
 
 ## The smoke
 
@@ -25,39 +43,50 @@ The smoke runs only in the `smoke` job.
 runs at the `http` layer. It re-runs itself with a `serve` argument to start a child process
 that imports the compiled `@terreno/admin-spa` by package name, registers `AdminSpaServeApp` on
 a plain Express app at `/console` with a custom `brandName`, and listens on a `freePort()`. The
-plugin serves the export from its default location (`admin-spa/dist`), the path a published
-package uses. The parent waits for `/console/app-config.json` with `spawnUntilReady`, then
-checks over real HTTP:
+plugin serves the export (`client/` + `server/`) from its default location (`admin-spa/dist`),
+the path a published package uses. The parent waits for `/console/app-config.json` with
+`spawnUntilReady`, then checks over real HTTP:
 
-1. `GET /console/` returns the HTML index with `window.__ADMIN_SPA_BASE__="/console"` injected.
-2. The entry bundle the index names under `/console/_expo/` returns JavaScript.
-3. `GET /console/app-config.json` returns the custom `brandName` merged over the default
+1. `GET /console/login` answers 200 HTML with `Cache-Control: no-store, private`, holding the
+   brand name, the login form and `window.__ADMIN_SPA_BASE__="/console"`.
+2. The entry bundle the page names under `/console/_expo/` returns JavaScript.
+3. `GET /_expo/loaders/login` (the site root) answers 200 with `Cache-Control: no-store, private`.
+4. `GET /console/app-config.json` returns the custom `brandName` merged over the default
    providers.
-4. A deep route, `GET /console/User/123`, falls back to the same index.
+5. An anonymous `GET /console/User/123` answers 302 to `/console/login?returnTo=%2FUser%2F123`.
 
 It prints one `✅` line per check and exits 1 on the first failure, after printing the
-server's output. Without `admin-spa/dist/index.html` it exits 1 at once and asks for
-`build:web`; otherwise the plugin would serve an empty index.
+server's output. Without `admin-spa/dist/server` it exits 1 at once and asks for `build:web`.
 
 The smoke needs no browser, MongoDB, or secrets. The browser flows belong to `test:e2e` and
 `test:integration`.
 
-**Measured time:** 4–4.5 s in two root `bun run smoke` runs after the build (the
-[guide's](../e2e.md#package-pages) figure), excluding the compile and the web export. Single runs
-took 6–8 s wall clock. Most of it is loading `@terreno/test` and starting the child;
-the checks take well under a second. Budget: 30 s.
+**Budget:** 30 s, excluding the compile and the web export.
+
+## The image smoke
+
+Job `admin-spa-image-smoke` builds `example-backend/Dockerfile`, runs the image with
+`ADMIN_SPA_ENABLED=true` against a mongo replica set, and runs
+`scripts/admin-spa-image-smoke.sh`. It passes only when `/console/login` is 200 HTML with the
+brand and `admin-spa-login-email`, and `/_expo/loaders/login` is 200 JSON with
+`Cache-Control: no-store, private`. It also runs the image with
+`ADMIN_SPA_DIST_DIR=/nonexistent` and expects the script to fail with a 503. See
+[Run tests locally](../../how-to/run-tests-locally.md) for the local commands.
 
 ## Add an e2e test
 
-- **New serve-plugin option or branch:** add a unit test in `admin-spa/src/serve.test.ts`. The
-  smoke stays one happy path; do not add edge cases to it.
-- **The serving contract changed** (the base-path global, the app-config route, or the SPA
-  fallback): update the checks in `admin-spa/e2e/smoke.ts` and keep it within 30 s.
+- **New serve-plugin option, gate rule or loader branch:** add a unit test in
+  `admin-spa/src/serve.test.ts`, `src/gate.test.ts`, or `src/server/` next to the loader's
+  tests. The smoke stays one happy path; do not add edge cases to it.
+- **The serving contract changed** (the SSR login page, the root loader path, cache headers,
+  the app-config route, the anonymous redirect): update the checks in `admin-spa/e2e/smoke.ts`
+  and keep it within 30 s.
 - **A screen or flow that needs no real backend:** add a Playwright spec to `admin-spa/e2e/`
   and extend the mocks in `e2e/serveTestApp.ts`. Follow
   [Write a Playwright spec](../e2e.md#write-a-playwright-spec) for `testID`s and waits.
-- **A flow that needs the real admin API or Better Auth** (CRUD, permissions): add a spec to
-  `admin-spa/e2e/integration/`.
+- **A server-rendered screen, or a flow that needs the real admin API or Better Auth:** add a
+  spec to `admin-spa/e2e/integration/`, one file per screen family
+  (`adminSpa.ssr.<area>.integration.spec.ts`).
 
 See [the `@terreno/admin-spa` reference](../../reference/admin-spa.md) for the plugin API and
 [the `@terreno/test` reference](../../reference/test.md#smoke-helpers) for the smoke helpers.

package/dist/docs/versioned/57.17.0/README.md

@@ -1,72 +0,0 @@
-# Terreno Documentation
-
-**Terreno is Django/Rails for TypeScript — with universal app support.**
-
-Terreno is Django/Rails for TypeScript — a batteries-included, full-stack
-framework where the undifferentiated 80% of an app is already written. On the
-backend you get Mongoose models, auto-generated REST APIs, permissions, an admin
-panel, authentication, and an AI service. On the frontend you get one universal
-app — a single React Native codebase that ships to iOS, Android, and web. It is
-built to be driven by AI coding agents from the first prompt to a production
-deploy.
-
-- Batteries included — auth, CRUD APIs, admin, permissions, AI, realtime,
-  feature flags, and consent are already built, so your code is business logic.
-- Universal by default — one React Native codebase ships to iOS, Android, and
-  web. Not a web framework with a mobile bolt-on.
-- AI-native — agents are a first-class client of the framework, not an
-  afterthought.
-
-Canonical copy, language rules, and the Django/Rails comparison:
-[Positioning](explanation/positioning.md).
-
-## Structure
-
-- **[Tutorials](tutorials/)** — Learning-oriented, hands-on lessons
-- **[How-to guides](how-to/)** — Problem-oriented, practical steps
-- **[Reference](reference/)** — Technical reference for APIs and packages
-- **[Explanation](explanation/)** — Concepts and context
-- **[Implementation Plans](implementationPlans/)** — Forward-looking plans for major features
-
-## Packages
-
-| Package | Description |
-|---------|-------------|
-| [@terreno/api](reference/api.md) | REST API framework for Express/Mongoose (modelRouter, auth, OpenAPI) |
-| [@terreno/test](reference/test.md) | Shared Bun test helpers, MongoDB preload utilities, and HTTP fixtures |
-| [@terreno/ui](reference/ui.md) | React Native UI component library for iOS, Android, and web |
-| [@terreno/syncdb](reference/syncdb.md) | Local-first data layer with TinyBase, durable outbox, and delta sync |
-| [@terreno/ai](reference/ai.md) | Provider-agnostic AI service with streaming chat, request logging, and admin tools |
-| [@terreno/blocks](reference/blocks.md) | Shared contracts for agent-driven UI: ask schemas and whole-reply block documents |
-| [@terreno/admin-backend](reference/admin-backend.md) | Admin panel backend plugin for `@terreno/api` |
-| [@terreno/admin-frontend](reference/admin-frontend.md) | Admin panel frontend screens for `@terreno/api` backends |
-| [@terreno/admin-spa](reference/admin-spa.md) | Standalone admin SPA (Expo Router web) plus Express serve plugin |
-| [@terreno/api-health](reference/api-health.md) | Health check plugin for `@terreno/api` |
-| [@terreno/announcements](reference/announcements.md) | In-app product update announcements |
-| [@terreno/comms](reference/comms.md) | Pluggable communications providers |
-| [@terreno/feature-flags](reference/feature-flags.md) | Feature flags and A/B testing plugin for `@terreno/api` |
-| [@terreno/mcp](reference/mcp-server.md) | MCP server that gives coding agents Terreno docs, codegen tools, and prompts |
-| [@terreno/cli](reference/cli.md) | `terreno` CLI for docs, codegen, bootstrap, and OpenAPI REST |
-| [create-terreno-app](reference/create-terreno-app.md) | CLI to scaffold a deployable full-stack Terreno app (`bunx create-terreno-app`) |
-
-### Legacy
-
-| Package | Description |
-|---------|-------------|
-| [@terreno/rtk](reference/legacy/rtk.md) (deprecated for data sync) | OpenAPI SDK, Better Auth Redux, and feature flags for Terreno frontends |
-
-**Migrating from RTK?** See [How to migrate from @terreno/rtk to @terreno/syncdb](how-to/migrate-rtk-to-syncdb.md).
-
-## Quick links
-
-- [Getting started](tutorials/getting-started.md)
-- [Set up Terreno MCP](how-to/set-up-terreno-mcp.md)
-- [Debug with MCP](how-to/debug-with-mcp.md)
-- [Create a Terreno app](how-to/create-a-terreno-app.md)
-- [Positioning](explanation/positioning.md)
-- [How admin interfaces are shaped](explanation/admin-interface.md)
-- [Build admin screens](how-to/build-admin-screens.md)
-- [Local-first data](explanation/local-first-data.md)
-- [AGENTS.md](../AGENTS.md) — Onboarding context for AI assistants
-- [ROADMAP.md](../ROADMAP.md) — Public roadmap
-- [CONTRIBUTING.md](../CONTRIBUTING.md) — Setup, tests, and pull requests