@terreno/mcp 57.8.0 → 57.10.1

package/src/docs/versioned/next/tasks/ai-observability.md

@@ -1,134 +1,263 @@
-# Tasks: AI observability (Langfuse-light)
+# Tasks: AI observability (Langfuse replacement)
 
 IP: [ai-observability](../implementationPlans/ai-observability.md)
+Design: [AI Observability Prototype](https://claude.ai/design/p/36eec001-f2f0-4161-b60d-20dace4d0a9c?file=AI+Observability+Prototype.dc.html)
 
 Supporting skills (every task as applicable): `mongoose-schema-safety`, `terreno-backend-api`, `backend-test-env`, `ai-prompt-governance`, `update-docs`, `generate-sdk`, `verify-ui-changes`, `terreno-ui`, `terreno-data-fetching`, `building-terreno-apps`.
 
-## Phase 1 — Contracts and tracer
+---
 
-- [ ] **Task 1.1**: Observability types + `ObservabilityApp` config validation
-  - Delivers: `ObservabilityPlugin` / sinks / `ObservabilityControlConfig`; boot throws on `experiments.primary !== datasets.primary` or `reviewQueue` ≠ `local`; throws if a primary’s plugin is missing
+# Phase 1 — MVP: prompt library, tracing, review queue
+
+Ships design screens: Prompts, Prompt editor (+ Playground), Traces, Trace detail, Review queue, Review item.
+
+## 1A — Contracts and tracer
+
+- [x] **Task 1.1**: Observability types + `ObservabilityApp` config validation
+  - Delivers: `ObservabilityPlugin` / `TraceSink` / `ScoreSink` / `PromptRegistry` / `ReviewQueue` interfaces; `ObservabilityControlConfig`; boot throws on `experiments.primary !== datasets.primary`, on `reviewQueue !== "local"`, and on a primary whose plugin is missing
   - Files: `ai/src/observability/types.ts`, `ai/src/observability/observabilityApp.ts`, `ai/src/observability/observabilityApp.test.ts`, `ai/src/index.ts`
   - Blocked by: none
-  - Docs: stub `docs/explanation/ai-observability.md` (two planes + config errors)
-  - Acceptance: unit tests for valid local-only, both plugins + defaults, illegal mix, langfuse reviewQueue, local-off + prompts primary local
+  - Docs: `docs/explanation/ai-observability.md` (two planes + config errors)
+  - Acceptance: unit tests for local-only valid config, illegal mix, langfuse review queue, missing plugin for a primary
 
-- [ ] **Task 1.2**: In-memory TraceSink/ScoreSink + `AIService` emit
-  - Delivers: every `generate*` exports a trace unless `skipTrace`; sink failures logged not thrown; `AIRequest` still written; `userId`/`sessionId`/`promptRef`/`priceMap` on options
+- [x] **Task 1.2**: In-memory sinks + `AIService` emit + production prompt resolve
+  - Delivers: `promptName` + `promptLabel` resolves via `PromptRegistry` **before the model call**, even when `skipTrace` is true or no trace sink is registered; missing registry/label/version returns 400 without a model call; resolved-version `sensitive` applies unless the generate option overrides it; every `generate*` exports a trace unless `skipTrace`; sink failures logged not thrown; `AIRequest` still written; `userId` / `sessionId` / `sensitive` / `promptRef` / `priceMap` on options; `costUsd` omitted for unpriced models
   - Files: `ai/src/observability/memorySinks.ts`, `ai/src/service/aiService.ts`, `ai/src/types/index.ts`, `ai/src/service/aiService.test.ts`
   - Blocked by: 1.1
-  - Docs: `docs/reference/ai.md` — `skipTrace`, `promptRef`, usage/cost fields
-  - Acceptance: tests: emit; skipTrace; throwing sink; costUsd from price map; missing model → no usd; multi-step still logs AIRequest
+  - Docs: `docs/reference/ai.md` — `skipTrace`, `promptName`, `promptLabel`, `sensitive`, usage/cost fields
+  - Acceptance: tests for emit, `skipTrace`, throwing sink, cost from price map, **no `costUsd` when the model is unpriced**, multi-step still logs `AIRequest`, resolved prompt body reaches the model with `skipTrace`, and missing registry/production label → 400 with no model call
 
-- [ ] **Task 1.3**: GPT routes pass identity
-  - Delivers: `/gpt/prompt` sets `userId` from `req.user` and `sessionId` from body/header `x-ai-session-id` into `AIService`
-  - Files: `ai/src/routes/gpt.ts`, `ai/src/routes/gpt.test.ts` (or existing route tests)
+- [x] **Task 1.3**: GPT routes pass identity and sensitivity
+  - Delivers: `/gpt/prompt` sets `userId` from `req.user`, `sessionId` from body/header `x-ai-session-id`, and may upgrade sensitivity with request `sensitive: true`; client `false` never downgrades a sensitive prompt version. Client `promptName` / `promptLabel` selection is admin-only; app routes select production prompts server-side
+  - Files: `ai/src/routes/gpt.ts`, route tests
   - Blocked by: 1.2
   - Docs: `docs/reference/ai.md` GPT section
-  - Acceptance: authenticated prompt produces a memory sink trace with that userId
+  - Acceptance: an authenticated prompt produces a memory-sink trace carrying that `userId` and `sessionId`
 
-## Phase 2 — Local telemetry + prompts
+## 1B — Local store: prompts, traces, review
 
-- [ ] **Task 2.1**: Local Mongo models for prompts + traces + spans + scores
-  - Delivers: schemas with descriptions, plugins, indexes from the IP; models registered only when local plugin constructs
-  - Files: `ai/src/observability/local/models/*.ts`, `ai/src/observability/local/types.ts`, model tests
+- [x] **Task 1.4**: Local Mongo models — prompts, versions, labels, traces, spans, scores
+  - Delivers: schemas with a description on every field, standard plugins, the indexes in the IP; models registered only when the local plugin constructs; `ObsPromptVersion` carries `variables[]` (`key`, `required`, `label?`, `reviewerNote?`), `inputSchema`, `outputSchema`, `outputFieldNotes`, **`sensitive` (default false)**, `config`; `ObsPrompt` carries `folder` + `tags[]`; `ObsTrace` carries `sensitive`, `errorSummary`, `prompts[]`; `ObsSpan` carries `status` and optional short `error`
+  - Files: `ai/src/observability/local/models/*.ts`, `ai/src/types/observability*.ts`, model tests
   - Blocked by: 1.1
   - Docs: `docs/reference/ai.md` model tables
-  - Acceptance: save/load version immutability; unique `(promptId, label)`; findExactlyOne/findOneOrNone only
+  - Acceptance: version immutability; unique `(promptId, label)`; `findExactlyOne` / `findOneOrNone` only; no unique index on scores
 
-- [ ] **Task 2.2**: Local `PromptRegistry` + prompt HTTP
-  - Delivers: create named prompt, immutable versions, move production/latest, compile `{{vars}}`, get by label
+- [x] **Task 1.5**: Local `PromptRegistry` + prompt HTTP + playground
+  - Delivers: create prompt in a folder; immutable `vN+1` create; move `production` / `staging` returning the outgoing version; `{{var}}` compile; `get({name, label})`; `POST /prompts/:name/playground` runs one `AIService` call and returns compiled messages, output, latency, tokens, cost without creating a version; list supports folder filter, search, and `include=usage7d`
   - Files: `ai/src/observability/local/promptStore.ts`, `ai/src/observability/routes/prompts.ts`, tests
-  - Blocked by: 2.1, 1.1
-  - Docs: `docs/how-to/observe-llm-calls.md` — create + pin production
-  - Acceptance: v1 unchanged after v2; get production returns v2; playground compile without new version
+  - Blocked by: 1.4, 1.1
+  - Docs: `docs/how-to/observe-llm-calls.md` — create a prompt and pin production
+  - Acceptance: v1 unchanged after v2; `get production` returns the labelled version; playground creates no version; usage rollup returns calls/cost for the last 7 days and `—` for a prompt with no production label
 
-- [ ] **Task 2.3**: Local `TraceSink`/`ScoreSink` + trace/session/user/cost HTTP
-  - Delivers: persist fan-in from ObservabilityApp; list/detail/filters; cost aggregates
+- [x] **Task 1.6**: Local `TraceSink` / `ScoreSink` + trace HTTP
+  - Delivers: persist the fan-in; list with time / prompt / status / user / session / `hasScore` / `sensitive` / `flaggedForDataset` filters and pagination; detail returns the span tree with kinds, offsets, durations, per-span I/O and cost; derive `errorSummary` from the first span where `status: "error"` using its `error`; `prompts[]` length drives the `N prompts` display; `POST /traces/:id/scores`
   - Files: `ai/src/observability/local/traceStore.ts`, `ai/src/observability/routes/traces.ts`, tests
-  - Blocked by: 2.1, 1.2
+  - Blocked by: 1.4, 1.2
   - Docs: `docs/reference/ai.md` routes
-  - Acceptance: filter user/session/prompt/minCost; nested spans round-trip; costs by model
-
-## Phase 3 — Evaluators, live sample, review
-
-- [ ] **Task 3.1**: Evaluator model + LLM-judge + code/json-assert
-  - Delivers: multidimensional scores; judge prompt from PromptRegistry production (or named constant in `prompts.ts` for the built-in judge wrapper); structured JSON via `generateJsonObject`
-  - Files: `ai/src/observability/local/evaluatorStore.ts`, `ai/src/observability/evaluate.ts`, `ai/src/service/prompts.ts` (judge wrapper constant), tests
-  - Blocked by: 2.2, 2.3
-  - Docs: explanation — dimensions; how-to — define evaluator
-  - Acceptance: numeric + boolean dimensions on one trace; judge parse failure records error score not throw to caller
-
-- [ ] **Task 3.2**: `sampleRate` live evals via BackgroundTask
-  - Delivers: default 0; >0 enqueues eval after trace export; still never throws from generate
-  - Files: `ai/src/observability/liveEval.ts`, `observabilityApp.ts`, tests
-  - Blocked by: 3.1
-  - Docs: how-to env `AI_OBS_SAMPLE_RATE`
-  - Acceptance: rate 0 → no task; rate 1 → BackgroundTask created (mock/stub runner ok)
-
-- [ ] **Task 3.3**: Local review queue API
-  - Delivers: enqueue from trace; list pending; submit scores; skip; assign
-  - Files: `ai/src/observability/local/reviewStore.ts`, `ai/src/observability/routes/review.ts`, tests
-  - Blocked by: 3.1
-  - Docs: how-to — review flow
-  - Acceptance: submit writes ScoreSink fan-out path; done item not in pending
-
-## Phase 4 — Datasets and local experiments
-
-- [ ] **Task 4.1**: Datasets + items (including from trace)
-  - Delivers: CRUD; add-from-trace copies input/output snapshot
-  - Files: dataset store + routes + tests
-  - Blocked by: 2.3
-  - Docs: how-to — gold set from traces
-  - Acceptance: item retains sourceTraceId; delete item does not delete trace
-
-- [ ] **Task 4.2**: Local ExperimentRunner always BackgroundTask
-  - Delivers: POST experiment compares 2–3 prompt versions on a dataset, runs evaluators, stores per-item results, aggregates; promote → production label
-  - Files: `ai/src/observability/local/experimentRunner.ts`, routes, tests
-  - Blocked by: 4.1, 3.1, 2.2
-  - Docs: how-to — run experiment + promote
-  - Acceptance: 1-item dataset still uses BackgroundTask; completed aggregates; promote moves label; failure sets experiment failed
-
-## Phase 5 — Langfuse adapter, OTel, admin, example
-
-- [ ] **Task 5.1**: Langfuse TraceSink + ScoreSink wrapping existing client
-  - Delivers: no second OTel SDK; best-effort; uses `getLangfuseClient()`
-  - Files: `ai/src/observability/langfuse/traceAdapter.ts`, tests (mock client)
+  - Acceptance: nested spans round-trip; every filter tested; a trace with two prompts reports both; unpriced model returns no `costUsd`
+
+- [x] **Task 1.7**: Human evaluator model + templates
+  - Delivers: `ObsEvaluator` with `type`, `target`, typed `dimensions[]`, `runModes`, `instructions`, `confidenceAlertBelow`; CRUD routes; `GET /evaluators/templates` returning the seeded set; `human` + `liveSampleRate > 0` rejected on save
+  - Files: `ai/src/observability/local/evaluatorStore.ts`, `ai/src/observability/evaluatorTemplates.ts`, `ai/src/observability/routes/evaluators.ts`, tests
+  - Blocked by: 1.4
+  - Docs: `docs/how-to/ai-feature-development.md` step 4 (human half)
+  - Acceptance: numeric / boolean / categorical dimensions persist; human evaluator with live sampling is a 400; a template installs by name
+
+- [x] **Task 1.8**: Local review queue API
+  - Delivers: `POST /traces/review` enqueues one or many traces against a human evaluator (`reason: "manual"`); `GET /review` by status with counts, oldest-first; `GET /review/:id` returns the item, its evaluator dimensions, and rendered `given` / `wrote` panels built from the prompt version's `variables[]` labels/notes and `outputSchema` + `outputFieldNotes`, falling back to raw keys; `POST /review/:id` submit / skip / assign
+  - Files: `ai/src/observability/local/reviewStore.ts`, `ai/src/observability/reviewPanels.ts`, `ai/src/observability/routes/review.ts`, tests
+  - Blocked by: 1.6, 1.7
+  - Docs: `docs/how-to/observe-llm-calls.md` — the review flow
+  - Acceptance: submit writes dimension-keyed scores through the ScoreSink fan-out and leaves pending; skip and assign move status; panels fall back cleanly for a prompt with no labels; a submitted item is absent from pending
+
+## 1C — Admin UI and example app
+
+- [x] **Task 1.9**: Admin chrome — nav group, breadcrumbs, status chip
+  - Delivers: one `AI Observability` sidebar group; `Admin / AI Observability / <Section> / <leaf>` breadcrumbs; status chip reading `GET /status` (local state, primaries); **Review queue hidden when the local plugin is off**; existing `AI Requests` untouched
+  - Files: `admin-frontend/src/widgets/aiObservability/shell/*.tsx`, `builtInWidgets.ts`
+  - Blocked by: 1.1
+  - Docs: `docs/reference/admin-frontend.md` screen names
+  - Acceptance: widget tests for chip states and for the hidden review entry
+
+- [x] **Task 1.10**: Prompts list + prompt editor + playground screens
+  - Delivers: folder rail with counts and search; table with type badge, latest, production, last used, calls 7d, cost 7d and the latest-vs-production tooltips; editor with full-width version history rows (`vN`, attached labels, creation time, selected highlight), Editor/Playground tabs, system + template + variables + schema summary, temperature preset and model hint, **Save as vN+1**, **Set vN as production…** confirm modal naming the outgoing version; playground variable inputs, Run once with a running state, compiled-messages disclosure, output, latency/tokens/cost, and a **Save this run to dataset** button disabled until phase 2
+  - Files: `admin-frontend/src/widgets/aiObservability/prompts/*.tsx`
+  - Blocked by: 1.5, 1.9
+  - Docs: `docs/how-to/ai-feature-development.md` steps 3 and 6
+  - Acceptance: widget tests empty/loading/loaded; no in-place save control exists; `verify-ui-changes` walks create → save v2 → set production
+
+- [x] **Task 1.11**: Traces list + trace detail screens
+  - Delivers: filter bar; row selection with the bulk bar (**Send to review queue**, sensitive-count warning, Clear; Add to dataset disabled until phase 2); status dot, `sensitive` badge, error line, `N prompts`, spans/tokens/cost/latency/scores columns; pagination footer; detail header with actions, left span list (kind badge, indent, duration bar), right span detail with **collapsed** sensitive I/O disclosures, and the scores panel with value + source
+  - Files: `admin-frontend/src/widgets/aiObservability/traces/*.tsx`
+  - Blocked by: 1.6, 1.8, 1.9
+  - Docs: `docs/how-to/ai-feature-development.md` steps 7–8 (observe half)
+  - Acceptance: widget tests for empty, error row, sensitive badge, and bulk bar; `verify-ui-changes` selects traces and sends them to the queue
+
+- [x] **Task 1.12**: Review queue + review item screens
+  - Delivers: tabs Pending / In progress / Done / Skipped with counts; oldest-first table (item, what the AI did, prompt, assignee, waiting, status); **Start reviewing — oldest first**; empty state naming both intake paths; item screen with "Item N of M pending" and prev/next, read-only given/wrote panels (long fields collapsed with word count, structured output as key/value rows, per-field reviewer notes), raw JSON disclosure, dimension-driven score form (slider / Pass-Fail / pills) headed by the evaluator instructions, comment, **Submit & next** / Skip / Assign to me, and a toast reporting the remaining count
+  - Files: `admin-frontend/src/widgets/aiObservability/review/*.tsx`
+  - Blocked by: 1.8, 1.9
+  - Docs: `docs/how-to/ai-feature-development.md` — the review loop
+  - Acceptance: widget tests for each dimension control type and the empty state; `verify-ui-changes` reviews an item end to end and reaches "Queue clear"
+
+- [x] **Task 1.13**: example-backend registration + seed + phase-1 docs
+  - Delivers: `ObservabilityApp` with the local plugin always on; `AI_OBS_PRICE_MAP_JSON` wiring; seed prompt `example-summarize` v1 labelled production in folder `examples`; seed the human evaluator template so the queue is walkable on a fresh DB; `.env.example`
+  - Files: `example-backend/src/server.ts`, seed script, `.env.example`, `docs/explanation/ai-observability.md`, `docs/how-to/observe-llm-calls.md`, `docs/how-to/ai-feature-development.md`, `docs/reference/ai.md`
+  - Blocked by: 1.10, 1.11, 1.12
+  - Docs: (this task)
+  - Acceptance: server boots local-only with no Langfuse keys; a stranger can follow the how-to to create a prompt, pin production, trace a call, and review it; `bun run --filter=@terreno/ai compile test lint` green
+
+---
+
+# Phase 2 — Evaluation loop: evaluators, datasets, experiments
+
+Ships design screens: Evaluators, Evaluator detail, New evaluator, Datasets, Dataset detail, Experiments, New experiment, Experiment results.
+
+- [x] **Task 2.1**: `llm-judge` + `json-assert` evaluator types with the schema contract
+  - Delivers: judge execution via `AIService.generateJsonObject` against a registry judge prompt; `json-assert` in-process assertion (`path`, `constraint`, plus the built-in "validate against the prompt version `outputSchema`" mode); **create/save rejects a judge whose prompt output schema omits a required dimension, naming that dimension**; parse failure records an error score rather than throwing to the caller
+  - Files: `ai/src/observability/evaluate.ts`, `ai/src/observability/local/evaluatorStore.ts`, `ai/src/observability/evaluatorTemplates.ts`, tests
+  - Blocked by: 1.7, 1.5
+  - Docs: explanation — dimensions and the schema contract; how-to step 4
+  - Acceptance: mismatch rejected with the dimension key in the message; a judge writes exactly its declared keys; `confidenceAlertBelow` defaults to 0.7; templates for correctness / hallucination / helpfulness / toxicity / schema-assert install (`*-human` variants preserved for the review queue)
+
+- [ ] **Task 2.2**: Live sampling execution
+  - Delivers: after trace export, roll each evaluator's `liveSampleRate` (capped by `AI_OBS_SAMPLE_RATE`) and enqueue the eval on the same `BackgroundTask` path experiments use; never throws from generate
+  - Files: `ai/src/observability/liveEval.ts`, `ai/src/observability/observabilityApp.ts`, tests
+  - Blocked by: 2.1, 1.2
+  - Docs: how-to — the sampling ceiling and its cost
+  - Acceptance: rate 0 → no task; rate 100 with ceiling 0 → no task; rate 100 with ceiling 100 → task created
+
+- [x] **Task 2.3**: Datasets and items with provenance
+  - Delivers: dataset CRUD with `inputSchemaPromptName` + `expectedOutputSchema`; item CRUD with `origin`, `proofread`, `annotatedBy`, `tags`, `outcomeClass`; `POST /datasets/:id/import` (**JSON** array of bare objects or structured rows, plus **CSV** via `Content-Type: text/csv` or `{format: "csv", content}`; validates against the bound input schema); `POST /traces/add-to-dataset` single and bulk copying the I/O snapshot with `sourceTraceId`; **a `sensitive` trace always lands `proofread: false`**
+  - Files: `ai/src/observability/local/datasetStore.ts`, `ai/src/observability/routes/datasets.ts`, tests
+  - Blocked by: 1.6
+  - Docs: how-to steps 1–2 and 8
+  - Acceptance: sensitive rule enforced; item keeps `sourceTraceId`; deleting an item does not touch the trace; import rejects schema-invalid rows with the failing path
+
+- [ ] **Task 2.4**: Synthetic dataset generation + playground save-to-dataset
+  - Delivers: `POST /datasets/:id/generate` drafting items via `AIService` and a named generator prompt, `origin: "synthetic"`, `proofread: false`; the phase-1 playground **Save this run to dataset** button wired
+  - Files: generate route, `ai/src/service/prompts.ts` generator constant, `admin-frontend/.../prompts/*`, tests
+  - Blocked by: 2.3, 1.5
+  - Docs: how-to step 1 (synthetic)
+  - Acceptance: generated items are excluded from a default experiment run; a playground run lands as an unproofread item with its input and output
+
+- [x] **Task 2.5**: Experiment runner, gates, and gate-blocked promote
+  - Delivers: `POST /experiments/estimate` (generations, USD, wall clock); `POST /experiments` comparing 2–3 versions on a dataset with optional `modelOverride`, always via `BackgroundTask` locally; per-item results per version; aggregates vs `thresholds[]` (defaulting to `SOP_DEFAULT_THRESHOLDS`); outlier and low-confidence item ids; failed items ordered first; `POST /experiments/:id/promote` returning **409 while any gate fails**
+  - Files: `ai/src/observability/local/experimentRunner.ts`, `ai/src/observability/routes/experiments.ts`, tests
+  - Blocked by: 2.3, 2.1, 1.5
+  - Docs: how-to steps 5–6
+  - Acceptance: a 1-item dataset still uses `BackgroundTask`; gate pass/fail computed for boolean (`trueRate`) and numeric (`mean`) dimensions; promote 409s on a failing gate and moves the label when gates pass; unproofread items excluded unless `includeUnproofread`
+
+- [ ] **Task 2.6**: Authenticated trace feedback
+  - Delivers: `POST /traces/:id/feedback` — thumbs / outcome class / flag-for-dataset; `source: "user-feedback"` score; owner (`trace.userId`) or admin; sets `flaggedForDataset`; optional review enqueue with `reason: "feedback"`
+  - Files: `ai/src/observability/routes/feedback.ts`, tests
+  - Blocked by: 1.6, 1.8
+  - Docs: how-to step 7; `docs/reference/ai.md`
+  - Acceptance: owner thumbs-down + flag succeeds; a stranger gets 403; an admin succeeds; the trace list `flaggedForDataset` filter finds it
+
+- [ ] **Task 2.7**: Evaluator screens (list, detail, new)
+  - Delivers: list with type badge, dimension summary, target, run-mode chips and the footnote; detail with the dimensions table, type-specific panel (judge prompt link + model + schema-match check, assertion path/constraint, reviewer instructions), run modes with the live-sampling slider and cost warning, and the **Used by** list with 30-day runs and cost; new-evaluator screen with type/target pickers, dimension builder, and the **inline schema mismatch error naming the undeclared dimension**
+  - Files: `admin-frontend/src/widgets/aiObservability/evaluators/*.tsx`
+  - Blocked by: 2.1, 1.9
+  - Docs: `docs/reference/admin-frontend.md`
+  - Acceptance: widget tests for each type's panel and for the mismatch error; `verify-ui-changes` creates a judge evaluator
+
+- [ ] **Task 2.8**: Dataset screens (list, detail)
+  - Delivers: list with the human/auto legend, provenance bar, source and updated columns; detail with human/auto counts, the input-schema binding line, Add item, Run experiment, tabs All / Human / Auto / **Needs review** (red count), and the items table with attribution lines and trace links
+  - Files: `admin-frontend/src/widgets/aiObservability/datasets/*.tsx`
+  - Blocked by: 2.3, 2.4, 1.9
+  - Docs: `docs/reference/admin-frontend.md`
+  - Acceptance: widget tests for each tab and the empty dataset; `verify-ui-changes` bulk-adds traces and sees them under Needs review
+
+- [ ] **Task 2.9**: Experiment screens (list, wizard, results)
+  - Delivers: list with status badge, running progress bar and cost, plus the BackgroundTask/Langfuse footnote; 4-step wizard with a completed-step rail, dataset picker showing counts and schema match, version checkboxes tagged latest/production/superseded, evaluator checkboxes with per-item cost, and a Review & run step showing the estimate; results with gate tiles, "N gates failing", the **Promote blocked** state naming the failing gate, the outliers list with reason chips, the side-by-side table with failed rows floated and highlighted, and the promote confirm modal
+  - Files: `admin-frontend/src/widgets/aiObservability/experiments/*.tsx`
+  - Blocked by: 2.5, 2.8, 1.9
+  - Docs: how-to steps 5–6
+  - Acceptance: widget tests for running progress, gate-fail results, and blocked promote; `verify-ui-changes` runs an experiment and promotes a passing version
+
+- [ ] **Task 2.10**: Phase-2 example seeds + docs
+  - Delivers: seed judge/assert evaluator templates and dataset `example-gold` with an input-schema binding; docs updated for evaluators, datasets, experiments, feedback, and `AI_OBS_SAMPLE_RATE`
+  - Files: `example-backend` seed, docs listed in the IP, `.rulesync/skills/ai-prompt-governance/SKILL.md` + `bun run skills:sync`
+  - Blocked by: 2.7, 2.8, 2.9, 2.6
+  - Docs: (this task)
+  - Acceptance: the full SOP loop is runnable from `docs/how-to/ai-feature-development.md` without Langfuse
+
+---
+
+# Phase 3 — Analytics and pluggable backends
+
+Ships design screens: Sessions, Session timeline, Users, User AI profile, Costs, plus sink health and Open in Langfuse.
+
+- [ ] **Task 3.1**: Session, user, and cost aggregates + screens
+  - Delivers: `GET /sessions`, `/sessions/:id`, `/users`, `/users/:id`, `/costs`; Sessions list with the Traces/Sessions toggle and the timeline detail; Users list subtitled "AI cost and usage per user — not the user CRUD table" and the profile with masked email, **no prompt content**, and 7d/30d/all-time rollups; Costs with the KPI row, by-model table showing **tokens only** for unpriced models plus the "no $0 lie" note, and the by-prompt table
+  - Files: `ai/src/observability/routes/analytics.ts`, `ai/src/observability/local/traceStore.ts`, `admin-frontend/src/widgets/aiObservability/analytics/*.tsx`, tests
+  - Blocked by: 1.6, 1.9
+  - Docs: `docs/reference/ai.md` routes; `docs/reference/admin-frontend.md`
+  - Acceptance: an unpriced model renders `tokens only` and never `$0`; no prompt or output text appears in any users response; widget tests for each screen
+
+- [ ] **Task 3.2**: Langfuse `TraceSink` + `ScoreSink`
+  - Delivers: adapters wrapping the existing `getLangfuseClient()`; **no second OTel SDK**; best-effort with logged failures
+  - Files: `ai/src/observability/langfuse/traceAdapter.ts`, tests with a mocked client
   - Blocked by: 1.2
-  - Docs: explanation — LangfuseApp vs ObservabilityApp
-  - Acceptance: generate with both sinks; langfuse mock throws → generate still ok; local row exists
+  - Docs: explanation — `LangfuseApp` vs `ObservabilityApp`
+  - Acceptance: one generate dual-writes; a throwing Langfuse mock leaves the local row intact and the caller unaffected
 
-- [ ] **Task 5.2**: Langfuse PromptRegistry / DatasetStore / ExperimentRunner adapters
-  - Delivers: primaries can be `langfuse`; Terreno HTTP facade delegates; Open in Langfuse URLs on status
+- [ ] **Task 3.3**: Langfuse `PromptRegistry` / `DatasetStore` / `ExperimentRunner` + primaries
+  - Delivers: `AI_OBS_*_PRIMARY` env wiring; Terreno HTTP facade delegating to the existing Langfuse admin routes; **Open in Langfuse** deep-link bases on `GET /status`
   - Files: `ai/src/observability/langfuse/*.ts`, tests
-  - Blocked by: 5.1, 2.2, 4.2
-  - Docs: how-to — switch primaries via env
-  - Acceptance: prompts primary langfuse list uses client; experiments primary langfuse does not start BackgroundTask
+  - Blocked by: 3.2, 1.5, 2.5
+  - Docs: how-to — switching primaries by env
+  - Acceptance: prompts primary `langfuse` reads through the client; experiments primary `langfuse` starts no `BackgroundTask`; the illegal mixes still fail boot
 
-- [ ] **Task 5.3**: `OtelTraceSink` OpenInference
-  - Delivers: OTLP HTTP JSON or SDK exporter; `openinference.span.kind=LLM`; optional endpoint
-  - Files: `ai/src/observability/otel/otelTraceSink.ts`, tests with mock exporter
+- [ ] **Task 3.4**: `OtelTraceSink` (OpenInference)
+  - Delivers: OTLP exporter behind `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`; `openinference.span.kind` set from the span kind; model and token attributes
+  - Files: `ai/src/observability/otel/otelTraceSink.ts`, tests with a mock exporter
   - Blocked by: 1.2
-  - Docs: how-to — Phoenix/collector endpoint
-  - Acceptance: mock exporter receives span with model + token attrs
+  - Docs: how-to — pointing at Phoenix or a generic collector
+  - Acceptance: the mock exporter receives an `LLM` span with model and token attributes
 
-- [ ] **Task 5.4**: `admin-frontend` screens
-  - Delivers: prompts, traces, costs, evaluators, datasets, experiments, review; status chip; hide review without local; wire `customScreens`
-  - Files: `admin-frontend/src/widgets/aiObservability/*.tsx`, `builtInWidgets.ts`, `ai/src` adminContribution
-  - Blocked by: 2.2, 2.3, 3.3, 4.2
-  - Docs: `docs/reference/admin-frontend.md` screen names
-  - Acceptance: widget tests empty/loading; `verify-ui-changes` on admin (login admin, open Prompts + Traces)
-
-- [ ] **Task 5.5**: example-backend register + seed + env
-  - Delivers: local plugin on; Langfuse sinks if keys; OTel if endpoint; seed prompt/evaluator; fail-fast config
-  - Files: `example-backend/src/server.ts`, seed script, `.env.example`
-  - Blocked by: 5.1, 5.3, 2.2, 3.1
-  - Docs: how-to uses example-backend snippet
-  - Acceptance: server boots with local-only; invalid primary mix prevents listen (test or documented script)
-
-- [ ] **Task 5.6**: Docs completeness + Langfuse IP pointer + prompt-governance skill note
-  - Delivers: explanation, how-to, reference complete; `docs/implementationPlans/terreno-langfuse-integration.md` see-also; skill: registry production prompts allowed
-  - Files: docs listed in IP; `.github/skills/ai-prompt-governance/SKILL.md` (source) + `bun run skills:sync` if required
-  - Blocked by: 5.5
+- [ ] **Task 3.5**: Sink health chip, example wiring, and phase-3 docs
+  - Delivers: last-error-per-sink on `GET /status`; the chip renders Langfuse health and active primaries and shows **Open in Langfuse** only when that plugin is registered; example-backend adds the Langfuse and OTel sinks conditionally; docs and the `terreno-langfuse-integration.md` pointer finalized
+  - Files: `ai/src/observability/routes/status.ts`, `admin-frontend/.../shell/*.tsx`, `example-backend/src/server.ts`, docs
+  - Blocked by: 3.1, 3.3, 3.4
+  - Docs: (this task)
+  - Acceptance: chip states covered by widget tests; a Langfuse-less boot shows no deep link; `verify-ui-changes` evidence attached
+
+---
+
+# Phase 4 — Domain-neutral RBAC and prompt relationship hub
+
+The user approved this phase on 2026-10-02. It adds no domain-specific vocabulary or
+in-app approval workflow. Supporting skills: `building-admin-interfaces`,
+`terreno-backend-api`, `terreno-ui`, `terreno-data-fetching`, `mongoose-schema-safety`,
+`update-docs`, and `verify-ui-changes`.
+
+- [x] **Task 4.1**: Observability RBAC vocabulary, compatibility grants, and route enforcement
+  - Delivers: resources `aiPrompt`, `aiTrace`, `aiReview`, `aiDataset`, `aiExperiment`, `aiEvaluator` with the actions approved in Q62/Q63; statement descriptions; full seeded Admin grants, read-only Auditor expansion, Super Admin wildcard; existing stored Admin roles gain only missing observability grants; optional `accessControl` on `ObservabilityApp`; every route maps to one resource/action while legacy admin fallback remains
+  - Files: `api/src/rbac/statements.ts`, `api/src/rbac/roleModel.ts`, `ai/src/observability/observabilityApp.ts`, `ai/src/observability/routes/*.ts`, route/RBAC tests, public exports as needed
+  - Blocked by: none
+  - Docs: `docs/reference/api.md`, `docs/reference/ai.md`, `docs/explanation/ai-observability.md`
+  - Acceptance: catalog and role tests prove exact actions; a read-only RBAC caller can GET but receives 403 for writes; denied mutations fail even when called directly; legacy admin tests remain green
+
+- [x] **Task 4.2**: Prompt descriptions and prompt-scoped relationship APIs
+  - Delivers: optional prompt `description` in Mongo/types/create/detail/list; trace list accepts prompt version in addition to name; experiment list accepts `promptName`; prompt detail API composes bounded recent traces and experiments for the selected prompt without client-side full-list scans
+  - Files: prompt/trace/experiment models, stores, routes, types, tests
+  - Blocked by: 4.1
+  - Docs: `docs/reference/ai.md`, `docs/explanation/ai-observability.md`
+  - Acceptance: description round-trips; prompt v2 filtering excludes v1-only traces; prompt detail relationships contain only the named prompt's traces and experiments; old prompts without descriptions still load
+
+- [x] **Task 4.3**: Caller-filtered navigation and read-only prompt hub
+  - Delivers: each custom screen declares its read/list permission; unauthorized screens disappear from `/admin/config`; prompt detail renders Overview, Versions, Traces, and Experiments with description/folder/tags; controls are hidden independently for denied `update`, `promote`, and `playground`; direct API calls remain protected by Task 4.1
+  - Files: `ai/src/observability/adminScreens.ts`, admin contribution tests, `admin-frontend/src/widgets/aiObservability/prompts/*`, permission helpers/tests
+  - Blocked by: 4.1, 4.2
+  - Docs: `docs/reference/admin-frontend.md`, `docs/how-to/ai-feature-development.md`
+  - Acceptance: admin-config tests cover mixed grants; widget tests prove a read-only prompt hub shows relationships and no write controls; an operator sees all controls; `verify-ui-changes` captures both roles
+
+- [x] **Task 4.4**: Phase-4 integration and documentation gate
+  - Delivers: `ObservabilityApp({accessControl: access})` in example-backend; example roles demonstrating read-only and operator composition; integration tests for admin vs viewer HTTP; all architecture/reference/how-to pages aligned; generated SDK and `verify-ui-changes` walkthrough when the parent closes UI gates
+  - Files: `example-backend` access/seed files, `observabilityRbac.integration.test.ts`, docs named above, generated SDK when required
+  - Blocked by: 4.3
   - Docs: (this task)
-  - Acceptance: a stranger can register ObservabilityApp from how-to without reading the IP
+  - Acceptance: example boots with existing admin access intact; a consumer can configure a domain-neutral read-only role from docs; example-backend RBAC unit + integration tests, compile, and lint pass; parent runs SDK regen and mandatory UI walkthrough before checking this task complete

package/src/docs/versioned/next/tasks/dashboard-chart-parity.md

@@ -0,0 +1,145 @@
+# Tasks: Dashboard chart parity
+
+IP: [dashboard-chart-parity.md](../implementationPlans/dashboard-chart-parity.md)  
+**Follows:** https://github.com/TerrenoLabs/terreno/issues/1175  
+**Roadmap issue:** set after IP approval
+
+**Feature profile:** false (full IP)
+
+## Phase 1 — Tracer (scorecard)
+
+- [x] **Task 1.1**: `SparklineChart` + dotted `comparisonData`
+  - Delivers: public plot-only sparkline; solid current series; dotted comparison aligned on `label`; no axes/grid/legend/tooltip row; empty input does not throw
+  - Files: `ui/src/SparklineChart.tsx`, `ui/src/SparklineChart.test.tsx`, `ui/src/charts/paths.ts` (reuse line path), `ui/src/Common.ts` (TypeDoc props), `ui/src/index.tsx` / `heavyOptionalExports.tsx`
+  - Blocked by: none
+  - Skills: `terreno-ui`, `update-docs`
+  - Docs: stub `SparklineChart` on `docs/reference/ui.md` Display list
+  - Acceptance: bun tests — N points → path present; comparison path present when `comparisonData` set; empty data shows nothing / emptyText if provided; `ui` compile green
+
+- [x] **Task 1.2**: `Scorecard` + `ChartCard`
+  - Delivers: KPI tile with title, formatted value, optional sparkline; `ChartCard` with title, period badge, optional `onPeriodPress`, optional `filterSummary`
+  - Files: `ui/src/Scorecard.tsx`, `ui/src/Scorecard.test.tsx`, `ui/src/ChartCard.tsx`, `ui/src/ChartCard.test.tsx`, lazy exports
+  - Blocked by: 1.1
+  - Skills: `terreno-ui`, `verify-ui-changes`, `update-docs`
+  - Docs: stub both on `docs/reference/ui.md`
+  - Acceptance: bun tests — value + title render; sparkline mounts when `sparklineData` set; badge press fires; period badge has no press handler when `onPeriodPress` omitted
+
+- [x] **Task 1.3**: Scorecard demo + first golden
+  - Delivers: demo story; gallery fixture `scorecard-sparkline-comparison` (five KPI tiles, static data); PNG golden
+  - Files: `demo/stories/Scorecard.stories.tsx`, `demo/stories/ChartCard.stories.tsx`, `demo/demoConfig.tsx`, `demo/chartVisual/fixtureCatalog.ts`, `demo/chartVisual/fixtures.tsx`, `demo/rendered-snapshots/scorecard-sparkline-comparison.png`
+  - Blocked by: 1.2
+  - Skills: `terreno-ui`, `review-chart-visuals`, `verify-ui-changes`
+  - Docs: `docs/how-to/compare-chart-rendered-snapshots.md` lists the new id
+  - Acceptance: fixture catalog test includes the id; `bun run ui:charts:compare` green for it; story registered
+
+## Phase 2 — Cartesian additive facades
+
+- [x] **Task 2.1**: `series` + `comparisonData` on `LineChart` / `AreaChart`
+  - Delivers: multi-series lines from `series`; single `data` still works; dotted comparison overlay; per-series legend; markers stay
+  - Files: `ui/src/LineChart.tsx`, `ui/src/AreaChart.tsx`, matching tests, `ui/src/charts/scales.ts` (shared x domain across series)
+  - Blocked by: 1.2
+  - Skills: `terreno-ui`, `update-docs`
+  - Docs: `docs/reference/ui.md` Line/Area props
+  - Acceptance: bun tests — three `series` → three paths + three legend labels; `{label,value}[]` only still draws one series; comparison path exists when set
+
+- [x] **Task 2.2**: `xTickPolicy` + bar comparison + header shortcuts
+  - Delivers: `"auto"` rotates when labels would truncate, stays horizontal for ≤7 categories; `BarChart` accepts `series` (grouped later if >1 — this slice: first series only unless a second is comparison) and `comparisonData`; `title` / `periodLabel` / `onPeriodPress` on cartesian facades
+  - Files: `ui/src/BarChart.tsx`, `ui/src/LineChart.tsx`, `ui/src/charts/layout.ts`, `ui/src/BarChart.test.tsx`, `ui/src/LineChart.test.tsx`
+  - Blocked by: 2.1
+  - Skills: `terreno-ui`
+  - Docs: tick policy in `docs/explanation/charts.md`
+  - Acceptance: bun tests — 14 long date labels with `auto` → rotated tick style; 7 weekday labels stay horizontal; facade title renders when set and `ChartCard` is not wrapping (documented exclusive use)
+
+- [x] **Task 2.3**: Cartesian goldens
+  - Delivers: fixtures `bar-time-rotated-ticks`, `line-three-series`, `bar-day-of-week`
+  - Files: `demo/chartVisual/**`, `demo/rendered-snapshots/*.png`
+  - Blocked by: 2.2
+  - Skills: `review-chart-visuals`, `verify-ui-changes`
+  - Docs: compare-how-to fixture table
+  - Acceptance: `bun run ui:charts:compare` green for the three ids
+
+## Phase 3 — Donut center and share legend
+
+- [x] **Task 3.1**: Donut hole + percent legend
+  - Delivers: `centerTitle`, `centerValue`; legend `swatch + label + formatShare` (default percent of total); 100% single-slice still draws a ring + center
+  - Files: `ui/src/DonutChart.tsx`, `ui/src/DonutChart.test.tsx`
+  - Blocked by: 1.2
+  - Skills: `terreno-ui`, `update-docs`
+  - Docs: `docs/reference/ui.md` Donut props
+  - Acceptance: bun tests — center copy present; legend includes `%`; one-slice data still has one path; existing slice color override still works
+
+- [x] **Task 3.2**: Donut goldens
+  - Delivers: `donut-center-and-share`, `donut-single-slice`
+  - Files: `demo/chartVisual/**`, goldens
+  - Blocked by: 3.1
+  - Skills: `review-chart-visuals`
+  - Docs: compare-how-to
+  - Acceptance: compare green for both ids
+
+## Phase 4 — Spanning grid
+
+- [x] **Task 4.1**: `DashboardGridItem` spans
+  - Delivers: eager `DashboardGridItem` with `span` per breakpoint; default span 1; children without a wrapper keep current equal-column behavior
+  - Files: `ui/src/DashboardGrid.tsx` and/or `ui/src/DashboardGridItem.tsx`, `ui/src/DashboardGrid.test.tsx`, `ui/src/index.tsx` (eager)
+  - Blocked by: 1.2
+  - Skills: `terreno-ui`, `update-docs`
+  - Docs: `docs/reference/ui.md` Layout
+  - Acceptance: bun tests — span 2 in a 4-column row yields ~2× cell width minus gap; unwrapped children still fill 1 column
+
+- [ ] **Task 4.2**: `hows-it-going-dashboard` chart composition
+  - Delivers: hard fixture matching the reference layout (5 scorecards, cost/conv bar, 3-line imp share, DOW bar, two donuts). Table slot may be a `ChartCard` placeholder until 5.2
+  - Files: `demo/chartVisual/fixtures.tsx`, `fixtureCatalog.ts`, golden
+  - Blocked by: 2.3, 3.2, 4.1
+  - Skills: `review-chart-visuals`, `verify-ui-changes`
+  - Docs: how-to example rewritten to this composition
+  - Acceptance: compare green; fixture uses `Scorecard` / `ChartCard` / `DashboardGridItem`, not ad-hoc `Box` widths only
+
+## Phase 5 — Table chrome
+
+- [ ] **Task 5.1**: Header groups, footer rows, range status, wrapLines
+  - Delivers: `headerGroups` second header row; `footerRows` sticky totals; range copy `start–end of total` with existing pagination; `wrapLines={2}` wraps cell text inside `rowHeight` (virtualization unchanged)
+  - Files: `ui/src/DataTable.tsx`, `ui/src/DataTable.test.tsx`, `ui/src/Pagination.tsx` (if range lives there), `ui/src/Common.ts`
+  - Blocked by: 4.1
+  - Skills: `terreno-ui`, `update-docs`
+  - Docs: `docs/reference/ui.md` DataTable; pointer from charts how-to
+  - Acceptance: bun tests — grouped header text visible; footer row present; range text `1–50 of 452` with page 1 size 50 total 452; pinned column scroll still syncs with groups/footer; wrapLines 2 does not change `rowHeight`
+
+- [ ] **Task 5.2**: Table card in the hard fixture
+  - Delivers: one `ChartCard` + `DataTable` in `hows-it-going-dashboard` (Search Terms or Ad Group Check-in static rows, totals, period headers)
+  - Files: `demo/chartVisual/fixtures.tsx`, update golden
+  - Blocked by: 5.1, 4.2
+  - Skills: `review-chart-visuals`, `verify-ui-changes`
+  - Docs: how-to shows table in the same grid
+  - Acceptance: compare green after intentional golden update; wrapped first-column names visible in the PNG
+
+## Phase 6 — Packaging and docs
+
+- [ ] **Task 6.1**: Lazy root + import regression
+  - Delivers: `Scorecard`, `SparklineChart`, `ChartCard`, chart facades stay on `heavyOptionalExports`; `DashboardGrid` / `DashboardGridItem` eager; root import does not evaluate chart files
+  - Files: `ui/src/lazyBoundaries/heavyOptionalExports.tsx`, `ui/src/index.tsx`, `ui/src/RootImportRegression.test.tsx`, `docs/reference/ui-performance.md`
+  - Blocked by: 4.1, 5.1, 2.1, 3.1
+  - Skills: `terreno-ui`, `update-docs`
+  - Docs: performance page lists new lazy exports
+  - Acceptance: `RootImportRegression` fails if new chart files load with the root import; subpath imports work
+
+- [ ] **Task 6.2**: Diátaxis + changelog + rules
+  - Delivers: how-to rewritten for easy-build composition; explanation covers scorecard, comparison stroke, additive facades vs grammar; reference complete; changelog fragment; ui rule export list; TypeDoc
+  - Files: `docs/how-to/charts-and-dashboards.md`, `docs/explanation/charts.md`, `docs/reference/ui.md`, `docs/how-to/README.md`, `.rulesync/rules/ui/00-ui.mdc`, `changelog/unreleased/dashboard-chart-parity.md`; `bun run website:generate` if needed
+  - Blocked by: 6.1, 5.2
+  - Skills: `update-docs`, `docs-audit`, `terreno-ui`
+  - Docs: the files listed above
+  - Acceptance: a stranger can copy the how-to into a screen that resembles the reference; explanation states grammar is not required; changelog `category: Added`; existing single-series how-to steps still valid as a subset
+
+## Verification mapping
+
+| Criterion | Method |
+|-----------|--------|
+| Scorecard + sparkline + comparison | bun tests + `scorecard-sparkline-comparison` PNG |
+| Multi-series line | bun tests + `line-three-series` PNG |
+| Rotated vs horizontal ticks | bun tests + `bar-time-rotated-ticks` / `bar-day-of-week` |
+| Donut center + share | bun tests + donut goldens |
+| Grid spans + reference layout | `hows-it-going-dashboard` PNG + grid unit tests |
+| Table chrome | bun tests + table region of hard golden |
+| Facade compatibility | existing `*Chart.test.tsx` stay green |
+| Lazy charts | `RootImportRegression` |
+| Frontend proof | `verify-ui-changes` on demo gallery; artifacts on the implementation PR |