@terreno/mcp 57.2.0 → 57.4.0

package/src/docs/versioned/next/implementationPlans/ai-agents-and-failover.md

@@ -0,0 +1,237 @@
+# Implementation plan: AI agents and provider failover
+
+**Status:** Draft  
+**Branch:** `cursor/ai-agents-and-failover-ip-97fd`  
+**Owner:** unassigned  
+**Created:** 2026-08-24  
+**Package:** `@terreno/ai`  
+**Related:** Laravel AI SDK comparison (items 1 and 3): named Agent reuse + LanguageModel failover on 429/503  
+
+## Goal
+
+Add two additive library seams to `@terreno/ai` so app code can:
+
+1. Define a reusable **Agent** (`name`, `instructions`, `tools`, optional `schema`, optional `middleware`) that runs through existing `AIService` logging.
+2. Wrap one or more Vercel AI SDK `LanguageModel`s in **`createFailoverModel`** so 429/502/503/overloaded failures try the next model. Every `AIService` method inherits failover without per-method changes.
+
+Destination: a consuming app can write a typed agent, pass a failover chain as `model`, and prove with mocks that the second model serves after the first throws a failoverable error.
+
+## Non-Goals
+
+- Embeddings, pgvector/Atlas Vector, RAG, `SimilaritySearch` (comparison item 2).
+- Image generation, TTS, STT, reranking, provider file/vector stores.
+- New HTTP routes, `AiApp` options, or GPT chat rewrite.
+- Human-in-the-loop tool approval / resume.
+- First-class sub-agents (keep existing `parentRequestId` / `subRequestIds` only).
+- Queued generation, Echo-style broadcast, `Agent.fake()` Laravel helpers.
+- Changing `AIService` constructor (`model` stays a single `LanguageModel`).
+- Mid-stream failover after the client has received text deltas.
+
+## Decisions
+
+Grow recorded these as executable defaults from the comparison plus current `@terreno/ai` shape (`AIService` + injected `LanguageModel` in `ai/src/service/aiService.ts`). Change a row before Approve if Pick should do something else.
+
+| ID | Question | Decision |
+| --- | --- | --- |
+| D1 | Scope | Agent API + failover wrapper only. No embeddings, no multimodal generation, no new routes. |
+| D2 | Agent shape | Class `Agent` wrapping an `AIService` instance. Not a replacement for `AIService`. Laravel-style PHP agent classes are the DX target, not the type system. |
+| D3 | HTTP | No new routes in v1. Chat stays `addGptRoutes`. Apps call `agent.run()` from their own services/routes. |
+| D4 | Failover seam | `createFailoverModel({models, isFailoverable?})` returns a `LanguageModel`. Pass it as `AIService` `model`. Do not add `models[]` to the constructor. |
+| D5 | Failover triggers | Default: HTTP 429, 502, 503, 529; SDK/provider errors whose `statusCode` matches; messages matching rate-limit / overloaded / capacity. Not 400/401/403/404. Not structured-output parse failures (`NoObjectGeneratedError`). |
+| D6 | Same-model retry | Each listed model is tried once per call. No extra retries on the same model. |
+| D7 | Streaming | Fail over only if `doStream` **rejects before returning a stream**. After the stream exists, errors propagate. No replay of partial output. |
+| D8 | Logging | Successful `AIRequest.aiModel` is the model that served. `metadata.failover` records `{attempts: [{modelId, error?}], servedModelId}`. Failures still never throw from logging. |
+| D9 | Instructions | Agent `instructions` are prompts. Named constants (app `prompts.ts` or `ai/src/service/prompts.ts`). `ai-prompt-governance` applies. |
+| D10 | Schema + tools | v1 allows tools on text/chat runs. `schema` uses `generateJsonObject` and **does not** combine with tools in the same `run`. Streaming `runStream` is text-only (no schema). |
+| D11 | Middleware | Array of `(input, next) => Promise<output>` around `run` / `runStream` only. Not Express middleware. |
+| D12 | Example app | Optional tracer in `example-backend` if two providers are already configured; otherwise tests-only tracer is enough. |
+| D13 | Compatibility | Additive exports only. Existing `new AIService({model})` and `AiApp` stay valid. |
+
+## Architecture
+
+```
+App
+  Agent({instructions, tools?, schema?, middleware?}, aiService)
+    → AIService.generateText | generateChatStream | generateJsonObject
+        → LanguageModel (optionally createFailoverModel)
+            → primary.doGenerate / doStream
+            → on failoverable error → next model
+        → AIRequest.logRequest (served model id + failover metadata)
+```
+
+`AIService` stays the only path that talks to the Vercel SDK for logged calls. Agent does not call `generateText` / `streamText` from `ai` directly.
+
+Failover is a `LanguageModel` Proxy (same idea as `withStrippedJsonFencesModel` in `aiService.ts`): intercept `doGenerate` and `doStream`, preserve other properties from the **primary** model (`specificationVersion`, `provider`, `supportedUrls`).
+
+`modelId` on the wrapper: `failover(<primaryId>)` for identification. `AIService` logging must read **served** id via a small well-known slot on the wrapper (for example `lastServedModelId`) so explorer rows are not all `failover(...)`.
+
+### Agent API (proposed)
+
+```typescript
+interface AgentContext {
+  userId?: mongoose.Types.ObjectId;
+}
+
+interface AgentRunOptions extends AgentContext {
+  prompt: string;
+  messages?: Array<{content: string; role: "user" | "assistant" | "system"}>;
+  temperature?: number;
+}
+
+interface AgentMiddleware<OUTPUT = string> {
+  (
+    input: AgentRunOptions,
+    next: (input: AgentRunOptions) => Promise<OUTPUT>
+  ): Promise<OUTPUT>;
+}
+
+interface AgentConfig<OUTPUT = string> {
+  name: string;
+  instructions: string | ((ctx: AgentContext) => string | Promise<string>);
+  tools?: ToolSet | ((ctx: AgentContext) => ToolSet | Promise<ToolSet>);
+  schema?: import("ai").FlexibleSchema<OUTPUT>;
+  middleware?: Array<AgentMiddleware<OUTPUT>>;
+  temperature?: number;
+  maxSteps?: number;
+  requestType?: string;
+}
+
+class Agent<OUTPUT = string> {
+  constructor(config: AgentConfig<OUTPUT>, service: AIService);
+  run(options: AgentRunOptions): Promise<OUTPUT | string>;
+  runStream(options: AgentRunOptions): AsyncGenerator<string>;
+}
+```
+
+Dispatch:
+
+| Config | Implementation |
+| --- | --- |
+| `schema` set | `AIService.generateJsonObject` with `systemPrompt` = resolved instructions. Ignore tools. |
+| `tools` set, no schema | Collect `generateChatStream` with `stopWhen: stepCountIs(maxSteps ?? 5)` (same default as `AiApp.maxSteps`). |
+| neither | `AIService.generateText` with `systemPrompt` = instructions. |
+
+`requestType` defaults to `"agent"` (extend `AIRequestType` union / explorer filters). Store `metadata.agentName`.
+
+Factory export `defineAgent(config)` that returns `(service) => new Agent(config, service)` is optional sugar; Pick may ship class-only if that is smaller.
+
+### Failover API (proposed)
+
+```typescript
+interface CreateFailoverModelOptions {
+  models: LanguageModel[];
+  isFailoverable?: (error: unknown) => boolean;
+}
+
+createFailoverModel(options: CreateFailoverModelOptions): LanguageModel;
+isFailoverableError(error: unknown): boolean;
+```
+
+- `models.length < 1` throws at construction (early return / guard).
+- `isFailoverable` defaults to `isFailoverableError`.
+- On failover: `logger.warn` with `fromModelId`, `toModelId`, `error` (no secrets).
+- If every model fails: throw the **last** error (do not wrap in a generic message that hides the provider error).
+- Structured-json path: `withStrippedJsonFencesModel(createFailoverModel(...))` still works because failover is a LanguageModel. Order: app wraps failover first, then `AIService` applies fence stripping on that wrapper for JSON methods.
+
+### `getModelId` / logging
+
+Update `getModelId` (or logging helpers) so if `model` has `lastServedModelId: string`, log that. Wrapper sets it on success. Tests assert `AIRequest.aiModel` equals the backup id after a primary 429.
+
+## Models
+
+No new Mongoose models. No schema migration.
+
+`AIRequest.metadata` (already Mixed) gains optional:
+
+```typescript
+interface FailoverLogMetadata {
+  attempts: Array<{modelId: string; error?: string}>;
+  servedModelId: string;
+}
+
+interface AgentLogMetadata {
+  agentName: string;
+}
+```
+
+`requestType` may be `"agent"` in addition to current defaults. Update admin explorer filters in `ai/src/routes/aiRequestsExplorer.ts`.
+
+## APIs
+
+No REST/OpenAPI changes.
+
+Public package exports from `ai/src/index.ts`:
+
+- `Agent`, type `AgentConfig`, type `AgentMiddleware`, type `AgentRunOptions`
+- `createFailoverModel`, `isFailoverableError`, type `CreateFailoverModelOptions`
+
+## Notifications
+
+None.
+
+## UI
+
+None. Admin explorer already shows `requestType` / `aiModel` / `metadata`; document the new metadata shape in reference docs only.
+
+## Phases
+
+1. **Failover LanguageModel** — `createFailoverModel` + tests (tracer).
+2. **Logging served model** — `lastServedModelId` into `AIRequest`.
+3. **Agent** — class + dispatch + middleware + tests.
+4. **Docs + agent rules** — reference, how-to, `.cursor/rules/ai`, prompt-governance mention.
+5. **Optional example-backend** — one agent if dual providers exist without new env requirements.
+
+## Feature Flags & Migrations
+
+None. Additive API. No feature flag.
+
+## Activity Log & User Updates
+
+`AIRequest` remains the audit log. No user-facing activity feed.
+
+## Not Included / Future Work
+
+- RAG / embeddings (comparison item 2).
+- Agent-as-tool / sub-agents.
+- Human tool approval.
+- `AiApp` `agent` option mounting a dedicated route.
+- Failover on mid-stream errors.
+- Combining structured schema with tool loops in one `run`.
+
+## Files to Create / Modify
+
+| File | Change |
+| --- | --- |
+| `ai/src/service/failoverModel.ts` | `createFailoverModel`, `isFailoverableError` |
+| `ai/src/service/failoverModel.test.ts` | Mock `doGenerate` / `doStream`; 429 then success; non-failoverable does not switch; empty list throws; stream reject-before-return vs after-return |
+| `ai/src/agent/agent.ts` | `Agent` class |
+| `ai/src/agent/agent.test.ts` | Instructions sent as system; schema path; tools path; middleware order |
+| `ai/src/service/aiService.ts` | Read `lastServedModelId` when logging; optional `requestType` passthrough only if needed (prefer Agent calling existing methods + metadata) |
+| `ai/src/types/index.ts` | Types; `"agent"` request type |
+| `ai/src/routes/aiRequestsExplorer.ts` | Filter includes `agent` |
+| `ai/src/index.ts` | Exports |
+| `docs/reference/ai.md` | Agent + failover sections |
+| `docs/how-to/define-an-ai-agent.md` | New how-to |
+| `docs/how-to/configure-ai-failover.md` | New how-to |
+| `docs/how-to/README.md` | Links |
+| `.cursor/rules/ai/00-ai.mdc` (source of truth for rules sync) | Public API notes |
+| `skills/ai-prompt-governance/SKILL.md` | Agent instructions = named constants |
+| `ai/README.md` | Short mention + link to reference |
+| `docs/tasks/ai-agents-and-failover.md` | This plan's task list |
+| `example-backend/src/api/ai.ts` | Optional Phase 5 only |
+
+## Task List
+
+[`docs/tasks/ai-agents-and-failover.md`](../tasks/ai-agents-and-failover.md)
+
+## Acceptance Criteria
+
+- [ ] `createFailoverModel` with two mocks: first `doGenerate` throws failoverable 429, second returns text; `AIService.generateText` returns the second text; `AIRequest.aiModel` is the second `modelId`; `metadata.failover.attempts` has length 2.
+- [ ] First mock throws 400; second is never called; error propagates.
+- [ ] `doStream` that **returns** a stream then errors does not call the next model.
+- [ ] `Agent` with named instruction constant: mock receives that string as `system` (or equivalent SDK field); `AIRequest` has `requestType: "agent"` and `metadata.agentName`.
+- [ ] `Agent` with `schema` returns a typed object via `generateJsonObject` (existing JSON normalization still applies).
+- [ ] Middleware runs in order; a middleware that returns without `next` short-circuits (no model call).
+- [ ] `docs/reference/ai.md` documents both APIs with one minimal example each.
+- [ ] `bun test` in `ai/` passes; `bun run compile` for `ai` succeeds.
+- [ ] No new Express routes; existing GPT tests still pass.

package/src/docs/versioned/next/implementationPlans/ai-dev-loop-boost.md

@@ -1,6 +1,6 @@
 # Implementation Plan: The AI Development Loop (Boost)
 
-**Status:** Draft — key decisions recorded (2026-07-29)
+**Status:** Approved — key decisions recorded (2026-07-29)
 **Roadmap issue:** https://github.com/FlourishHealth/terreno/issues/1014
 **Priority:** High
 **Effort:** Big batch

package/src/docs/versioned/next/implementationPlans/ai-observability.md

@@ -0,0 +1,348 @@
+# Implementation Plan: AI observability (Langfuse-light, pluggable)
+
+**Status:** Draft  
+**Created:** 2026-08-24  
+**Owner:** unassigned  
+**Branch:** `cursor/ai-observability-ip-64ca`  
+**Task list:** [docs/tasks/ai-observability.md](../tasks/ai-observability.md)  
+**Linear:** none  
+**Roadmap issue:** none yet (handoff after Approved)
+
+**Depends on:** current `@terreno/ai` (`AIService`, `AIRequest`, `LangfuseApp`, `AIAdminApp`). The requested `ai-agents-and-failover` IP is **not in this repo**; this plan follows shipped AI code, not that missing doc.
+
+**Related:** [terreno-langfuse-integration.md](./terreno-langfuse-integration.md) (`LangfuseApp` stays). [FH-48](https://linear.app/flourish-health/issue/FH-48/ai-improvements) (prompts in-app vs Langfuse).
+
+## Goal
+
+Ship **Langfuse-like** prompt versioning, nested traces (user / session / cost), multidimensional evaluators (LLM-as-judge + structured I/O), datasets, experiments, and a **local human review queue** — **inside Terreno**, as **plugins on `@terreno/ai`**, with an operator UI in **`admin-frontend` only**.
+
+Apps can:
+
+- Run **fully local** (Mongo) with zero Langfuse keys.
+- Keep **`LangfuseApp` as a second plugin** (not deleted, not merged).
+- Fan out **traces and scores** to local + Langfuse + **OTLP/OpenInference** (Phoenix, Laminar, Arize AX, generic collectors).
+- Point **prompts, datasets, experiments** at local **or** Langfuse independently (with one mix rule).
+
+## Non-goals
+
+- Removing or rewriting `LangfuseApp`.
+- `admin-spa` screens (spa already embeds `admin-frontend` widgets).
+- First-class Braintrust, Helicone, LangSmith, Weave, Opik, Portkey adapters (document how they fit; do not code stubs).
+- Langfuse **annotation queues** as `reviewQueue.primary` (local Mongo queue only in this IP).
+- Dual-write of prompt versions, datasets, or review items.
+- Retention / redaction / retry **jobs** (full I/O stored; Q9=C, Q13=A).
+- Replacing `AIRequest` (cheap call log remains).
+- Pixel-clone of Langfuse Cloud chrome.
+
+## Decisions
+
+| ID | Question | Choice |
+| --- | --- | --- |
+| Q1 | System of record | Mongo **local plugin** + **pluggable** backends (Langfuse and others), not wrap-only |
+| Q2 | Package | Grow `@terreno/ai` |
+| Q3 | Existing Langfuse | **Two plugins**: keep `LangfuseApp`; add local observability plugin |
+| Q4 | Operator UI | **`admin-frontend` only** |
+| Q5 | IP shape | **One IP**, full destination, **phased tasks** |
+| Q6 / Q11 | Dual receive | **Traces + scores + OTel** fan-out. Not prompts/datasets/queue |
+| Q7 | SDK | Capability interfaces + `capabilities` (comms-style) |
+| Q8 | v1 adapters | `local` (full) + `langfuse` (mapped) + **`otel` TraceSink** |
+| Q9 | PHI / I/O | Persist **full I/O**; **no** retention job in this IP |
+| Q10 | Langfuse UI | Terreno screens + **Open in Langfuse** for playground / Langfuse experiments |
+| Q12 / Q16 / Q21 | Primaries | Four flags: `prompts`, `datasets`, `experiments`, `reviewQueue` |
+| Q22 | Mix rule | `experiments.primary` **must equal** `datasets.primary` (fail config) |
+| Q23 | Defaults | All four primaries **`local`** when both plugins exist; Langfuse/OTel are extra **sinks** |
+| Q24 | Local off | **Allowed** (Langfuse-only app; no Mongo observability collections) |
+| Q25 | Review queue | **Local-only**; `reviewQueue.primary: "langfuse"` is a **config error** |
+| Q13 | Secondary down | Local (or primary write) **succeeds**; Langfuse/OTel errors **logged, never thrown** |
+| Q14 / Q17 | Experiments | Runner follows **`experiments.primary`**: local → always **`BackgroundTask`**; Langfuse → Langfuse experiment API |
+| Q15 | Mongo | **New collections**; keep `AIRequest` |
+| Q18 | Live evals | Interceptor with **`sampleRate` default `0`** |
+| Q19 | Cost | `inputTokens` / `outputTokens` / `model`; **USD from config price map** (missing model → tokens only) |
+| Q20 | Instrumentation | If observability is registered, **`AIService` emits traces** on every `generate*` (**opt-out per call**) |
+
+### Competitor coverage (why the SDK is split)
+
+| Backend | TraceSink | PromptRegistry | Dataset / experiment | ScoreSink | ReviewQueue |
+| --- | --- | --- | --- | --- | --- |
+| Local Mongo | yes | yes | yes | yes | yes (v1) |
+| Langfuse | yes | yes | yes | yes | no in v1 |
+| OTLP / OpenInference | yes | no | no | no | no |
+| Phoenix / Laminar / Datadog | via OTel | no | no | no | no |
+| Helicone / Portkey | later (gateway) | no | no | no | no |
+| Braintrust / LangSmith | later | later | later | later | later |
+
+Traces use **OpenTelemetry + OpenInference** span kinds (`LLM`, `AGENT`, `CHAIN`, `TOOL`, `EVALUATOR`, `PROMPT`). That is the honest “works with competitors” layer. Prompt/dataset/queue APIs stay vendor CRUD behind Terreno interfaces.
+
+## Architecture
+
+```
+AIService.generate*
+        │
+        ├─ AIRequest.logRequest          (unchanged cheap log)
+        └─ if observability registered and !opts.skipTrace
+              Observability.traceSinks[]  →  local | langfuse | otel   (fan-out, best-effort)
+
+PromptRegistry.get({name, label})     → primary prompts plugin only
+DatasetStore / ExperimentRunner       → their primaries (must match each other)
+ReviewQueue                           → local plugin only
+ScoreSinks[]                          → fan-out with traces
+```
+
+**Two planes**
+
+| Plane | Plugins | Write | Read (admin) |
+| --- | --- | --- | --- |
+| Telemetry | all registered `TraceSink` / `ScoreSink` | fan-out | **local** if local plugin on; else Langfuse |
+| Control | primary per capability | **one** writer | that primary |
+
+Mirrors `@terreno/comms`: interfaces in `ai/src/observability/types.ts`; adapters do not put extra vendor SDKs on core beyond what `LangfuseApp` already uses. OTLP exporter is an optional adapter file.
+
+### Public types (normative)
+
+```typescript
+type ObservabilityCapability =
+  | "traces"
+  | "prompts"
+  | "scores"
+  | "datasets"
+  | "experiments"
+  | "reviewQueue";
+
+type ControlPrimary = "local" | "langfuse";
+
+interface ObservabilityControlConfig {
+  datasets: ControlPrimary;
+  experiments: ControlPrimary;
+  prompts: ControlPrimary;
+  reviewQueue: "local"; // langfuse rejected at register()
+}
+
+interface ObservabilityPlugin {
+  readonly capabilities: ReadonlySet<ObservabilityCapability>;
+  readonly id: string;
+  datasetStore?: DatasetStore;
+  experimentRunner?: ExperimentRunner;
+  promptRegistry?: PromptRegistry;
+  reviewQueue?: ReviewQueue;
+  scoreSink?: ScoreSink;
+  traceSink?: TraceSink;
+}
+
+interface TraceRecord {
+  id: string;
+  name: string;
+  startedAt: string;
+  endedAt?: string;
+  userId?: string;
+  sessionId?: string;
+  status: "ok" | "error";
+  input?: unknown;
+  output?: unknown;
+  metadata?: Record<string, unknown>;
+  usage?: {
+    inputTokens?: number;
+    outputTokens?: number;
+    model?: string;
+    costUsd?: number;
+  };
+  prompt?: {name: string; version: number; label?: string};
+  spans: SpanRecord[];
+}
+
+interface ScoreRecord {
+  name: string;
+  dataType: "numeric" | "categorical" | "boolean";
+  value: number | string | boolean;
+  comment?: string;
+  source: "human" | "llm-judge" | "code";
+  evaluatorId?: string;
+  traceId: string;
+  spanId?: string;
+}
+```
+
+`ObservabilityApp` (new `TerrenoPlugin`):
+
+- Validates config (Q22, Q25, plugin present for each primary).
+- Holds the sink lists and primaries.
+- Contributes admin screens when local and/or Langfuse control APIs are on.
+- Does **not** replace `LangfuseApp.register()` — example-backend may `.register(new LangfuseApp(...)).register(new ObservabilityApp({plugins: [local, langfuseAdapter, otel]}))`. The Langfuse **adapter** wraps the existing client from `LangfuseApp` / `getLangfuseClient()`; it must not start a second OTel SDK if `LangfuseApp` already did.
+
+### `AIService` hook
+
+After each generate path (same place as `logRequest`):
+
+1. Skip if no `ObservabilityApp` or `options.skipTrace === true`.
+2. Build a root span (`CHAIN` or `LLM`) plus children for tool rounds when present.
+3. Attach `userId`, `sessionId` from generate options (GPT routes copy from `req.user` / body).
+4. Attach prompt name/version when `options.promptRef` is set.
+5. Compute `costUsd` from `options.priceMap ?? observability.priceMap` keyed by `model`.
+6. `Promise.allSettled` on each `TraceSink.export`; log rejections; **never throw**.
+7. If `sampleRate > 0` and `Math.random() < sampleRate`, enqueue live evaluators (same BackgroundTask path as experiments, one trace).
+
+Parent/child workflows: `AIRequest` already has `parentRequestId` / `subRequestIds`. Traces use `traceId` + `parentSpanId`. Multi-agent `logMultiAgentRequest` should also emit a parent `AGENT`/`CHAIN` span wrapping children when observability is on.
+
+## Models (local plugin)
+
+All new models: `createdUpdatedPlugin`, `isDeletedPlugin`, `findOneOrNone`, `findExactlyOne`, **description on every field**. Types in `ai/src/types/` (five-type pattern). Connection: default mongoose connection (same as `AIRequest`). If local plugin is not registered, **do not** register these models.
+
+| Model | Role |
+| --- | --- |
+| `ObsPrompt` | Named prompt (`name` unique). Tags. |
+| `ObsPromptVersion` | Immutable version: `type` text\|chat, body, `variables[]`, `inputSchema`/`outputSchema` (JSON Schema objects), `config` (temperature preset name, model hint). |
+| `ObsPromptLabel` | Movable labels: `production`, `latest`, optional `staging`. Unique `(promptId, label)`. |
+| `ObsTrace` | Root trace: user, session, status, usage, prompt ref, timestamps. |
+| `ObsSpan` | Nested spans: `traceId`, `parentSpanId`, OpenInference `kind`, input/output, model, tokens, cost. |
+| `ObsScore` | Scores on trace/span. |
+| `ObsEvaluator` | Named grader: dimensions[], type human\|llm-judge\|json-assert, judge `promptName` + schema. |
+| `ObsDataset` / `ObsDatasetItem` | Items: input vars, optional expected output, `sourceTraceId`. |
+| `ObsExperiment` / `ObsExperimentItem` | Run metadata + per-item outputs/scores. Status queued\|running\|completed\|failed. `backgroundTaskId`. |
+| `ObsReviewItem` | Queue: pending\|in_progress\|done\|skipped, evaluatorId, traceId, assigneeId, scores payload. |
+
+**Do not** put a unique index on fields that can duplicate (scores per trace are many). Index `{created: -1, userId: 1}`, `{sessionId: 1, created: -1}`, `{promptName: 1, promptVersion: 1}`, `{status: 1, created: 1}` on review items.
+
+No backfill: new collections. `AIRequest` unchanged.
+
+## APIs
+
+Admin-only unless noted (`Permissions.IsAdmin`). OpenAPI via `createOpenApiBuilder`. Base path `/ai/observability` (stable, not `/admin/langfuse`).
+
+| Method | Path | Behavior |
+| --- | --- | --- |
+| GET | `/ai/observability/status` | Plugin ids, capabilities, primaries, sink health (last error) |
+| GET/POST | `/ai/observability/prompts` | List/create via **prompt primary** |
+| GET/POST | `/ai/observability/prompts/:name/versions` | Immutable versions; POST creates `vN+1` |
+| POST | `/ai/observability/prompts/:name/labels` | Move `production` / `staging` |
+| POST | `/ai/observability/prompts/:name/playground` | Compile + one `AIService` call (not a new version) |
+| GET | `/ai/observability/traces` | Filters: time, user, session, prompt, status, minCost, hasScore |
+| GET | `/ai/observability/traces/:id` | Tree + scores |
+| GET | `/ai/observability/sessions/:sessionId` | Timeline |
+| GET | `/ai/observability/users/:userId` | Totals 7d/30d/all + recent traces |
+| GET | `/ai/observability/costs` | Aggregates by model, prompt, user |
+| CRUD | `/ai/observability/evaluators` | Dimension definitions |
+| POST | `/ai/observability/traces/:id/scores` | Human or code score → **all ScoreSinks** |
+| CRUD | `/ai/observability/datasets` + items | Dataset **primary** |
+| POST | `/ai/observability/experiments` | Creates run; local → `BackgroundTask`; Langfuse → vendor API |
+| GET | `/ai/observability/experiments/:id` | Status + aggregates |
+| GET/POST | `/ai/observability/review` | Local queue only |
+
+GPT routes: pass `userId` / `sessionId` / optional `promptRef` into `AIService`.
+
+Langfuse-primary prompt/dataset/experiment routes **proxy** existing Langfuse admin routes where they already exist (`langfuseRoutesPrompts.ts` etc.) so we do not duplicate HTTP. ObservabilityApp may mount a thin facade that delegates.
+
+## UI (`admin-frontend`)
+
+New screens in `AI_ADMIN_WIDGETS` / `customScreens` (same pattern as `AIRequestsScreenWidget`):
+
+| Screen | Name | Notes |
+| --- | --- | --- |
+| Prompts | `ai-prompts` | List + editor (new version, labels, JSON Schema I/O, playground) |
+| Traces | `ai-traces` | List + detail waterfall |
+| Sessions | `ai-sessions` | Optional v1: filter traces by session from detail |
+| Costs | `ai-costs` | KPI + breakdown tables |
+| Evaluators | `ai-evaluators` | Dimension builder |
+| Datasets | `ai-datasets` | |
+| Experiments | `ai-experiments` | Wizard; poll BackgroundTask; promote version |
+| Review | `ai-review` | Split view; **hide** if local plugin off |
+| Status chip | chrome | Active primaries; **Open in Langfuse** when Langfuse plugin id present |
+
+Reuse `@terreno/ui` (`Page`, `Box`, `DataTable`, `Tabs`, `Modal`, `TextField`). Keep **AI Requests** explorer as-is (`AIRequest`).
+
+Disabled tabs: if prompt primary is Langfuse, editor still works via API facade; if local off, Prompts/Datasets/Experiments/Review either hide or show Langfuse-backed lists + deep links (Q10). Review hidden when local off (Q25).
+
+## Example app
+
+`example-backend`:
+
+- Register `ObservabilityApp` with **local plugin always**.
+- If `LANGFUSE_*` already used for `LangfuseApp`, also pass Langfuse TraceSink/ScoreSink (and prompt adapter only if env `AI_OBS_PROMPTS_PRIMARY=langfuse`).
+- Optional `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` → `OtelTraceSink`.
+- Seed one prompt `example-summarize` v1 production, one evaluator `faithfulness` (numeric 0–1) + `safety` (boolean), empty dataset.
+
+`example-frontend`: no product UI required (admin-only). Regen SDK if example-frontend calls new routes (likely admin-only; skip consumer SDK if unused).
+
+## Feature flags & env
+
+No OpenFeature flag required: **registration is the flag**.
+
+Env (example-backend):
+
+| Variable | Role |
+| --- | --- |
+| `AI_OBS_PROMPTS_PRIMARY` | `local` \| `langfuse` (default `local`) |
+| `AI_OBS_DATASETS_PRIMARY` | default `local` |
+| `AI_OBS_EXPERIMENTS_PRIMARY` | must equal datasets |
+| `AI_OBS_SAMPLE_RATE` | `0`–`1`, default `0` |
+| `AI_OBS_PRICE_MAP_JSON` | `{ "gemini-2.5-flash": { "inputPerMTok": 0.1, "outputPerMTok": 0.4 } }` |
+| existing Langfuse keys | unchanged |
+| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | optional OTLP |
+
+Validate dataset/experiment primary equality and reject `reviewQueue` langfuse at boot with a clear `APIError` / logger.error + throw (fail fast like missing SendGrid key).
+
+## Activity log
+
+Do **not** add `UserUpdate` spam for every trace. Admin audit: prompt **production** label moves and experiment **promote** (reuse `onAdminAudit` if AdminApp is present; otherwise `logger.info` with prompt name + version).
+
+## Testing
+
+- Bun tests, `expect`, `@terreno/test` Mongo preload. **Never mock `@terreno/api` or models.**
+- Fake TraceSink / ScoreSink in unit tests (in-memory).
+- `AIService` tests: emit called; `skipTrace`; sink throw does not fail generate; `AIRequest` still written.
+- Config tests: illegal experiment/dataset mix; langfuse reviewQueue; local-off + prompt primary local fails.
+- Local experiment: POST creates `BackgroundTask`, items scored, aggregates.
+- Langfuse adapter tests: mock `getLangfuseClient` (existing pattern in `langfuseClient.test.ts`).
+- Admin widget tests: render list empty/loading (existing admin-frontend patterns).
+- `verify-ui-changes` on admin-frontend screens before Brew.
+
+## Documentation (same slices as code)
+
+| Page | Kind |
+| --- | --- |
+| `docs/explanation/ai-observability.md` | Why two planes, plugins, vs `LangfuseApp` / `AIRequest` |
+| `docs/how-to/observe-llm-calls.md` | Register plugins, primaries, price map, sampleRate, Open in Langfuse |
+| `docs/reference/ai.md` | Models, routes, `AIService` trace options, env |
+| `terreno-langfuse-integration.md` | Pointer: native path is this IP; LangfuseApp remains the vendor plugin |
+| `ai-prompt-governance` skill | Prompt bodies for judges still named constants **or** `PromptRegistry` production label — not inline strings in `AIService` |
+
+## Phases
+
+1. **Contracts + ObservabilityApp + AIService emit + in-memory sinks** (tracer).
+2. **Local Mongo plugin** (prompts, traces, scores, costs, users/sessions).
+3. **Evaluators + live sampleRate + review queue**.
+4. **Datasets + local experiments via BackgroundTask**.
+5. **Langfuse adapter** (trace/score/prompt/dataset/experiment) + OTel TraceSink + admin UI + example-backend + docs.
+
+## Acceptance criteria
+
+- [ ] `AIService.generateText` with ObservabilityApp registered writes a nested-capable trace to **every** TraceSink; a throwing sink does **not** fail the generate; `AIRequest` still exists.
+- [ ] Creating prompt v2 does not mutate v1; moving `production` is an explicit label API; `get({name, label: "production"})` returns v2.
+- [ ] Trace list filters by `userId`, `sessionId`, prompt name/version; detail shows cost USD when price map has the model.
+- [ ] Evaluator with two dimensions (numeric + boolean) can score a trace via LLM-judge (`AIService` + named judge prompt) and via human review queue.
+- [ ] Dataset of N prior traces; experiment of production vs draft always uses `BackgroundTask` when local; promote sets production label.
+- [ ] Langfuse plugin on: same generate dual-writes traces/scores; prompts stay on configured primary; boot fails if `experiments.primary !== datasets.primary` or `reviewQueue` is langfuse.
+- [ ] OTel sink: one test asserts an OTLP payload / exporter mock received an OpenInference `LLM` span.
+- [ ] Admin: prompts, traces, costs, review, experiments usable as admin; Review hidden without local plugin.
+- [ ] Docs pages above match shipped behavior; `bun run --filter=@terreno/ai compile test lint` green.
+
+## Risks
+
+| Risk | Mitigation |
+| --- | --- |
+| Langfuse experiment API vs Terreno dataset | Config equality; no sync job |
+| Double OTel SDK vs LangfuseApp | Adapter shares existing tracing init |
+| PHI in Mongo | Document; retention is a follow-on IP |
+| Live evals cost | `sampleRate` default 0 |
+| Missing failover IP | Follow current `AIService` / parent-child `AIRequest` |
+| Admin SPA drift | Widgets in admin-frontend automatically appear in spa shell |
+
+## Files to create / modify (expected)
+
+- `ai/src/observability/**` (types, app, local models/stores, otel sink, langfuse adapter)
+- `ai/src/service/aiService.ts` — emit traces
+- `ai/src/routes/gpt.ts` — pass user/session
+- `ai/src/aiAdminApp.ts` / new `observabilityAdmin.ts` — screens
+- `ai/src/index.ts` — exports
+- `admin-frontend/src/widgets/*` — new screens + registry
+- `example-backend/src/server.ts` — register ObservabilityApp
+- docs listed above
+- tests colocated