@terreno/mcp 57.9.0 → 57.11.0

package/src/docs/versioned/next/reference/ai.md

@@ -15,6 +15,8 @@ AI service layer for Terreno backends: provider-agnostic chat via the Vercel AI
 - [Route registrars](#route-registrars)
 - [AiApp plugin](#aiapp-plugin)
 - [LangfuseApp plugin](#langfuseapp-plugin)
+- [Observability](#observability)
+- [Durable agent harness](#durable-agent-harness)
 - [Langfuse integration](#langfuse-integration)
 - [FileStorageService](#filestorageservice)
 - [MCPService](#mcpservice)
@@ -64,13 +66,15 @@ src/
     project.ts             # GPT project + memories
   routes/
     gpt.ts                 # Streaming chat, remix, tools, ratings
-    gptHistories.ts        # History CRUD
+    gptHistories.ts        # History CRUD, headless pendingAsks and turn actions
     aiRequestsExplorer.ts  # Admin request explorer
     files.ts               # File upload/signed URL/delete
     projects.ts            # Project CRUD + memories
     mcp.ts                 # MCP server status and tools
   service/
     aiService.ts           # Provider-agnostic AI service
+    asks.ts                # Ask tools, reserved ask_ names, paused-turn replay
+    chatTurn.ts            # Chat turn runner behind /gpt/prompt (SSE sink) and turn (buffered JSON)
     fileStorage.ts         # GCS upload helper
     getMCPTools.ts         # modelRouter MCP tools as Vercel AI SDK tools
     mcpService.ts          # MCP client connections
@@ -92,8 +96,10 @@ src/
 - **Structured output:** `parseAiJson`, `normalizeLlmJsonTextForStructuredOutput`, re-exported `Output`, `jsonSchema`, `JSONValue`, `FlexibleSchema` from `ai`
 - **Langfuse:** `initLangfuseClient`, `getLangfuseClient`, `shutdownLangfuseClient`, `compilePrompt`, `createPrompt`, `getPrompt`, `createTelemetryConfig`, `preparePromptForAI`, `initTracing`, `shutdownTracing`, `LangfuseCache`, cache helpers
 - **Gemini / Vertex:** `listGeminiApiModels`, `normalizeGeminiModelId`, `GEMINI_API_BASE_URL`, `createVertexProvider`, `listEnabledVertexModels`, `verifyVertexModelsEnabled`, `assertVertexModelsEnabled`, `isVertexModelAllowed`, `normalizeVertexModelId`, `DEFAULT_VERTEX_LOCATION`
-- **Prompts:** `CONTENT_SUMMARY_PROMPT`, `DEFAULT_GPT_MEMORY`, `JSON_VALUE_SYSTEM_PROMPT`, `REMIX_PROMPT`, `TITLE_GENERATION_PROMPT`, `TRANSLATION_PROMPT`
+- **Prompts:** `COMPACT_SURFACE_SYSTEM_PROMPT`, `CONTENT_SUMMARY_PROMPT`, `DEFAULT_GPT_MEMORY`, `JSON_VALUE_SYSTEM_PROMPT`, `REMIX_PROMPT`, `TERRENO_ASKS_SYSTEM_PROMPT`, `TITLE_GENERATION_PROMPT`, `TRANSLATION_PROMPT`
+- **Asks:** `createAskTools({kinds, surface?})`, `TERRENO_ASKS_SYSTEM_PROMPT`, `COMPACT_SURFACE_SYSTEM_PROMPT`, types `AsksOptions`, `ApprovalAskInput`, `AskOrigin`, `GptHistoryPendingAsk`, `GptHistoryPromptAsk`, `GptHistoryAskStatus`, and `Ask`, `AskKind`, `AskResponse`, `AskValidationError`, `SimpleCard`, `SimpleCardButton` re-exported from `@terreno/blocks` ([Agent UI Asks](agent-ui-asks.md))
 - **Web search:** `WebSearchProvider`, `WebSearchResult` types
+- **Harness (subpath `@terreno/ai/harness`):** `Harness`, `defineTask`, `defineAgent`, `defineTool`, `defineExtension`, `section`, `hook`, `wrapTool`, `HarnessExtensionError`, `HARNESS_HOOK_KINDS`, `HarnessConversationHandle`, `HarnessConversationBusyError`, `HarnessConversationOwnedError`, `HARNESS_EVENT_TYPES`, `HARNESS_SUBMIT_DISPOSITIONS`, `HARNESS_WHEN_BUSY`, `HarnessModelCallError`, `HarnessSubagentError`, `isRetryableModelError`, `AGENT_TURN_TASK_NAME`, `AGENT_TOOL_TASK_NAME`, `HARNESS_AGENT_DEFAULT_MAX_STEPS`, `HARNESS_CONVERSATION_STATUSES`, `HARNESS_INTERRUPT_ACTIONS`, `HARNESS_MESSAGE_ROLES`, `HARNESS_MODEL_RETRY_DEFAULTS`, `InProcessRunner`, `HarnessCommitConflictError`, `HARNESS_RESOLVE_ACTIONS`, `HARNESS_RETRY_DEFAULTS`, `HARNESS_TASK_STATUSES`, `HARNESS_WAIT_KINDS`, `HARNESS_WAIT_POLICIES`, `HARNESS_WAIT_RESOLUTIONS`, `IN_PROCESS_RUNNER_ROLES`, `approvalGate`, `approvalTaskInput`, `HarnessApp`, `HarnessApprovalConflictError`, `HARNESS_APPROVAL_STATUSES`, `HARNESS_DEFAULT_APPROVERS` — see [AI harness reference](ai-harness.md)
 
 ## AIService
 
@@ -131,18 +137,26 @@ const aiService = new AIService({
 | `generateText(options)` | Non-streaming text generation; logs as `requestType: "general"` |
 | `generateJsonValue(options)` | Any JSON value via `Output.json()`; logs as `"json_value"` |
 | `generateJsonObject(options)` | Typed object from schema/Zod via `Output.object()`; logs as `"json_object"` |
+| `generateBlocks(options)` | One block document via `Output.object(blocksJsonSchema)`, then `validateBlocks`. Temperature is always `TemperaturePresets.DETERMINISTIC` (0). Logs as `"ui_blocks"`. |
 | `generateJsonArray(options)` | Typed array via `Output.array()`; logs as `"json_array"` |
 | `generateTextStream(options)` | Async generator of text chunks; logs full response after stream completes |
 | `generateRemix(options)` | Reword text using `REMIX_PROMPT` at `TemperaturePresets.BALANCED` |
 | `generateSummary(options)` | Summarize text using `CONTENT_SUMMARY_PROMPT` at `TemperaturePresets.LOW` |
 | `translateText(options)` | Translate text using `TRANSLATION_PROMPT` at `TemperaturePresets.LOW` |
-| `buildMessages(prompts)` | Convert `GptHistoryPrompt[]` to Vercel AI SDK `ModelMessage[]` (skips tool-call/result entries) |
+| `buildMessages(prompts)` | Convert `GptHistoryPrompt[]` to Vercel AI SDK `ModelMessage[]`. Skips host tool-call/result rows and approval ask rows (`ask.origin: "approval"`). Keeps each answered or cancelled ask as an assistant tool call plus its tool result; consecutive ask calls share one assistant message. Skips asks still waiting for an answer. |
 | `generateChatStream(options)` | Stream multi-turn chat with optional tools; logs prompt as joined message text |
 
 All generation methods log to `AIRequest` via private `logRequest()`. Logging failures never throw.
 
 ## Structured JSON output
 
+`generateBlocks({prompt, systemPrompt?, userId?, repair?})` asks for one block document:
+
+- Uses `TERRENO_UI_BLOCKS_SYSTEM_PROMPT` when `systemPrompt` is omitted.
+- Temperature is `TemperaturePresets.DETERMINISTIC` (0).
+- Checks the object with `validateBlocks`. When `repair` is omitted or true, one retry appends the error list to the user prompt. `repair: false` skips that retry.
+- A second validation failure throws `APIError` 422 (`title: "Block document failed validation"`, `meta.fields` keyed by error code) and stores `metadata.errorCodes` on the `AIRequest`. A model or network error throws 502 (`title: "Block generation failed"`) and is not repaired.
+
 `generateJsonValue`, `generateJsonObject`, and `generateJsonArray`:
 
 - Default to `TemperaturePresets.DETERMINISTIC` when `temperature` is omitted.
@@ -186,13 +200,13 @@ Logs all AI calls for monitoring and admin explorer.
 |-------|------|-------------|
 | `aiModel` | string | Model identifier (field name avoids Mongoose `model` conflict) |
 | `prompt` | string | Input prompt |
-| `requestType` | string | e.g. `general`, `remix`, `summarization`, `translation`, `json_value`, `json_object`, `json_array` |
+| `requestType` | string | e.g. `general`, `remix`, `summarization`, `translation`, `json_value`, `json_object`, `json_array`, `ui_action`, `ui_blocks` |
 | `response` | string? | Response text |
 | `responseTime` | number? | Milliseconds |
 | `tokensUsed` | number? | Total tokens |
 | `userId` | ObjectId? | Requesting user |
 | `error` | string? | Error message |
-| `metadata` | Mixed? | Extra data (e.g. structured-output debug) |
+| `metadata` | Mixed? | Extra data (e.g. structured-output debug; `ask` and `nextAsk` for [asks](agent-ui-asks.md#stored-state)) |
 | `parentRequestId` | ObjectId? | Parent in multi-agent workflow |
 | `subRequestIds` | ObjectId[]? | Child request refs |
 | `totalResponseTime` | number? | Combined sub-request time |
@@ -209,11 +223,12 @@ Conversation history with multi-modal prompts.
 | Field | Type | Description |
 |-------|------|-------------|
 | `userId` | ObjectId | Owner (required) |
-| `title` | string? | Auto-generated on first `/gpt/prompt` response when empty |
+| `title` | string? | Auto-generated on the first chat turn's reply (`/gpt/prompt` or `turn`) when empty |
 | `projectId` | ObjectId? | Optional project association |
-| `prompts` | array | Messages: `text`, `type` (`user` \| `assistant` \| `system` \| `tool-call` \| `tool-result`), optional `content` parts, `model`, `rating`, tool fields, `status`, `streamId`. `text` is required unless `content` has parts or `status` is set; an image-only assistant response saves `text: ""` |
+| `prompts` | array | Messages: `text`, `type` (`user` \| `assistant` \| `system` \| `tool-call` \| `tool-result`), optional `content` parts, `model`, `rating`, tool fields (`toolCallId`, `toolName`, `args`, `result`), `ask: {kind, status}` on ask `tool-call` rows (`status`: `pending` \| `answered` \| `cancelled`), and on assistant replies `status` (`streaming` \| `complete` \| `error`) plus `streamId`. `text` is required unless `content` has parts or `status` is set; an image-only assistant response saves `text: ""` |
+| `pendingAsk` | object? | The ask the conversation waits on: `toolCallId`, `kind`, `input`, `simple`, `promptIndex`, `responseMessages`, `created`, and for an approval ask `origin`, `approvalId`, `toolName`. `/gpt/histories` responses leave out `promptIndex` and `responseMessages`. Only a chat turn (`/gpt/prompt` or the `turn` action) sets and clears it; see [Agent UI Asks](agent-ui-asks.md#stored-state). |
 
-Assistant replies from `/gpt/prompt` carry `streamId` and `status`: `streaming` while partial text is persisted, then `complete` or `error`. `buildMessages` skips `streaming` replies and `error` replies with no output. Attachment `content` parts carry an optional `gcsKey` when the file was uploaded to durable storage.
+Assistant replies from `/gpt/prompt` carry `streamId` and `status`: `streaming` while partial text is persisted, then `complete` or `error`. A paused ask's reply is `complete`, and `pendingAsk` is set beside it. `buildMessages` skips `streaming` replies and `error` replies with no output. Attachment `content` parts carry an optional `gcsKey` when the file was uploaded to durable storage.
 
 **Virtual:** `ownerId` aliases `userId` for `Permissions.IsOwner`.
 
@@ -251,13 +266,61 @@ GPT project with persistent context and memories.
 
 | Endpoint | Method | Auth | Description |
 |----------|--------|------|-------------|
-| `/gpt/prompt` | POST | `IsAuthenticated` | SSE streaming chat; body: `prompt`, optional `historyId`, `systemPrompt`, `attachments`, `model`, `projectId` |
+| `/gpt/prompt` | POST | `IsAuthenticated` | SSE streaming chat turn; [body](#gptprompt-body) and [events](#sse-events) below |
 | `/gpt/remix` | POST | `IsAuthenticated` | Non-streaming text remix; body: `{text}` |
 | `/gpt/histories/:id/rating` | PATCH | `IsAuthenticated` | Rate a prompt; body: `{promptIndex, rating: "up" \| "down" \| null}` |
 | `/gpt/histories/:id/stream` | GET | `IsAuthenticated` (owner) | SSE resume of an in-flight reply; query: optional `streamId`, `offset` |
-| `/gpt/tools` | GET | `IsAuthenticated` | List builtin + MCP tools |
+| `/gpt/tools` | GET | `IsAuthenticated` | List builtin + MCP tools (ask tools are not listed) |
+| `/gpt/datasets/:id` | GET | owner (`IsOwner`; another user is 404) | Read a stored dataset. Mounted only when `uiBlocks` is on. Query: `grain` (`hour` \| `day` \| `week` \| `month`), `limit` (default 500, max 1000), `page`. Response `data`: `{columns, rows, rowCount, page, more}`. `grain` buckets the first date column in UTC. An offset is converted before `startOf`. A date with no zone is that UTC day. Null date cells are skipped. Number columns are summed. Without `page`, a series longer than `limit` is LTTB-downsampled and `more` is false. With `page`, rows are a page and `more` is true when another page remains. |
+| `/gpt/actions` | POST | `IsAuthenticated` plus history owner (another user is 403) | Run a host callback. Mounted only when `uiBlocks` is on, on the `/gpt` path. Body: `{historyId, messageId, blockId, elementId, name, payload?}`. Unknown `name` is 404. A payload that fails the host schema is 400 with `meta.fields`. The handler has 10 seconds (`actionTimeoutMs` can set another cap) and then 504. Response `data`: `{text?, blocks?, replace?}`. An invalid `blocks` document is 500. Logged as `AIRequest` `requestType: "ui_action"`. |
+
+Generated images (image-output models such as `gemini-3-pro-image`) arrive as SSE `image` events: `{image: {mimeType, url}}` with a base64 data URL. Each image is sent once, even when the model reports it both as a stream file part and in the final `result.files`. The saved assistant prompt stores one `image` content part per image and `text: ""` when there is no text. On later turns, `buildMessages` sends an image-only assistant prompt to the model as the text `[Generated image]`, because providers reject empty assistant turns.
+
+AI resolution order: `x-ai-api-key` header + `createModelFn` → `createServerModelFn(modelId)` → configured `aiService`. When none resolves, `/gpt/prompt` streams a canned demo reply and `/gpt/remix` returns it. This happens whether or not `demoMode` is set.
+
+Pass `asks: true` (or `{kinds: ["choice"]}`) to let the model ask the user typed questions in the chat. Asks are off by default; with them off, tools, system prompt, and SSE events are unchanged. See [Agent UI Asks](agent-ui-asks.md).
 
-Generated images (image-output models such as `gemini-3-pro-image`) arrive as SSE `image` events: `{image: {mimeType, url}}` with a base64 data URL. Each image is sent once, even when the model reports it both as a stream file part and in the final `result.files`. The saved assistant prompt stores one `image` content part per image. On later turns, `buildMessages` sends an image-only assistant prompt to the model as the text `[Generated image]`, because providers reject empty assistant turns.
+Pass `uiBlocks: true` (or `{hostActions, html, imageHosts, repair, datasetTtlDays, datasetMaxRows}`) to require each assistant reply to be a block document. `html: true` allows `html` blocks and sanitizes them before they are stored. `imageHosts` lists hostnames allowed on `https` image sources. The client receives that document once, after missing action ids are filled, `repair: true` rewrites it, and html is sanitized, and before `{ask}`. `{replace: "text", text}` is sent only when text was already streamed and then changed. `datasetTtlDays` defaults to `0` (keep the dataset). `datasetMaxRows` defaults to 50,000. Off by default; with it off, the system prompt, SSE events, `/gpt/datasets`, and `/gpt/actions` are unchanged. When it is on, the system prompt gains `TERRENO_UI_BLOCKS_SYSTEM_PROMPT` (host callback names included when `hostActions` is set). After the final text, the route validates it and sends `{blocks: {ok, errors, warnings}}` before `{done}`. `hostActions` is the callback allowlist: a name outside it fails with `UNKNOWN_HOST_ACTION`. `{repair: true}` runs one repair call when validation fails and stores that reply. A document that is still invalid is stored with a `Block validation errors:` note so the next turn sees it. See [Validate a block document locally](../how-to/agent-ui-blocks.md).
+
+With asks on, a host tool with the AI SDK's `needsApproval: true` runs only after the user approves it: the turn pauses on a server-made `confirm` ask. `asks.approvals` sets that ask's input per tool name, as `(input) => ConfirmAskInput` (`ApprovalAskInput`); without an entry, the ask is "Allow <toolName>?" with the tool's description. With asks off, such a tool never runs. See [Approval asks](agent-ui-asks.md#approval-asks).
+
+With `asks` on, pass the same options to `addGptHistoryRoutes` as `chat` to add the non-streaming [headless endpoints](#addgpthistoryroutesrouter-options) for clients that do not read server-sent events.
+
+#### `/gpt/prompt` body
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `prompt` | string | The user's message. Required unless `askResponse` is sent. |
+| `historyId` | string? | Continue this conversation; omit to start one. Required with `askResponse`. |
+| `askResponse` | object? | The answer to the pending ask: `{toolCallId, action, content?, reason?}`. Read only when `asks` is on. Send it instead of `prompt`, without `attachments`. See [Answer an ask](agent-ui-asks.md#answer-an-ask). |
+| `surface` | `"full"` \| `"compact"`? | Where the user reads and answers. Default `"full"`. `"compact"` is a watch or another small screen: the model gets only button-sized asks and is asked for replies of at most two short sentences. Any other value returns 400. See [Compact surface](agent-ui-asks.md#compact-surface). |
+| `systemPrompt` | string? | System prompt for this turn; project context and the Langfuse prompt are prepended |
+| `attachments` | array? | `{type: "image" \| "file", url, mimeType, filename?}` items added to the user message |
+| `model` | string? | Model id passed to `createModelFn` or `createServerModelFn` |
+| `projectId` | string? | Project whose context and memories are prepended to the system prompt; saved on a history that has none |
+
+#### SSE events
+
+`/gpt/prompt` streams `data: <json>` lines, one event object per line. `{askResolved}` is sent when an answer is stored, before the model runs. `{started}` is sent when the model produces its first part and the reply row is saved, before any text from that part. On a prompt that is not answering an ask, `{started}` is first. `{ask}` and `{done}` come last. With `uiBlocks` on, `{text}` is the finished document and comes before `{ask}`.
+
+| Event | Shape | When |
+|-------|-------|------|
+| `{started}` | `{started: true, historyId, streamId}` | Sent when the model produces its first part and the reply row is saved. A reload can resume this `streamId`. It comes before text from that part. A model that fails before any part sends no `{started}`. The demo reply does not send it. |
+| `{askResolved}` | `{askResolved: {toolCallId, action}}` | The turn answered the pending ask, or cancelled it because a new `prompt` arrived. Asks only. |
+| `{text}` | `{text: string}` | A step's text, sent when the step ends. Text from a step that calls a tool is dropped, unless that text parses as a block document and the turn has no other text. A trailing JSON `"action"` blob is stripped. With `uiBlocks` on, this is the final document: sent once, after id-fill, repair, and sanitizing, and before `{ask}`. |
+| `{toolCall}` | `{toolCall: {toolCallId, toolName, args}}` | The model called a host tool (route, request, or MCP). Never sent for ask tools. |
+| `{file}` | `{file: {filename, mimeType, url}}` | A host tool result had a `fileData` data URL. Sent before its `{toolResult}`. `filename` defaults to `document` and `mimeType` to `application/octet-stream`. |
+| `{toolResult}` | `{toolResult: {toolCallId, toolName, result}}` | A host tool returned. `fileData` is removed from `result`. For a tool whose approval was denied, `result` is `{approved: false, reason}`. Never sent for ask tools. |
+| `{image}` | `{image: {mimeType, url}}` | The model generated an image; `url` is a `data:` URL. Each data URL is sent once. |
+| `{ask}` | `{ask: {toolCallId, kind, input, simple, origin?, toolName?}, historyId}` | The turn paused on an ask. Sent after the turn is saved. `historyId` is the conversation that waits on the ask, so a new chat's ask can be answered before `{done}`. An [approval ask](agent-ui-asks.md#approval-asks) adds `origin: "approval"` and the host `toolName`. Asks only. |
+| `{error}` | `{error: string}` | The model stream reported an error, or the turn failed after the stream started. `{done}` still follows. |
+| `{replace}` | `{replace: "text", text}` | The assistant document after the server changed text the client already has: a missing actions `id` was filled in, `repair: true` rewrote the document, or `uiBlocks.html` sanitized it. Sent before `{blocks}`. With `uiBlocks` on, the first send is `{text}` of that finished document, so `{replace}` is not sent for it. |
+| `{blocks}` | `{blocks: {ok, errors, warnings}}` | The final assistant text checked as a block document. Sent after the text and before `{done}`, only when `uiBlocks` is on and the turn produced text. |
+| `{done}` | `{done: true, historyId?, title?, pendingAsk?}` | Last event of every turn that started streaming, also after `{error}`. `historyId` is missing only in the demo response and when a failed new chat could not be saved. `title` is set once the conversation has one. `pendingAsk: {toolCallId}` when the turn waits on an ask. |
+
+When a turn fails after the stream starts, before the model's first chunk or partway through, the stream sends `{error}` then `{done}` with `historyId`. The turn keeps what the client already saw: the user's message, host tool rows, and any partial assistant text, saved with `status: "error"`. An empty placeholder is removed. An ask the failed stream had started is dropped. When the model call after an answer fails before the client gets any text, tool result, or ask, the answer is undone: the stream is `{askResolved}`, `{error}`, `{done, pendingAsk}`, the ask is pending again, and the same answer can be sent again. When it fails later, the stream is `{askResolved}`, `{error}`, `{done}` and the answer is kept: the ask stays answered and sending it again returns 409. Send a new `prompt` to continue. See [Agent UI Asks](agent-ui-asks.md#answer-an-ask).
+
+Errors raised before the stream starts return JSON `{status, title, detail, fields?}` instead: 400 for an invalid body, 403 for another user's history, 404 for an unknown `historyId`, 409 for an answer to an ask that is not pending, and 500 otherwise. A `prompt` never gets 409: when another request resolved the ask it meant to cancel, it goes ahead as a normal message. [Agent UI Asks error responses](agent-ui-asks.md#error-responses) lists the ask cases.
 
 #### Attachments
 
@@ -269,11 +332,11 @@ When `fileStorageService` is set, `data:` attachments are uploaded with `FileSto
 
 #### Stream events and resume
 
-`/gpt/prompt` saves the user turn and a `status: "streaming"` assistant placeholder before streaming. While the reply streams, partial text is persisted about every second (`streamPersistIntervalMs`), plus a heartbeat every 10 seconds. The final reply replaces the placeholder with `status: "complete"`. On failure, partial text is kept with `status: "error"`; an empty placeholder is removed.
+`/gpt/prompt` saves the user turn and a `status: "streaming"` assistant placeholder when the model produces its first part. Waiting until then lets two turns on one history both load it before either writes. While the reply streams, partial text is persisted about every second (`streamPersistIntervalMs`), plus a heartbeat every 10 seconds. A text reply replaces that placeholder in place with `status: "complete"`. On failure, partial text is kept with `status: "error"`; an empty placeholder is removed.
 
 | Event | Sent by | Meaning |
 |-------|---------|---------|
-| `{historyId, started: true, streamId}` | prompt | First event; the turn is saved and resumable |
+| `{historyId, started: true, streamId}` | prompt | The reply row is saved and resumable; before any text from the model's first part |
 | `{historyId, resumed: true, streamId?}` | resume | First event; `streamId` is absent when nothing is streaming |
 | `{text}` | both | Text delta |
 | `{replace: true, text}` | resume | Authoritative whole reply: sent first when the client provides `offset`, and whenever persisted text was rewritten (for example, a step became a tool call) |
@@ -284,8 +347,6 @@ When `fileStorageService` is set, `data:` attachments are uploaded with `FileSto
 
 `GET /gpt/histories/:id/stream` re-attaches after a reload or remount. Pass `offset` as the number of characters the client already shows, usually the stored placeholder `text`. The endpoint polls the stored history (`streamResumePollIntervalMs`, default 500 ms), so it works across server instances. A `streaming` reply with no update for `streamStaleAfterMs` (default 60 s) is marked `error`.
 
-AI resolution order: `x-ai-api-key` header + `createModelFn` → `createServerModelFn(modelId)` → configured `aiService` → demo SSE response when `demoMode` and none available.
-
 ### addGptHistoryRoutes(router, options?)
 
 CRUD at `/gpt/histories` via `modelRouter`:
@@ -295,7 +356,16 @@ CRUD at `/gpt/histories` via `modelRouter`:
 | Create, List | `IsAuthenticated` |
 | Read, Update, Delete | `IsOwner` |
 
-Query filtered by `userId`; sort `-updated`; query fields `userId`, `projectId`.
+Query filtered by `userId`; sort `-updated`; query fields `userId`, `projectId`. Create sets `userId` to the caller, so `{}` is a valid create body even when the app validates request bodies. Create and update bodies drop `pendingAsk` (including dotted `pendingAsk.*` paths), so only a chat turn writes it; the OpenAPI spec marks it `readOnly` on create and update.
+
+Pass `chat`, the options given to `addGptRoutes`, to add two headless actions for clients that do not read server-sent events, such as a watch app. Both exist only when `chat` turns `asks` on: without `chat`, or with `asks` off, neither action exists, so a host that never turned asks on gets no new endpoints. `AiApp` adds them when its `asks` option is set.
+
+| Endpoint | Method | Auth | Description |
+|----------|--------|------|-------------|
+| `/gpt/histories/pendingAsks` | GET | `IsAuthenticated` | The caller's pending asks with their simple cards, newest first |
+| `/gpt/histories/:id/turn` | POST | `IsOwner`; admins who do not own the history get 403 | Runs one chat turn to completion with the `chat` options and returns it as JSON. Body: one of `{prompt}`, `{askResponse}`, or `{toolCallId, buttonId}`, plus `surface`. The turn finishes and saves even if the client disconnects. |
+
+Bodies, results, and errors: [Headless endpoints](agent-ui-asks.md#headless-endpoints).
 
 ### addProjectRoutes(router, options?)
 
@@ -311,9 +381,11 @@ Requires `fileStorageService` and `gcsBucket` (registered by `AiApp` when both a
 
 | Endpoint | Method | Auth | Description |
 |----------|--------|------|-------------|
-| `/files/upload` | POST | `IsAuthenticated` | Multipart upload (`file` field); allowed MIME: images, PDF, plain text, CSV, JSON. Returns `403` when `fileUploadsEnabled` is off |
-| `/files/*gcsKey` | GET | None | Returns signed read URL (1 hour) |
-| `/files/*gcsKey` | DELETE | `IsAuthenticated` (owner) | Soft-delete attachment and remove from GCS |
+| `/files/upload` | POST | `IsAuthenticated` | Multipart upload (`file` field); allowed MIME: images, PDF, plain text, CSV, JSON. Capped at `maxFileSize` (default 10 MB). Returns `403` when `fileUploadsEnabled` is off. Returns `{data: {id, filename, gcsKey, mimeType, size, url}}`; send `id` as the `fileId` of a [`files` ask](agent-ui-asks.md#files) answer. |
+| `/files/*gcsKey` | GET | `IsAuthenticated` (owner) | Returns `{data: {url}}`, a signed read URL (1 hour), for the caller's own upload. Another user's file returns 404, the same as a missing one, so keys cannot be probed; admins get no exception. |
+| `/files/*gcsKey` | DELETE | `IsAuthenticated` (owner) | Soft-delete attachment and remove from GCS. 404 for a missing file, 403 for another user's. |
+
+`*gcsKey` is the full key with its slashes, such as `uploads/<userId>/<ms>-<name>`.
 
 ### addMcpRoutes(router, options)
 
@@ -356,10 +428,11 @@ new AiApp({
 | Option | Description |
 |--------|-------------|
 | `aiService` | Pre-configured server-wide AI service |
+| `asks` | Let the model ask the user typed questions in chat: `true` or `{approvals, kinds, maxFileSizeBytes}`. `approvals` sets the [approval ask](agent-ui-asks.md#approval-asks) for host tools with `needsApproval`. Passed to `addGptRoutes`, and adds the headless `pendingAsks` and `turn` actions to `/gpt/histories`; see [Agent UI Asks](agent-ui-asks.md). `maxFileSizeBytes` also caps `/files/upload`. |
 | `createModelFn` | Build model from per-request `x-ai-api-key` |
 | `createServerModelFn` | Server-side model factory (e.g. Vertex ADC) without per-request key |
-| `demoMode` | Return canned responses when no AI service resolves |
-| `fileStorageService` + `gcsBucket` | Enable file upload routes and durable `/gpt/prompt` attachments |
+| `demoMode` | Not read. The routes send a canned demo reply whenever no AI service resolves |
+| `fileStorageService` + `gcsBucket` | Enable file upload routes, durable `/gpt/prompt` attachments, and `files` ask answers that name uploads by `fileId` |
 | `fileUploadsEnabled` | `false` or a function returning `false` rejects uploads and chat attachments with `403`. Omit to leave uploads enabled |
 | `mcpService` | Enable MCP routes and tool discovery in chat |
 | `tools` | Static Vercel AI SDK tool definitions for chat |
@@ -402,6 +475,158 @@ Client construction failures log a warning and skip the plugin so the API proces
 
 Calls `shutdownLangfuseClient()` and `shutdownTracing()` on `SIGTERM`.
 
+## Observability
+
+In-app prompt versions, nested traces, evaluators, datasets, experiments, review queue, and in-app feedback. Operator loop: [Develop an AI feature](../how-to/ai-feature-development.md). Register plugins: [Observe LLM calls](../how-to/observe-llm-calls.md). Why two planes: [AI observability](../explanation/ai-observability.md). Locked design: [implementation plan](../implementationPlans/ai-observability.md).
+
+Register `ObservabilityApp` with at least a local plugin. Construction throws if `experiments.primary !== datasets.primary`, if `reviewQueue` is not `local`, or if a control primary has no matching plugin. Defaults for all four primaries are `local`. Construction also registers the app as the process singleton (`getObservabilityApp()`) through the dependency-free observability registry, so routes do not import the plugin class that registers them. Call `resetObservabilityApp()` in tests. `createLocalObservabilityPlugin()` registers the local Mongo models (`ObsPrompt`, `ObsPromptVersion`, `ObsPromptLabel`, `ObsTrace`, `ObsSpan`, `ObsScore`) on the default connection.
+
+The example backend always registers `createLocalObservabilityPlugin()` and passes the
+validated `AI_OBS_PRICE_MAP_JSON` object as `priceMap`. `bun run backend:seed` idempotently
+creates `examples/example-summarize` with production on v1 and an experimental v2, installs
+`correctness-human` and `schema-assert`, and creates a two-item proofread `example-gold`
+dataset bound to the prompt input schema. The same seed creates `examples/chat-safety-screen`
+(production v1 scores the new message alone; v2 reads earlier turns), `examples/chat-safety-judge`,
+the `chat-safety-agreement` llm-judge, and the 12-item proofread synthetic dataset
+`chat-safety-synthetic`. Each row is one new message plus earlier turns from a two-person chat,
+labeled for 988 vs care-team routing, toxicity, a privacy leak, and a dismissive reply.
+Admin → Scripts → `seedChatSafetyDataset`, or `bun run script seedChatSafetyDataset --wet` from
+`example-backend`, loads that set into an already-running database. `SEED_DEFAULTS=true` loads
+it on boot, including PR preview. Invalid price JSON or negative/non-numeric prices
+fail startup with `AI_OBS_PRICE_MAP_JSON` in the error.
+
+`POST /ai/example-summarize` (example backend) runs that seeded prompt with
+`promptLabel: "production"`, `userId`, and `sessionId` from `x-ai-session-id`. It uses the
+server `AIService` when configured, otherwise a request-scoped service built from
+`x-ai-api-key`, and returns **503** when neither exists. The example frontend calls it from
+**Todos → Summarize**.
+
+### Local observability models
+
+| Model | Role |
+| --- | --- |
+| `ObsPrompt` | Named prompt (`name` unique) with `folder` and `tags[]` |
+| `ObsPromptVersion` | Immutable `vN` body, `variables[]`, schemas, `sensitive` (default false), `config` |
+| `ObsPromptLabel` | Movable labels; unique `(promptId, label)` |
+| `ObsTrace` | Root trace: user, session, status, `errorSummary`, `sensitive`, `prompts[]`, usage |
+| `ObsSpan` | Nested span with `kind`, `status`, optional `error`, offsets, usage |
+| `ObsScore` | Scores on a trace/span; many per trace, **no unique index** |
+| `ObsEvaluator` | Evaluator: `type` (`human` \| `llm-judge` \| `json-assert`), `target`, `dimensions[]`, `runModes`, `instructions`, `judgePromptName` (judge), `assertion` (json-assert), `confidenceAlertBelow` (default 0.7) |
+| `ObsReviewItem` | Review queue item: status, evaluator, trace, reason, scores, comment |
+| `ObsDataset` | Named dataset with optional `inputSchemaPromptName` and `expectedOutputSchema` |
+| `ObsDatasetItem` | Item with `input`, `expectedOutput`, `origin`, `proofread`, `tags`, `outcomeClass`, `sourceTraceId`, `metadata` |
+| `ObsExperiment` | Compares 2–3 prompt versions on a dataset with thresholds and aggregates |
+| `ObsExperimentItem` | Per dataset row: outputs per version, evaluator score maps, gate failure flags |
+
+`POST /ai/observability/traces/review` requires a `human` `ObsEvaluator`. Its
+`dimensions[]` render as reviewer score fields and `instructions` render above the review form.
+Automatic evaluator types return **400** instead of entering the human queue. Submitting a review
+requires every dimension marked `required`; omitted optional dimensions do not create empty score
+rows. The local score store remains the fallback when no external score sink is configured.
+
+`AIService` generate methods:
+
+| Option | Default | Behavior |
+| --- | --- | --- |
+| `promptName` | unset | Resolve `PromptRegistry.get({name, label})` **before** the model call, even when `skipTrace` is true |
+| `promptLabel` | `"production"` | Label used with `promptName` |
+| `skipTrace` | `false` | Skip `TraceSink.export` only; prompt resolve and `AIRequest` still run |
+| `sensitive` | inherited | Explicit value wins; otherwise the resolved prompt version's `sensitive` |
+| `sessionId` / `userId` | unset | Copied onto the exported trace |
+| `priceMap` | app `priceMap` | Per-call override; `costUsd` is omitted when the model is unpriced |
+
+Missing registry, missing prompt, or missing label throws `APIError` 400 and does not call the model. Sink `export` failures are logged and never fail generate.
+
+**GPT tool spans:** `/gpt/prompt` collects streamed `tool-call` / `tool-result` events into `TOOL` child spans (name = tool name, input = args, output = cleaned result with large `fileData` stripped). When any tool span is present, `AIService.recordGenerate` exports one trace with a `CHAIN` root plus `TOOL` children linked by `parentSpanId`. Ordinary `generateText` / JSON helpers without `childSpans` still emit a single `LLM` root span.
+
+`ObservabilityApp.exportTrace(trace)` fans out to every `TraceSink` (best-effort) and returns the first persisted `{id}` from sinks that support it (for example `LocalTraceSink`). `TraceSink.export` may return `TraceExportResult` (`{id?: string}`) or `void`; `MemoryTraceSink` remains in-memory only.
+
+When `prompts.primary` is `local`, `ObservabilityApp.register` mounts admin-only prompt routes at `/ai/observability`. Pass `aiService` on `ObservabilityApp` for playground runs and the multi-stage trace smoke endpoint. Apps that let an admin supply a per-request provider key may instead set `requestAiServiceFactory`; both the playground and the multi-stage smoke endpoint read the key from `x-ai-api-key`, while a configured server `aiService` remains preferred. `GET /ai/observability/status` is always mounted so admin chrome can read plugin ids, capabilities, primaries, `localOn`, and `playgroundAi.source` (`server` \| `request-key` \| `unavailable`). Admin UIs use that field to distinguish “save a provider key” from true backend misconfiguration.
+
+| Method | Path | Behavior |
+| --- | --- | --- |
+| GET | `/ai/observability/status` | Admin chrome. `{plugins, primaries, localOn, playgroundAi}` — drives the status chip, hides Review when `localOn` is false, and tells the prompt playground whether AI comes from `ObservabilityApp.aiService` (`server`), per-request `x-ai-api-key` via `requestAiServiceFactory` (`request-key`), or is not configured (`unavailable`) |
+
+| Method | Path | Behavior |
+| --- | --- | --- |
+| GET | `/ai/observability/prompts` | List. Query `folder`, `search`, `include=usage7d` (7-day calls/cost). `production` is `"—"` until a production label exists |
+| POST | `/ai/observability/prompts` | Create prompt in a folder as immutable v1 (`latest` label) |
+| GET | `/ai/observability/prompts/:name` | Prompt + versions + labels |
+| POST | `/ai/observability/prompts/:name/versions` | Create `vN+1`; never mutates an existing version |
+| POST | `/ai/observability/prompts/:name/labels` | Move `production` or `staging`; `outgoingVersion` is the previous pointer |
+| POST | `/ai/observability/prompts/:name/playground` | Compile `{{var}}` + one `AIService` call; returns compiled messages, output, latency, tokens, cost; creates no version. Uses `ObservabilityApp.aiService`, or `requestAiServiceFactory({apiKey, modelId})` when the server service is absent (`apiKey` comes from `x-ai-api-key`) |
+
+`PromptRegistry.get({name, label})` (default label `production`) reads the labelled local version. `createLocalObservabilityPlugin()` wires `LocalPromptStore` as that registry and local `TraceSink` / `ScoreSink`.
+
+| Method | Path | Behavior |
+| --- | --- | --- |
+| GET | `/ai/observability/traces` | Admin list. Query `from`, `to`, `prompt`, `status`, `userId`, `sessionId`, `hasScore`, `sensitive`, `flaggedForDataset`, `page`, `limit`. Body is `{data, page, limit, more, total}` so pagination survives RTK `{data}` unwrap. Each row includes `spanCount` and `scoreCount`. `prompts.length` is the `N prompts` count |
+| GET | `/ai/observability/traces/:id` | Span tree (kind, offsets, durations, I/O, cost) plus scores. `errorSummary` is the first span with `status: "error"` |
+| POST | `/ai/observability/traces/:id/scores` | Persist a score and fan out to every `ScoreSink` |
+| POST | `/ai/observability/traces/test-multi-stage` | Admin-only smoke workflow, registered only with the local trace sink. Uses `ObservabilityApp.aiService`, or `requestAiServiceFactory({apiKey})` when the server service is absent (`apiKey` comes from `x-ai-api-key`); answers **503** with the same missing-key title as playground when neither exists. Body `{input?: string}` (defaults to a built-in sample). Runs two `AIService.generateJsonObject` calls with `skipTrace: true` and named JSON output schemas (`obs-test-multi-stage-call-1` / `call-2`), a deterministic local `text-metrics` `TOOL` stage, then a final `generateJsonObject` synthesis against `obs-test-multi-stage-final`; exports exactly one parent trace with ordered child spans `LLM`, `LLM`, `TOOL`, `LLM` under a `CHAIN` root. LLM span input includes `outputSchema`. Returns `{traceId, output, stages[]}` where `output` is the final schema object (`sentence`, `phrase`, `keywords`, `metrics`). Child LLM failures export an error trace then rethrow |
+
+`createLocalObservabilityPlugin()` registers `ObsEvaluator` with the other local models.
+
+| Method | Path | Behavior |
+| --- | --- | --- |
+| GET | `/ai/observability/evaluators/templates` | Seeded templates: `llm-judge` (`correctness`, `hallucination`, `helpfulness`, `toxicity`), `json-assert` (`schema-assert`), and human queue variants (`correctness-human`, …) |
+| POST | `/ai/observability/evaluators/templates/:name` | Install a template by name as an immutable-named evaluator |
+| GET/POST | `/ai/observability/evaluators` | List / create. Create accepts `target: "full trace"` only (`generation span` and `dataset item` → 400). Seeded template install can still store other targets. `llm-judge` requires `judgePromptName`; create rejects when the judge prompt `outputSchema` omits a required dimension (400 names the key). `json-assert` supports `assertion` (`path` + `constraint`) or built-in output-schema mode. Human + `liveSampleRate > 0` → 400. Numeric dimension `range` is `min-max` (for example `0-1`); categorical `range` is `label|label` |
+| GET/PATCH/DELETE | `/ai/observability/evaluators/:id` | Read / update / soft-delete |
+| POST | `/ai/observability/traces/review` | Enqueue one or many traces against a human evaluator (`reason: "manual"`) |
+| GET | `/ai/observability/review` | Queue by `status` with counts; oldest-first. Response includes `more: false` so RTK preserves the count envelope. Rows include `traceName`, `promptName`, assignee, reason, and enqueue time |
+| GET | `/ai/observability/review/:id` | Item + evaluator dimensions + `given` / `wrote` panels and `rawInput` / `rawOutput` for the Raw JSON disclosure |
+| POST | `/ai/observability/review/:id` | `submit` (scores via ScoreSinks, status `done`), `skip`, or `assign` |
+
+`createLocalObservabilityPlugin()` wires `LocalDatasetStore` and `LocalExperimentRunner` when datasets/experiments primaries are `local`.
+
+| Method | Path | Behavior |
+| --- | --- | --- |
+| GET/POST | `/ai/observability/datasets` | List (includes `humanCount` for proofread items, `autoCount` for unreviewed trace or synthetic items, `needsReviewCount`) / create |
+| GET/PATCH/DELETE | `/ai/observability/datasets/:id` | Detail (with counts) / update / soft-delete. PATCH `null` clears optional dataset fields; omitted fields stay unchanged |
+| GET/POST | `/ai/observability/datasets/:id/items` | List / create items |
+| PATCH/DELETE | `/ai/observability/datasets/:id/items/:itemId` | Update labels (`expectedOutput`, `proofread`, `tags`, `outcomeClass`) / delete (does not touch the source trace). PATCH `null` clears optional item fields |
+| POST | `/ai/observability/datasets/:id/import` | **JSON:** body is an array of bare input objects, or structured rows with `input` / `expectedOutput` / `proofread` / `tags` / `outcomeClass` / `metadata`. **CSV:** `Content-Type: text/csv` with raw CSV body, or JSON `{format: "csv", content: "..."}`. Plain columns map to `input`; plain `input` / `expectedOutput` cells accept JSON values; `input.foo` and `expectedOutput.foo` nest fields; reserved `proofread`, `tags`, `outcomeClass` map metadata. Nested paths reject `__proto__`, `constructor`, and `prototype` segments. Rows validate against the dataset's bound prompt `inputSchema` when `inputSchemaPromptName` is set; 400 reports row number and JSON path |
+| POST | `/ai/observability/traces/add-to-dataset` | `{datasetId, traceId \| traceIds[]}`. Copies span I/O; `origin: "trace"`; `sourceTraceId` set; **sensitive traces always `proofread: false`** |
+
+| Method | Path | Behavior |
+| --- | --- | --- |
+| POST | `/ai/observability/experiments/estimate` | `{datasetId, promptName, versions[], evaluatorIds[], modelOverride?}` → generation count, USD, wall-clock estimate |
+| GET/POST | `/ai/observability/experiments` | List / create. Body: dataset, 2–3 version numbers, evaluator ids, optional `thresholds[]` (defaults to `SOP_DEFAULT_THRESHOLDS`), `modelOverride`, `includeUnproofread` (default false). Evaluator ids must be `llm-judge` or `json-assert` (human → 400). Local primary always enqueues `BackgroundTask` (even one item) |
+| GET | `/ai/observability/experiments/:id` | Status, progress, per-version aggregates, gate pass/fail (`gates[].version`), `outlierItemIds`, `lowConfidenceItemIds`, per-item side-by-side (**failed rows first**) |
+| POST | `/ai/observability/experiments/:id/promote` | `{version}` moves the `production` label when **that version's** gates pass; **409** while any gate for the selected version fails |
+
+Authenticated `POST /ai/observability/traces/:id/feedback` records thumbs, outcome class, and flag-for-dataset (phase 2.6).
+
+## Durable agent harness
+
+`@terreno/ai/harness` runs multi-phase tasks that survive restarts. Each phase commits a
+checkpoint and its `ObsSpan` audit record in one Mongo transaction, so it needs a replica
+set and `createLocalObservabilityPlugin()`. Owner and task leases resume crashed work on a
+fresh process; phases that are not replay-safe park as `interrupted` until an operator calls
+`resolveInterrupted`. A thrown phase retries with exponential backoff before it fails.
+Phases can start child tasks (`rt.createTask`), wait on them (`rt.waitForTasks`), and
+`harness.abort` stops a whole ownership tree bottom-up, running compensation handlers.
+`defineAgent` / `defineTool` run durable agent conversations: each turn is a task whose
+model requests retry 429/5xx and fall back to other models, and whose tool calls run as
+child tasks with per-tool replay rules. `rt.runAgent` runs an agent as a subagent from a
+phase (a task-owned conversation) and returns its text or schema-checked output.
+`defineExtension` bundles prompt sections, tools, tool wraps, and hooks
+(`beforeModelRequest`, `beforeTool` with block or rewrite, `afterTool`); `rt.memo` stores
+first-write-wins decisions that survive restarts. `rt.waitFor(event, {timeout})` and
+`rt.sleep(duration)` park a task without a lease; `harness.sendEvent` stores the event
+durably and wakes the task, even when no runner is up. `rt.approval(key, {...})` waits for
+a human decision; `approvalGate` requires one before named tool calls; `HarnessApp` serves
+`GET /harness/approvals` (only what the caller may approve) and `approve` / `reject`
+instance actions, each decision audited in an `approval:<key>` span. Clients watch runs
+over SSE (`GET /harness/conversations/:id/events`, `GET /harness/tasks/:id/events`),
+resumable from any instance with `Last-Event-ID`; turns stream coalesced text deltas, and
+`POST /harness/conversations/:id/submit` queues or steers a message while a turn runs.
+API: [AI harness reference](ai-harness.md).
+Why: [Durable agent harness](../explanation/durable-agent-harness.md).
+How-to: [Build a durable workflow](../how-to/build-a-durable-workflow.md),
+[Ship a new task version](../how-to/ship-a-new-task-version.md).
+
 ## Langfuse integration
 
 Low-level exports (also used by `addGptRoutes` when `langfuseSystemPromptName` is set):
@@ -423,7 +648,8 @@ const storage = new FileStorageService({
   storageOptions: {}, // optional @google-cloud/storage options
 });
 
-await storage.upload({buffer, filename, mimeType, userId});
+await storage.upload({buffer, filename, mimeType, userId}); // {id, filename, gcsKey, mimeType, size, url}
+await storage.download(gcsKey);      // the upload's bytes, as a Buffer
 await storage.getSignedUrl(gcsKey);  // 1-hour v4 signed URL
 await storage.delete(gcsKey);        // GCS delete + soft-delete FileAttachment
 ```
@@ -505,7 +731,7 @@ new TerrenoApp({userModel: User})
   .start();
 ```
 
-Legacy `setupServer` pattern: call `addGptHistoryRoutes`, `addGptRoutes`, etc. inside `addRoutes`.
+Legacy `setupServer` pattern: call `addGptHistoryRoutes`, `addGptRoutes`, etc. inside `addRoutes`. With `asks` on, pass the chat options to `addGptHistoryRoutes` as `chat` to keep the headless endpoints.
 
 ## Environment variables
 
@@ -513,7 +739,7 @@ Legacy `setupServer` pattern: call `addGptHistoryRoutes`, `addGptRoutes`, etc. i
 |----------|---------|-------------|
 | `GOOGLE_VERTEX_PROJECT` | `createVertexProvider` | GCP project for Vertex models |
 | `GOOGLE_VERTEX_LOCATION` | `createVertexProvider` | Vertex region (default `global`) |
-| `LANGFUSE_PUBLIC_KEY` | `LangfuseApp` | Langfuse public key |
+| `AI_OBS_PRICE_MAP_JSON` | `ObservabilityApp` | JSON model map with non-negative `inputPerMTok` / `outputPerMTok`; omitted models have tokens but no USD cost || `LANGFUSE_PUBLIC_KEY` | `LangfuseApp` | Langfuse public key |
 | `LANGFUSE_SECRET_KEY` | `LangfuseApp` | Langfuse secret key |
 | `LANGFUSE_BASE_URL` | Langfuse client | Langfuse host URL |
 
@@ -532,7 +758,7 @@ GCS credentials use standard Google Cloud Application Default Credentials for `F
 
 - Framework: `bun test` with preload `./src/tests/bunSetup.ts`
 - HTTP: supertest against real routes
-- DB: memory Mongo via `@terreno/test` (`TERRENO_TEST_USE_MEMORY_MONGO` or `TERRENO_TEST_MONGODB_URI`)
+- DB: in-memory single-node replica set via `@terreno/test` (transactions work); `TERRENO_TEST_MONGODB_URI` overrides it and must point at a replica set
 - Mock AI model: implement `doGenerate` and `doStream` on a fake `LanguageModel`
 
 ```typescript

package/src/docs/versioned/next/reference/api.md

@@ -41,7 +41,7 @@ REST API framework built on Express and Mongoose. Provides modelRouter (CRUD end
 - Validation: `configureOpenApiValidator`, `validateRequestBody`, `validateQueryParams`, `createValidator`
 - Middleware: `openApiEtagMiddleware`, `sentryAppVersionMiddleware`
 - Extensibility: `TerrenoPlugin` interface
-- Notifiers: `sendToSlack`, `formatSlackUserMention`, `lookupSlackUserIdByEmail`, `sendToGoogleChat`, `sendToZoom`
+- Notifiers: `sendToSlack`, `formatSlackUserMention`, `lookupSlackUserIdByEmail`, `createSlackPrivateChannel`, `inviteSlackUsersToChannel`, `postSlackMessage`, `findSlackChannelByName`, `sendToGoogleChat`, `sendToZoom`
 - HTTP client: `createAuthenticatedClient`, `withApiErrorHandling`, `normalizeApiError`, `markRetryUnsafe`
 
 ## Server Setup
@@ -224,6 +224,30 @@ export const todoRouter = modelRouter("/todos", Todo, {
 Do not add `endpoints: (router) => { router.get(...) }` when an action fits.
 Action config supports `allowAnonymous: true` as a per-action override; it does not change authentication for CRUD or sibling actions.
 
+## modelRouter list queries
+
+`GET /{path}` accepts filters on fields listed in `queryFields` (qs syntax, so operators work:
+`?status[$in][0]=open&created[$gte]=2026-01-01`), plus the reserved `limit`, `page`, and `sort`.
+`limit` is clamped to `maxLimit` (default 500); the response echoes the applied `limit` with
+`{data, limit, more, page, total}`.
+
+A filter on a field missing from `queryFields` is rejected before query validation runs, so
+it can never be stripped and silently ignored:
+
+```json
+{
+  "status": 400,
+  "code": "query-param-not-allowed",
+  "title": "Query parameter not allowed",
+  "detail": "title is not allowed as a query param. Add \"title\" to queryFields on the Todo modelRouter to filter on it. Allowed: completed, created, ownerId.",
+  "source": {"parameter": "title"},
+  "meta": {"queryParam": "title", "model": "Todo", "allowedQueryFields": ["completed", "created", "ownerId"]}
+}
+```
+
+`@terreno/syncdb` raises this as `QueryFieldNotAllowedError` (see
+[Query windows](syncdb.md#query-windows)).
+
 ## modelRouter array operations
 
 When a model has at least one array field, `modelRouter` provides subroutes for adding,
@@ -235,6 +259,10 @@ updating, and removing array items:
 | `PATCH` | `/:id/:field/:itemId` | Replace or merge an item |
 | `DELETE` | `/:id/:field/:itemId` | Remove an item |
 
+`:id` must be an ObjectId. Any other id leaves this router so a later route on the same
+prefix can handle it. `POST /ai/observability/evaluators` is not an array push on a model
+mounted at `/ai`.
+
 The `field` path must resolve to an array on the document. Requests for scalar, object,
 or unknown fields run the normal update transform and `preUpdate` hook first so consumer
 authorization remains authoritative, then return `400` with code
@@ -834,6 +862,48 @@ await Membership.create({
 });
 ``````
 
+### AI observability RBAC
+
+`terrenoStatements` includes domain-neutral resources for `@terreno/ai` `ObservabilityApp`
+routes. Pass `accessControl` from `createAccess` on `ObservabilityApp` to enforce them; omit
+`accessControl` to keep legacy `user.admin` / `Permissions.IsAdmin` behavior.
+
+| Resource | Actions |
+| --- | --- |
+| `aiPrompt` | `list`, `read`, `create`, `update`, `promote`, `playground` |
+| `aiTrace` | `list`, `read` |
+| `aiReview` | `list`, `read`, `score`, `assign` |
+| `aiDataset` | `create`, `list`, `read`, `update`, `delete` |
+| `aiEvaluator` | `create`, `list`, `read`, `update`, `delete` |
+| `aiExperiment` | `list`, `read`, `create`, `promote` |
+
+Human-readable labels for the admin role editor live in `terrenoStatementDescriptions`
+(optional on `createAccess({ statementDescriptions })`).
+
+| Role | Observability grants |
+| --- | --- |
+| `superadmin` | `*` (every action on every resource) |
+| `admin` (seeded) | Full grants on all six resources |
+| `auditor` | Read-only sentinel expands to `list` and `read` on each observability resource (no `admin:access`) |
+| `aiObservabilityViewer` (example-backend only) | `admin:access` plus `list` and `read` on each observability resource |
+| `aiObservabilityOperator` (example-backend only) | `admin:access` plus full grants on each observability resource |
+
+The seeded `auditor` role does **not** include `admin:access`. Observability `list` / `read`
+grants alone do not open the admin shell — compose a consumer role (for example
+`auditor` + `admin: ["access"]`, or a custom role) when operators should browse observability
+screens.
+
+Re-seeding default roles does not overwrite customized unsealed roles, except seeded
+`admin` and `auditor`: on each `seedDefaults`, Terreno **merges** missing observability
+actions into stored permissions (`admin` — full grants; `auditor` — `list` and `read` per
+resource) without removing consumer-defined grants on other resources.
+
+When `accessControl` is configured, each observability HTTP route checks `admin:access` and
+the mapped resource action. Callers with `user.admin === true` still receive full access
+(legacy compatibility). RBAC-only operators need `admin:access` plus the exact action for
+mutations. Route mapping (for example prompt label moves → `aiPrompt:promote`, review
+submit → `aiReview:score`) is documented in [AI reference — Observability](ai.md#observability).
+
 ## Request Validation
 
 @terreno/api provides runtime validation of incoming requests against OpenAPI schemas using [AJV](https://ajv.js.org/). Validation is opt-in and can be enabled globally or per-route.
@@ -1535,6 +1605,44 @@ await sendToSlack(`${formatSlackUserMention("U012ABCDEF")} Deployment complete`,
 });
 ``````
 
+Incoming webhooks are fixed to the channel chosen when the webhook was created.
+Private channels, invites, and messages to a channel id use the bot token for
+one Slack workspace (`SLACK_BOT_TOKEN`):
+
+| Function | Slack method | Bot scopes |
+| --- | --- | --- |
+| `createSlackPrivateChannel` | `conversations.create`, `conversations.invite`, `chat.postMessage` | `groups:write`, `groups:write.invites`, `chat:write` |
+| `inviteSlackUsersToChannel` | `conversations.invite` | `groups:write.invites` for private channels, `channels:write.invites` for public |
+| `postSlackMessage` | `chat.postMessage` | `chat:write` |
+| `findSlackChannelByName` | `conversations.list` | `channels:read`, `groups:read` |
+| `lookupSlackUserIdByEmail` | `users.lookupByEmail` | `users:read.email` |
+
+The bot can post and invite only in channels it belongs to. Creating a private
+channel makes the bot a member. An existing private channel must already
+include the app; a bot cannot join a private channel by itself. Store the
+returned channel id. Names are unique, mutable, and limited to lowercase
+letters, numbers, hyphens, and underscores.
+
+``````typescript
+import {createSlackPrivateChannel, inviteSlackUsersToChannel} from "@terreno/api";
+
+const channel = await createSlackPrivateChannel({
+  name: "case-12345",
+  userIds: ["U012ABCDEF"],
+  initialMessage: "Case opened.",
+  mentionUserIds: ["U012ABCDEF"],
+});
+
+await inviteSlackUsersToChannel({
+  channelId: channel.channelId,
+  userIds: ["U098765432"],
+});
+``````
+
+A taken name throws status 409. `findSlackChannelByName({name})` returns a
+public channel, or a private channel the bot is already in. It returns
+`undefined` when that channel is not visible to the bot.
+
 ### Google Chat Notifications
 
 ``````typescript
@@ -1867,7 +1975,7 @@ Complete reference of environment variables used by @terreno/api:
 | Variable | Required | Default | Description |
 |----------|----------|---------|-------------|
 | `SLACK_WEBHOOKS` | No | — | JSON object mapping names to Slack webhook URLs: `{"default":"https://..."}` |
-| `SLACK_BOT_TOKEN` | No | — | Bot token (`xoxb-…`) with `users:read.email` for `lookupSlackUserIdByEmail` |
+| `SLACK_BOT_TOKEN` | No | — | Bot token (`xoxb-…`) for one workspace. Scopes: `users:read.email`, `groups:write`, `groups:write.invites`, `channels:write.invites`, `chat:write`, `channels:read`, `groups:read` |
 | `GOOGLE_CHAT_WEBHOOKS` | No | — | JSON object mapping names to Google Chat webhook URLs |
 | `ZOOM_CHAT_WEBHOOKS` | No | — | JSON object mapping names to Zoom webhook URLs |
 | `WEBHOOK_SECRET` | No | — | Secret for validating incoming webhook signatures |