@terreno/mcp 57.17.0 → 57.19.2

package/src/docs/versioned/57.17.0/how-to/agent-ui-blocks.md

@@ -0,0 +1,288 @@
+# Validate a block document locally
+
+`terreno-blocks` checks one whole-reply YAML or JSON document before you send it.
+
+1. Compile the package so the bin exists: `cd blocks && bun run compile`.
+2. Run `terreno-blocks validate path/to/document.yaml`.
+3. A valid document exits 0. Each problem prints as `path  CODE  message — fix` and the
+   process exits 1.
+4. Pass `-` to read the document from stdin.
+
+The same check is `validateBlocks` in `@terreno/blocks`. Field tables are in the
+[blocks reference](../reference/blocks.md).
+
+## Preview a document in the playground
+
+1. Open the component demo and choose **BlocksPlayground**.
+2. Start from a preset: **All blocks** (one of each block type), **Block type** (one preset per
+   block type, with every chart kind, callout status, and action kind), Layout, Invalid, or
+   Sunday roast. Or paste your own document into the text area.
+3. The preview updates as you type. An invalid document shows the error banner and keeps
+   the raw text collapsed.
+4. Press the controls. The playground has no server: a stepper's − and + and a checklist tick
+   run on the device with `scaleStepperBlock` and `applyChecklistState` from `@terreno/blocks`
+   (the helpers `scaleStepperHostAction` and `toggleChecklistHostAction` use), and a toast names
+   the callback and its result, such as "scaleStepper → 6 People". A reply, open, select, or
+   other callback shows a toast describing the event instead. Copy writes the clipboard.
+   Editing the text drops the local results.
+
+## Require documents from the model
+
+Pass `uiBlocks: true` to `addGptRoutes`. The model is told that its whole reply is one document. When the turn's text finishes, the route checks it and sends `{blocks: {ok, errors, warnings}}` before `{done}`.
+
+```ts
+addGptRoutes(router, {
+  aiService,
+  uiBlocks: {
+    hostActions: {export_csv: {}},
+    repair: true,
+  },
+});
+```
+
+`hostActions` is the callback allowlist. A callback outside it fails with `UNKNOWN_HOST_ACTION`. `repair: true` runs one repair call and stores that document. A document that is still invalid is stored with a `Block validation errors:` note so the next turn sees the codes.
+
+An actions block that omits `id` gets one before storage. A block document written in the same step as a tool call is kept when no later step has text, so the first reply still shows. The chat receives that document after ids are filled, repair runs, and html is sanitized, so an invalid draft is not shown. `{replace: "text"}` is sent only when text was already streamed and then changed.
+
+Outside a chat turn, `AIService.generateBlocks({prompt})` returns one validated document. Temperature is 0. A failed check is repaired once. Pass `repair: false` to skip that retry. A second validation failure throws 422 and logs `metadata.errorCodes`. A model or network error throws 502 and is not repaired.
+
+## Store rows for a ref chart
+
+1. Turn `uiBlocks` on. `addGptRoutes` then mounts `GET /gpt/datasets/:id`.
+2. From a server tool, call `registerAiDataset({userId, historyId, columns, rows})`. The return type is `RegisteredDataset` (`datasetId`, `columns`, `rowCount`, `preview`, `stats`).
+3. Hand the model the returned `datasetId`. A `ref` dataset uses that id. The model cannot create one.
+4. The owner reads rows with `GET /gpt/datasets/:id?grain=week&limit=40`. Leave `page` off to downsample a line or area with LTTB. Pass `page` to paginate a table; `more` is true when another page remains.
+5. `datasetTtlDays: 0` (the default) stores no `expiresAt`. `7` sets `expiresAt` to seven days after `created`. More than `datasetMaxRows` (default 50,000) returns 413. Another user's id returns 404.
+
+## Register a server callback
+
+1. Pass `hostActions` on `uiBlocks`. Each name has a Zod `payload` and a `handler`.
+2. The client posts `{historyId, messageId, blockId, elementId, name, payload}` to `POST /gpt/actions`.
+3. Return `{replace: "block", blocks}` to swap the block, or `{text}` to append an assistant message. `blocks` must be a valid document (`v` then `blocks`).
+4. An unknown name is 404. A payload that fails the schema is 400 with `meta.fields`. Another user's history is 403. The handler stops at 10 seconds with 504. An invalid document from the handler is 500.
+
+```ts
+addGptRoutes(router, {
+  aiService,
+  uiBlocks: {
+    hostActions: {
+      export_csv: {
+        payload: z.object({format: z.literal("csv")}).strict(),
+        handler: async () => ({
+          replace: "block",
+          blocks: {v: 1, blocks: [{type: "badge", text: "Exporting", status: "info"}]},
+        }),
+      },
+    },
+  },
+});
+```
+
+The path is `/gpt/actions`, so a limiter on `/gpt` covers it. Each call is stored as an `AIRequest` with `requestType: "ui_action"`.
+
+## Add a stepper callback
+
+A `stepper` is a − value + control. Each tap goes to a host action, and the action returns the
+replacement stepper. Use it when a number changes other numbers, such as guests changing a
+shopping list.
+
+1. Register a stepper action. `scaleStepperHostAction` from `@terreno/ai` scales every item
+   linearly from the stepper the agent wrote:
+
+   ```ts
+   import {addGptRoutes, scaleStepperHostAction} from "@terreno/ai";
+
+   addGptRoutes(router, {
+     aiService,
+     uiBlocks: {hostActions: {scaleStepper: scaleStepperHostAction}},
+   });
+   ```
+
+2. Leave `uiBlocks.richBlocks` unset (it defaults to `true`). The prompt offers `stepper` only
+   when some host action has `handles: "stepper"`, and names those actions as its callbacks.
+3. List the same name in `GPTChat` `hostActions` (`["scaleStepper"]`). `onBlockCallback` posts
+   to `POST /gpt/actions` and returns the result, as in [Wire the example chat](#wire-the-example-chat).
+4. The agent writes a stepper whose `callback.name` is that action:
+
+   ```yaml
+   - type: stepper
+     id: guests
+     label: Number of people
+     unit: People
+     value: 5
+     min: 1
+     max: 20
+     callback: {name: scaleStepper}
+     items:
+       - {label: Bone-in leg of lamb, amount: 2, unit: kg, decimals: 1}
+       - {label: Carrots, amount: 8, round: up}
+   ```
+
+5. Press +. The client sends `payload: {...callback.payload, value: 6}`, with `blockId: guests`
+   and `elementId: guests_increase`. The action returns `{replace: "block", blocks}` and
+   `GPTChat` swaps the stepper: lamb reads "2.4 kg", carrots "10". Both buttons are disabled
+   until the reply lands, so a fast second tap is dropped.
+
+A stepper whose `callback.name` is not a `handles: "stepper"` action fails `UNKNOWN_HOST_ACTION`.
+`value` outside `min`–`max`, or off the `step` grid, returns 400.
+
+**Stored prompts are owner-writable.** The history owner can change any stored prompt with
+`PATCH /gpt/histories/:id`, including the stepper `scaleStepperHostAction` reads. When the numbers
+matter (prices, stock, dosage), write your own action and compute them from your own data:
+
+```ts
+import {findAgentBlock, type HostAction} from "@terreno/ai";
+import {z} from "@terreno/api";
+
+const priceGuests: HostAction = {
+  handles: "stepper",
+  logResponse: false, // keep the returned block out of the AIRequest log
+  payload: z.object({value: z.number().int().min(1).max(20)}).passthrough(),
+  handler: async ({blockId, history, messageId, payload, user}) => {
+    const stepper = findAgentBlock({blockId, history, messageId, type: "stepper"});
+    const {value} = payload as {value: number};
+    const items = await quoteItems({guests: value, userId: user._id}); // your data, not the stored block
+    return {replace: "block", blocks: {v: 1, blocks: [{...stepper, items, value}]}};
+  },
+};
+```
+
+`findAgentBlock` reads `history.prompts[n]` for `messageId: msg-<n>`, otherwise the only assistant
+prompt holding that block: several matches throw 409, none 404. Field rules are in
+[Stepper](../reference/blocks.md#stepper) and [Host actions](../reference/ai.md#host-actions).
+
+## Add a checklist callback
+
+A `checklist` without a `callback` ticks on the device only. To send ticks to the server:
+
+1. Register `toggleChecklistHostAction` (or your own `handles: "checklist"` action) under a name,
+   such as `toggleChecklist`, and list it in `GPTChat` `hostActions`.
+2. The prompt then tells the model to set `callback: {name: toggleChecklist}` on a checklist.
+3. A tick sends `{...callback.payload, itemId, checked, state}`, where `state` holds every item's
+   tick after this one. The whole checklist is locked until the reply lands, and the "n of m"
+   counter changes when the returned checklist arrives.
+
+`toggleChecklistHostAction` returns the stored checklist with `checked` from `state` and saves
+nothing. An unknown item id returns 400. An app that records progress writes its own action.
+
+## Copy a block's text
+
+Add a button with `action: {kind: copy, target: <block id>}` (or `text` for a literal). The
+target is a `stepper`, `checklist`, `list`, `table`, or `text` block. `BlocksView` copies the
+block as it is shown now, so a stepper at 6 copies the scaled amounts. Nothing reaches the
+server, so `hostActions` does not apply. See [Copy](../reference/blocks.md#copy).
+
+## Keep the old catalog for shipped native builds
+
+Rich blocks (`stepper`, `checklist`, `gallery`, `list`, `card.eyebrow`, `copy`) are in the prompt
+by default. A native build released before them shows an error banner for a reply that uses
+them. Until those builds update, turn them off:
+
+```ts
+addGptRoutes(router, {aiService, uiBlocks: {hostActions, richBlocks: false}});
+```
+
+The prompt is then the same as before rich blocks, even with a stepper action registered.
+Validation still accepts the new blocks.
+
+## Smoke-test a real model
+
+`bun run blocks:smoke` (from the repo root) sends "Give me a plan for a sunday lamb roast, I'm
+having friends over still figuring out numbers tbh" through one real chat turn with the example
+backend's `uiBlocks` options and the `findPhotos` tool.
+
+| Check | Passes when |
+| --- | --- |
+| Document | The reply parses and passes `validateBlocks` |
+| Blocks | It holds a `gallery`, `list`, `stepper`, and `checklist`, and a `copy` action |
+| Photos | Every image `src` is a `file:` id that `findPhotos` returned in that turn |
+
+| Variable | Use |
+| --- | --- |
+| `GOOGLE_VERTEX_PROJECT` (with Application Default Credentials) or `GEMINI_API_KEY` | The model. With neither, the script prints that it skipped and exits 0 |
+| `BLOCKS_SMOKE_MODEL` | Optional model id instead of the server default |
+| `MONGO_URI`, `MONGO_DB_NAME` | The database holding the photo library ([Give the agent photos](#give-the-agent-photos)) |
+| `BLOCKS_SMOKE_OUT_DIR` or `--out <dir>` | Where `reply.yaml` and `result.json` go (default: a new temp directory) |
+
+It exits 1 when any check fails. The turn runs as `blocks-smoke@system.invalid`, and its history
+is deleted afterwards.
+
+## Wire the example chat
+
+1. Turn `uiBlocks` on in `addGptRoutes` so `GET /gpt/datasets/:id` and `POST /gpt/actions` are in the OpenAPI spec.
+2. Regenerate the frontend SDK: `cd example-frontend && bun run sdk`.
+3. Set `uiBlocks` and `hostActions` on `GPTChat`. `reply` already calls `onSubmit`.
+4. `onBlockCallback` calls `usePostGptActionsMutation`. Return that result so a `{replace: "block"}` swaps the block.
+5. `resolveDataset` calls `useLazyGetGptDatasetsByIdQuery` and returns `{columns, rows, source: "inline"}`.
+6. An `open` action uses `router.push` for a `/…` route and `Linking.openURL` for an `https` URL.
+7. When the `{blocks}` event is `ok`, set `blockNote` to the top-level block count, such as `3 components`. Recompute that caption from stored assistant YAML when a history is loaded again.
+
+## Turn blocks on in the example backend
+
+`example-backend` passes `uiBlocks` (`exampleUiBlocksOptions` in `src/ai/hostActions.ts`) with
+`repair: true`, `html: true`, and three host callbacks:
+
+| Name | Handles | Does |
+| --- | --- | --- |
+| `exportDataset` | buttons | Payload `{dataset: string}`. Returns a badge that names that dataset |
+| `scaleStepper` | `stepper` | `scaleStepperHostAction` |
+| `toggleChecklist` | `checklist` | `toggleChecklistHostAction` |
+
+`richBlocks` is left at its default (`true`). Dataset TTL stays at the default, so stored rows are kept.
+
+The per-request tool `todoStats` counts the signed-in user's open and completed todos, calls `registerAiDataset`, and returns `datasetId`. It stores the rows on `req.body.historyId` when that id belongs to the caller. With no history id it uses the caller's newest history, and with no history it returns `{datasetId: null, rowCount: 0}`. A chart dataset uses `source: ref` and that id. Columns are `status` and `count`.
+
+`GPTChat` lists `exportDataset`, `scaleStepper`, and `toggleChecklist` in `hostActions`, so those callbacks stay enabled. The e2e mock also allows `export_csv`.
+
+## Give the agent photos
+
+The model cannot make photos on the fly, and an `https` image needs `imageHosts`. The example
+app generates a photo library once, and the agent cites its photos as `file:` ids.
+
+1. Generate the library, once per environment. From `example-backend`, with `GOOGLE_VERTEX_PROJECT`,
+   Application Default Credentials for Vertex, and `GCS_BUCKET` set, run `bun run photos:generate`.
+   Each prompt in `src/scripts/photoPrompts.ts` becomes one `PhotoLibraryEntry`. A re-run skips
+   prompts that already have an entry; pass `--force` to regenerate them. `PHOTO_IMAGE_MODEL`
+   (default `imagen-4.0-fast-generate-001`) and `GOOGLE_VERTEX_LOCATION` (default `us-central1`)
+   are optional.
+2. Register the `findPhotos` tool. `createFindPhotosTool()` from `src/ai/tools.ts` is in the
+   example's per-request tools. `findPhotos({query, count})` matches the query words against each
+   entry's `tags` and `alt`. `count` is 1–6. It returns `{photos: [{src: "file:<entry id>", alt}], note}`.
+   Entries that match more words come first; ties keep entry id order. Soft-deleted entries are
+   left out. When nothing matches, `photos` is empty and `note` tells the model not to invent an
+   id. The tool description tells the model to put each `src` into a `gallery`, `list`, or
+   `image` block exactly as returned.
+3. Serve the URLs. `photoLibraryRouter` (`src/api/photoLibrary.ts`) mounts `/photoLibrary`.
+   List and read need a signed-in user; create, update, and delete return 405.
+   `GET /photoLibrary/:id/url` returns `{url}`, a signed read URL that lasts one hour, to any
+   signed-in user. The library is shared, so the per-user `GET /files/*` route does not fit.
+   An unknown or soft-deleted id is 404. Without a bucket (`GCS_BUCKET` unset and none saved in
+   Profile) it returns 503 `Photo storage is not configured`.
+4. Regenerate the frontend SDK: `cd example-frontend && bun run sdk`. This adds
+   `useLazyPhotoLibraryUrlQuery`.
+5. Pass `resolveImage` to `GPTChat`. The example builds it with `createPhotoImageResolver`
+   (`example-frontend/lib/photoLibraryImages.ts`) around `openapi.useLazyPhotoLibraryUrlQuery`.
+   It fetches each id once while the URL is fresh (50 minutes). A failed lookup returns
+   `undefined`, so the block keeps its labelled placeholder, and a later render tries again.
+
+In the example app, every `file:` image id in a block is a photo library entry id. Nothing else
+in the example writes `file:` image ids, so the ids carry no extra prefix. A host that also cites
+its own files must tell the two apart in its `resolveImage`. An id that is not in the library,
+such as `file:roast-lamb` in the reference examples, resolves to `undefined` and shows the
+placeholder.
+
+## Render documents in chat
+
+Set `uiBlocks` on `GPTChat`. Assistant `content` is the YAML document. While `isStreaming`
+is true, finished top-level blocks render and a spinner marks the block still arriving.
+
+- `reply` calls `onSubmit` with the button's text.
+- `open` and `select` call `onBlockAction`.
+- `callback` calls `onBlockCallback`. Return `{replace: "block", blocks}` to swap that
+  block, or `{text}` to append an assistant message. The button shows loading until the
+  promise settles.
+- Stepper taps and checklist ticks with a callback also call `onBlockCallback`. The block is
+  disabled until the promise settles.
+- `copy` writes the clipboard on the device and calls nothing.
+- Pass `hostActions` to disable callbacks the host does not run, `resolveDataset` to
+  load `ref` datasets, and `resolveImage` to turn a `file:` image id into a URL.

package/src/docs/versioned/57.17.0/how-to/ai-feature-development.md

@@ -0,0 +1,161 @@
+# Develop an AI feature
+
+Work this loop in **Terreno admin** when observability primaries are `local`. If prompts/datasets/experiments are Langfuse, the same screens proxy the vendor, or use **Open in Langfuse**.
+
+Architecture: [AI observability](../explanation/ai-observability.md). Routes and models: [AI reference](../reference/ai.md). Register plugins: [Observe LLM calls](./observe-llm-calls.md).
+
+Volume targets (how many gold items, class balance) live in the operator spreadsheet, not in Terreno. Use **tags** and **outcome class** so you can filter for balance.
+
+For a walkable example, run `bun run backend:seed`: it installs
+`examples/example-summarize` with production on v1 and candidate v2, the human
+`correctness-human` and automatic `schema-assert` evaluators, and the proofread
+`example-gold` dataset. It also installs `chat-safety-synthetic`, twelve proofread
+synthetic chats between two people. Each item is the new message plus a few earlier
+turns. Gold labels cover 988 (immediate danger to self), care-team follow-up,
+toxicity, a privacy leak, and a dismissive reply. Compare `chat-safety-screen` v1
+(new message only) with v2 (earlier turns) using evaluator `chat-safety-agreement`.
+These fixtures are proofread on purpose so experiments include them. The Generate
+button still lands drafts as `proofread: false` until a human accepts them. Load the
+same set into a running database, including a PR preview, with Admin → Scripts →
+`seedChatSafetyDataset` (wet run).
+
+## 1. Gather a gold dataset
+
+1. Open **Datasets** → create a named set. Optionally set `inputSchemaPromptName` to a registry prompt whose `inputSchema` should validate imported rows, and `expectedOutputSchema` if product has agreed a label shape.
+2. Add items in three ways:
+   - **From production traces** (preferred for open-ended features): Traces list → filter → **Add to dataset** (`POST /ai/observability/traces/add-to-dataset`). Sensitive traces always land `proofread: false`.
+   - **Import JSON or CSV** (`POST /ai/observability/datasets/:id/import`): JSON can be an array of bare input objects or structured rows with `input`, `expectedOutput`, `proofread`, `tags`, `outcomeClass`, and `metadata`. CSV accepts `Content-Type: text/csv` or `{format: "csv", content: "..."}`; plain columns map to `input`, `input.foo` nests fields, `expectedOutput.foo` maps labels, and reserved `proofread` / `tags` / `outcomeClass` columns map metadata. Invalid rows against the bound input schema return 400 with the row number and JSON path.
+   - **Manual**: `POST /datasets/:id/items` with a production-shaped `input`.
+   - **Synthetic** (phase 2.4): **Generate** drafts. They land as `origin: synthetic` and `proofread: false`. Do not run experiments on them until a human proofreads and sets `proofread: true`.
+3. Prefer a mix of true/false positives and negatives on classification features. Tag critical cases.
+
+In-app **thumbs** / **false positive|negative** on a live feature write feedback scores and can **flag for dataset** (step 7). Those traces show up with `flaggedForDataset`.
+
+## 2. Label expected output
+
+1. Open the dataset item.
+2. Fill `expectedOutput` to match the schema from step 1.
+3. Set `outcomeClass` (`tp` / `fp` / `tn` / `fn`) when the feature is a flag.
+4. Add `tags` for categories you will filter on later.
+5. Set `proofread: true` on synthetic items you accept.
+
+Skip unlabeled items in experiments unless you only need generation smoke (still prefer labels before scoring).
+
+## 3. Create the prompt
+
+1. Open **Admin → AI Observability → Prompts**. The folder rail shows counts; search filters name and folder. If your role lacks `aiPrompt:create`, **Create prompt** is hidden (API writes still 403).
+2. **Create prompt** (folder, name, system, user template with `{{variables}}`). That write is immutable **v1**. Open the row to the prompt hub (**Overview**, **Versions**, **Traces**, **Experiments**).
+3. On **Versions**, edit system/template/variables/temperature when `aiPrompt:update` is allowed, then **Save as vN+1**. There is no in-place save. **Playground → Run once** (when `aiPrompt:playground` is allowed) compiles and calls the model; it does **not** create a version. **Traces** and **Experiments** show recent items for this prompt from prompt detail relationships.
+4. Do not paste that string into app routes. Apps call `AIService` with `promptName` + `promptLabel` (step 6).
+
+Read-only auditors with `aiPrompt:list` and `read` can review overview metadata and relationship tabs but do not see create, save, promote, or playground controls. Operators with the full `aiPrompt` action set see those controls. UI gating follows `permissions` on `GET /ai/observability/status`; assign roles in **Admin → Roles** per [API reference — AI observability RBAC](../reference/api.md#ai-observability-rbac).
+
+Judge prompts for evaluators are named constants or registry prompts — never inline in `AIService`.
+
+## 4. Set up evaluators
+
+**Evaluators** → **Create from template** (`POST /ai/observability/evaluators/templates/:name`) or custom (`POST /ai/observability/evaluators`):
+
+| Feature shape | Template | Score |
+| --- | --- | --- |
+| Boolean flags (urgent, etc.) | `correctness` (`llm-judge`) or `correctness-human` (review queue) | boolean `correct` |
+| Open-ended text | `hallucination` / `hallucination-human` | boolean `hallucinated` / `contains_hallucination` |
+| Open-ended text | `helpfulness` / `helpfulness-human` | numeric 0–1 |
+| Open-ended text | `toxicity` / `toxicity-human` | boolean `is_toxic` / `toxic` |
+| Structured JSON output | `schema-assert` (`json-assert`) | boolean `schema_valid` |
+| Open-ended text (if useful) | `conciseness` | numeric 0–1 (human) |
+
+`llm-judge` templates require a registry judge prompt (`eval-judge-*`) whose `outputSchema` declares every required dimension. Create returns 400 naming any missing key. `confidenceAlertBelow` defaults to **0.7**.
+
+A human evaluator with `runModes.liveSampleRate > 0` is rejected (400). Live sampling for judges is task 2.2, capped by `AI_OBS_SAMPLE_RATE`.
+
+Add more dimensions with product/engineering. Attach them to the experiment in step 5.
+
+## 5. Run experiments
+
+1. **Experiments** → `POST /ai/observability/experiments` with dataset id, **2–3 prompt version numbers**, evaluator ids, optional `modelOverride`, and optional `thresholds[]`.
+2. Thresholds default to the SOP gates (override per run if product disagrees):
+
+| Dimension | Gate |
+| --- | --- |
+| correctness true-rate | = 100% |
+| hallucination true-rate | = 0% |
+| helpfulness mean | ≥ 90% |
+| toxicity true-rate | = 0% |
+
+3. Wait for the run (`BackgroundTask` when local — even for a one-item dataset).
+4. `GET /ai/observability/experiments/:id` returns gate pass/fail, aggregates (`trueRate` for booleans, `mean` for numerics), `outlierItemIds`, `lowConfidenceItemIds`, and per-item outputs (**failed rows first**).
+5. Skim **per-item outputs**. If they are nonsense, fix the prompt before trusting aggregates.
+6. Read **pass/fail vs thresholds**. Open outliers and low-confidence items (default alert below 0.7). Use those items to rewrite the prompt.
+7. Save a new prompt version. Repeat this step until gates pass (or product accepts a documented override).
+8. `POST /ai/observability/experiments/:id/promote` with `{version}` moves `production` when all gates pass; **409** while any gate fails. Experiments never auto-promote without this call.
+
+Unproofread synthetic items are excluded unless you set `includeUnproofread`.
+
+## 6. Deploy to production
+
+1. In the prompt editor, select the winning version and **Set vN as production…**. The confirm modal names the **outgoing** production version (or none). Experiments never auto-promote.
+2. In the app, resolve that label at call time:
+
+```typescript
+await aiService.generateText({
+  prompt: text,
+  promptLabel: "production",
+  promptName: "example-summarize",
+  userId: req.user?.id,
+});
+```
+
+Missing `production` fails the request (400). Do not fall back to a hardcoded string.
+
+3. Bound cost before users reach the AI routes. `@terreno/ai` has no per-user token or spend
+   limit, and a chat reply keeps running after the client disconnects. Enable `TerrenoApp`
+   `rateLimit` and set a provider budget or quota alert; see
+   [Cost and abuse limits](../reference/ai.md#cost-and-abuse-limits).
+
+## 7. Production tracing and feedback
+
+Do this in parallel with steps 1–6 once the feature is reachable.
+
+1. Register `ObservabilityApp`. Every `generate*` emits a trace unless `skipTrace`.
+2. Pass `userId` and `sessionId` from the request.
+3. From the product UI, call `POST /ai/observability/traces/:id/feedback` as the signed-in user:
+   - `kind: "thumbs"` + `up` / `down` (helpfulness of suggestions)
+   - `kind: "outcome"` + `tp` / `fp` / `tn` / `fn` (flags)
+   - `flagDataset: true` to queue the trace as a dataset candidate
+4. In **Admin → AI Observability → Traces**, filter by time, prompt dropdown, status, user, session, score presence, or data sensitivity. Score and sensitivity dropdowns include both positive and negative states. Rows show a status dot, `sensitive` badge, error line, `N prompts`, tokens, cost, and latency. Open a row for the span tree (kind, indent, duration bar). Sensitive I/O disclosures start **collapsed**.
+5. Select one or more traces. The bulk bar warns when any selected row is `sensitive`. Choose **Send to human review**, read the queue explanation, and pick a human evaluator. The evaluator's dimensions become the score fields and its instructions guide the reviewer. Confirming posts `POST /ai/observability/traces/review` with `reason: "manual"`. Automatic (`llm-judge` / `json-assert`) evaluators cannot enter this queue.
+6. Open **Review queue**. Tabs show Pending / In progress / Done / Skipped counts, and pending
+   items are oldest-first. Select **Start reviewing — oldest first**.
+7. On each item, compare **What the AI was given** with **What the AI wrote**. Long fields begin
+   collapsed with a word count; structured outputs retain field labels and reviewer notes. Use
+   **Raw JSON** only when the field view is insufficient.
+8. Score every required evaluator dimension: numeric dimensions use the declared range slider,
+   boolean dimensions use Pass / Fail, and categorical dimensions use pills. Add an optional
+   comment, then **Submit & next**. **Skip** records no score. For the second phase-1 intake path,
+   use **Assign to me** to manually claim an item and move it to In progress. Continue until the
+   toast says **Queue clear**.
+
+Feedback is a score (`source: "user-feedback"`) fanned out to every ScoreSink. Non-owners who are not admin get 403.
+
+## 8. Add weak production traces back to the dataset
+
+1. **Traces**: filter `thumbs=down`, `flaggedForDataset`, `failedEval`, or `lowConfidence` (filters land with task 2.6 feedback).
+2. **Add to dataset** (`POST /ai/observability/traces/add-to-dataset`, bulk allowed).
+3. Label the new items (step 2).
+4. Repeat steps 5–8.
+
+Phase 1 also supports **Send to review queue** from the Traces bulk bar (step 7) when human scoring is enough before dataset intake.
+
+## Checklist (one feature)
+
+- [ ] Gold set exists with mixed origins and labels
+- [ ] Prompt versions in the registry, not inlined in routes
+- [ ] SOP (or custom) evaluators attached
+- [ ] Experiment gates green or explicitly overridden
+- [ ] `production` label on the winning version
+- [ ] App calls `promptName` + `promptLabel`
+- [ ] Live traces include user/session
+- [ ] Product can send thumbs / outcome / flag-for-dataset
+- [ ] Human review queue reaches Queue clear
+- [ ] Weak traces can be added back to the dataset

package/src/docs/versioned/57.17.0/how-to/audit-log.md

@@ -0,0 +1,85 @@
+# Enable the framework audit log
+
+Register `AuditApp` and set `audit: true` on routers you want logged. Successful HTTP mutations write append-only `AuditEvent` rows. Default retention is forever.
+
+## Steps
+
+### 1. Register the plugin
+
+```typescript
+import {AuditApp, TerrenoApp} from "@terreno/api";
+
+new TerrenoApp({userModel: User}).register(new AuditApp()).start();
+```
+
+Importing `@terreno/api` does not register `AuditEvent`. HTTP list/read is admin-only at `GET /audit-events` and `GET /admin/audit-events`. Create/update/delete over HTTP are disabled (405), including the admin remount.
+
+### 2. Opt in per router
+
+```typescript
+modelRouter("/todos", Todo, {
+  audit: true, // or {redact: ["ssn"]}
+  permissions: {
+    /* ... */
+  },
+});
+```
+
+`audit: true` without `AuditApp` still returns 2xx and logs an error once per process.
+
+## Operations
+
+| HTTP | `verb` | `operation` | Diff |
+| --- | --- | --- | --- |
+| POST `/` | `created` | `create` | `after` only |
+| PATCH `/:id` | `updated` | `update` | Changed fields in `before` / `after` |
+| DELETE `/:id` | `deleted` | `delete` | `before` only |
+| POST `/:id/:field` | `updated` | `arrayPush` | Changed fields; `recordId` set |
+| PATCH `/:id/:field/:itemId` | `updated` | `arrayUpdate` | Changed fields; `recordId` set |
+| DELETE `/:id/:field/:itemId` | `updated` | `arrayRemove` | Changed fields; `recordId` set |
+
+`source` is `modelRouter` for these writes. Secrets (`password`, `hash`, `salt`, `token`,
+`secret`, `refreshToken`) are omitted at every object and array depth, including compound
+keys such as `tokenHash`. Extra `redact` names merge with that list as exact field names.
+
+### 3. Admin mutations
+
+When `AuditApp` is registered, AdminApp writes the same `AuditEvent` collection with
+`source: "admin"` after successful admin POST/PATCH/DELETE. You do not need `onAdminAudit` for
+that. `onAdminAudit` remains an extra sink if the app still wants a second destination.
+Admin diffs also omit each model's `hiddenFields` and `excludeFields`.
+
+RBAC mutations fan into the same collection when `createAccess({auditSink: persistRbacAuditToAuditEvent})` is set. `source` is `rbac`.
+
+`organizationId` is copied onto the event when `req.organization` is set (`id` or `_id`), otherwise from the request org AsyncLocalStorage, then `X-Organization-Id`, then the mutated document's `organizationId` string. Socket admin-window mutates attach `organizationId` on the payload so non-tenant models still land in the selected organization's audit log. The field is omitted when none of those exist. Admin list/read for `AuditEvent` is not organization-scoped, so unscoped rows stay visible when no organization is selected.
+
+`actorId` comes from `req.user.id` (or `_id`). Only a 24-hex Mongo ObjectId is stored;
+mongoose's looser `isValidObjectId` (12-character strings) is not used. Any other id
+shape still writes the event with `actorId` omitted and a warning logged.
+
+Update and delete snapshot `before` only when `audit` is on. If that `toJSON` throws, the
+mutation still completes and the event is recorded without a `before` snapshot.
+
+### 4. Fire-and-forget persist
+
+HTTP handlers do not await Mongo. Diffs still run before the response so `before` / `after` are captured. Failures log; the mutation stays 2xx.
+
+To persist completely off the API process, set Cloud Tasks env (`GCP_PROJECT`, `GCP_LOCATION`, `GCP_TASKS_AUDIT_QUEUE`, `AUDIT_TASKS_URL`, `AUDIT_TASKS_SECRET`) and `bun add @google-cloud/tasks`. `AuditApp` then enqueues the write and handles `POST /internal/audit-events` with header `X-Terreno-Audit-Secret`. Or pass `enqueue` yourself:
+
+```typescript
+new AuditApp({
+  enqueue: async (write) => {
+    await myQueue.push(write);
+  },
+  processQueuePath: "/internal/audit-events",
+  processQueueSecret: process.env.AUDIT_TASKS_SECRET,
+});
+```
+
+Authenticated malformed queue payloads are acknowledged without persistence. They cannot
+become valid on retry, and returning a non-2xx response would make Cloud Tasks retry the
+same poison task indefinitely.
+
+### 5. Retention
+
+Omit `retentionDays` or set `0` to keep events forever (no TTL index). `new AuditApp({retentionDays: 90})` creates a Mongo TTL index on `created` with `expireAfterSeconds = 90 * 86400`, replacing the default `{created: 1}` field index so Mongo does not reject a duplicate key pattern. Mongo expires documents in the background; lowering or removing TTL later requires dropping that index yourself (`db.auditevents.dropIndex(...)`) — Mongoose will not remove it.