@terreno/mcp 57.9.0 → 57.10.1

package/src/docs/versioned/next/explanation/agent-ui-blocks.md

@@ -0,0 +1,67 @@
+# Agent UI blocks
+
+When a chat turns UI blocks on, the assistant reply is one YAML document of Terreno
+components, not free HTML or JSX. This page explains why the catalog is closed and why the
+whole reply is the document. Field tables and error codes are in the
+[reference](../reference/blocks.md). Asks, which return a typed answer to the agent, stay
+in [Agent UI Asks](agent-ui-asks.md).
+
+The design record is
+[agent-ui-blocks.md](../implementationPlans/agent-ui-blocks.md). Pick executes that work as
+Phase 5 of [Agent UI Asks](../implementationPlans/agent-ui-asks.md).
+
+## Why a closed catalog
+
+An agent that can emit arbitrary components can also emit styles, scripts, and layout the
+app theme does not own. The catalog is the list of `@terreno/ui` components the chat can
+paint: heading, text, metric, badge, divider, context, chart, table, actions, and the two
+layout blocks `columns` and `card`. Unknown
+keys and unknown block types fail validation, so a reply cannot smuggle a `style` object
+or a hex color.
+
+The same pure functions run wherever a document is checked. `parseBlocks` and
+`validateBlocks` live in `@terreno/blocks`, which has no React and no Express. The renderer
+and the chat route will call them; they do not grow a second grammar.
+
+## Why the whole reply is YAML
+
+The document is the reply. There is no prose before it and no fenced block inside
+markdown. A leading or trailing fence is stripped so a model that adds one still parses.
+JSON is accepted because it is a YAML 1.2 subset and is what a structured-output call
+emits.
+
+Key order is `v`, then `datasets`, then `blocks`. Datasets come first so a streaming
+renderer can know the data before the first block. `parseBlocksPartial` returns the
+top-level blocks that have already finished and marks the cut tail pending. The finished
+document is still checked with `validateBlocks`.
+
+Text in the same step as a tool call is not the reply, except when that text is a block document and no later step has text. An actions block that left off `id` receives one before storage, and the chat is sent the corrected document.
+
+A reply that is not a mapping with `v` is not a document. The chat can still show it:
+`wrapAsTextDocument` turns the raw text into one `text` block. The error
+`NOT_A_DOCUMENT` is how the caller tells that case apart from a document that failed a
+field check.
+
+YAML anchors, aliases, and explicit tags are rejected. They let one node stand in for
+another, which makes a small document able to expand in ways the limits cannot see.
+
+## Ownership
+
+| Layer | Owns |
+| --- | --- |
+| `@terreno/blocks` | Schema, parse, partial parse, validate, JSON Schema, prompt section, CLI, limits, error codes |
+| `@terreno/ui` | `BlocksView` for heading, text, metric, badge, divider, context, chart, table, actions, columns, and card. A select action changes the target chart or table dataset in that view. A segmented control starts on the option that matches that dataset. A callback stays disabled when `hostActions` is set and does not include its name. `GPTChat` `uiBlocks` renders assistant messages through `BlocksView`, including a spinner while the reply is still streaming. A `file:` image stays as alt text until `resolveImage` returns a URL. |
+| `@terreno/ai` | When `uiBlocks` is on: the chat system prompt, the post-stream `{blocks}` check, `AIDataset` storage, and `POST /gpt/actions`. `AIService.generateBlocks` returns one validated document outside a chat turn. |
+
+Asks use the same package under `src/asks/`. A block shows something. An ask collects an
+answer and returns it to the agent.
+
+A callback button is the Block Kit `block_actions` path. The document names the callback.
+The host owns the code.
+
+| | Block Kit | Terreno |
+| --- | --- | --- |
+| Who runs the click | Slack posts `block_actions` to the app | The client posts `POST /gpt/actions` |
+| Payload check | The app trusts the platform body | The host Zod schema runs before the handler |
+| What comes back | `chat.update` replaces the message | `{replace, blocks}` replaces the block, or `{text}` appends a message |
+| Time limit | The platform's request window | 10 seconds, then 504 |

package/src/docs/versioned/next/explanation/ai-observability.md

@@ -0,0 +1,101 @@
+# AI observability
+
+Terreno ships Langfuse-like prompt versioning, nested traces, evaluators, datasets, experiments, and a local human review queue **inside `@terreno/ai`**, with operator UI in **`admin-frontend` only**.
+
+## Why not Langfuse-only
+
+Apps must iterate prompts without a required vendor account. `LangfuseApp` stays a second plugin. Traces and scores **fan out** to every registered sink (local Mongo, Langfuse, OTLP/OpenInference). Prompts, datasets, experiments, and the review queue have **one writer** each (a primary). The review queue is **local-only**.
+
+That split is the two planes:
+
+| Plane | Write | Admin read |
+| --- | --- | --- |
+| Telemetry (traces, scores) | All sinks, best-effort | Local if the local plugin is on, else Langfuse |
+| Control (prompts, datasets, experiments, queue) | One primary per capability | That primary |
+
+`experiments.primary` must equal `datasets.primary`. Defaults are all `local`.
+
+`ObservabilityApp` fails boot (constructor throws) when:
+
+| Error | Cause |
+| --- | --- |
+| `experiments.primary must equal datasets.primary` | Mixed dataset/experiment writers |
+| `reviewQueue.primary must be local` | `reviewQueue` set to `langfuse` |
+| `<capability> primary "<id>" has no plugin` | No registered plugin with that `id`, capability, and store |
+
+Telemetry sinks still fan out even when a control primary is local-only.
+
+Admin chrome lives in `admin-frontend`. One sidebar group **AI Observability** holds Prompts,
+Traces, Evaluators, Datasets, Experiments, and Review (Review is omitted when the local plugin is
+off). Existing **AI Requests** stays under **Screens**. Every observability screen shows
+breadcrumbs `Admin / AI Observability / <Section> / <leaf>` and a status chip from
+`GET /ai/observability/status`. Prompt detail is a hub (Overview, Versions, Traces, Experiments);
+versions are immutable in admin (`Save as vN+1`); apps never inline the string.
+
+## Why this shape for product work
+
+Flourish AI features follow an 8-step loop: gold dataset → labels → prompt versions → evaluators → experiments with gates → `production` label → live traces and in-app feedback → weak traces back into the dataset.
+
+That loop is the product requirement, not an optional dashboard. Operator steps: [Develop an AI feature](../how-to/ai-feature-development.md). Registration and env: [Observe LLM calls](../how-to/observe-llm-calls.md). Models and routes: [AI reference](../reference/ai.md). Design lock: [implementation plan](../implementationPlans/ai-observability.md).
+
+## RBAC (domain-neutral)
+
+Observability HTTP routes use Terreno RBAC resources (`aiPrompt`, `aiTrace`, `aiReview`,
+`aiDataset`, `aiExperiment`, `aiEvaluator`) instead of domain-specific approval vocabulary.
+Pass the same `accessControl` instance on `ObservabilityApp` and `AdminApp` so
+`GET /ai/observability/status` permissions match `/admin/config` screen filtering. Admin widgets
+hide write controls from status flags (fail closed while status is loading or errored). Seeded
+`admin` receives every observability action; `auditor` receives `list` and
+`read` on each observability resource through the read-only sentinel (but not `admin:access` —
+pair auditor with a shell grant in a composed consumer role); `superadmin` receives `*`. Existing
+customized `admin` and `auditor` roles gain only missing observability actions on re-seed. Legacy
+`user.admin` remains a full-access fallback when RBAC is enabled. See
+[API reference](../reference/api.md#ai-observability-rbac). The example backend registers
+`accessControl: access` on `ObservabilityApp` and seeds optional `aiObservabilityViewer` and
+`aiObservabilityOperator` roles (not auto-assigned) in `example-backend/src/rbacRoles.ts`.
+
+`AIRequest` remains the cheap per-call log. Observability traces are the nested, scored, user/session/cost record used in the SOP.
+
+## Prompt hub relationships (phase 4)
+
+Prompts may carry an optional domain-neutral `description` (folder semantics stay consumer-defined). `GET /ai/observability/prompts/:name` composes bounded recent **traces** and **experiments** for that prompt name so the admin hub does not scan full trace or experiment lists on the client. Trace evidence is version-aware: list filters accept `prompt` plus optional `promptVersion`, and relationship trace rows include the matching `promptVersion` from `ObsTrace.prompts[]`. The same optional positive integer `promptVersion` on prompt detail filters hub traces with `$elemMatch` on `prompts` (name and version). Omit it to keep the unfiltered prompt-name match. Invalid or non-positive versions return **400**. Experiment relationships filter server-side by `promptName` on `GET /ai/observability/experiments?promptName=…` (omit or pass an empty value for no filter). The experiment list endpoint returns summary rows without per-item hydration; use experiment detail for full item results. Reading relationships uses the same `aiPrompt:read` grant as prompt detail.
+
+## Phase 1 reference loop
+
+The example backend always registers the local plugin. Its idempotent seed creates
+`examples/example-summarize` with production on v1 and an experimental v2, the human
+`correctness-human` evaluator, the automatic `schema-assert` evaluator, and a two-item
+`example-gold` dataset. It also creates `chat-safety-synthetic`, a proofread set of
+two-person chats for routing, toxicity, privacy, and dismissive-reply experiments.
+This makes the review and experiment loops walkable without Langfuse:
+resolve the production prompt → emit a trace → inspect spans and sensitive I/O → send the
+trace to Review → record a human score.
+
+The example frontend supplies that first step from real product usage: **Todos → Summarize**
+calls `/ai/example-summarize`, which resolves `example-summarize` by name and `production`
+label server-side. Prompt selection stays on the server, while the client only contributes
+identity (`x-ai-session-id`) and, when the backend has no provider credentials, the user's own
+key.
+
+`AI_OBS_PRICE_MAP_JSON` belongs to deployment configuration because prices change
+independently of prompt versions. A missing model price preserves token counts and omits
+USD cost; it never invents `$0`.
+
+## Phase 2 backend (evaluators, datasets, experiments)
+
+With the local plugin registered and primaries set to `local`, the control plane now includes:
+
+- **`llm-judge` and `json-assert` evaluators** — judges call `AIService.generateJsonObject` through a named registry prompt (`judgePromptName`); create/update rejects a judge when the prompt `outputSchema` omits a required dimension (the error names that key). `json-assert` supports path/constraint checks and a built-in mode that validates output against the prompt version `outputSchema`. Parse or generation failures record an error outcome instead of throwing to the caller. Custom creates score a full trace; generation span and dataset item targets are reserved. Seeded templates install `correctness` / `hallucination` / `helpfulness` / `toxicity` as `llm-judge`, `schema-assert` as `json-assert`, and `*-human` variants for the review queue.
+- **Human review evaluators** — sending traces to review requires a `human` evaluator. Its dimensions become the review item's score fields and its instructions tell the reviewer how to apply them. `llm-judge` and `json-assert` evaluators run automatically and are rejected from the human queue.
+- **Datasets** — CRUD plus item provenance (`origin`, `proofread`, `sourceTraceId`, tags, outcome class). Import accepts **JSON** (bare objects or structured rows) and **CSV** (quoted fields; `input.*` / `expectedOutput.*` column prefixes). Rows validate against the bound prompt input schema when configured. Adding from a trace copies I/O; sensitive traces always land `proofread: false`. Deleting an item never mutates the trace.
+- **Experiments** — compare 2–3 prompt versions on a dataset with optional `modelOverride` (requires `aiServiceFactory` on `ObservabilityApp`), evaluator thresholds (defaulting to the SOP gates), **per-version gate tiles** (`gates[].version`), outlier/low-confidence item ids, and version-scoped promote (**409** when the selected version's gates fail). Unproofread items are excluded unless `includeUnproofread` is true. Local runs compile exact prompt versions and call `AIService.generateText` with compiled `prompt` + `systemPrompt` (no `promptName`/`promptLabel`). `ObservabilityApp` wires the local experiment runner from `aiService` at register time.
+
+Operator UI for datasets and experiments ships in tasks 2.7–2.9; routes and stores are live for API clients and tests today.
+
+## GPT tool spans and nested traces
+
+`/gpt/prompt` streams tool rounds to the client **and** records them in observability traces. Each provider `tool-call` / `tool-result` pair becomes a `TOOL` span (input args, cleaned output, timing, `parentSpanId` on the chain root). When tools run, the exported trace uses a `CHAIN` root wrapping those `TOOL` children instead of a lone `LLM` span. Plain `AIService.generate*` calls without tool children still log a single `LLM` root.
+
+## Admin multi-stage trace smoke test
+
+`POST /ai/observability/traces/test-multi-stage` (admin-only) exercises a real multi-stage workflow through the configured `ObservabilityApp.aiService`: two `generateJsonObject` LLM passes (`skipTrace: true`) with named output schemas, a deterministic local `text-metrics` tool stage, then a final synthesis `generateJsonObject` call against a combining schema. System prompts live in `OBS_TEST_MULTI_STAGE_*` constants in `ai/src/service/prompts.ts` (never inline). Each LLM span records `{prompt, schemaName, outputSchema}` as input and the parsed object as output. The handler exports one parent `TraceRecord` via `ObservabilityApp.exportTrace` with child span order `LLM`, `LLM`, `TOOL`, `LLM` and returns `{traceId, output, stages}` so operators can open the persisted trace in admin. Missing `aiService` → **503**; child failure exports an error trace then propagates the error.