@terreno/mcp 57.20.0 → 57.22.0

package/src/docs/versioned/next/reference/README.md

@@ -8,7 +8,7 @@ Technical reference for Terreno packages and APIs. Information-oriented, precise
 - [Account security](account-security.md) — `AccountSecurityApp`: per-user security summary, recovery actions (2FA reset, passkeys, providers, sessions, password reset, disable, sign-in links), security audit events
 - [@terreno/ui](ui.md) — Components, theming, layout
 - [@terreno/syncdb](syncdb.md) — Local-first data layer (reads, writes, offline sync)
-- [@terreno/ai](ai.md) — AI service, GPT routes, Langfuse integration
+- [@terreno/ai](ai.md) — AI service, GPT routes, observability
 - [@terreno/ai/harness](ai-harness.md) — Durable multi-phase tasks with transactional checkpoints and audit spans
 - [Agent UI Asks](agent-ui-asks.md) — `@terreno/blocks` ask schemas, simple cards, the compact surface, limits, error codes, SSE events, the headless `pendingAsks` and `turn` endpoints, and JSON Schemas for native clients
 - [UI blocks](blocks.md) — whole-reply YAML for leaf, layout, and interactive blocks (stepper, checklist, gallery, list, copy): grammar, limits, and error codes

package/src/docs/versioned/next/reference/admin-backend.md

@@ -113,6 +113,29 @@ When `JobsApp` is already registered, the run is enqueued as job `admin/script`
 `defineAdminScriptJob` on the worker process with the same script list. The CLI
 (`runScriptCli`) does not enqueue.
 
+## MCP screens
+
+`mcpAdminModels(options?)` returns model configs for an MCP server's admin screens, in the `MCP` sidebar group:
+
+| Route | Model | Allowed | Notes |
+| --- | --- | --- | --- |
+| `/mcp-service-tokens` | `McpServiceToken` | list, read, delete | Delete sets `revokedAt` |
+| `/mcp-oauth-clients` | `McpOAuthClient` | list, read, delete | Delete removes the registration and revokes every token issued to it |
+| `/mcp-oauth-connections` | `McpOAuthToken` | list, read, delete | One row per token pair; delete revokes the whole family (connection) and keeps the rows |
+| `/mcp-request-log` | `McpRequestLog` | list, read | Needs `mcpRequestLog` on `TerrenoApp` |
+| `/mcp-tools` | `McpToolSetting` | list, read, update | Only `enabled` is writable. Needs `mcpToolSettings` on `TerrenoApp` |
+
+Options: `group` (default `"MCP"`); `requestLog: true` and `toolSettings: true` add those screens. Both default to `false`; set them to match `TerrenoApp`'s `mcpRequestLog` and `mcpToolSettings`. Hash fields are hidden. Each config sets `adminAccess.resource`; with `accessControl`, spread `mcpAdminStatements` into your `createAccess({statements})`.
+
+`mcpAdminScripts()` adds `mcp-sign-out-everywhere` (`--user <id or email>`), which revokes every OAuth connection and service token the user holds. A dry run reports the counts. Email lookup uses the model registered as `User`.
+
+```typescript
+new AdminApp({
+  models: [...appModels, ...mcpAdminModels({requestLog: true, toolSettings: true})],
+  scripts: [...appScripts, ...mcpAdminScripts()],
+});
+```
+
 ## Config Endpoint
 
 `GET {basePath}/config` returns metadata for all registered models:

package/src/docs/versioned/next/reference/admin-frontend.md

@@ -682,10 +682,34 @@ plugin is on. `AI Requests` (`ai-requests`) stays ungrouped. Widgets also regist
 `ai-prompt-editor`, `ai-trace-detail`, `ai-review-item`, `ai-evaluator-detail`, `ai-evaluator-new`,
 `ai-dataset-detail`, `ai-experiment-new`, and `ai-experiment-results`.
 
-Every observability screen wraps `AiObservabilityChrome`: breadcrumbs
-`Admin / AI Observability / <Section> / <leaf>` and a status chip from
-`GET /ai/observability/status` (`Local on|off` plus active primaries). Review queue nav and the
-review screen body hide when `localOn` is false.
+Every observability screen wraps `AiObservabilityChrome`: a breadcrumb row with a quiet status
+chip from `GET /ai/observability/status` (**Local observability on** / **Local observability
+off**), then one left-aligned page header (title, one-sentence description, optional meta badges,
+and header actions on the right). There is no back arrow. List screens end the breadcrumbs at the
+section (`Admin / AI Observability / <Section>`); detail screens add the leaf
+(`Admin / AI Observability / <Section> / <leaf>`). Review queue nav and the review screen body
+(including header actions) hide when `localOn` is false.
+
+#### Shared patterns
+
+The shell components live in `widgets/aiObservability/shell/`:
+
+| Component | Use |
+| --- | --- |
+| `ObservabilityPageHeader` | The one page header: 32px title, one sentence of purpose, optional `meta`, and `actions` that wrap under the title on phones. `AiObservabilityChrome` renders it from `title`, `description`, `meta`, and `actions` |
+| `ObservabilityStatusBadge` | The one status scale (`observabilityStatus.ts`): pending neutral, in progress / running info, done / completed / ok success, skipped neutral, failed / error error. Unknown statuses fall back to a neutral badge with a readable label |
+| `AgreementMeter`, `ProgressMeter`, `MeterBar` | `AgreementMeter` shows judge-vs-people agreement as percent, bar, and badge from `AGREEMENT_THRESHOLDS`: ≥ 90% **Matches**, 75–89% **Close**, < 75% **Off**, otherwise **No answers yet** (`compact` drops the bar). `ProgressMeter` shows "8 of 10" with a bar. `MeterBar` is the thin bar both use |
+| `ObservabilityEmptyState`, `ObservabilityErrorState` | Empty list: icon, short heading, the next step, and an optional outline action. Load failure: alert banner in place of the content with an outline **Try again** |
+| `EvalStepper` | Numbered stepper: done steps show a check, the current step is filled, later steps are outlined. Used for the four-step eval plan and both New experiment wizards |
+
+Rules:
+
+- One primary action per screen, in the header. Other header actions are outline.
+- Colour means state, never kind. Kinds (evaluator type, prompt folder, experiment kind) use
+  neutral badges or `SegmentedControl` filters.
+- Filters and pickers with a few fixed choices use `SegmentedControl`. Status tabs show counts as
+  badges.
+- Labels are readable words, not API keys (for example **Has a score**, **Agrees with people**).
 
 Each observability list screen is contributed with `adminAccess: {resource, action: "list"}` so
 `/admin/config` omits screens the caller cannot list (`aiPrompt`, `aiTrace`, `aiEvaluator`,
@@ -697,24 +721,26 @@ action). Legacy admin users and apps without `accessControl` receive all actions
 callers receive effective flags for the signed-in user. Pass the same `accessControl` to
 `ObservabilityApp` and `AdminApp` so status permissions match sidebar filtering. Admin widgets
 read the payload through `resolvePromptActionPermissions` and `observabilityActionAllowed` (fail
-closed while status is loading, errored, or missing `permissions`). List screens gate **Create
-prompt** on `aiPrompt:create`, **Create evaluator** on `aiEvaluator:create`, and **New dataset** on
-`aiDataset:create`; experiment lists hide **New experiment** without `aiExperiment:create`.
+closed while status is loading, errored, or missing `permissions`). List screens gate the header
+**New prompt** on `aiPrompt:create`, **New evaluator** on `aiEvaluator:create`, and **New dataset**
+on `aiDataset:create`; experiment lists hide **New experiment** without `aiExperiment:create`.
 Dataset detail hides **Add item** without `aiDataset:update` and **Run experiment** without
 `aiExperiment:create`; dataset lists hide **Import** without
 `aiDataset:update`. Judge schema checks use the production prompt version, or the latest version
 when production is unset. Widgets do not couple to host Redux.
 
-`ai-prompts` lists prompts with a folder rail, search, type badge, latest vs production columns
-(tooltips), 7-day usage, and **Create prompt** when `aiPrompt:create` is allowed. The create modal
-includes an optional **Description** (saved on the prompt record and shown on the hub Overview tab).
-`ai-prompt-editor?name=` is a prompt hub with **Overview**, **Versions**, **Traces**, and
-**Experiments** tabs. Overview shows optional description, folder, tags, and latest vs production
-badges. Traces and Experiments render bounded relationship rows from
+`ai-prompts` lists prompts with a folder `SegmentedControl` (All plus each folder, with counts),
+search, type, Production and Newest version columns, 7-day calls, last used, and a header **New
+prompt** when `aiPrompt:create` is allowed. The create modal
+includes an optional **Description** (saved on the prompt record and shown on the hub Versions screen).
+`ai-prompt-editor?name=` is a prompt hub with **Versions**, **Traces**, and
+**Experiments** tabs. Versions shows optional description, folder, tags, latest vs production
+badges, and the version editor on one screen. System and template fields include helper text.
+Traces and Experiments render bounded relationship rows from
 `GET /ai/observability/prompts/:name?promptVersion=` (the hub pins the selected version, defaulting
 to latest after the first load, so related traces and experiments are filtered server-side while the
-full immutable version list stays on the detail payload). The **Versions** tab
-hosts the versioned editor: full-width version history rows show `vN`, every label attached to
+full immutable version list stays on the detail payload). The **Versions** screen
+hosts the versioned editor under that metadata: full-width version history rows show `vN`, every label attached to
 that version, and its creation time on one line. The selected row is highlighted. Editor /
 Playground tabs appear on Versions when allowed; **Save as vN+1**, **Set vN as production…**, and
 playground **Run once** hide when `aiPrompt:update`, `aiPrompt:promote`, or `aiPrompt:playground`
@@ -729,17 +755,21 @@ trimmed key is available. When `playgroundAi.source` is `server`, no key is requ
 `requestAiServiceFactory`. A stale playground **503** with the missing-key title is remapped to the
 same hint for `request-key` hosts so operators are not told to fix server configuration. The
 example admin supplies the Gemini key saved from Profile. **Save this run to dataset** stays
-disabled until phase 2.
-
-`ai-traces` lists traces with a filter bar: from/to date fields; dropdowns for prompt, status,
-score presence (**All traces / Has a score / No scores**), and data sensitivity
-(**All traces / Sensitive only / Not sensitive**); plus user and session text fields.
+disabled until phase 2. The editor's schema line lists variables and output fields: for a JSON
+Schema `outputSchema` it names each entry in `properties` with its `type` (for example
+`faithful (boolean)`); for a plain field map it lists the keys.
+
+`ai-traces` lists traces in a `DataTable` sorted on the server (`sort` query param): Trace,
+Status, Started, Tokens, Latency, Cost, and Scores are sortable, newest first by default.
+Filters live in the column headers: Trace (prompt), Status (OK / Error), Started (date range), and
+Scores (**Has a score**), plus additional filters for Sensitive, User, and Session. Changing a sort or
+filter returns to page 1 and clears the selection.
 Checkbox selection opens a bulk bar with **Send to human review**, a sensitive-count
 warning, **Clear**, and **Add to dataset** (opens a dataset picker modal; sensitive traces show a
 warning before bulk add). **Send to human review** opens a modal that explains how human
 evaluators define the score fields and reviewer instructions, then requires a human evaluator
-before creating one review-queue item per trace. When local trace storage is on,
-**Run multi-stage trace test** calls the admin-only smoke endpoint and
+before creating one review-queue item per trace. When local trace storage is on, the header
+**Run a test trace** action calls the admin-only smoke endpoint and
 opens the resulting detail: two schema-validated LLM stages, one deterministic tool span, and a
 final schema-validated combining LLM stage under one CHAIN root. LLM span input includes
 `outputSchema`. That run needs AI the same way the playground does: hosts may pass `apiKey`,
@@ -747,28 +777,45 @@ final schema-validated combining LLM stage under one CHAIN root. LLM span input
 `x-ai-api-key`. When `GET /ai/observability/status` reports `playgroundAi.source: "request-key"`
 and no trimmed key is available, the button is disabled and the hint explains where to save one;
 `unavailable` instead reports that the backend must configure `aiService` or
-`requestAiServiceFactory`. The example admin supplies the Gemini key saved from Profile. Rows show a status dot (primary for successful runs, accent for failed runs),
-`sensitive` badge, error line, numeric prompt count, span count, tokens, cost, latency, score count, and
-**Open**. Pagination uses `page` / `limit` / `more` / `total`.
+`requestAiServiceFactory`. The example admin supplies the Gemini key saved from Profile. Each row's
+Trace cell (click to open) shows the name, a **Sensitive** badge, and a subline; Status shows the
+shared status badge and the error line for failed runs; then tokens, latency, cost, and score
+count. Pagination uses `page` / `limit` / `more` / `total`.
 `ai-trace-detail?id=` shows the header, left span list (kind badge, indent, duration bar),
 right span detail with **collapsed** sensitive I/O, and scores (value + source). The two columns
 size from the row width rather than their content, so a wide span value (multi-stage LLM spans embed
 `outputSchema` JSON) wraps inside the detail column instead of pushing it below the span list.
 
-`ai-review` shows Pending / In progress / Done / Skipped tabs with counts. Each tab is
-oldest-first and lists the trace action, prompt, assignee, waiting time, and status.
-**Start reviewing — oldest first** opens the first pending item; the empty state names both
-Traces intake and manual **Assign to me** assignment.
-
-`ai-review-item?id=` shows "Item N of M pending", previous/next navigation, read-only
-**What the AI was given** / **What the AI wrote** panels, collapsed long fields with word
-counts, reviewer notes, and a collapsed Raw JSON disclosure. Score controls come from evaluator
-dimensions (numeric slider, boolean Pass / Fail, categorical pills). Actions are
-**Submit & next**, **Skip**, and **Assign to me**; completion toasts report the remaining count
-or **Queue clear**.
-
-`ai-evaluators` lists evaluators with type badge, dimension summary, target, and run-mode chips.
-**Create evaluator** opens `ai-evaluator-new` with the name and optional purpose first, then how
+`ai-review` shows Pending / In progress / Done / Skipped as a `SegmentedControl` with counts.
+Each tab is oldest-first and lists the output (click to open), question set, prompt, assignee,
+waiting time, and status. Header actions are outline **Review batches** and the primary **Review
+oldest pending**, which opens the first pending item; the empty state points to Traces and review
+batches.
+
+`ai-review-batches` lists batches with answered-by-people progress and the best judge's agreement.
+The search field and **More filters** button sit on the same baseline. Pick a question set and optional
+prompt, then **Save batch** on the New batch card creates
+a batch from the 10 newest successful traces. `ai-review-batch?id=` shows progress, **Answer N remaining**, **Run on N items** for a
+chosen judge, and **Which judge matches people?** with an `AgreementMeter` per judge.
+
+`ai-review-item?id=` is titled "Item N of M" (or "Review item" when not pending), with status and
+source badges under the title and outline **Previous** / **Next** in the header. **What the AI was
+given** and **What the AI wrote** are headings above equal columns; each card holds the field
+labels and values, and the model output card uses the `ai` surface.
+Long fields start collapsed with a word count; reviewer notes and a collapsed Raw JSON disclosure
+stay available. Every question is on one screen: the segmented **Yes** / **No** control (or a
+numeric slider or categorical pills) is a fixed-width column on the left and the question is on
+the right. A numeric or categorical score with no `question` spans the full row under its
+humanized key (for example `tone_rating` reads "Tone rating"). On web, `Y` / `N` answer the highlighted question as soon as the item opens, including
+when a Yes/No segment is focused, `Enter`
+submits, and `S` skips. Actions are **Submit and open next**, **Assign to me**, and **Skip**;
+completion toasts report the remaining count or **Queue clear** (**Batch complete** inside a
+batch). The nested admin stack hides its header (`headerShown: false`); `AdminScreenPage` owns
+the back arrow.
+
+`ai-evaluators` lists evaluators with a kind `SegmentedControl` (All / Human / LLM judge / JSON
+assert), kind, scores, looks at, runs in, and **Agrees with people** (`AgreementMeter`). The header
+**New evaluator** opens `ai-evaluator-new` with the name and optional purpose first, then how
 scoring happens (human / JSON assert / LLM judge). New evaluators score a **full trace** only;
 generation span and dataset item targets are not offered yet. Score fields depend on data type:
 boolean has no extra control, numeric asks for min and max, and categorical is a list you add to.
@@ -779,18 +826,21 @@ model-call impact. The optional purpose is saved as the evaluator description an
 detail screen. LLM judge schema feedback stays idle until a prompt is selected, then shows
 loading/error states and only checks dimensions after its production schema loads.
 `ai-evaluator-detail?id=` leads with the evaluator name, description, and type/target/run-mode
-badges, then explains the saved scores, how scoring works, where it runs, and a **Used by** list
-derived from recent experiments. A failed experiments request shows **Could not load experiments.**
+badges, then explains the saved scores, how scoring works, where it runs, and a **Used in the last 30 days**
+table derived from recent experiments. Each experiment name opens that experiment's results.
+A failed experiments request shows **Could not load experiments.**
 instead of an empty 30-day list. Its dimension and usage rows use the shared `ObservabilityTable` instead of
 `DataTable`, which sizes to a height-constrained parent and collapses inside a scrolling page. LLM
 judge details show loading or load-failed feedback while resolving the judge prompt; schema
 mismatches appear only after its production schema loads.
 
 `ai-datasets` lists datasets with item counts, provenance bar, input-schema binding, and updated
-time. **New dataset** creates a dataset; **Import** on each row accepts `.json` or `.csv` via
+time. When any dataset has human answers, a **People's answers** column summarizes each dataset's
+`judgments`: **N answers out of date** when questions were reworded since they were answered,
+**All answered** when every question set answered every item, otherwise **N of M answered**. The header **New dataset** creates a dataset; **Import** on each row accepts `.json` or `.csv` via
 `FilePickerButton` (local URI read) or paste, posting `{rows}` for JSON or `{format:'csv',content}`
 for CSV. `ai-dataset-detail?id=` shows counts, schema binding, tabs **All / Human / Auto / Needs
-review**. Human is proofread items. Auto is unreviewed trace or synthetic items, so a proofread
+review** (a `SegmentedControl`). Human is proofread items. Auto is unreviewed trace or synthetic items, so a proofread
 trace is not listed in both tabs. The screen includes an items table (input, expected, provenance,
 trace link), **Add item**, and **Run experiment**. The items table uses `ObservabilityTable`: rows grow with their content
 and cells wrap to three lines before truncating, so long inputs and expected outputs no longer
@@ -808,13 +858,26 @@ cell renders truncated text and a node cell renders as-is. Its
 bordered shell shrink-wraps the rows inside flex/scroll parents so its bottom border cannot stretch
 into the following section.
 
-`ai-experiments` lists experiments with status, running progress, and cost. **New experiment**
-opens a four-step wizard (dataset with counts, prompt versions tagged latest/production/superseded,
-llm-judge and json-assert evaluators, review & run with estimate). A slower in-flight estimate is ignored after the selection changes, so it cannot replace a newer estimate or its error. Human evaluators are omitted. On the prompt-versions step, **Next** and the later
-wizard rail buttons stay disabled until 2–3 versions are selected. `includeUnproofread` and optional model override are on
-the wizard. `ai-experiment-results?id=` polls while pending/running, shows gate tiles per version,
-failing gate count, outliers, a side-by-side per-item output table (failed rows first from the
-API), and **Promote to production** with a confirm modal; promote is blocked when gates fail (409).
+`ai-experiments` lists experiments with a kind `SegmentedControl` (All / Judge calibration /
+Prompt comparison), status badge, running progress (`ProgressMeter`), result, and cost. The screen
+loads the newest 100 experiments (the route's maximum page); sorting and filters apply to those,
+and when the server reports more, a note under the table (`ai-experiments-truncated-note`) says so. The header
+**New experiment** opens `ai-experiment-new`, where a `SegmentedControl` picks **Calibrate a
+judge** or **Compare prompt versions**. Each pipeline is a four-step `EvalStepper` wizard with
+ghost **Back** and primary **Continue** (the last step shows the run button). Compare prompt
+versions: Dataset (with counts), Prompt versions (tagged latest/production/superseded),
+Evaluators (llm-judge and json-assert), Review and run (with estimate). Calibrate a judge:
+Judged dataset, Question set, Judge and versions, Review and run. A slower in-flight estimate is
+ignored after the selection changes, so it cannot replace a newer estimate or its error. Human
+evaluators are omitted. On the prompt-versions step, **Continue** and later stepper steps stay
+disabled until 2–3 versions are selected. `includeUnproofread` and optional model override are on
+the wizard. `ai-experiment-results?id=` polls while pending/running (`ProgressMeter`), shows one
+verdict banner, one **Gates** table (Gate, Needs, one Pass / Fail column per version), outliers, a
+side-by-side per-item output table (failed rows first from the API), and header actions outline
+**Run again** and primary **Promote vN to production** with a confirm modal; promote is blocked
+when gates fail (409). The promoted version defaults to the newest version that passes every gate
+(the newest version while gates are pending) and follows the gates as they arrive, until the
+operator picks one in **Version to promote**, which appears once gates exist and a version passes.
 
 ### AI Harness approvals inbox
 

package/src/docs/versioned/next/reference/admin-spa.md

@@ -139,6 +139,15 @@ new TerrenoApp({userModel: User})
 
 If no bundle exists at `distDir/index.html`, the plugin logs a warning suggesting `bun run --filter '@terreno/admin-spa' build:web` or `devProxyTarget`.
 
+## MCP OAuth consent route
+
+`${basePath}/oauth/consent?request=<id>` is the consent screen for `@terreno/api` MCP OAuth. Set `mcpOAuth.consentUrl` to `${issuer}${basePath}/oauth/consent` on a backend that serves the SPA from the same origin.
+
+- Any signed-in user can open it, not only admins. Admin providers (sync client, org context) do not start on this route.
+- A signed-out user is sent to `${basePath}/login?returnTo=<path>` and returned after sign-in. `returnTo` must start with a single `/` and contain no `\`, scheme, whitespace, or control characters; anything else is ignored.
+- The screen calls `GET /oauth/authorize/:request` and `POST /oauth/authorize/:request/approve` or `/deny` with the Better Auth session cookie, then follows `data.redirectUrl`.
+- States: loading, missing `request`, expired (400), already decided (409), not found, signed out, and error. Test ids start with `admin-spa-oauth-consent-`.
+
 ## Mount path constraint
 
 The default web export is built with `experiments.baseUrl: "/console"`, so `basePath` must match. To mount elsewhere, rebuild with a matching base (e.g. `EXPO_BASE_URL=/admin-ui bun run build:web`) and set `basePath` accordingly. Mounting at `/` requires a root-base build and `basePath: "/"`.

package/src/docs/versioned/next/reference/ai.md

@@ -1,6 +1,6 @@
 # @terreno/ai
 
-AI service layer for Terreno backends: provider-agnostic chat via the Vercel AI SDK, request logging, GPT history, projects, file uploads, MCP tools, and optional Langfuse integration.
+AI service layer for Terreno backends: provider-agnostic chat via the Vercel AI SDK, request logging, GPT history, projects, file uploads, MCP tools, and built-in observability (prompt registry, traces, evaluators).
 
 > `@terreno/ai` is provider-agnostic via the Vercel AI SDK (the `ai` package and its `@ai-sdk/*` providers). That is a model library, unrelated to Vercel hosting; on this page "Vercel" means the AI SDK unless it says hosting. To host on Vercel, see [Deploy to Vercel](../how-to/deploy-to-vercel.md).
 
@@ -17,10 +17,8 @@ AI service layer for Terreno backends: provider-agnostic chat via the Vercel AI
 - [Route registrars](#route-registrars)
 - [AiApp plugin](#aiapp-plugin)
 - [Cost and abuse limits](#cost-and-abuse-limits)
-- [LangfuseApp plugin](#langfuseapp-plugin)
 - [Observability](#observability)
 - [Durable agent harness](#durable-agent-harness)
-- [Langfuse integration](#langfuse-integration)
 - [FileStorageService](#filestorageservice)
 - [MCPService](#mcpservice)
 - [Gemini and Vertex helpers](#gemini-and-vertex-helpers)
@@ -56,12 +54,6 @@ bun run lint:fix   # Fix lint issues
 src/
   index.ts                 # Public exports
   aiApp.ts                 # AiApp TerrenoPlugin
-  langfuseApp.ts           # LangfuseApp TerrenoPlugin
-  langfuseClient.ts        # Langfuse SDK client lifecycle
-  langfuseCache.ts         # Prompt/trace caching
-  langfusePrompts.ts       # Prompt compile/fetch helpers
-  langfuseTracing.ts       # OpenTelemetry tracing setup
-  langfuseVercelAi.ts      # Bridge Langfuse prompts to Vercel AI SDK
   models/
     aiRequest.ts           # AI request logging model
     gptHistory.ts          # Conversation history model
@@ -91,7 +83,7 @@ src/
 
 ## Key exports
 
-- **Plugins:** `AiApp`, `LangfuseApp`
+- **Plugins:** `AiApp`, `ObservabilityApp`
 - **Service:** `AIService`, `TemperaturePresets`, `FileStorageService`, `MCPService`,
   `getMCPTools`, `runBufferedChatTurn` (one chat turn to completion without HTTP: the same system
   prompt, tools, and block checks as `/gpt/histories/:id/turn`; takes `{body: {historyId, prompt},
@@ -99,7 +91,6 @@ src/
 - **Models:** `AIRequest`, `GptHistory`, `FileAttachment`, `Project` (lazy; prefer `getProjectModel()`)
 - **Routes:** `addGptRoutes`, `addGptHistoryRoutes`, `addAiRequestsExplorerRoutes`, `addFileRoutes`, `addProjectRoutes`, `addMcpRoutes`
 - **Structured output:** `parseAiJson`, `normalizeLlmJsonTextForStructuredOutput`, re-exported `Output`, `jsonSchema`, `JSONValue`, `FlexibleSchema` from `ai`
-- **Langfuse:** `initLangfuseClient`, `getLangfuseClient`, `shutdownLangfuseClient`, `compilePrompt`, `createPrompt`, `getPrompt`, `createTelemetryConfig`, `preparePromptForAI`, `initTracing`, `shutdownTracing`, `LangfuseCache`, cache helpers
 - **Gemini / Vertex:** `listGeminiApiModels`, `normalizeGeminiModelId`, `GEMINI_API_BASE_URL`, `createVertexProvider`, `listEnabledVertexModels`, `verifyVertexModelsEnabled`, `assertVertexModelsEnabled`, `isVertexModelAllowed`, `normalizeVertexModelId`, `DEFAULT_VERTEX_LOCATION`
 - **Prompts:** `COMPACT_SURFACE_SYSTEM_PROMPT`, `CONTENT_SUMMARY_PROMPT`, `DEFAULT_GPT_MEMORY`, `JSON_VALUE_SYSTEM_PROMPT`, `REMIX_PROMPT`, `TERRENO_ASKS_SYSTEM_PROMPT`, `TITLE_GENERATION_PROMPT`, `TRANSLATION_PROMPT`
 - **Block host actions:** `scaleStepperHostAction`, `toggleChecklistHostAction`, `findAgentBlock`, types `HostAction`, `HostActionContext`, `HostActionResult` ([Host actions](#host-actions))
@@ -350,7 +341,7 @@ With `asks` on, pass the same options to `addGptHistoryRoutes` as `chat` to add
 | `historyId` | string? | Continue this conversation; omit to start one. Required with `askResponse`. |
 | `askResponse` | object? | The answer to the pending ask: `{toolCallId, action, content?, reason?}`. Read only when `asks` is on. Send it instead of `prompt`, without `attachments`. See [Answer an ask](agent-ui-asks.md#answer-an-ask). |
 | `surface` | `"full"` \| `"compact"`? | Where the user reads and answers. Default `"full"`. `"compact"` is a watch or another small screen: the model gets only button-sized asks and is asked for replies of at most two short sentences. Any other value returns 400. See [Compact surface](agent-ui-asks.md#compact-surface). |
-| `systemPrompt` | string? | System prompt for this turn; project context and the Langfuse prompt are prepended |
+| `systemPrompt` | string? | System prompt for this turn; project context and the registry `systemPromptName` prompt are prepended |
 | `attachments` | array? | `{type: "image" \| "file", url, mimeType, filename?}` items added to the user message |
 | `model` | string? | Model id passed to `createModelFn` or `createServerModelFn` |
 | `projectId` | string? | Project whose context and memories are prepended to the system prompt; saved on a history that has none |
@@ -495,6 +486,8 @@ new AiApp({
 | `toolChoice` | `"auto"` \| `"none"` \| `"required"` (default `"auto"` when tools present) |
 | `maxSteps` | Max tool-calling steps (default 5) |
 | `titleModelId` | Cheaper model for conversation title generation |
+| `systemPromptName` | Observability prompt registry name used as the default system prompt. Its body is prepended to the request's system prompt, and the trace records the version. Skipped when no `ObservabilityApp` is registered, the label has no version, or an admin request selects `promptName` |
+| `systemPromptLabel` | Registry label read for `systemPromptName` (default `production`) |
 | `openApiOptions` | Passed to route OpenAPI builders |
 | `projects` | Default `true`. `false` skips the `/gpt/projects` routes, so `AiApp` never registers the `Project` model |
 
@@ -537,40 +530,6 @@ Before exposing AI routes to users, do both:
    budget alert and Vertex AI or Gemini API quotas), so a runaway caller or bug is caught by the
    provider even when the app's own limits are loose.
 
-## LangfuseApp plugin
-
-Optional Langfuse admin UI and tracing:
-
-```typescript
-import {LangfuseApp} from "@terreno/ai";
-
-new LangfuseApp({
-  publicKey: process.env.LANGFUSE_PUBLIC_KEY!,
-  secretKey: process.env.LANGFUSE_SECRET_KEY!,
-  baseUrl: process.env.LANGFUSE_BASE_URL,
-  adminPath: "/admin/langfuse",
-  enableTracing: true,
-  enableAdminUI: true,
-  evaluation: {enabled: true, scoringFunctions: [...]},
-}).register(app);
-```
-
-| Option | Default | Description |
-|--------|---------|-------------|
-| `publicKey`, `secretKey` | — | Langfuse API keys (required) |
-| `baseUrl` | — | Langfuse host |
-| `adminPath` | `"/admin/langfuse"` | Admin route prefix |
-| `organization` | maintainer default | Langfuse organization slug — set your own |
-| `project` | `"terreno"` | Langfuse project slug — set your own |
-| `enableTracing` | `true` | OpenTelemetry via `@langfuse/otel` |
-| `enableAdminUI` | `true` | Prompt, trace, playground, evaluation routes |
-| `evaluation.enabled` | — | Register evaluation scoring routes |
-| `cache` | — | Prompt/trace TTL overrides |
-
-Client construction failures log a warning and skip the plugin so the API process still listens. Tracing init failures log and leave admin routes mounted. example-backend does not register `LangfuseApp` when `MONGO_DB_NAME` is a PR preview database (`terreno-example-pr-*`). GitHub Actions preview deploys omit Langfuse secrets and use `secrets_update_strategy: overwrite` so a previous revision cannot merge those keys into the preview.
-
-Calls `shutdownLangfuseClient()` and `shutdownTracing()` on `SIGTERM`.
-
 ## Observability
 
 In-app prompt versions, nested traces, evaluators, datasets, experiments, review queue, and in-app feedback. Operator loop: [Develop an AI feature](../how-to/ai-feature-development.md). Register plugins: [Observe LLM calls](../how-to/observe-llm-calls.md). Why two planes: [AI observability](../explanation/ai-observability.md). Locked design: [implementation plan](../implementationPlans/ai-observability.md).
@@ -608,14 +567,20 @@ server `AIService` when configured, otherwise a request-scoped service built fro
 | `ObsSpan` | Nested span with `kind`, `status`, optional `error`, offsets, usage |
 | `ObsScore` | Scores on a trace/span; many per trace, **no unique index** |
 | `ObsEvaluator` | Evaluator: `type` (`human` \| `llm-judge` \| `json-assert`), `target`, `dimensions[]`, `runModes`, `instructions`, `judgePromptName` (judge), `assertion` (json-assert), `confidenceAlertBelow` (default 0.7) |
-| `ObsReviewItem` | Review queue item: status, evaluator, trace, reason, scores, comment |
+| `ObsReviewItem` | Review queue item: status, evaluator, reason, scores, comment, optional `batchId` and `datasetItemId`. `traceId` is optional: unset when reviewers judge a dataset item that has no source trace |
+| `ObsReviewBatch` | Review batch: `name`, human question-set `evaluatorId`, `traceIds[]`, optional `promptName` and `description`; dataset batches add `datasetId` and `questionsFingerprint` |
+| `ObsJudgeRun` | One LLM judge's answers for every item in a batch: `batchId`, `evaluatorId`, `modelId`, `results[]` (`traceId?`, `datasetItemId?`, `scores`, `confidence`, `error`) |
 | `ObsDataset` | Named dataset with optional `inputSchemaPromptName` and `expectedOutputSchema` |
-| `ObsDatasetItem` | Item with `input`, `expectedOutput`, `origin`, `proofread`, `tags`, `outcomeClass`, `sourceTraceId`, `metadata` |
-| `ObsExperiment` | Compares 2–3 prompt versions on a dataset with thresholds and aggregates |
-| `ObsExperimentItem` | Per dataset row: outputs per version, evaluator score maps, gate failure flags |
+| `ObsDatasetItem` | Item with `input`, `expectedOutput`, `origin`, `proofread`, `tags`, `outcomeClass`, `sourceTraceId`, `metadata`, and `humanAnswers` (keyed by question-set evaluator id: `{fingerprint, scores, reviewItemId, userId?, answeredAt}`) |
+| `ObsExperiment` | `kind: "prompt"` (default) compares 2–3 prompt versions with thresholds; `kind: "judge"` compares 2–3 judge prompt versions by agreement with human answers (`questionSetId`, `judgeEvaluatorId`, `targetAgreement`, `questionsFingerprint`) |
+| `ObsExperimentItem` | Per dataset row: outputs per version, evaluator score maps, gate failure flags; judge experiments add `humanScores` |
 
 `POST /ai/observability/traces/review` requires a `human` `ObsEvaluator`. Its
 `dimensions[]` render as reviewer score fields and `instructions` render above the review form.
+Each dimension may set `question`, which the review screen shows instead of the key. Yes/no
+questions render as a segmented **Yes** / **No** control in a fixed left column, with the
+question on the right. On web, `Y` / `N` answer the highlighted question as soon as the item
+opens, `Enter` submits, and `S` skips.
 Automatic evaluator types return **400** instead of entering the human queue. Submitting a review
 requires every dimension marked `required`; omitted optional dimensions do not create empty score
 rows. The local score store remains the fallback when no external score sink is configured.
@@ -641,22 +606,24 @@ When `prompts.primary` is `local`, `ObservabilityApp.register` mounts admin-only
 
 | Method | Path | Behavior |
 | --- | --- | --- |
-| GET | `/ai/observability/status` | Admin chrome. `{plugins, primaries, localOn, playgroundAi}` — drives the status chip, hides Review when `localOn` is false, and tells the prompt playground whether AI comes from `ObservabilityApp.aiService` (`server`), per-request `x-ai-api-key` via `requestAiServiceFactory` (`request-key`), or is not configured (`unavailable`) |
+| GET | `/ai/observability/status` | Admin chrome. `{plugins, primaries, localOn, playgroundAi}` — drives the status chip (**Local observability on** / **off**), hides Review when `localOn` is false, and tells the prompt playground whether AI comes from `ObservabilityApp.aiService` (`server`), per-request `x-ai-api-key` via `requestAiServiceFactory` (`request-key`), or is not configured (`unavailable`) |
 
 | Method | Path | Behavior |
 | --- | --- | --- |
 | GET | `/ai/observability/prompts` | List. Query `folder`, `search`, `include=usage7d` (7-day calls/cost). `production` is `"—"` until a production label exists |
 | POST | `/ai/observability/prompts` | Create prompt in a folder as immutable v1 (`latest` label) |
-| GET | `/ai/observability/prompts/:name` | Prompt + versions + labels |
+| GET | `/ai/observability/prompts/:name` | Prompt + versions + labels. Each version includes `inputSchema`, `outputSchema`, and `outputFieldNotes` when set, so judge schema checks can see required dimensions such as `helpfulness` |
 | POST | `/ai/observability/prompts/:name/versions` | Create `vN+1`; never mutates an existing version |
 | POST | `/ai/observability/prompts/:name/labels` | Move `production` or `staging`; `outgoingVersion` is the previous pointer |
 | POST | `/ai/observability/prompts/:name/playground` | Compile `{{var}}` + one `AIService` call; returns compiled messages, output, latency, tokens, cost; creates no version. Uses `ObservabilityApp.aiService`, or `requestAiServiceFactory({apiKey, modelId})` when the server service is absent (`apiKey` comes from `x-ai-api-key`) |
 
 `PromptRegistry.get({name, label})` (default label `production`) reads the labelled local version. `createLocalObservabilityPlugin()` wires `LocalPromptStore` as that registry and local `TraceSink` / `ScoreSink`.
 
+Prompt references: a `system` or `template` may contain `{{> <name>}}`, with an optional `label=<label>` (default `production`) or `version=<n>`, not both: `{{> base-rules}}`, `{{> base-rules label=staging}}`, `{{> base-rules version=3}}`. `PromptRegistry.get` returns `body` with every reference resolved (nested up to 5 levels) and `includes`: one `{includedBy, label?, name, version}` per referenced version. The trace records `prompts: [<called prompt>, ...includes]`; harness prompt refs record `includes` the same way. Every reference must resolve when it is saved: create, version, and label routes return 400 `Invalid prompt reference` for a malformed tag, an unknown prompt or version, or a label the referenced prompt does not have yet, 400 `Prompt reference cycle` when the save or label move would close a cycle (a new version counts as `latest`), and 400 `Prompt references nested too deeply` when it would nest any prompt that includes it past 5 levels. Prompt detail returns the stored, unresolved text, plus `usedBy`: `{items, limit, total}` for other prompts whose saved `system` or `template` contains `{{> this name}}`. Each item has `folder`, `name`, and the `versions` that include the tag. `items` is capped at 20, the same bound as trace and experiment relationships. The prompt editor's **Insert prompt reference** button opens a modal that lists other prompts and appends `{{> name}}` (the production label) to the template, or to the system prompt on a chat prompt. See [Share instructions across prompts](../how-to/observe-llm-calls.md#share-instructions-across-prompts).
+
 | Method | Path | Behavior |
 | --- | --- | --- |
-| GET | `/ai/observability/traces` | Admin list. Query `from`, `to`, `prompt`, `status`, `userId`, `sessionId`, `scope`, `hasScore`, `sensitive`, `flaggedForDataset`, `page`, `limit`. Body is `{data, page, limit, more, total}` so pagination survives RTK `{data}` unwrap. Each row includes `spanCount` and `scoreCount`. `prompts.length` is the `N prompts` count |
+| GET | `/ai/observability/traces` | Admin list. Query `from`, `to`, `prompt` (one name, or a list such as `prompt[0]=a&prompt[1]=b` to match any; `promptVersion` needs exactly one), `status`, `userId`, `sessionId`, `scope`, `hasScore`, `sensitive`, `flaggedForDataset`, `sort`, `page`, `limit`. `sort` is one of `startedAt`, `name`, `status`, `tokens`, `cost`, `latency`, `spans`, `scores`, with a leading `-` for descending; default `-startedAt` (newest first). Sorting runs in a Mongo aggregation before skip/limit, so it holds across pages: latency is `endedAt - startedAt`, tokens is `usage.inputTokens + usage.outputTokens`, cost is `usage.costUsd`, and traces missing those values list last in both directions; spans and scores sort by row counts. An unknown key is **400**. Body is `{data, page, limit, more, total}` so pagination survives RTK `{data}` unwrap. Each row includes `spanCount` and `scoreCount`. `prompts.length` is the `N prompts` count |
 | GET | `/ai/observability/traces/:id` | Span tree (kind, offsets, durations, I/O, cost) plus scores. `errorSummary` is the first span with `status: "error"` |
 | POST | `/ai/observability/traces/:id/scores` | Persist a score and fan out to every `ScoreSink` |
 | POST | `/ai/observability/traces/test-multi-stage` | Admin-only smoke workflow, registered only with the local trace sink. Uses `ObservabilityApp.aiService`, or `requestAiServiceFactory({apiKey})` when the server service is absent (`apiKey` comes from `x-ai-api-key`); answers **503** with the same missing-key title as playground when neither exists. Body `{input?: string}` (defaults to a built-in sample). Runs two `AIService.generateJsonObject` calls with `skipTrace: true` and named JSON output schemas (`obs-test-multi-stage-call-1` / `call-2`), a deterministic local `text-metrics` `TOOL` stage, then a final `generateJsonObject` synthesis against `obs-test-multi-stage-final`; exports exactly one parent trace with ordered child spans `LLM`, `LLM`, `TOOL`, `LLM` under a `CHAIN` root. LLM span input includes `outputSchema`. Returns `{traceId, output, stages[]}` where `output` is the final schema object (`sentence`, `phrase`, `keywords`, `metrics`). Child LLM failures export an error trace then rethrow |
@@ -672,7 +639,28 @@ When `prompts.primary` is `local`, `ObservabilityApp.register` mounts admin-only
 | POST | `/ai/observability/traces/review` | Enqueue one or many traces against a human evaluator (`reason: "manual"`) |
 | GET | `/ai/observability/review` | Queue by `status` with counts; oldest-first. Response includes `more: false` so RTK preserves the count envelope. Rows include `traceName`, `promptName`, assignee, reason, and enqueue time |
 | GET | `/ai/observability/review/:id` | Item + evaluator dimensions + `given` / `wrote` panels and `rawInput` / `rawOutput` for the Raw JSON disclosure |
-| POST | `/ai/observability/review/:id` | `submit` (scores via ScoreSinks, status `done`), `skip`, or `assign` |
+| POST | `/ai/observability/review/:id` | `submit` (scores via ScoreSinks when the item has a trace, status `done`), `skip`, or `assign`. Submitting an item with `datasetItemId` writes `humanAnswers[evaluatorId]` on the dataset item (replacing an earlier answer) with the question set's current fingerprint and the reviewer's id. `skip` writes nothing |
+| GET | `/ai/observability/review/batches` | Batches with `counts` (`done`, `pending`, `skipped`, `total`), `judgeCount`, `bestJudge`, and for dataset batches `datasetId`, `datasetName`, `questionsFingerprint`, `questionsStale` (`aiReview:list`) |
+| POST | `/ai/observability/review/batches` | Create a batch for a human evaluator from `traceIds`, or from the newest `limit` (default 10, max 50) successful traces with output not already batched for that evaluator, optionally filtered by `promptName`. Enqueues one review item per trace. 404 `Unknown trace ids` when any id is missing; a failed enqueue removes the batch so its traces stay available. **From a dataset:** `{evaluatorId, datasetId, name?, description?}` (400 with `traceIds` or `limit`) enqueues one item per dataset item without a current answer from that question set and not already pending in another batch, with `datasetItemId` and `traceId = sourceTraceId`. 400 `Every item already has current answers`, or `Every unanswered item is already waiting in an open review batch` (`aiReview:assign`) |
+| GET | `/ai/observability/review/batches/:id` | Batch detail: question `dimensions`, `items[]` (input, output, status, human `scores`, `judgeAnswers` by judge id, `datasetItemId?`, `traceId?`), and `judges[]` with `agreement` (`rate`, `matches`, `compared`, `itemsCompared`, `errors`, per-question `questions[]`, `disagreements[]`) sorted best first (`aiReview:read`) |
+| POST | `/ai/observability/review/batches/:id/judgeRuns` | Run an `llm-judge` evaluator over every batch item (five at a time) and store an `ObsJudgeRun`. 400 when the judge shares no dimension keys with the question set; 503 without an AI client (`aiReview:assign`) |
+
+Reviewers and judges see the same thing for a dataset item: its source trace's input and output
+when the trace still exists, otherwise the item's `input` and `expectedOutput` (shown as
+"Dataset item"; `traceId` is unset and no `ObsScore` rows are written).
+
+Judge agreement compares only items with `done` human answers and only dimension keys the judge
+also returns. Booleans and categories must match exactly (case-insensitive for categories);
+numbers match within 10% of the dimension's range (default 0–1). A judge error or missing answer
+counts as a disagreement. Answers pair by `datasetItemId` when set, otherwise by `traceId`;
+`disagreements[]` carry both. `computeJudgeAgreement` and `answersAgree` are exported for custom
+reports.
+
+**Question fingerprint.** `questionSetFingerprint(dimensions)` (exported) is a sha256 over each
+dimension's `key`, `dataType`, `question`, and `range`, sorted by key (`required` and order do not
+count). A human answer is *current* while its fingerprint matches the question set now and
+*stale* after any of those fields change. `loadCurrentHumanAnswers({datasetId, questionSetId})`
+returns the dataset items with current answers. `GET /ai/observability/review` also accepts `batchId` to scope the queue to one batch.
 
 `createLocalObservabilityPlugin()` wires `LocalDatasetStore` and `LocalExperimentRunner` when datasets/experiments primaries are `local`.
 
@@ -680,17 +668,49 @@ When `prompts.primary` is `local`, `ObservabilityApp.register` mounts admin-only
 | --- | --- | --- |
 | GET/POST | `/ai/observability/datasets` | List (includes `humanCount` for proofread items, `autoCount` for unreviewed trace or synthetic items, `needsReviewCount`) / create |
 | GET/PATCH/DELETE | `/ai/observability/datasets/:id` | Detail (with counts) / update / soft-delete. PATCH `null` clears optional dataset fields; omitted fields stay unchanged |
-| GET/POST | `/ai/observability/datasets/:id/items` | List / create items |
+| GET/POST | `/ai/observability/datasets/:id/items` | List / create items. Items include `humanAnswers` keyed by question-set id (`answeredAt` as ISO) |
+
+Dataset list and detail include `judgments[]`: one row per question set that answered any item or
+has a review batch built from the dataset, `{evaluatorId, evaluatorName, answered, current,
+stale, total}`. `current` answers match the question set's fingerprint now; `stale` answers were
+given to an older version of the questions; `total` is the dataset's item count.
 | PATCH/DELETE | `/ai/observability/datasets/:id/items/:itemId` | Update labels (`expectedOutput`, `proofread`, `tags`, `outcomeClass`) / delete (does not touch the source trace). PATCH `null` clears optional item fields |
 | POST | `/ai/observability/datasets/:id/import` | **JSON:** body is an array of bare input objects, or structured rows with `input` / `expectedOutput` / `proofread` / `tags` / `outcomeClass` / `metadata`. **CSV:** `Content-Type: text/csv` with raw CSV body, or JSON `{format: "csv", content: "..."}`. Plain columns map to `input`; plain `input` / `expectedOutput` cells accept JSON values; `input.foo` and `expectedOutput.foo` nest fields; reserved `proofread`, `tags`, `outcomeClass` map metadata. Nested paths reject `__proto__`, `constructor`, and `prototype` segments. Rows validate against the dataset's bound prompt `inputSchema` when `inputSchemaPromptName` is set; 400 reports row number and JSON path |
 | POST | `/ai/observability/traces/add-to-dataset` | `{datasetId, traceId \| traceIds[]}`. Copies span I/O; `origin: "trace"`; `sourceTraceId` set; **sensitive traces always `proofread: false`** |
 
 | Method | Path | Behavior |
 | --- | --- | --- |
-| POST | `/ai/observability/experiments/estimate` | `{datasetId, promptName, versions[], evaluatorIds[], modelOverride?}` → generation count, USD, wall-clock estimate |
-| GET/POST | `/ai/observability/experiments` | List / create. Body: dataset, 2–3 version numbers, evaluator ids, optional `thresholds[]` (defaults to `SOP_DEFAULT_THRESHOLDS`), `modelOverride`, `includeUnproofread` (default false). Evaluator ids must be `llm-judge` or `json-assert` (human → 400). Local primary always enqueues `BackgroundTask` (even one item) |
-| GET | `/ai/observability/experiments/:id` | Status, progress, per-version aggregates, gate pass/fail (`gates[].version`), `outlierItemIds`, `lowConfidenceItemIds`, per-item side-by-side (**failed rows first**) |
-| POST | `/ai/observability/experiments/:id/promote` | `{version}` moves the `production` label when **that version's** gates pass; **409** while any gate for the selected version fails |
+| POST | `/ai/observability/experiments/estimate` | `{datasetId, promptName, versions[], evaluatorIds[], modelOverride?}` → generation count, USD, wall-clock estimate. Judge experiments: `{kind: "judge", datasetId, questionSetId, versions[]}` counts items with current human answers |
+| GET/POST | `/ai/observability/experiments` | List (query `promptName`, `kind` = `prompt` \| `judge`, `page`, `limit`) / create. Prompt body: dataset, 2–3 version numbers, evaluator ids, optional `thresholds[]` (defaults to `SOP_DEFAULT_THRESHOLDS`; `aggregate: "agreement"` → 400), `modelOverride`, `includeUnproofread` (default false). Evaluator ids must be `llm-judge` or `json-assert` (human → 400). Judge body: see below. Local primary always enqueues `BackgroundTask` (even one item) |
+| GET | `/ai/observability/experiments/:id` | Status, progress, per-version aggregates, gate pass/fail (`gates[].version`), `outlierItemIds`, `lowConfidenceItemIds`, per-item side-by-side (**failed rows first**). Every view has `kind`, `datasetName`; judge views add `questionSetId`, `questionSetName`, `judgeEvaluatorId`, `judgeName`, `targetAgreement`, `questionsFingerprint`, `questionsStale`, `results.agreement`, and per-item `humanScores` |
+| POST | `/ai/observability/experiments/:id/promote` | `{version}` moves the `production` label of the experiment's `promptName` (the judge prompt for judge experiments) when **that version's** gates pass; **409** while any gate for the selected version fails |
+
+**Judge calibration experiments.** `POST /ai/observability/experiments` with
+`{kind: "judge", name, datasetId, questionSetId, judgeEvaluatorId, versions, targetAgreement?, modelOverride?}`
+runs 2–3 versions of the judge evaluator's `judgePromptName` (stored as `promptName`) over every
+dataset item with a current human answer from `questionSetId`. Each version runs with its own
+resolved `system` (or `template` when `system` is empty) as the system prompt, its own
+`outputSchema`, and the judge's usual user prompt, on what reviewers saw (source trace, else
+item input / expected output). 400 titles: `questionSetId must be a human question set`,
+`judgeEvaluatorId must be an llm-judge evaluator with a judge prompt`, `Judge answers none of the
+question set's questions`, `Dataset has no current human answers for this question set`,
+`Unknown judge prompt version`, `Judge prompt version has no output schema`, and
+`judge experiments must compare 2–3 judge prompt versions`. Results:
+
+- `items[].versionResults[v].evaluatorScores[judgeEvaluatorId]` holds each version's answers;
+  `items[].humanScores` the reviewers' answers. A row is `failed` when any version errors or
+  disagrees on a compared question.
+- `results.agreement[v]` is a `JudgeAgreement` (overall `rate`, per-question `questions[]`,
+  `disagreements[]` with `datasetItemId`).
+- `results.gates` has one gate per version:
+  `{evaluatorName: <judge>, dimension: "agreement", aggregate: "agreement", op: "gte", value: targetAgreement (default 0.9), actual, passed, version}`.
+
+**Judge calibration on evaluators.** `GET /ai/observability/evaluators` and `/:id` add
+`calibration` to `llm-judge` evaluators that have a completed judge experiment:
+`{experimentId, experimentName, datasetName, questionSetName, version, rate, passed, stale, targetAgreement}`
+from the latest one. `version` is the judge prompt's production version when the experiment
+compared it, otherwise the best-agreeing version. `stale` is true when the question set's
+fingerprint changed since the run.
 
 Authenticated `POST /ai/observability/traces/:id/feedback` records thumbs, outcome class, and flag-for-dataset (phase 2.6).
 
@@ -723,17 +743,6 @@ Why: [Durable agent harness](../explanation/durable-agent-harness.md).
 How-to: [Build a durable workflow](../how-to/build-a-durable-workflow.md),
 [Ship a new task version](../how-to/ship-a-new-task-version.md).
 
-## Langfuse integration
-
-Low-level exports (also used by `addGptRoutes` when `langfuseSystemPromptName` is set):
-
-- **Client:** `initLangfuseClient`, `getLangfuseClient`, `isLangfuseInitialized`, `shutdownLangfuseClient`
-- **Prompts:** `getPrompt`, `createPrompt`, `compilePrompt`, `invalidatePromptCache`, `preparePromptForAI`
-- **Tracing:** `initTracing`, `shutdownTracing`, `createTelemetryConfig`
-- **Cache:** `LangfuseCache`, `getCached`, `setCached`, `invalidateCache`
-
-Subpath imports for tree-shaking: `@terreno/ai/langfuseClient`, `@terreno/ai/langfuseApp`.
-
 ## Subpath exports
 
 Import these instead of `dist/` paths. Deep `dist/` imports break when a bundler (such as
@@ -842,19 +851,13 @@ Env fallbacks: `GOOGLE_VERTEX_PROJECT`, `GOOGLE_VERTEX_LOCATION` (default `globa
 
 ```typescript
 import {TerrenoApp} from "@terreno/api";
-import {AiApp, AIService, LangfuseApp} from "@terreno/ai";
+import {AiApp, AIService} from "@terreno/ai";
 import {google} from "@ai-sdk/google";
 
 const aiService = new AIService({model: google("gemini-3.8-flash")});
 
 new TerrenoApp({userModel: User})
   .register(new AiApp({aiService, openApiOptions: {}}))
-  .register(
-    new LangfuseApp({
-      publicKey: process.env.LANGFUSE_PUBLIC_KEY!,
-      secretKey: process.env.LANGFUSE_SECRET_KEY!,
-    })
-  )
   .start();
 ```
 
@@ -866,9 +869,7 @@ Legacy `setupServer` pattern: call `addGptHistoryRoutes`, `addGptRoutes`, etc. i
 |----------|---------|-------------|
 | `GOOGLE_VERTEX_PROJECT` | `createVertexProvider` | GCP project for Vertex models |
 | `GOOGLE_VERTEX_LOCATION` | `createVertexProvider` | Vertex region (default `global`) |
-| `AI_OBS_PRICE_MAP_JSON` | `ObservabilityApp` | JSON model map with non-negative `inputPerMTok` / `outputPerMTok`; omitted models have tokens but no USD cost || `LANGFUSE_PUBLIC_KEY` | `LangfuseApp` | Langfuse public key |
-| `LANGFUSE_SECRET_KEY` | `LangfuseApp` | Langfuse secret key |
-| `LANGFUSE_BASE_URL` | Langfuse client | Langfuse host URL |
+| `AI_OBS_PRICE_MAP_JSON` | `ObservabilityApp` | JSON model map with non-negative `inputPerMTok` / `outputPerMTok`; omitted models have tokens but no USD cost |
 
 GCS credentials use standard Google Cloud Application Default Credentials for `FileStorageService`.