@terreno/mcp 57.9.0 → 57.11.0

package/dist/docs/versioned/next/implementationPlans/durable-agent-harness.md

@@ -0,0 +1,340 @@
+# Implementation Plan: Durable agent harness (`@terreno/ai/harness`)
+
+**Status:** Approved  
+**Roadmap:** Area=`ai`, Target=`Next`, Impact=`Feature`  
+**Branch:** `durable-agent-harness` (stacked on `cursor/ai-observability-ip-64ca`)  
+**Owner:** unassigned  
+**Created:** 2026-10-02  
+**Task list:** [docs/tasks/durable-agent-harness.md](../tasks/durable-agent-harness.md)  
+**Inspiration:** [Pi Durable](https://earendil.com/posts/pi-durable/) (concepts only; no dependency)  
+**Depends on:** [ai-observability.md](./ai-observability.md) (`ObsTrace` / `ObsSpan`, `ObservabilityApp`), [job-queues.md](./job-queues.md) (`@terreno/jobs`, optional)  
+**Supersedes in part:** [ai-agents-and-failover.md](./ai-agents-and-failover.md) — its `Agent` and failover ideas become harness primitives; its "no human-in-the-loop / no resume / no sub-agents" non-goals are reversed here.
+
+## Goal
+
+One easy-to-use harness inside `@terreno/ai` that runs long-lived agentic work durably:
+deterministic multi-phase workflows, LLM agent loops, tools, subagents, human approvals,
+retries, and crash recovery — all audited. It must be strong enough to build an
+OpenClaw/Hermes-class personal agent, a coding agent, and a regulated clinical automation
+on the same primitives.
+
+Today a chat turn lives inside one HTTP request (`ai/src/routes/gpt.ts`), history is saved
+only after the stream ends, and a crash loses the turn. `@terreno/jobs` is durable but a
+job is one opaque handler with no checkpoints. The harness fills that gap.
+
+## Non-Goals
+
+- Depending on or wrapping `@earendil-works/pi-durable`.
+- Journaled/replayed `step.run` workflows (Temporal/Inngest style). Phases only.
+- Socket.io streaming for agent output. Streaming is SSE only.
+- Auto-migrating in-flight runs across task versions.
+- Pluggable non-Mongo storage (SQLite/JSONL/memory). Mongo for every runner.
+- Field-level encryption, retention/redaction policies, e-signatures (future work).
+- Changing `/gpt/prompt` behavior before Phase 4.
+
+## Decisions
+
+| Question | Decision |
+|----------|----------|
+| Build our own harness or adopt pi-durable? | Own harness on Mongo + Vercel `ai` SDK, copying Pi's concepts (tasks, checkpoints, replay-safe tools, hooks, memos, documents, compaction, forks). |
+| Where should the harness live? | Inside `@terreno/ai`, as subpath `@terreno/ai/harness`. |
+| How should the `@terreno/jobs` dependency be wired? | Optional peer. Jobs integration is optional; a single process can own execution. |
+| Who executes a run? | Pluggable `Runner`: `InProcessRunner` (single owner, default) and `JobsRunner` (leased, multi-instance). |
+| How do durable code steps resume after a crash? | Explicit phases (Pi-style `defineTask` state machine); each phase commits a checkpoint. |
+| Where does state live for the single-process owner? | Mongo for both runners; switching runners needs no migration. |
+| What compliance guarantees ship in v1? | Immutable step/event audit log, built on the AI observability branch's `ObsTrace` / `ObsSpan`. |
+| How does the audit log relate to `ObsTrace` / `ObsSpan`? | `ObsSpan` **is** the audit log for harness work. |
+| How do we guarantee a span exists for every checkpoint? | Checkpoint commit and its `ObsSpan` write happen in one Mongo transaction. Langfuse/OTel stay best-effort after commit. |
+| How does this sequence against the observability branch? | Branch off `cursor/ai-observability-ip-64ca` and rebase as it lands. |
+| What happens on resume when replay was not declared? | Never auto-replay. Interrupted tools are reported to the model; interrupted phases go to `interrupted`. Opt in with `replay: "safe"`. |
+| Which features land in Phase 1 with the clinical tracer? | Tasks, phases, agent loop, tools + replay, retries, audit, **hooks + memos + approvals**, **subagents + ownership tree**. |
+| How do clients watch runs live? | SSE only, resumable via `Last-Event-ID`; a separate submit endpoint sends input/steer. |
+| How does a clinician sign off on an approval? | An approval inbox in `@terreno/admin-frontend`. |
+| Who can see and act on an approval? | Per-approval `approvers` (modelRouter permission shape); inbox lists only what the user may approve; default `IsAdmin`; mountable outside admin chrome. |
+| What happens to `/gpt/prompt`? | Untouched in Phase 1; rebuilt on the harness in Phase 4. |
+| Where do coding-agent execution environments land? | Phase 1 defines the `ExecutionEnv` interface only; Phase 5 ships Node env, coding tools, sandbox adapter. |
+| Task definition changes while runs are in flight? | Runs pin `name@version`; old versions must stay registered; startup fails loudly otherwise. |
+
+### Recorded assumptions (low-risk, chosen by convention)
+
+- **Interrupted model requests are re-requested.** A model call has no side effect, so a
+  crash mid-stream marks the partial response aborted and the turn retries the request.
+- **Model-call retries:** 429/5xx/network errors retry with exponential backoff + jitter
+  (default 3 attempts), then an optional `fallbackModels` list. A thrown phase retries
+  under the task's `retry` policy (default `maxAttempts: 3`); exhausted → `failed`.
+- **Token deltas are persisted, coalesced** (~250 ms) as `HarnessEvent` rows with a TTL so
+  any instance can serve an SSE resume. Committed messages are permanent.
+- **Cross-instance SSE** tails `HarnessEvent` with a Mongo change stream (replica set is
+  already required).
+- **Non-local trace sinks** (Langfuse, OTel) receive the full trace once a root task is
+  terminal, best-effort, via `ObservabilityApp.exportTrace`.
+- **Approvals inbox filtering** evaluates `approvers` per pending approval in memory
+  (pending sets are small); pagination applies after filtering.
+
+## Architecture
+
+```
+            app code: defineTask / defineAgent / defineTool / defineExtension
+                                     |
+                          Harness (registry + storage)
+         +-------------+-------------+--------------+----------------+
+         |             |             |              |                |
+     Task engine   Agent loop     Hooks/memos    Approvals       Event log
+     (phases,      (built-in      (extensions)   (wait + inbox)  (SSE source)
+      checkpoints,  task)
+      ownership)
+         |
+     runtime.commit()  ── one Mongo transaction ──>  HarnessTask/Message/Memo/Event
+                                                     + ObsSpan / ObsTrace (audit)
+         |
+       Runner:  InProcessRunner (owner lease)  |  JobsRunner (@terreno/jobs, optional)
+```
+
+### Core concepts
+
+| Concept | What it is |
+|---------|-----------|
+| **Task** | Durable unit of work: `name@version`, `input`, `state`, `phase`, status, ownership. Everything (model request, tool call, workflow, subagent) is a task. |
+| **Phase** | A named async function of a task. It does work and calls `rt.commit()` with the next phase or a terminal outcome. A phase is the replay boundary. |
+| **Checkpoint** | The committed `{phase, state}`. Resume always starts the current phase from its checkpoint. |
+| **Conversation** | A transcript plus agent config (model, instructions, tools, extensions). Root, or owned by a task (subagent). |
+| **Agent** | `defineAgent` config. Running an agent = the built-in `terreno.agent.turn` task on a conversation. |
+| **Tool** | `defineTool` with zod params and `replay: "safe" | "never"` (default `never`). Runs as a child task of the turn. |
+| **Extension** | Named bundle of prompt sections, tools, hooks, and tool wraps. |
+| **Hook** | `beforeModelRequest`, `beforeTool`, `afterTool`. Can rewrite, block, or annotate. |
+| **Memo** | `rt.memo(key, value?)` — durable decision storage scoped to a task, survives restarts. |
+| **Approval** | `rt.approval(key, {...})` — creates a `HarnessApproval`, waits for decision, audits the decider. |
+| **Event** | Append-only `HarnessEvent` rows with per-conversation/task `seq`; the SSE source. |
+| **Runner** | Decides who executes runnable tasks and when. |
+
+### Commit and audit
+
+`rt.commit(fn)` opens a Mongo transaction that, fenced by the task's lease token:
+
+1. Updates `HarnessTask` (`phase`, `state`, status, `attempt`, outcome).
+2. Inserts any `HarnessMessage`, `HarnessMemo`, `HarnessApproval`, `HarnessEvent` rows.
+3. Inserts the `ObsSpan` for the finished work and updates the root `ObsTrace`.
+
+If the lease token no longer matches (another owner took over) the transaction aborts and
+the phase result is discarded. `Harness` construction throws when the local observability
+plugin (`createLocalObservabilityPlugin`) is not registered or the connection is not a
+replica set.
+
+Span mapping: root task → `ObsTrace` + `CHAIN` root span; phase → `CHAIN`; model request →
+`LLM` (usage, cost via `priceMap`); tool → `TOOL`; subagent → `AGENT`; approval decision →
+`CHAIN` named `approval:<key>` with `decidedBy`, `decision`, `reason` in `output`. Spans
+are never updated after insert except `endedAt`/`status` on the same commit.
+
+### Replay and interruption
+
+On resume, a task whose status is `running` with an expired lease was interrupted:
+
+| Interrupted thing | `replay: "safe"` | default (`never`) |
+|---|---|---|
+| Phase | Re-run the phase from its checkpoint | Status `interrupted`; await `resolveInterrupted` (`retry` / `abort` / `complete`) |
+| Tool call | Re-run the tool | Tool result = "interrupted, not retried"; agent loop continues |
+| Model request | Always re-requested (assumption above) | — |
+
+### Runners
+
+| Runner | Ownership | Use |
+|---|---|---|
+| `InProcessRunner` (default) | One process holds the `HarnessOwner` lease (heartbeat). A second process gets `standby` and takes over on lease expiry. Polls runnable tasks; wakes on `runAt` and events. | Local agents, single-instance apps, tests |
+| `JobsRunner` (`@terreno/ai/harness/jobsRunner`, optional peer `@terreno/jobs`) | Each runnable phase → job `terreno.harness.phase`, idempotency key `taskId:phase:attempt:claims` (`claims` counts claims, so a phase revisited by an event wake or crash replay gets a new job; jobs idempotency keys are permanent). Handler acquires the task lease (fenced, heartbeat) and runs one phase. The task lease, not the job lock, is the authority. | Multi-instance Cloud Run |
+
+### Ownership tree
+
+Tasks are owned by a conversation or a task. `abort(taskId)` aborts owned tasks
+bottom-up, running each task's `abort` handler (compensations) before marking it
+`aborted`. `background: true` tasks belong to a conversation but are not part of its
+current turn; a turn abort leaves them running.
+
+### Versioning
+
+Each task records `name` + `version`. The runner only resumes a task on a registered
+handler for exactly that version. `Harness.start()` scans non-terminal tasks and throws
+listing every unregistered `name@version`.
+
+## Models
+
+All in `ai/src/harness/models/`, types in `ai/src/types/harness.ts`. Every field has
+`description`; `strict: "throw"`.
+
+| Model | Key fields |
+|---|---|
+| `HarnessTask` | `name`, `version`, `input`, `state`, `phase`, `status` (`pending` \| `running` \| `waiting` \| `interrupted` \| `completed` \| `failed` \| `aborted`), `ownership {kind: conversation\|task\|root, id}`, `background`, `attempt`, `retry {maxAttempts, backoffMs, maxBackoffMs}`, `runAt`, `lease {owner, token, expiresAt}`, `waiting {kind: event\|tasks\|sleep, key, taskIds, policy, timeoutAt}`, `outcome {status, result, error}`, `traceId`, `rootTaskId`, `requestId` (unique sparse), `userId` |
+| `HarnessConversation` | `ownership`, `agent {name, model, instructions, tools[], extensions[]}`, `status` (`idle` \| `busy`), `activeTurnTaskId`, `queued[]`, `userId`, `seq` |
+| `HarnessMessage` | `conversationId`, `seq`, `role`, `parts[]`, `toolCallId`, `toolName`, `turnTaskId`, `aborted` |
+| `HarnessMemo` | `taskId`, `key`, `value` — unique `(taskId, key)` |
+| `HarnessApproval` | `taskId`, `rootTaskId`, `definitionKey` (`name@version:key`), `title`, `summary`, `payload`, `status` (`pending` \| `approved` \| `rejected` \| `expired`), `decidedBy`, `decidedAt`, `reason`, `expiresAt` |
+| `HarnessEvent` | `streamId` (conversation or task), `seq`, `type`, `payload`, TTL for `delta` events |
+| `HarnessOwner` | singleton lease for `InProcessRunner`: `owner`, `expiresAt` |
+
+## APIs
+
+### Library (`@terreno/ai/harness`)
+
+```typescript
+import {
+  Harness, defineTask, defineAgent, defineTool, defineExtension, hook, section,
+  approvalGate, InProcessRunner,
+} from "@terreno/ai/harness";
+
+const summarizer = defineAgent({
+  name: "clinic.summarizer",
+  model: {provider: "anthropic", modelId: "claude-sonnet-5-5"},
+  instructions: "Summarize the chart for a clinician. Cite sources.",
+  tools: [],
+  fallbackModels: [{provider: "anthropic", modelId: "claude-haiku-4-5-20251001"}],
+});
+
+const intakeSummary = defineTask<{patientId: string}, IntakeState, {status: string}>({
+  name: "clinic.intakeSummary",
+  version: 1,
+  initial: () => ({phase: "fetch"}),
+  retry: {maxAttempts: 3},
+  phases: {
+    fetch: {
+      replay: "safe",
+      run: async (task, rt) => {
+        const chart = await ehr.getChart(task.input.patientId);
+        await rt.commit({phase: "summarize", state: {chart}});
+      },
+    },
+    summarize: {
+      replay: "safe",
+      run: async (task, rt) => {
+        const summary = await rt.runAgent(summarizer, {input: task.state.chart, output: SummarySchema});
+        await rt.commit({phase: "review", state: {...task.state, summary}});
+      },
+    },
+    review: {
+      replay: "safe",
+      run: async (task, rt) => {
+        // Approvers are declared on the definition: approvals: {"clinician-signoff": {approvers: [Permissions.IsAdmin, isClinician]}}
+        const decision = await rt.approval("clinician-signoff", {
+          title: "Sign off intake summary",
+          payload: task.state.summary,
+          timeout: {hours: 24},
+        });
+        await rt.commit(decision.approved ? {phase: "write"} : {terminal: {status: "completed", result: {status: "rejected"}}});
+      },
+    },
+    write: {
+      // default replay: "never" — an interrupted EHR write goes to `interrupted`
+      run: async (task, rt) => {
+        await ehr.writeNote(task.input.patientId, task.state.summary, {idempotencyKey: `note-${task.id}`});
+        await rt.commit({terminal: {status: "completed", result: {status: "filed"}}});
+      },
+    },
+  },
+  abort: async (task, rt) => { /* compensations */ },
+});
+
+const harness = await Harness.open({
+  models: createModelFn,
+  registry: [intakeSummary, summarizer, ClinicExtension],
+  runner: new InProcessRunner(),
+});
+await harness.start();
+const run = await harness.createTask(intakeSummary, {patientId}, {requestId: `intake-${patientId}`});
+```
+
+Runtime (`rt`) surface in Phase 1: `commit`, `memo`, `runAgent`, `createTask`,
+`waitForTasks(ids, {policy: "all" | "failFast"})`, `waitFor(event, {timeout})`,
+`sleep(duration)`, `approval`, `output` (stream text to the event log), `signal`, `env`
+(`ExecutionEnv`, interface only in Phase 1).
+
+Harness surface: `open`, `start`, `stop`, `createTask`, `conversation(id)`,
+`createConversation`, `sendEvent(taskId, event, payload)`, `abort(taskId)`,
+`resolveInterrupted(taskId, action)`.
+
+### HTTP (`HarnessApp` plugin)
+
+| Method | Path | Behavior |
+|---|---|---|
+| POST | `/harness/conversations/:id/submit` | instanceAction. `{content, requestId, whenBusy: "queue" \| "steer"}`; idempotent on `requestId` |
+| GET | `/harness/conversations/:id/events` | SSE; honors `Last-Event-ID`; replays then tails |
+| GET | `/harness/tasks/:id/events` | SSE for a task subtree |
+| POST | `/harness/tasks/:id/abort` | instanceAction; owner or admin |
+| POST | `/harness/tasks/:id/resolveInterrupted` | instanceAction; admin; `{action: "retry" \| "abort" \| "complete", reason}` |
+| GET | `/harness/approvals` | list pending approvals the caller may approve |
+| POST | `/harness/approvals/:id/approve` \| `/reject` | instanceActions; `{reason}` (required on reject); audited span |
+
+SSE is a listed exception to the modelRouter-actions rule; everything else is
+`modelRouter` collection/instance actions.
+
+## Notifications
+
+None in Phase 1. `rt.approval` accepts an optional `notify` callback so apps can send via
+`@terreno/comms`; the harness does not depend on comms.
+
+## UI
+
+- **Approvals inbox** (`@terreno/admin-frontend`, `widgets/harness/`): list of pending
+  approvals (title, task, age, expires), detail with `payload` rendered as JSON/markdown,
+  Approve / Reject with reason. Registered as admin custom screen `harness-approvals`
+  (group "AI Harness"), and exported as `HarnessApprovalInbox` for mounting outside admin
+  chrome. Uses `@terreno/ui` only.
+- **Runs screen** (Phase 2): task list/tree, status filters, abort, resolve interrupted.
+- Spans render in the existing observability Traces screens.
+
+## Phases
+
+| Phase | Ships | Proof |
+|---|---|---|
+| **1 — Durable clinical tracer** | Engine, phases, checkpoints, `InProcessRunner`, transactional `ObsSpan` audit, replay semantics, retries, agent loop, tools, subagents + ownership tree, extensions/hooks/memos, waits/events, approvals + inbox, SSE + submit, version pinning, example-backend `clinic.intakeSummary` | Crash-and-resume e2e; inbox Playwright screenshots; span tree assertions |
+| **2 — Multi-instance** | `JobsRunner`, two-worker fencing, Runs admin screen | Two-process race tests; screenshots |
+| **3 — Long context** | Background compaction, `reset`/handoff, conversation forks, `defineDoc` documents (rewindable, fork `asOf`) | Bun tests on 10k-message conversations; fork isolation tests |
+| **4 — Durable chat + multiplayer** | `/gpt` rebuilt on harness, `GptHistory` mapping, steering, foreground/background abort, `viewState` over SSE, rtk `useHarnessConversation`, `GPTChat` adapter | Kill backend mid-chat, reconnect resumes; two clients see same view |
+| **5 — Coding agents + malleability** | `NodeExecutionEnv`, `CodingTools` (read/write/edit/bash), remote sandbox adapter, registry hot-replace | Coding agent fixes a failing test in a fixture repo after a crash |
+
+## Feature Flags & Migrations
+
+- No feature flag; the harness is opt-in by registering `HarnessApp`.
+- New collections only. Phase 4 adds a migration mapping `GptHistory` → `HarnessConversation`/`HarnessMessage` (human gate at Phase 4 start: keep dual-read or one-shot migrate).
+
+## Activity Log & User Updates
+
+Audit lives in `ObsSpan`. Changelog entry per phase in `changelog/`.
+
+## Not Included / Future Work
+
+Field encryption, retention/redaction, e-signature approvals, Socket.io streaming,
+non-Mongo storage, journaled steps, in-flight version migration.
+
+## Files to Create / Modify
+
+| Path | Change |
+|---|---|
+| `ai/src/harness/` | new: `harness.ts`, `defineTask.ts`, `runtime.ts`, `commit.ts`, `agentLoop.ts`, `defineAgent.ts`, `defineTool.ts`, `extensions.ts`, `approvals.ts`, `events.ts`, `modelCall.ts`, `runners/inProcessRunner.ts`, `runners/jobsRunner.ts`, `harnessApp.ts`, `routes/*.ts`, `models/*.ts` |
+| `ai/src/types/harness.ts` | new model + API types |
+| `ai/package.json` | `./harness`, `./harness/jobsRunner` exports; optional peer `@terreno/jobs` |
+| `admin-frontend/src/widgets/harness/` | new approvals inbox (Phase 1), runs screen (Phase 2) |
+| `example-backend/src/harness/` | `clinic.intakeSummary`, fake EHR, registration in `server.ts` |
+| `example-frontend` | admin route for inbox; `bun run sdk` |
+| `docs/explanation/durable-agent-harness.md` | new |
+| `docs/reference/ai-harness.md` | new |
+| `docs/how-to/build-a-durable-workflow.md` | new |
+| `docs/reference/ai.md`, `docs/reference/admin-frontend.md` | link + inbox section |
+
+## Task List
+
+[docs/tasks/durable-agent-harness.md](../tasks/durable-agent-harness.md)
+
+## Acceptance Criteria
+
+| # | Criterion | Verification |
+|---|---|---|
+| AC1 | A two-phase task runs to completion and every checkpoint has exactly one `ObsSpan`, written in the same transaction | bun test: abort the transaction mid-commit → neither task update nor span exists |
+| AC2 | Killing the owner mid-run resumes from the last checkpoint on a fresh `Harness` | bun test: throw-on-signal simulation + new harness instance; e2e: kill example-backend during `summarize`, restart, run completes |
+| AC3 | Default-replay phases and tools never re-run after interruption; `replay: "safe"` ones do | bun test matrix over phase/tool × replay |
+| AC4 | Thrown phases retry with backoff and land `failed` after `maxAttempts`; model 429/5xx retries then falls back | bun test with fake model provider |
+| AC5 | Aborting a root task aborts owned subtasks bottom-up and runs `abort` handlers | bun test ordering assertion |
+| AC6 | `beforeTool` hook can block a tool; memo decision survives restart | bun test |
+| AC7 | `rt.approval` blocks until a permitted user approves; unpermitted users get 403 and do not see it in the inbox; decision span records `decidedBy` | supertest |
+| AC8 | Approvals inbox lists, approves, and rejects in admin-frontend | Playwright run with screenshots in `/opt/cursor/artifacts/` |
+| AC9 | SSE reconnect with `Last-Event-ID` replays missed events exactly once, from any instance | bun test with two `HarnessApp` instances on one replica set |
+| AC10 | Starting with an in-flight unregistered `name@version` throws listing it | bun test |
+| AC11 | `JobsRunner`: two workers never run the same phase concurrently; stale lease owner's commit is rejected | bun test (Phase 2) |
+| AC12 | Docs explain, reference, and how-to every public API in the slice that ships it | `docs-audit` skill run shows no undocumented harness exports; Roast reads each page against the shipped API |

package/dist/docs/versioned/next/implementationPlans/terreno-langfuse-integration.md

@@ -3,7 +3,7 @@
 **Roadmap issue:** https://github.com/TerrenoLabs/terreno/issues/1093  
 **Status:** Complete
 
-> **See also:** Native in-app observability (prompts, traces, evals, experiments, review queue) is specified in [ai-observability.md](./ai-observability.md). This IP remains the **Langfuse Cloud/self-host vendor plugin**. Both plugins may be registered; traces/scores fan out (see that IP).
+> **See also:** Native in-app observability (prompts, traces, evals, experiments, review queue) is specified in [ai-observability.md](./ai-observability.md). Operator loop: [Develop an AI feature](../how-to/ai-feature-development.md). This IP remains the **Langfuse Cloud/self-host vendor plugin**. Both plugins may be registered; traces/scores fan out (see the observability IP).
 
 A TerrenoApp plugin that integrates [Langfuse](https://langfuse.com) into Terreno backends and frontends. Provides prompt management with MongoDB caching, OpenTelemetry tracing, Vercel AI SDK helpers, React hooks, and an admin UI — all wired up via `.install()`.
 

package/dist/docs/versioned/next/reference/README.md

@@ -8,6 +8,9 @@ Technical reference for Terreno packages and APIs. Information-oriented, precise
 - [@terreno/ui](ui.md) — Components, theming, layout
 - [@terreno/syncdb](syncdb.md) — Local-first data layer (reads, writes, offline sync)
 - [@terreno/ai](ai.md) — AI service, GPT routes, Langfuse integration
+- [@terreno/ai/harness](ai-harness.md) — Durable multi-phase tasks with transactional checkpoints and audit spans
+- [Agent UI Asks](agent-ui-asks.md) — `@terreno/blocks` ask schemas, simple cards, the compact surface, limits, error codes, SSE events, the headless `pendingAsks` and `turn` endpoints, and JSON Schemas for native clients
+- [UI blocks](blocks.md) — whole-reply YAML for leaf and layout blocks: grammar, limits, and error codes
 - [@terreno/admin-backend](admin-backend.md) — Auto-generated admin CRUD endpoints
 - [@terreno/admin-frontend](admin-frontend.md) — Admin panel UI components
 - [@terreno/admin-spa](admin-spa.md) — Standalone admin SPA + Express serve plugin

package/dist/docs/versioned/next/reference/admin-config.md

@@ -40,6 +40,8 @@
 `AdminAction` has `id`, `label`, optional `confirm`, `background`, and `patchKeys`.
 
 `AdminContribution` may provide `models`, `customScreens`, `homeWidgets`, and `scripts`.
+`customScreens[].group` becomes a named sidebar heading (`AI Observability`). Screens without
+`group` stay under **Screens** — that is where **AI Requests** remains.
 Model contributions contain `model`, `routePath`, and `admin`. Duplicate `routePath`s
 throw for registered routers; the same Mongoose model at two paths gets unique config
 `name`s so the admin UI can route to each.

package/dist/docs/versioned/next/reference/admin-frontend.md

@@ -434,3 +434,219 @@ import {CommsDashboardScreenWidget, CommsMessageDetail} from "@terreno/admin-fro
 The list screen persists filters in the URL (`channel`, `provider`, `status`, `errorClass`, `q`,
 `startDate`, `endDate`, `page`) and calls `/comms/messages`, `/comms/stats`, and
 `/comms/messages/retryMany`.
+
+### AI Observability chrome
+
+`ObservabilityApp` contributes grouped custom screens (`group: "AI Observability"`): `ai-prompts`,
+`ai-traces`, `ai-evaluators`, `ai-datasets`, `ai-experiments`, and `ai-review` when the local
+plugin is on. `AI Requests` (`ai-requests`) stays ungrouped. Widgets also register detail routes:
+`ai-prompt-editor`, `ai-trace-detail`, `ai-review-item`, `ai-evaluator-detail`, `ai-evaluator-new`,
+`ai-dataset-detail`, `ai-experiment-new`, and `ai-experiment-results`.
+
+Every observability screen wraps `AiObservabilityChrome`: breadcrumbs
+`Admin / AI Observability / <Section> / <leaf>` and a status chip from
+`GET /ai/observability/status` (`Local on|off` plus active primaries). Review queue nav and the
+review screen body hide when `localOn` is false.
+
+Each observability list screen is contributed with `adminAccess: {resource, action: "list"}` so
+`/admin/config` omits screens the caller cannot list (`aiPrompt`, `aiTrace`, `aiEvaluator`,
+`aiDataset`, `aiExperiment`, `aiReview`). Detail routes stay registered on the host; route
+handlers still enforce Task 4.1 RBAC.
+
+`GET /ai/observability/status` includes a `permissions` map (per observability resource and
+action). Legacy admin users and apps without `accessControl` receive all actions `true`. RBAC
+callers receive effective flags for the signed-in user. Pass the same `accessControl` to
+`ObservabilityApp` and `AdminApp` so status permissions match sidebar filtering. Admin widgets
+read the payload through `resolvePromptActionPermissions` and `observabilityActionAllowed` (fail
+closed while status is loading, errored, or missing `permissions`). List screens gate **Create
+prompt** on `aiPrompt:create`, **Create evaluator** on `aiEvaluator:create`, and **New dataset** on
+`aiDataset:create`; experiment lists hide **New experiment** without `aiExperiment:create`.
+Dataset detail hides **Add item** without `aiDataset:update` and **Run experiment** without
+`aiExperiment:create`; dataset lists hide **Import** without
+`aiDataset:update`. Judge schema checks use the production prompt version, or the latest version
+when production is unset. Widgets do not couple to host Redux.
+
+`ai-prompts` lists prompts with a folder rail, search, type badge, latest vs production columns
+(tooltips), 7-day usage, and **Create prompt** when `aiPrompt:create` is allowed. The create modal
+includes an optional **Description** (saved on the prompt record and shown on the hub Overview tab).
+`ai-prompt-editor?name=` is a prompt hub with **Overview**, **Versions**, **Traces**, and
+**Experiments** tabs. Overview shows optional description, folder, tags, and latest vs production
+badges. Traces and Experiments render bounded relationship rows from
+`GET /ai/observability/prompts/:name?promptVersion=` (the hub pins the selected version, defaulting
+to latest after the first load, so related traces and experiments are filtered server-side while the
+full immutable version list stays on the detail payload). The **Versions** tab
+hosts the versioned editor: full-width version history rows show `vN`, every label attached to
+that version, and its creation time on one line. The selected row is highlighted. Editor /
+Playground tabs appear on Versions when allowed; **Save as vN+1**, **Set vN as production…**, and
+playground **Run once** hide when `aiPrompt:update`, `aiPrompt:promote`, or `aiPrompt:playground`
+is denied (fields stay read-only). Playground **Run once** does not create a version; hosts may pass `apiKey` to
+`AiPromptEditorScreenWidget`, which forwards it as `x-ai-api-key` without putting the key in the
+request body. Pass `apiKeyLoading` while the host reads a saved key (for example from
+`useStoredState`) so the playground waits instead of showing a missing-key message. Pass
+`playgroundApiKeyHint` when `GET /ai/observability/status` reports
+`playgroundAi.source: "request-key"`; the editor blocks **Run once** and shows that hint until a
+trimmed key is available. When `playgroundAi.source` is `server`, no key is required. When it is
+`unavailable`, the editor explains that the backend must configure `aiService` or
+`requestAiServiceFactory`. A stale playground **503** with the missing-key title is remapped to the
+same hint for `request-key` hosts so operators are not told to fix server configuration. The
+example admin supplies the Gemini key saved from Profile. **Save this run to dataset** stays
+disabled until phase 2.
+
+`ai-traces` lists traces with a filter bar: from/to date fields; dropdowns for prompt, status,
+score presence (**All traces / Has a score / No scores**), and data sensitivity
+(**All traces / Sensitive only / Not sensitive**); plus user and session text fields.
+Checkbox selection opens a bulk bar with **Send to human review**, a sensitive-count
+warning, **Clear**, and **Add to dataset** (opens a dataset picker modal; sensitive traces show a
+warning before bulk add). **Send to human review** opens a modal that explains how human
+evaluators define the score fields and reviewer instructions, then requires a human evaluator
+before creating one review-queue item per trace. When local trace storage is on,
+**Run multi-stage trace test** calls the admin-only smoke endpoint and
+opens the resulting detail: two schema-validated LLM stages, one deterministic tool span, and a
+final schema-validated combining LLM stage under one CHAIN root. LLM span input includes
+`outputSchema`. That run needs AI the same way the playground does: hosts may pass `apiKey`,
+`apiKeyLoading`, and `apiKeyHint` to `AiTracesScreenWidget`, which forwards the key as
+`x-ai-api-key`. When `GET /ai/observability/status` reports `playgroundAi.source: "request-key"`
+and no trimmed key is available, the button is disabled and the hint explains where to save one;
+`unavailable` instead reports that the backend must configure `aiService` or
+`requestAiServiceFactory`. The example admin supplies the Gemini key saved from Profile. Rows show a status dot (primary for successful runs, accent for failed runs),
+`sensitive` badge, error line, numeric prompt count, span count, tokens, cost, latency, score count, and
+**Open**. Pagination uses `page` / `limit` / `more` / `total`.
+`ai-trace-detail?id=` shows the header, left span list (kind badge, indent, duration bar),
+right span detail with **collapsed** sensitive I/O, and scores (value + source). The two columns
+size from the row width rather than their content, so a wide span value (multi-stage LLM spans embed
+`outputSchema` JSON) wraps inside the detail column instead of pushing it below the span list.
+
+`ai-review` shows Pending / In progress / Done / Skipped tabs with counts. Each tab is
+oldest-first and lists the trace action, prompt, assignee, waiting time, and status.
+**Start reviewing — oldest first** opens the first pending item; the empty state names both
+Traces intake and manual **Assign to me** assignment.
+
+`ai-review-item?id=` shows "Item N of M pending", previous/next navigation, read-only
+**What the AI was given** / **What the AI wrote** panels, collapsed long fields with word
+counts, reviewer notes, and a collapsed Raw JSON disclosure. Score controls come from evaluator
+dimensions (numeric slider, boolean Pass / Fail, categorical pills). Actions are
+**Submit & next**, **Skip**, and **Assign to me**; completion toasts report the remaining count
+or **Queue clear**.
+
+`ai-evaluators` lists evaluators with type badge, dimension summary, target, and run-mode chips.
+**Create evaluator** opens `ai-evaluator-new` with the name and optional purpose first, then how
+scoring happens (human / JSON assert / LLM judge). New evaluators score a **full trace** only;
+generation span and dataset item targets are not offered yet. Score fields depend on data type:
+boolean has no extra control, numeric asks for min and max, and categorical is a list you add to.
+LLM judge picks the judge prompt from the existing prompt list. Human setup points operators to
+**Send to human review** and omits live sampling because it never applies; automatic evaluators
+explain experiment/manual availability and production sampling, while LLM judges call out the billed
+model-call impact. The optional purpose is saved as the evaluator description and appears on its
+detail screen. LLM judge schema feedback stays idle until a prompt is selected, then shows
+loading/error states and only checks dimensions after its production schema loads.
+`ai-evaluator-detail?id=` leads with the evaluator name, description, and type/target/run-mode
+badges, then explains the saved scores, how scoring works, where it runs, and a **Used by** list
+derived from recent experiments. A failed experiments request shows **Could not load experiments.**
+instead of an empty 30-day list. Its dimension and usage rows use the shared `ObservabilityTable` instead of
+`DataTable`, which sizes to a height-constrained parent and collapses inside a scrolling page. LLM
+judge details show loading or load-failed feedback while resolving the judge prompt; schema
+mismatches appear only after its production schema loads.
+
+`ai-datasets` lists datasets with item counts, provenance bar, input-schema binding, and updated
+time. **New dataset** creates a dataset; **Import** on each row accepts `.json` or `.csv` via
+`FilePickerButton` (local URI read) or paste, posting `{rows}` for JSON or `{format:'csv',content}`
+for CSV. `ai-dataset-detail?id=` shows counts, schema binding, tabs **All / Human / Auto / Needs
+review**. Human is proofread items. Auto is unreviewed trace or synthetic items, so a proofread
+trace is not listed in both tabs. The screen includes an items table (input, expected, provenance,
+trace link), **Add item**, and **Run experiment**. The items table uses `ObservabilityTable`: rows grow with their content
+and cells wrap to three lines before truncating, so long inputs and expected outputs no longer
+overlap adjacent rows. Input and Expected receive 2.5× the flexible width of metadata columns.
+Selecting a row opens a scrollable modal with the complete input, expected output, provenance,
+annotation ids, tags, timestamps, metadata, and an **Open source trace** action when linked.
+Opening the source trace dismisses the item modal before navigation.
+Item-query loading and failures render dedicated states with Retry; they never appear as an empty
+dataset.
+
+`ObservabilityTable` (`widgets/aiObservability/shell/ObservabilityTable.tsx`) is the shared
+flow-height table for these screens. Columns take a `title`, optional `minWidth`, and optional
+relative `grow`; rows take a `key`, `cells`, and optional click/accessibility properties. A string
+cell renders truncated text and a node cell renders as-is. Its
+bordered shell shrink-wraps the rows inside flex/scroll parents so its bottom border cannot stretch
+into the following section.
+
+`ai-experiments` lists experiments with status, running progress, and cost. **New experiment**
+opens a four-step wizard (dataset with counts, prompt versions tagged latest/production/superseded,
+llm-judge and json-assert evaluators, review & run with estimate). A slower in-flight estimate is ignored after the selection changes, so it cannot replace a newer estimate or its error. Human evaluators are omitted. On the prompt-versions step, **Next** and the later
+wizard rail buttons stay disabled until 2–3 versions are selected. `includeUnproofread` and optional model override are on
+the wizard. `ai-experiment-results?id=` polls while pending/running, shows gate tiles per version,
+failing gate count, outliers, a side-by-side per-item output table (failed rows first from the
+API), and **Promote to production** with a confirm modal; promote is blocked when gates fail (409).
+
+### AI Harness approvals inbox
+
+`HarnessApp` (`@terreno/ai/harness`) contributes custom screen `harness-approvals`
+(`group: "AI Harness"`, label **Approvals**) through `adminContribution()`, so registering
+`HarnessApp` before `AdminApp` is enough on the backend. `HARNESS_ADMIN_WIDGETS["harness-approvals"]`
+(`HarnessApprovalsScreenWidget`) is in the built-in screen registry, so `/admin/harness-approvals`
+renders through `AdminScreenRouter` with no host route file. See
+[AI harness approvals](ai-harness.md#approvals).
+
+The inbox lists the pending approvals the signed-in user may decide
+(`GET /harness/approvals`, oldest first), each with title, task (`name@version` from
+`definitionKey`), age, and expiry. Selecting one shows its `summary` (rendered with `MarkdownView`
+when it looks like markdown) and `payload` (pretty JSON; a string payload that looks like
+markdown renders as markdown), plus a **Reason** field and
+**Approve** / **Reject**.
+
+| Behavior | Detail |
+| --- | --- |
+| Approve | `POST /harness/approvals/:id/approve` with `{reason}` when one was typed, else `{}`. |
+| Reject | Requires a non-blank reason (checked before the request); `POST .../reject` with `{reason}` (trimmed). |
+| Success | Back to the list with a confirmation; the list refetches. |
+| 409 | "Already decided, expired, or its task ended" notice; back to the list and refetch. |
+| Other errors | The API error `title` under the buttons; the detail stays open. |
+| States | Loading spinner, error with **Try again**, empty (**Nothing to approve.**), **Refresh** button. |
+| Page size | The oldest 100 (`?limit=100`). When the response has `more`, a note says so; deciding items brings the rest into view. |
+| In flight | Both buttons and the reason field are disabled until the decision returns, so it is sent once. |
+
+Requests use the host-bound admin request client (`AdminProvider` `getAuthHeaders` / `apiOrigin`),
+never RTK `injectEndpoints`. Test ids: `harness-approvals` (root), `harness-approvals-list`,
+`harness-approvals-empty`, `harness-approvals-error`, `harness-approvals-loading`,
+`harness-approvals-refresh`, `harness-approvals-notice`, `harness-approval-row-<id>-clickable`,
+`harness-approval-detail`, `harness-approval-payload`, `harness-approval-reason`,
+`harness-approval-approve`, `harness-approval-reject`, `harness-approval-error`,
+`harness-approval-back` (standalone only; in admin the page back chevron closes the detail).
+
+#### Mount outside admin chrome
+
+`HarnessApprovalInbox` is exported for app screens (for example a clinician view). Inside an
+`AdminProvider` it needs no props. Elsewhere, pass a request client from `bindAdminRequest`:
+
+```typescript
+import {bindAdminRequest, HarnessApprovalInbox} from "@terreno/admin-frontend";
+import {baseUrl} from "@terreno/rtk";
+import {Page} from "@terreno/ui";
+
+// getAuthHeaders: the host's `() => ({Authorization: "Bearer …"})`, as passed to AdminProvider.
+const request = bindAdminRequest({getAuthHeaders, origin: baseUrl});
+
+export const SignOffScreen: React.FC = () => (
+  <Page title="Sign-offs" scroll>
+    <HarnessApprovalInbox request={request} />
+  </Page>
+);
+```
+
+| Prop | Type | Default | Description |
+| --- | --- | --- | --- |
+| `request` | `(args: AdminRequestArgs) => Promise<unknown>` | `AdminProvider` client | Request client, usually `bindAdminRequest({getAuthHeaders, origin})`. |
+| `baseUrl` | `string` | `/harness` | Where `HarnessApp` is mounted (its `basePath`). Relative paths resolve against the client's `origin`. |
+| `selected` | `HarnessApprovalRow \| undefined` | internal | Controlled selection: the approval whose detail is open. |
+| `onSelectedChange` | `(approval \| undefined) => void` | — | With `selected`, lets the host's back control close the detail; the inbox then hides its own **All approvals** button. The admin screen uses this so the page chevron returns to the list. |
+| `showHeading` | `boolean` | `true` | Show the **Approvals** heading; the admin screen passes `false` because `AdminScreenPage` already titles it. |
+| `testID` | `string` | `harness-approvals` | Root test id. |
+
+The built-in admin screen always uses `/harness`. A host that mounts `HarnessApp` at another
+`basePath` registers its own `widgets.screens["harness-approvals"]` that renders
+`<HarnessApprovalInbox baseUrl="/ops/harness" />`.
+
+The example backend registers `HarnessApp` with a demo task, `demo.approvalDemo@1`. Start
+approvals with the admin script `startHarnessApprovalDemo` (args `count`, default 2, and `prefix`)
+from the admin Scripts screen or `bun run script startHarnessApprovalDemo --wet`. It needs Mongo
+running as a replica set; without one the backend logs `Harness disabled` and skips the routes.
+`example-frontend/e2e/harness-approvals.spec.ts` approves one and rejects one.