@terreno/mcp 57.20.0 → 57.22.0

package/dist/docs/versioned/57.21.0/implementationPlans/deploy-to-vercel.md

@@ -0,0 +1,222 @@
+# Implementation Plan: Deploy to Vercel + `deploy-vercel` Skill
+
+**Status:** Implemented — V1/V2/V3 decided 2026-10-11; Vercel runs in parallel with Cloud Run + Netlify for an evaluation period
+**Roadmap issue:** https://github.com/TerrenoLabs/terreno/issues/1012
+**Priority:** High
+**Effort:** Big batch
+**Owner:** unassigned
+**Created:** 2026-07-27
+**Program:** [OSS launch](oss-launch-program.md)
+**Depends on:** [`deployment-foundation`](deployment-foundation.md)
+**RTK deprecation flag:** **Partial** — the web build and client configuration sections depend on the frontend data layer; the syncdb websocket transport also has implications for which parts of the app can be served from serverless functions. Marked tasks must be re-verified after PR #869.
+
+## Goal
+
+Give Terreno the fastest possible "my app is on the internet" path, and a skill that executes it. Vercel is where most JavaScript developers expect to deploy, and Expo Router has first-class Vercel support (`vercel.json` with `expo export -p web`, plus `expo-server/adapter/vercel` for server output). Terreno currently has zero Vercel documentation — the only occurrences of "Vercel" in the repo refer to the Vercel **AI SDK** inside `@terreno/ai`, which is a completely different thing and is itself a source of confusion worth addressing.
+
+**Topology is not finalized.** A pre-implementation spike must answer whether a single-platform Vercel deploy (web + `@terreno/api` + file storage) is viable, or whether the documented path remains split (static web on Vercel, long-lived backend elsewhere). Until the spike completes, this IP documents the **interim split topology** and tracks open questions as TODOs.
+
+## Non-Goals
+
+- Committing to a Vercel backend topology before the spike (see **Open TODOs** below).
+- Vercel-hosted MongoDB (does not exist; Atlas is the answer).
+- Native app distribution (that is EAS; see the `expo-deployment` skill).
+
+## Blocking questions
+
+**Recorded 2026-07-29** (defaults accepted where not marked open).
+
+| # | Question | Decision |
+|---|----------|----------|
+| V1 | Backend host | **Decided 2026-10-11 (human): run all-in-one on Vercel in parallel with the existing split deploy.** `terreno-backend` (Vercel Functions, Fluid compute) serves `@terreno/api` with Socket.IO; `terreno-frontend` and `terreno-demo` are static exports. Both topologies are documented; split stays the conservative default for apps that need background workers or GCS uploads. Uploads (GCS) and the jobs worker are disabled on Vercel during the evaluation. |
+| V2 | Document `server` output? | **Decided: A** — advanced section only (repo is on Expo SDK 57); both apps keep `static` output; SSE buffering caveat documented. |
+| V3 | Commit `vercel.json`? | **Decided: A** — committed in `example-backend`, `example-frontend`, and `demo`, wired to real projects in the `terrenolabs` Vercel team (owner: Josh) and deployed by CircleCI (`vercel-deploy` job, `terreno-vercel` context). |
+| V4 | Preview deployments | **A** — include CORS + Better Auth `trustedOrigins` for preview URLs |
+| V5 | Disambiguate Vercel AI SDK vs hosting | **A** — consistent phrasing: "the Vercel AI SDK" vs "Vercel (hosting)" |
+
+## Spike outcome (2026-10-11)
+
+Answered by deploying the examples, not a scratch spike. See `docs/how-to/deploy-to-vercel.md` and `docs/how-to/vercel-function-websockets.md`.
+
+- `@terreno/api` runs on Vercel Functions (Node 24, Fluid compute) by exporting an `http.Server` that `TerrenoApp` attaches to (`example-backend/src/app.ts`, `start(false, server)`). WebSocket upgrades work; change-stream watchers run per instance.
+- The backend must be fully bundled with Bun (`build:vercel`): Vercel's launcher refuses `require()` of ESM, Bun inlines `NODE_ENV`, and the Express preset needs an `import "express"` marker.
+- Clients must use WebSocket-only Socket.IO on Vercel; `@terreno/rtk` `useSocketConnection({transports})` and `@terreno/syncdb` `createSyncDb({socketTransports})` were added.
+- Preview origins are matched with `CORS_ORIGINS` wildcards (`toCorsOrigin`), and Better Auth gets the raw patterns.
+
+## Open TODOs (pre-implementation spike)
+
+Complete these before Phase 1 of the how-to guide ships:
+
+- [x] **V1-todo:** Can `@terreno/api` run on Vercel with Socket.io sessions and MongoDB change streams? Use [Vercel Functions WebSockets (Public Beta)](https://vercel.com/docs/functions/websockets) as the source of truth. Document runtime limits, Fluid compute, cold starts, max-duration disconnects, and that a connection is pinned to one function instance (not session affinity across reconnects).
+- [x] **V1-todo:** Write operator docs for websocket usage on Vercel Functions (see **Vercel Functions WebSockets** below). Link the beta page; do not copy it. Re-verify every config key and limitation against that page before the how-to ships.
+- [x] **V1-todo:** (Deferred — Terreno does not route AI through Expo server output; the guide tells operators to keep `/gpt/prompt` on the `@terreno/api` backend.) Can `@terreno/ai` SSE streaming work through `expo-server/adapter/vercel` without unacceptable buffering?
+- [x] **V1-todo:** (Deferred, owner Josh — GCS uploads are off on Vercel during the evaluation; Blob vs external GCS is future work.) Where do user file uploads land on an all-in-one Vercel deploy (Blob, external GCS, or other)?
+- [x] **V1-todo:** Compare all-in-one Vercel vs split (Vercel web + long-running backend) on cost, ops complexity, and preview-deployment ergonomics.
+- [x] **V2-todo:** If `server` output is documented, confirm Expo SDK ≥ 55+ requirements and list which Terreno features break under static/server export.
+- [x] **V3-todo:** If committing `vercel.json`, confirm CI/deploy wiring for `example-frontend` and `example-backend` and who owns the Vercel project.
+
+## Architecture
+
+### Interim topology (document until spike closes)
+
+Until V1 is decided, document this split layout — it is known to work:
+
+```mermaid
+flowchart LR
+  U["Users<br/>web + native"]
+  V["Vercel<br/>Expo web export<br/>(single output)"]
+  B["Long-running host<br/>@terreno/api + Socket.io"]
+  A["MongoDB Atlas<br/>replica set"]
+  S["Object storage<br/>uploads"]
+  U -->|"web"| V
+  U -->|"native"| B
+  V -->|"XHR + websocket"| B
+  B --> A
+  B --> S
+```
+
+The web deployment is static; dynamic operations go to the backend. Vercel needs SPA rewrites and build-time `EXPO_PUBLIC_API_URL`.
+
+### Candidate topology (if spike succeeds)
+
+```mermaid
+flowchart LR
+  U["Users<br/>web + native"]
+  VC["Vercel<br/>web + @terreno/api + storage"]
+  A["MongoDB Atlas<br/>replica set"]
+  U --> VC
+  VC --> A
+```
+
+**Do not document this as the recommended path until all V1 TODOs are checked.**
+
+### `vercel.json` for `single` output
+
+Per Expo's published configuration, the SPA case needs rewrites so client-side routes resolve:
+
+```json
+{
+  "buildCommand": "expo export -p web",
+  "outputDirectory": "dist",
+  "devCommand": "expo",
+  "cleanUrls": true,
+  "framework": null,
+  "rewrites": [{ "source": "/:path*", "destination": "/" }]
+}
+```
+
+### `vercel.json` for `server` output (advanced)
+
+Server output changes the shape: `dist/client` is served statically and a function handles everything else via `expo-server/adapter/vercel`, with `includeFiles` pulling in `dist/server/**`. Document only after V2 TODOs close. Flag the streaming caveat for `@terreno/ai` chat.
+
+### The preview-deployment problem
+
+Vercel gives every PR a unique origin. That breaks two things:
+
+1. **CORS** — `corsOrigin` in `setupServer` must accept the preview origin. Document a pattern (a function or regex matching `https://<project>-*.vercel.app`) rather than `corsOrigin: true`, and say plainly that `true` is not acceptable in production.
+2. **Better Auth `trustedOrigins`** — same problem, separate config, and OAuth redirect URIs must be registered per provider.
+
+This section ships regardless of V1 outcome.
+
+### `deploy-vercel` skill
+
+New skill at `.rulesync/skills/deploy-vercel/SKILL.md`:
+
+1. **Detect** — is this an Expo Router web app; what is `web.output` in the app config; is there an existing `vercel.json`.
+2. **Configure** — write or update `vercel.json` for the detected output mode; add the `vercel-build` script when using server output.
+3. **Wire the backend** — determine the backend URL (per finalized V1 decision), set `EXPO_PUBLIC_API_URL` as a Vercel environment variable per environment (production/preview/development), and print the CORS and `trustedOrigins` changes the user must make on the backend.
+4. **Deploy** — `vercel` / `vercel --prod`, with a confirmation gate before the first production deploy.
+5. **Verify** — load the deployed URL, confirm the app boots, confirm an authenticated request reaches the backend, and confirm the websocket connects.
+6. **Troubleshoot** — blank page (missing rewrites), 404 on refresh (same), API calls to localhost (`EXPO_PUBLIC_API_URL` not set at build time), CORS failure, websocket failure, buffered SSE on server output.
+
+The verification step must include the websocket check. A Terreno web app can look completely fine while realtime and live feature flags are silently broken.
+
+### Vercel Functions WebSockets (Public Beta) — required how-to
+
+Vercel now documents WebSockets on Vercel Functions as **Public Beta**: [WebSockets](https://vercel.com/docs/functions/websockets). Phase 0 and the how-to must treat that page as the vendor contract. The Terreno page is operator docs for `@terreno/api` (`RealtimeApp` / Socket.io) and `@terreno/syncdb` on that runtime — not a reprint of Vercel’s examples.
+
+**Ship** `docs/how-to/vercel-function-websockets.md`. Link it from `docs/how-to/deploy-web-to-vercel.md` and from `docs/how-to/websocket-integration.md`. Mark the feature **beta**; tell operators to re-read the Vercel page before production. Do **not** present all-in-one Vercel hosting as the recommended Terreno topology until the V1 TODOs close.
+
+The published page must cover the following (cite the Vercel URL; keep examples Terreno-shaped).
+
+#### What Vercel guarantees (as of the beta page)
+
+| Fact | Terreno implication |
+|------|---------------------|
+| A WebSocket starts as HTTP `GET` with `Upgrade`. Routing Middleware, rewrites, Firewall, and rate limits apply **before** the upgrade. | Document Firewall rules and rate limits on the Socket.io path. Better Auth / JWT handshake auth still runs on the upgrade, same as today. |
+| After upgrade, messages stay on the **same function instance** for the life of that connection. Fluid compute can multiplex many connections on one instance. | In-memory `socket.join` rooms work for one connection’s instance only. Cross-instance fan-out still needs the Redis adapter (`RealtimeApp` `adapter: "redis"` + `VALKEY_URL` / `REDIS_URL`). |
+| New connections are **not** guaranteed to hit the same instance. After a deploy, **new** connections may land on the new deployment while **existing** connections stay on the old one until they close. | Presence, rooms, counters, and pub/sub must live in Redis (or equivalent), not process memory. Change-stream watchers are per instance — spike must record cost and fan-out via Redis. |
+| The connection closes when the function hits **max duration**. Clients must reconnect, resubscribe, and reload state. | Document exponential backoff. `@terreno/rtk` `useSocketConnection` already reconnects; `@terreno/syncdb` must resubscribe after reconnect. Name the debug flag (`WEBSOCKETS_DEBUG` at time of writing). |
+| Socket.IO is supported if the **client uses the WebSocket transport only** (`transports: ['websocket']`). Long-polling is not the Vercel path. | Terreno already documents `transports: ["websocket"]` in `docs/how-to/websocket-integration.md`. The Vercel page must restate that long-polling is forbidden on Functions. |
+| Express (and Hono, etc.) can serve sockets by attaching `ws` / Socket.IO to a Node HTTP server and **`export default server`**. | Spike must map this to `TerrenoApp.build()` (or equivalent) vs `start()`. Today the process listens; a Function entry exports the HTTP server. Record the exact adapter file (`api/server.ts` or Expo `expo-server/adapter/vercel` is a **different** path — do not conflate web SSR with the API process). |
+| Fluid compute is required. It is the default for new Vercel projects created on or after 2025-04-23. | Document how to confirm Fluid is on. Without it, do not host Terreno sockets on Functions. |
+| Usage is billed as Function time while the socket is open, plus Fast Data Transfer / Fast Origin Transfer. | Include a cost warning in the all-in-one vs split comparison (V1 cost TODO). Long-lived sync sockets are not “free HTTP”. |
+
+#### Required operator sections
+
+1. **When to use this** — hosting `@terreno/api` (or a dedicated socket process) on Vercel Functions. When **not** to: static Expo `single` output on Vercel talking to a long-running backend (interim topology); native EAS; treating the Vercel **AI SDK** as this feature.
+2. **Prerequisites** — Fluid compute; Function max duration set high enough for the product; Redis if more than one instance or rolling deploys; MongoDB Atlas replica set still required for change streams (unchanged).
+3. **Socket.IO on a Function** — export an HTTP server with Socket.IO attached; client `path` and `transports: ['websocket']`; Terreno `RealtimeApp` + `SyncApp` still registered. Link Vercel’s Socket.IO snippet as the vendor shape, then show the Terreno registration, not a second copy of `ws` echo servers.
+4. **Reconnects** — max-duration close; backoff; resubscribe; “looks fine while sync/flags are dead”.
+5. **State** — no in-memory rooms across instances; Redis; deploy split-brain (old + new deployments both alive).
+6. **Limits and pricing** — pointer to Vercel’s limits/pricing on that page; Function duration while connected.
+7. **Verify** — DevTools Network → WS; `WEBSOCKETS_DEBUG`; confirm the client did **not** fall back to polling.
+
+#### Explicitly out of scope for this page
+
+- Next.js `experimental_upgradeWebSocket` from `@vercel/functions` (not Terreno’s server).
+- Bun `Bun.serve()` / Python FastAPI examples (link the Vercel page; do not re-home them).
+- Replacing MongoDB change streams with Vercel Queues or Vercel KV pub/sub as the Terreno protocol.
+
+## Models / APIs / Notifications / UI
+
+None.
+
+## Phases
+
+0. **Spike** — close all **Open TODOs** above; record decision on V1/V2/V3 in this file; write `docs/how-to/vercel-function-websockets.md` from the beta contract (may ship before V1 is decided, as constraints — not as a topology recommendation).
+1. **How-to guide** — `single` output, clone → public URL (topology per V1 outcome); link the websocket Functions page.
+2. **Preview deployments and origins** — CORS, `trustedOrigins`, OAuth redirects.
+3. **Advanced: server output** — only if V2 is approved.
+4. **Skill** — author and generate mirrors.
+5. **Validate** — deploy examples per V3 outcome.
+
+## Feature Flags & Migrations
+
+None.
+
+## Not Included / Future Work
+
+- EAS native distribution (separate skill).
+- Multi-region Vercel edge configuration.
+
+## Files to Create / Modify
+
+**Create**
+
+- `docs/how-to/deploy-to-vercel.md` (named for both web and backend; the task list's `deploy-web-to-vercel.md`)
+- `docs/how-to/vercel-function-websockets.md` (Vercel Functions WebSockets beta; may ship during Phase 0)
+- `.rulesync/skills/deploy-vercel/SKILL.md`
+
+**Modify (after V3 decision)**
+
+- `example-frontend/vercel.json` (candidate)
+- `example-backend/vercel.json` (candidate — only if all-in-one path wins)
+
+**Modify (with Task 0.2)**
+
+- `docs/how-to/README.md`
+- `docs/how-to/websocket-integration.md`
+
+## Task List
+
+See [`docs/tasks/deploy-to-vercel.md`](../tasks/deploy-to-vercel.md).
+
+## Acceptance Criteria
+
+- [ ] All **Open TODOs** are closed with a written decision on V1, V2, and V3.
+- [ ] `docs/how-to/deploy-to-vercel.md` matches the decided topology (not the interim split doc if all-in-one wins, and vice versa).
+- [ ] `docs/how-to/vercel-function-websockets.md` exists, cites [Vercel Functions WebSockets](https://vercel.com/docs/functions/websockets), covers Fluid compute, websocket-only Socket.IO, instance pinning, max-duration reconnects, Redis for cross-instance state, Function billing while connected, and Terreno `RealtimeApp` / syncdb — without recommending all-in-one until V1 is decided.
+- [ ] Preview-deployment CORS and `trustedOrigins` guidance is present.
+- [ ] `deploy-vercel` skill includes websocket verification in its checklist.
+- [ ] Vercel AI SDK vs Vercel hosting disambiguation appears in both `@terreno/ai` and deployment docs.

package/dist/docs/versioned/57.21.0/implementationPlans/durable-agent-harness.md

@@ -0,0 +1,340 @@
+# Implementation Plan: Durable agent harness (`@terreno/ai/harness`)
+
+**Status:** Approved  
+**Roadmap:** Area=`ai`, Target=`Next`, Impact=`Feature`  
+**Branch:** `durable-agent-harness` (stacked on `cursor/ai-observability-ip-64ca`)  
+**Owner:** unassigned  
+**Created:** 2026-10-02  
+**Task list:** [docs/tasks/durable-agent-harness.md](../tasks/durable-agent-harness.md)  
+**Inspiration:** [Pi Durable](https://earendil.com/posts/pi-durable/) (concepts only; no dependency)  
+**Depends on:** [ai-observability.md](./ai-observability.md) (`ObsTrace` / `ObsSpan`, `ObservabilityApp`), [job-queues.md](./job-queues.md) (`@terreno/jobs`, optional)  
+**Supersedes in part:** [ai-agents-and-failover.md](./ai-agents-and-failover.md) — its `Agent` and failover ideas become harness primitives; its "no human-in-the-loop / no resume / no sub-agents" non-goals are reversed here.
+
+## Goal
+
+One easy-to-use harness inside `@terreno/ai` that runs long-lived agentic work durably:
+deterministic multi-phase workflows, LLM agent loops, tools, subagents, human approvals,
+retries, and crash recovery — all audited. It must be strong enough to build an
+OpenClaw/Hermes-class personal agent, a coding agent, and a regulated clinical automation
+on the same primitives.
+
+Today a chat turn lives inside one HTTP request (`ai/src/routes/gpt.ts`), history is saved
+only after the stream ends, and a crash loses the turn. `@terreno/jobs` is durable but a
+job is one opaque handler with no checkpoints. The harness fills that gap.
+
+## Non-Goals
+
+- Depending on or wrapping `@earendil-works/pi-durable`.
+- Journaled/replayed `step.run` workflows (Temporal/Inngest style). Phases only.
+- Socket.io streaming for agent output. Streaming is SSE only.
+- Auto-migrating in-flight runs across task versions.
+- Pluggable non-Mongo storage (SQLite/JSONL/memory). Mongo for every runner.
+- Field-level encryption, retention/redaction policies, e-signatures (future work).
+- Changing `/gpt/prompt` behavior before Phase 4.
+
+## Decisions
+
+| Question | Decision |
+|----------|----------|
+| Build our own harness or adopt pi-durable? | Own harness on Mongo + the Vercel AI SDK (`ai`), copying Pi's concepts (tasks, checkpoints, replay-safe tools, hooks, memos, documents, compaction, forks). |
+| Where should the harness live? | Inside `@terreno/ai`, as subpath `@terreno/ai/harness`. |
+| How should the `@terreno/jobs` dependency be wired? | Optional peer. Jobs integration is optional; a single process can own execution. |
+| Who executes a run? | Pluggable `Runner`: `InProcessRunner` (single owner, default) and `JobsRunner` (leased, multi-instance). |
+| How do durable code steps resume after a crash? | Explicit phases (Pi-style `defineTask` state machine); each phase commits a checkpoint. |
+| Where does state live for the single-process owner? | Mongo for both runners; switching runners needs no migration. |
+| What compliance guarantees ship in v1? | Immutable step/event audit log, built on the AI observability branch's `ObsTrace` / `ObsSpan`. |
+| How does the audit log relate to `ObsTrace` / `ObsSpan`? | `ObsSpan` **is** the audit log for harness work. |
+| How do we guarantee a span exists for every checkpoint? | Checkpoint commit and its `ObsSpan` write happen in one Mongo transaction. Langfuse/OTel stay best-effort after commit. |
+| How does this sequence against the observability branch? | Branch off `cursor/ai-observability-ip-64ca` and rebase as it lands. |
+| What happens on resume when replay was not declared? | Never auto-replay. Interrupted tools are reported to the model; interrupted phases go to `interrupted`. Opt in with `replay: "safe"`. |
+| Which features land in Phase 1 with the clinical tracer? | Tasks, phases, agent loop, tools + replay, retries, audit, **hooks + memos + approvals**, **subagents + ownership tree**. |
+| How do clients watch runs live? | SSE only, resumable via `Last-Event-ID`; a separate submit endpoint sends input/steer. |
+| How does a clinician sign off on an approval? | An approval inbox in `@terreno/admin-frontend`. |
+| Who can see and act on an approval? | Per-approval `approvers` (modelRouter permission shape); inbox lists only what the user may approve; default `IsAdmin`; mountable outside admin chrome. |
+| What happens to `/gpt/prompt`? | Untouched in Phase 1; rebuilt on the harness in Phase 4. |
+| Where do coding-agent execution environments land? | Phase 1 defines the `ExecutionEnv` interface only; Phase 5 ships Node env, coding tools, sandbox adapter. |
+| Task definition changes while runs are in flight? | Runs pin `name@version`; old versions must stay registered; startup fails loudly otherwise. |
+
+### Recorded assumptions (low-risk, chosen by convention)
+
+- **Interrupted model requests are re-requested.** A model call has no side effect, so a
+  crash mid-stream marks the partial response aborted and the turn retries the request.
+- **Model-call retries:** 429/5xx/network errors retry with exponential backoff + jitter
+  (default 3 attempts), then an optional `fallbackModels` list. A thrown phase retries
+  under the task's `retry` policy (default `maxAttempts: 3`); exhausted → `failed`.
+- **Token deltas are persisted, coalesced** (~250 ms) as `HarnessEvent` rows with a TTL so
+  any instance can serve an SSE resume. Committed messages are permanent.
+- **Cross-instance SSE** tails `HarnessEvent` with a Mongo change stream (replica set is
+  already required).
+- **Non-local trace sinks** (Langfuse, OTel) receive the full trace once a root task is
+  terminal, best-effort, via `ObservabilityApp.exportTrace`.
+- **Approvals inbox filtering** evaluates `approvers` per pending approval in memory
+  (pending sets are small); pagination applies after filtering.
+
+## Architecture
+
+```
+            app code: defineTask / defineAgent / defineTool / defineExtension
+                                     |
+                          Harness (registry + storage)
+         +-------------+-------------+--------------+----------------+
+         |             |             |              |                |
+     Task engine   Agent loop     Hooks/memos    Approvals       Event log
+     (phases,      (built-in      (extensions)   (wait + inbox)  (SSE source)
+      checkpoints,  task)
+      ownership)
+         |
+     runtime.commit()  ── one Mongo transaction ──>  HarnessTask/Message/Memo/Event
+                                                     + ObsSpan / ObsTrace (audit)
+         |
+       Runner:  InProcessRunner (owner lease)  |  JobsRunner (@terreno/jobs, optional)
+```
+
+### Core concepts
+
+| Concept | What it is |
+|---------|-----------|
+| **Task** | Durable unit of work: `name@version`, `input`, `state`, `phase`, status, ownership. Everything (model request, tool call, workflow, subagent) is a task. |
+| **Phase** | A named async function of a task. It does work and calls `rt.commit()` with the next phase or a terminal outcome. A phase is the replay boundary. |
+| **Checkpoint** | The committed `{phase, state}`. Resume always starts the current phase from its checkpoint. |
+| **Conversation** | A transcript plus agent config (model, instructions, tools, extensions). Root, or owned by a task (subagent). |
+| **Agent** | `defineAgent` config. Running an agent = the built-in `terreno.agent.turn` task on a conversation. |
+| **Tool** | `defineTool` with zod params and `replay: "safe" | "never"` (default `never`). Runs as a child task of the turn. |
+| **Extension** | Named bundle of prompt sections, tools, hooks, and tool wraps. |
+| **Hook** | `beforeModelRequest`, `beforeTool`, `afterTool`. Can rewrite, block, or annotate. |
+| **Memo** | `rt.memo(key, value?)` — durable decision storage scoped to a task, survives restarts. |
+| **Approval** | `rt.approval(key, {...})` — creates a `HarnessApproval`, waits for decision, audits the decider. |
+| **Event** | Append-only `HarnessEvent` rows with per-conversation/task `seq`; the SSE source. |
+| **Runner** | Decides who executes runnable tasks and when. |
+
+### Commit and audit
+
+`rt.commit(fn)` opens a Mongo transaction that, fenced by the task's lease token:
+
+1. Updates `HarnessTask` (`phase`, `state`, status, `attempt`, outcome).
+2. Inserts any `HarnessMessage`, `HarnessMemo`, `HarnessApproval`, `HarnessEvent` rows.
+3. Inserts the `ObsSpan` for the finished work and updates the root `ObsTrace`.
+
+If the lease token no longer matches (another owner took over) the transaction aborts and
+the phase result is discarded. `Harness` construction throws when the local observability
+plugin (`createLocalObservabilityPlugin`) is not registered or the connection is not a
+replica set.
+
+Span mapping: root task → `ObsTrace` + `CHAIN` root span; phase → `CHAIN`; model request →
+`LLM` (usage, cost via `priceMap`); tool → `TOOL`; subagent → `AGENT`; approval decision →
+`CHAIN` named `approval:<key>` with `decidedBy`, `decision`, `reason` in `output`. Spans
+are never updated after insert except `endedAt`/`status` on the same commit.
+
+### Replay and interruption
+
+On resume, a task whose status is `running` with an expired lease was interrupted:
+
+| Interrupted thing | `replay: "safe"` | default (`never`) |
+|---|---|---|
+| Phase | Re-run the phase from its checkpoint | Status `interrupted`; await `resolveInterrupted` (`retry` / `abort` / `complete`) |
+| Tool call | Re-run the tool | Tool result = "interrupted, not retried"; agent loop continues |
+| Model request | Always re-requested (assumption above) | — |
+
+### Runners
+
+| Runner | Ownership | Use |
+|---|---|---|
+| `InProcessRunner` (default) | One process holds the `HarnessOwner` lease (heartbeat). A second process gets `standby` and takes over on lease expiry. Polls runnable tasks; wakes on `runAt` and events. | Local agents, single-instance apps, tests |
+| `JobsRunner` (`@terreno/ai/harness/jobsRunner`, optional peer `@terreno/jobs`) | Each runnable phase → job `terreno.harness.phase`, idempotency key `taskId:phase:attempt:claims` (`claims` counts claims, so a phase revisited by an event wake or crash replay gets a new job; jobs idempotency keys are permanent). Handler acquires the task lease (fenced, heartbeat) and runs one phase. The task lease, not the job lock, is the authority. | Multi-instance Cloud Run |
+
+### Ownership tree
+
+Tasks are owned by a conversation or a task. `abort(taskId)` aborts owned tasks
+bottom-up, running each task's `abort` handler (compensations) before marking it
+`aborted`. `background: true` tasks belong to a conversation but are not part of its
+current turn; a turn abort leaves them running.
+
+### Versioning
+
+Each task records `name` + `version`. The runner only resumes a task on a registered
+handler for exactly that version. `Harness.start()` scans non-terminal tasks and throws
+listing every unregistered `name@version`.
+
+## Models
+
+All in `ai/src/harness/models/`, types in `ai/src/types/harness.ts`. Every field has
+`description`; `strict: "throw"`.
+
+| Model | Key fields |
+|---|---|
+| `HarnessTask` | `name`, `version`, `input`, `state`, `phase`, `status` (`pending` \| `running` \| `waiting` \| `interrupted` \| `completed` \| `failed` \| `aborted`), `ownership {kind: conversation\|task\|root, id}`, `background`, `attempt`, `retry {maxAttempts, backoffMs, maxBackoffMs}`, `runAt`, `lease {owner, token, expiresAt}`, `waiting {kind: event\|tasks\|sleep, key, taskIds, policy, timeoutAt}`, `outcome {status, result, error}`, `traceId`, `rootTaskId`, `requestId` (unique sparse), `userId` |
+| `HarnessConversation` | `ownership`, `agent {name, model, instructions, tools[], extensions[]}`, `status` (`idle` \| `busy`), `activeTurnTaskId`, `queued[]`, `userId`, `seq` |
+| `HarnessMessage` | `conversationId`, `seq`, `role`, `parts[]`, `toolCallId`, `toolName`, `turnTaskId`, `aborted` |
+| `HarnessMemo` | `taskId`, `key`, `value` — unique `(taskId, key)` |
+| `HarnessApproval` | `taskId`, `rootTaskId`, `definitionKey` (`name@version:key`), `title`, `summary`, `payload`, `status` (`pending` \| `approved` \| `rejected` \| `expired`), `decidedBy`, `decidedAt`, `reason`, `expiresAt` |
+| `HarnessEvent` | `streamId` (conversation or task), `seq`, `type`, `payload`, TTL for `delta` events |
+| `HarnessOwner` | singleton lease for `InProcessRunner`: `owner`, `expiresAt` |
+
+## APIs
+
+### Library (`@terreno/ai/harness`)
+
+```typescript
+import {
+  Harness, defineTask, defineAgent, defineTool, defineExtension, hook, section,
+  approvalGate, InProcessRunner,
+} from "@terreno/ai/harness";
+
+const summarizer = defineAgent({
+  name: "clinic.summarizer",
+  model: {provider: "anthropic", modelId: "claude-sonnet-5-5"},
+  instructions: "Summarize the chart for a clinician. Cite sources.",
+  tools: [],
+  fallbackModels: [{provider: "anthropic", modelId: "claude-haiku-4-5-20251001"}],
+});
+
+const intakeSummary = defineTask<{patientId: string}, IntakeState, {status: string}>({
+  name: "clinic.intakeSummary",
+  version: 1,
+  initial: () => ({phase: "fetch"}),
+  retry: {maxAttempts: 3},
+  phases: {
+    fetch: {
+      replay: "safe",
+      run: async (task, rt) => {
+        const chart = await ehr.getChart(task.input.patientId);
+        await rt.commit({phase: "summarize", state: {chart}});
+      },
+    },
+    summarize: {
+      replay: "safe",
+      run: async (task, rt) => {
+        const summary = await rt.runAgent(summarizer, {input: task.state.chart, output: SummarySchema});
+        await rt.commit({phase: "review", state: {...task.state, summary}});
+      },
+    },
+    review: {
+      replay: "safe",
+      run: async (task, rt) => {
+        // Approvers are declared on the definition: approvals: {"clinician-signoff": {approvers: [Permissions.IsAdmin, isClinician]}}
+        const decision = await rt.approval("clinician-signoff", {
+          title: "Sign off intake summary",
+          payload: task.state.summary,
+          timeout: {hours: 24},
+        });
+        await rt.commit(decision.approved ? {phase: "write"} : {terminal: {status: "completed", result: {status: "rejected"}}});
+      },
+    },
+    write: {
+      // default replay: "never" — an interrupted EHR write goes to `interrupted`
+      run: async (task, rt) => {
+        await ehr.writeNote(task.input.patientId, task.state.summary, {idempotencyKey: `note-${task.id}`});
+        await rt.commit({terminal: {status: "completed", result: {status: "filed"}}});
+      },
+    },
+  },
+  abort: async (task, rt) => { /* compensations */ },
+});
+
+const harness = await Harness.open({
+  models: createModelFn,
+  registry: [intakeSummary, summarizer, ClinicExtension],
+  runner: new InProcessRunner(),
+});
+await harness.start();
+const run = await harness.createTask(intakeSummary, {patientId}, {requestId: `intake-${patientId}`});
+```
+
+Runtime (`rt`) surface in Phase 1: `commit`, `memo`, `runAgent`, `createTask`,
+`waitForTasks(ids, {policy: "all" | "failFast"})`, `waitFor(event, {timeout})`,
+`sleep(duration)`, `approval`, `output` (stream text to the event log), `signal`, `env`
+(`ExecutionEnv`, interface only in Phase 1).
+
+Harness surface: `open`, `start`, `stop`, `createTask`, `conversation(id)`,
+`createConversation`, `sendEvent(taskId, event, payload)`, `abort(taskId)`,
+`resolveInterrupted(taskId, action)`.
+
+### HTTP (`HarnessApp` plugin)
+
+| Method | Path | Behavior |
+|---|---|---|
+| POST | `/harness/conversations/:id/submit` | instanceAction. `{content, requestId, whenBusy: "queue" \| "steer"}`; idempotent on `requestId` |
+| GET | `/harness/conversations/:id/events` | SSE; honors `Last-Event-ID`; replays then tails |
+| GET | `/harness/tasks/:id/events` | SSE for a task subtree |
+| POST | `/harness/tasks/:id/abort` | instanceAction; owner or admin |
+| POST | `/harness/tasks/:id/resolveInterrupted` | instanceAction; admin; `{action: "retry" \| "abort" \| "complete", reason}` |
+| GET | `/harness/approvals` | list pending approvals the caller may approve |
+| POST | `/harness/approvals/:id/approve` \| `/reject` | instanceActions; `{reason}` (required on reject); audited span |
+
+SSE is a listed exception to the modelRouter-actions rule; everything else is
+`modelRouter` collection/instance actions.
+
+## Notifications
+
+None in Phase 1. `rt.approval` accepts an optional `notify` callback so apps can send via
+`@terreno/comms`; the harness does not depend on comms.
+
+## UI
+
+- **Approvals inbox** (`@terreno/admin-frontend`, `widgets/harness/`): list of pending
+  approvals (title, task, age, expires), detail with `payload` rendered as JSON/markdown,
+  Approve / Reject with reason. Registered as admin custom screen `harness-approvals`
+  (group "AI Harness"), and exported as `HarnessApprovalInbox` for mounting outside admin
+  chrome. Uses `@terreno/ui` only.
+- **Runs screen** (Phase 2): task list/tree, status filters, abort, resolve interrupted.
+- Spans render in the existing observability Traces screens.
+
+## Phases
+
+| Phase | Ships | Proof |
+|---|---|---|
+| **1 — Durable clinical tracer** | Engine, phases, checkpoints, `InProcessRunner`, transactional `ObsSpan` audit, replay semantics, retries, agent loop, tools, subagents + ownership tree, extensions/hooks/memos, waits/events, approvals + inbox, SSE + submit, version pinning, example-backend `clinic.intakeSummary` | Crash-and-resume e2e; inbox Playwright screenshots; span tree assertions |
+| **2 — Multi-instance** | `JobsRunner`, two-worker fencing, Runs admin screen | Two-process race tests; screenshots |
+| **3 — Long context** | Background compaction, `reset`/handoff, conversation forks, `defineDoc` documents (rewindable, fork `asOf`) | Bun tests on 10k-message conversations; fork isolation tests |
+| **4 — Durable chat + multiplayer** | `/gpt` rebuilt on harness, `GptHistory` mapping, steering, foreground/background abort, `viewState` over SSE, rtk `useHarnessConversation`, `GPTChat` adapter | Kill backend mid-chat, reconnect resumes; two clients see same view |
+| **5 — Coding agents + malleability** | `NodeExecutionEnv`, `CodingTools` (read/write/edit/bash), remote sandbox adapter, registry hot-replace | Coding agent fixes a failing test in a fixture repo after a crash |
+
+## Feature Flags & Migrations
+
+- No feature flag; the harness is opt-in by registering `HarnessApp`.
+- New collections only. Phase 4 adds a migration mapping `GptHistory` → `HarnessConversation`/`HarnessMessage` (human gate at Phase 4 start: keep dual-read or one-shot migrate).
+
+## Activity Log & User Updates
+
+Audit lives in `ObsSpan`. Changelog entry per phase in `changelog/`.
+
+## Not Included / Future Work
+
+Field encryption, retention/redaction, e-signature approvals, Socket.io streaming,
+non-Mongo storage, journaled steps, in-flight version migration.
+
+## Files to Create / Modify
+
+| Path | Change |
+|---|---|
+| `ai/src/harness/` | new: `harness.ts`, `defineTask.ts`, `runtime.ts`, `commit.ts`, `agentLoop.ts`, `defineAgent.ts`, `defineTool.ts`, `extensions.ts`, `approvals.ts`, `events.ts`, `modelCall.ts`, `runners/inProcessRunner.ts`, `runners/jobsRunner.ts`, `harnessApp.ts`, `routes/*.ts`, `models/*.ts` |
+| `ai/src/types/harness.ts` | new model + API types |
+| `ai/package.json` | `./harness`, `./harness/jobsRunner` exports; optional peer `@terreno/jobs` |
+| `admin-frontend/src/widgets/harness/` | new approvals inbox (Phase 1), runs screen (Phase 2) |
+| `example-backend/src/harness/` | `clinic.intakeSummary`, fake EHR, registration in `server.ts` |
+| `example-frontend` | admin route for inbox; `bun run sdk` |
+| `docs/explanation/durable-agent-harness.md` | new |
+| `docs/reference/ai-harness.md` | new |
+| `docs/how-to/build-a-durable-workflow.md` | new |
+| `docs/reference/ai.md`, `docs/reference/admin-frontend.md` | link + inbox section |
+
+## Task List
+
+[docs/tasks/durable-agent-harness.md](../tasks/durable-agent-harness.md)
+
+## Acceptance Criteria
+
+| # | Criterion | Verification |
+|---|---|---|
+| AC1 | A two-phase task runs to completion and every checkpoint has exactly one `ObsSpan`, written in the same transaction | bun test: abort the transaction mid-commit → neither task update nor span exists |
+| AC2 | Killing the owner mid-run resumes from the last checkpoint on a fresh `Harness` | bun test: throw-on-signal simulation + new harness instance; e2e: kill example-backend during `summarize`, restart, run completes |
+| AC3 | Default-replay phases and tools never re-run after interruption; `replay: "safe"` ones do | bun test matrix over phase/tool × replay |
+| AC4 | Thrown phases retry with backoff and land `failed` after `maxAttempts`; model 429/5xx retries then falls back | bun test with fake model provider |
+| AC5 | Aborting a root task aborts owned subtasks bottom-up and runs `abort` handlers | bun test ordering assertion |
+| AC6 | `beforeTool` hook can block a tool; memo decision survives restart | bun test |
+| AC7 | `rt.approval` blocks until a permitted user approves; unpermitted users get 403 and do not see it in the inbox; decision span records `decidedBy` | supertest |
+| AC8 | Approvals inbox lists, approves, and rejects in admin-frontend | Playwright run with screenshots in `/opt/cursor/artifacts/` |
+| AC9 | SSE reconnect with `Last-Event-ID` replays missed events exactly once, from any instance | bun test with two `HarnessApp` instances on one replica set |
+| AC10 | Starting with an in-flight unregistered `name@version` throws listing it | bun test |
+| AC11 | `JobsRunner`: two workers never run the same phase concurrently; stale lease owner's commit is rejected | bun test (Phase 2) |
+| AC12 | Docs explain, reference, and how-to every public API in the slice that ships it | `docs-audit` skill run shows no undocumented harness exports; Roast reads each page against the shipped API |