@ultimat3/cli 29.4.0 → 30.0.0

package/wiki/MCP-And-AI.md

@@ -0,0 +1,753 @@
+# MCP and AI
+
+The differentiator. Not a chat widget, not an "AI SDK integration" — the framework is built so an agent can read it, drive it, and verify its own work, and so the apps it generates have the same property.
+
+`As of 2026-08`. Stable API — semver from here ([Upgrading](Upgrading)). The MCP registry, wire protocol, dev-tool catalog, read-only SQL guard, and action projection are built, and so are the four that used to be contracted: `llm()` is an action factory ([`packages/ai/src/llm.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/ai/src/llm.ts)), prompts are versioned, `postgresVectorStore()` fuses pgvector cosine with Postgres FTS via RRF, and evals gate on a committed baseline inside `x verify`'s `eval` step.
+
+## Built-in MCP dev server
+
+`x mcp serve` is the dev-tool server, over stdio or HTTP; `x dev` does not start it. Point any MCP client at it and the agent stops guessing. A new app's `.mcp.json` already names it — [Client config](#client-config).
+
+The whole catalog, spelled exactly as each tool must be called; `bunx x mcp tools --json` reads it off the server rather than a count here. No aliases; renaming one is a major.
+
+| Tool | Introspects / does | Replaces the agent's usual guess |
+|---|---|---|
+| `routes.list` | the `x routes --json` table: pages (render mode, hydrate, offline, budget), then every action, query and `runtime.ts` route | grepping a router directory |
+| `schema.describe` | entities with columns, types and invariants | reading migration files in order |
+| `policies.list` | every `policy`: permission, subject, where it is enforced | "is this endpoint protected?" |
+| `actions.describe` | every action **and query**: input/output schema, policy, cache tags, MCP exposure | reading `api/` by hand |
+| `jobs.inspect` | job definitions, retry policy and steps; omit `name` for all | reading `jobs.ts` and guessing the retry |
+| `queue.depth` | pending, running and failed counts per queue | tailing a worker to see if it keeps up |
+| `manifest.read` | the whole `x.manifest.json`, as text | ten separate reads |
+| `errors.explain` | `X_*` code → cause, exact fix command, docs URL | web search |
+| `graph.query` | `{ node }` (a file path, a primitive name or a node id) → every route, action, query, job, MCP tool, published operation, test and budget a change to it reaches, each with the provenance of the edges that reached it — `x impact`'s answer. `{ from, to }` → the chain of edges linking two nodes — `x graph --from --to`'s. An unknown name is `X_GRAPH_NODE_UNKNOWN` naming the nearest three; any other combination of arguments is an `isError` naming the two shapes. Rebuilt per call, so it answers for the code as it is now | grepping importers before an edit, and missing the route that reads it through a query |
+| `db.query` | **read-only** SQL, 100-row default and 1000-row maximum, `EXPLAIN` on request | inventing a query and hoping |
+| `db.migrate` | apply pending migrations **in a branch DB only** | mutating the dev database |
+| `tests.run` | run the suite or a substring filter, structured results | parsing terminal output |
+| `verify.run` | the whole gate; `fix: true` applies safe autofixes | guessing whether the work is shippable |
+| `logs.tail` | last N structured log lines, filterable by runtime role | scrollback archaeology |
+| `ui.shot` | photograph one route at a named viewport (`phone`/`tablet`/`desktop`, or `width`+`height`) in `light` or `dark` (`theme`: `prefers-color-scheme` is emulated **and** the scheme is stored as the visitor's choice under `THEME_STORAGE_KEY` before navigation, so an app with `theme.defaultMode: 'dark'` still captures light when asked — `As of 2026-09-19`), against the running `x dev`; returns the PNG **path** and the same verdict `x shot` writes — console, page errors, refused requests, whether every island mounted. Refuses a route with no `budget.js` | a hand-written puppeteer script, and a picture nobody judged |
+| `ui.island` | `x shot --island <name> [--state]` as a tool: every declared state photographed and judged | the same, per component |
+| `ui.inspect` | DOM, computed-style and accessibility facts for up to 20 selectors in **one** navigation of a route: per match the tag, text (≤200 chars), box, visibility, attributes (≤20), the computed `styles` you name (≤32, kebab-case; the rest land in `droppedStyles`) and with `a11y: true` the browser-computed role and name; plus `title`, `data-theme`, the focused element, the island count, console and page errors. ≤25 matches per selector, 64 KB on the wire (matches are dropped from the LAST selectors first, `truncated: true`); an unparsable selector answers `valid: false`. Takes the same PNG and verdict as `ui.shot` under `inspect/`; same budget gate, same `theme` semantics (the reported `data-theme` is the one asked for, on a dark-default app too) | `page.evaluate` scripts written per question, and a browser launched per question |
+| `ui.interact` | drive one route through at most 12 steps — `{click}`, `{type: {selector, text}}` (≤500 chars), `{press}` (a key chord), `{focus}`, `{wait: ms ≤ 5000 \| selector}` (waited visible) — then photograph it (`fullPage` defaults to **false**: a dialog is judged on the fold) and, with an `inspect` block, read the same facts `ui.inspect` reads, in **one** navigation. Same `theme` semantics as `ui.shot`. After every step the islands settle again and one poll interval passes for the CSS transition; per step you get `{ index, kind, ms, navigated, url }`. Refuses whole, never trims: a list over the bounds (`X_UI_INTERACT_STEPS_INVALID`), typing into `<input type="password">` before any keystroke (`X_UI_INTERACT_SECRET_FIELD`), a step that leaves the dev server's origin (`X_UI_INTERACT_LEFT_APP`; a same-origin navigation is allowed and reported `navigated: true`), a step the driver refused (`X_UI_INTERACT_STEP_FAILED`, `meta.step` names it). PNG and verdict under `interact-<hash of the steps>/`, never over `ui.shot`'s; `ok` is the verdict's, except that a navigation a step caused is not the redirect the verdict fails a capture for | a puppeteer script per scene, and a picture of the closed state |
+| `ui.diff` | compare two PNGs the other `ui.*` tools wrote — `before`/`after` relative to the app root and confined to `.x/shot/` (`X_UI_DIFF_PATH_OUTSIDE` otherwise, symlinks resolved) — pixel by pixel: `changedPixels`, `changedPercent` (two decimals), `changedBox` (the bounding box, or `null`) and the path of a diff PNG written beside `after` (`diff-<hash8 of before>.png`, or `out`): the `after` capture faded to a quarter over grey, changed pixels solid red. A pixel is changed when any channel moved by more than `threshold × 255` (default `0.1`); no anti-alias detection. Two sizes are `X_UI_DIFF_SIZE_MISMATCH`, a missing file `X_UI_DIFF_FILE_MISSING`. **No browser, no dependency** | a pixelmatch install, or two pictures eyeballed |
+
+| Class | Tools | Exposure |
+|---|---|---|
+| read | `routes.list`, `schema.describe`, `policies.list`, `actions.describe`, `jobs.inspect`, `queue.depth`, `manifest.read`, `errors.explain`, `graph.query`, `ui.diff` | scope `dev:read`, unrestricted in dev — `ui.diff` reads files, and sits here because it reads only `.x/shot/` (resolve + prefix check, symlinks resolved) and launches nothing |
+| gated read | `db.query`, `logs.tail` | scope `db:read` / `dev:logs` |
+| executes code | `tests.run`, `verify.run`, `ui.shot`, `ui.island`, `ui.inspect`, `ui.interact` | scope `dev:test`; all six declare `destructive: true`, so none is metered as read chatter — the four `ui.*` tools launch a browser |
+| write | `db.migrate` | scope `db:migrate`, **branch environments only** |
+
+None of them is exposed in `ROLE=web`. `db.query` accepts one statement, whose leading keyword must be `SELECT`/`WITH`/`EXPLAIN`/`SHOW`/`TABLE`/`VALUES` — necessary, never sufficient. Batches, any write keyword at statement level (a data-modifying CTE included), locking clauses (`FOR UPDATE`/`FOR SHARE`), `EXPLAIN ANALYZE`, and whole function families matched by prefix of the called name, quoted and schema-qualified spellings included — file access (`pg_read_*`, `pg_ls_*`, `lo_*`, `dblink`), locks (`pg_advisory_*`), session settings (`set_config`), sleeps (`pg_sleep*`), catalog writes (`pg_import_*`) and the functions that run a query handed to them as text (`query_to_xml*`, `cursor_to_xml*`, `table_to_xml*`, `schema_to_xml*`, `database_to_xml*`, `ts_stat`, `ts_rewrite`) — are **refused**, not discouraged — `X_MCP_QUERY_REJECTED`, enforced before the host sees the string. What the scan cannot read is refused, never decoded: a Unicode-escaped quoted identifier (`U&"…"`) is `X_MCP_QUERY_REJECTED` (`As of 2026-10`). A keyword is matched as Postgres lexes an identifier, so a column `set2` is a column, not `SET`. Its Postgres SELECT-only role is conditional on the connection's own rights; the answer's `guards` array names the defences that engaged. `db.migrate` refuses a target that is not a branch database — `X_MCP_NOT_BRANCH_DB`.
+
+
+### Client config
+
+`x new` writes one client config, `.mcp.json` at the app root — the format several MCP clients read — and no per-vendor directory: a second copy is a second thing to drift.
+
+```json
+{
+  "mcpServers": {
+    "myapp": {
+      "command": "bunx",
+      "args": ["@ultimat3/cli", "mcp", "serve", "--transport", "stdio"]
+    }
+  }
+}
+```
+
+| Client wants | Give it |
+|---|---|
+| `.mcp.json` | nothing: it reads this file |
+| a command and its arguments | `bunx @ultimat3/cli mcp serve --transport stdio`, started in the app's root — the command needs an app (`X_NOT_IN_APP` otherwise) |
+| a URL | `bunx x mcp serve --transport http --port 9229`, then `http://127.0.0.1:9229/mcp` and the bearer token it prints |
+| its own config file | the same command, in that file's own syntax; the framework generates none |
+
+**`bunx @ultimat3/cli`, never `bunx x`, in a client config.** A client may spawn the command outside the installed app, and there `bunx x` fetches an unrelated npm package named `x`. Inside an app's shell, `bunx x` resolves the app's own CLI. `scaffold-mcp-json.test.ts` pins the spelling, and CI's `scaffold-smoke` spawns the file's own command and asks it `initialize` and `tools/list` on every push.
+
+### Rate limits on the HTTP transport
+
+**Enforced `As of 2026-08-24`.** `mcpHttpRoute` meters itself, per caller, per class, per minute. It has to: all MCP traffic is one `POST /mcp`, so the HTTP pipeline's own buckets — which key on the route — cannot tell an `initialize` handshake from a migration.
+
+| | Default | Spent by |
+|---|---|---|
+| `read` | 120 / minute | every method that is not `tools/call`, plus any tool that does not declare `destructive: true` |
+| `write` | 20 / minute | a `tools/call` naming a `destructive: true` tool, and **any call this server cannot resolve** — fail-closed, so a probing client never gets the cheap bucket |
+| `unauthenticated` | 20 / minute, per **address** | a missing token, or one `resolveToken` answers `null`. Past it the address is refused `429` **before** `resolveToken` runs — a valid guess included, so the answer says nothing about the token (`As of 2026-10`). Keyed on `handle(request, { address })` — the host's resolved client address; a host that passes none is not metered here |
+
+| Rule | Detail |
+|---|---|
+| Metered after the parse, before the tool | the class comes out of the body, so it cannot move above the JSON parse. An unauthenticated caller is answered `401` four lines earlier and can never spend an actor's allowance |
+| The key is the ACTOR | `mcp:<class>\|actor:<id>`, and it never reaches the caller: a 429 is provokable by anyone holding a valid token, so an actor or org id in one is a leak wearing a throttle's clothes |
+| Refusal | `X_MCP_RATE_LIMITED`, 429, with `Retry-After`. Its **own** code and not `X_RATE_LIMITED`, because that one's `fix:` names the HTTP pipeline's buckets, which do not govern this route — a fix line that runs and changes nothing is worse than none |
+| The knob | `mcpHttpRoute({ rateLimits: { read, write, unauthenticated } })`, or `defineAppMcp({ rateLimits })` |
+| Behind N replicas | pass `rateLimitStore: postgresRateLimitStore({ executor })` as well. The default is a per-process memory store — honest for `x mcp serve`, and a lie for N replicas behind one URL, each enforcing the full allowance on its own |
+
+
+### Protocol revisions: modern and legacy, one endpoint
+
+`As of 2026-11`, read from the spec on 2026-11-02. Every server is **dual-era**: the era is decided per request, never per connection.
+
+| Request | Era | Answered |
+|---|---|---|
+| `params._meta["io.modelcontextprotocol/protocolVersion"]` in `MCP_MODERN_VERSIONS` (`2026-07-28`) | modern, stateless | `server/discover` and the seven shared methods. Every result carries `resultType: 'complete'` and `_meta["io.modelcontextprotocol/serverInfo"]`; the lists, `resources/read` and `server/discover` add `ttlMs: 0`, `cacheScope: 'private'`. No `initialize`, no `ping` |
+| no `_meta` version, or one in `MCP_LEGACY_VERSIONS` (`2025-11-25`, `2025-06-18`, `2025-03-26`) | legacy | `initialize` echoes an offered legacy revision, else answers `2025-11-25`; `ping`; the seven shared methods |
+| any other `_meta` version | — | `-32022` `X_MCP_PROTOCOL_UNSUPPORTED`, `data: { supported, requested }` |
+
+`MCP_PROTOCOL_VERSION` is gone (`As of 2026-11`): name the era — `MCP_MODERN_VERSIONS[0]`, or `MCP_LEGACY_VERSIONS`. `resources/templates/list` is `{ resourceTemplates: [] }` on both eras. A resource not found, hidden or absent, is one answer: `-32601` legacy, `-32602` modern. Not served and not advertised: `subscriptions/listen`, multi-round-trip input, the tasks extension, `x-mcp-header`, SSE responses; every `listChanged` is `false`.
+
+Over HTTP:
+
+| Request | Answer |
+|---|---|
+| `Origin` present and core's `proveSameOrigin` cannot show it is this app: not the route's public origin, not listed in `http.cors.origins` (CSRF's, the sync upgrade's and `/_x`'s rule) | `403` `X_MCP_ORIGIN_REFUSED`, **before** `resolveToken` runs. No `Origin` (a client process) passes. A page rebound to this server's address IS same-origin and passes here: `x mcp serve` refuses a non-loopback Host first (`421` `X_MCP_HOST_REFUSED`), and `x new`'s development token is honoured only with no `Origin` at all |
+| modern, `MCP-Protocol-Version` / `Mcp-Method` / `Mcp-Name` (`tools/call`, `resources/read`, `prompts/get`) absent or unequal to the body (`=?base64?…?=` decoded first) | `400`, `-32020` `X_MCP_HEADER_MISMATCH` |
+| `MCP-Protocol-Version` naming no served revision | `400`, `-32022` |
+| modern, a method this server does not answer | `404`, `-32601` |
+| `GET` or `DELETE` on the endpoint — `x mcp serve` and an app's own mounted `/mcp` alike | `405` `X_METHOD_NOT_ALLOWED`, `Allow: POST` — `@ultimat3/http`'s, the pipeline's own |
+| `x mcp serve`: a Host that is not `localhost`, `*.localhost` or a loopback address | `421` `X_MCP_HOST_REFUSED`, before anything else — `/_x`'s rule |
+
+## Every action is an MCP tool
+
+```ts
+mcp: { expose: true, description: 'Publish a draft post' },
+```
+
+That line is the entire integration. From the existing declaration:
+
+| MCP requirement | Source |
+|---|---|
+| tool name | the action's export name, **verbatim** — `publishPost`, never `publish_post`. The one name `tools/call` accepts, `scopes:` is keyed on, and every published catalog spells. `As of 2026-08`: `openapi.json`'s `x-ultimate.mcpTool` and `describe().mcp.tool` published a snake_case name until then, and an agent that trusted either called a tool the server answers ToolNotFound for |
+| JSON Schema for input | the `input` schema (Standard Schema → JSON Schema, via `introspect()`) |
+| output schema | `output` — published as the tool's `outputSchema` when its root is an object ([below](#what-a-client-is-told-beyond-the-schema)) |
+| description | `mcp.description`, ending `Declared refusals: <codes>.` when the primitive declares `errors:` — the same sentence on the MCP tool and the in-app agent's tool, on both protocol eras, so an agent plans around a refusal before it calls ([Actions](Actions#declared-refusals-errors)). `As of 2026-11` |
+| title | `mcp.title` — omitted, none is published |
+| annotations | derived from the kind, `mcp.annotations` overrides key by key ([below](#what-a-client-is-told-beyond-the-schema)) |
+| **authorization** | the action's `policy` — unchanged, unwrapped, identical |
+| idempotency | `idempotent: true` adds one optional argument, `idempotencyKey` (string, 1–255), to the tool's `inputSchema`. `tools/call` removes it from the arguments and passes it to `invoke` as the key — the `Idempotency-Key` header's MCP twin, filed under action, caller and key — so a retry with the same key replays the first result and the same key with other arguments is `X_IDEMPOTENCY_CONFLICT`. The action's input never sees it. An argument, not `params._meta`: a model-driven client cannot set `_meta`, and the retry that needs the key is the agent's |
+| audit trail | the same OTel span and log line as an HTTP call |
+
+The projected tool's `run` **is** `invoke` — the same entry point the HTTP route calls (`packages/mcp/src/projectable.ts`). Policy runs inside it, so there is nothing to keep in sync. The projection itself adds **no** MCP scope: a second gate hard-coded into the projection would sit in front of the only gate that matters, and the two would eventually disagree. `defineAppMcp`'s `scopes:` map may still attach one from outside — a property of the connection's token, never invented by the projection.
+
+No MCP-specific permission table, no service account with broad rights. Exposure is opt-in; silence exposes nothing.
+
+The user's own agents can therefore operate the user's product — refund an order, re-run an import, publish a post — with the exact permissions that user has in the UI. See [Admin dashboard](Admin-Dashboard) and [Actions](Actions).
+
+### What a client is told beyond the schema
+
+`As of 22.10`, MCP 2025-06-18. Every piece is optional for the app; the defaults come from the
+declaration already written.
+
+| Field | Where it comes from | Default |
+|---|---|---|
+| `initialize` → `instructions` | `defineAppMcp({ instructions })` — a string, or `(caller) => string \| undefined` for per-population advice (the same shape as `surface`) | none sent. A function that throws or answers blank sends none; the handshake still answers |
+| tool `title` | `mcp: { title }` on the action/query; `title` on a hand-written tool | none |
+| `annotations.readOnlyHint` | the kind | query `true`, action `false` |
+| `annotations.destructiveHint` | the kind | action `true` — the spec's own default made explicit: the framework cannot tell an insert from a delete. Declare `false` for an additive write |
+| `annotations.idempotentHint` | `idempotent: true` on the action | `false` |
+| `annotations.openWorldHint` | `mcp.annotations` only | not published (only the author knows a write sends mail) |
+| `outputSchema` | an action's `output` whose root is an object; a query's declared `rows` as `{ rows: [<row>] }` — or the row itself for a `single: true` read, which answers one row or `X_NOT_FOUND` as its route does (`As of 2026-10`) | none — text only |
+| `structuredContent` | every successful call of a tool that publishes `outputSchema`: the serialized answer read back (a `Date` is its string). A LIST query's rows sit under `rows`; a `single: true` query's answer is the row object itself, with no wrapper | absent |
+
+```ts
+mcp: {
+  expose: true,
+  description: 'Tag a post. Adds, never removes.',
+  title: 'Tag post',
+  annotations: { destructiveHint: false },
+},
+```
+
+An `outputSchema` is **structure only** — `type`, `properties`, `required`, `items`, `enum`,
+`const`, `anyOf`. A client validates `structuredContent` against it and refuses the call on a
+miss, so a bound, a pattern or `additionalProperties: false` on rows read from the database would
+be a promise nothing enforces on the way out. The meta tools publish their own hints:
+`list_resources` / `describe_resource` read-only, `manage_resource` a write (it reaches every
+grouped tool); `manage_resource` answers the inner tool's `structuredContent` byte for byte.
+
+**Split by effect** (`surface: 'meta-split'`, `As of 2026-10-18`): `read_resource` (queries,
+`readOnlyHint: true`), `write_resource` (`mcp: { effect: 'write' }`) and `delete_resource`
+(`mcp: { effect: 'delete' }`, `destructiveHint: true`; `{ resource, action, params, live,
+confirmed_by_user }` — without `live: true` it answers a preview (the framework's runs nothing; an action declaring `live` answers its own), with it it runs
+only on `confirmed_by_user: true`) replace `manage_resource`, for clients and directories that
+refuse a dispatcher which both reads and writes. A grouped write without an effect fails boot
+(`X_MCP_EFFECT_UNDECLARED`); an action called on the wrong dispatcher is `X_MCP_WRONG_TOOL`
+naming the right one; an unconfirmed commit is `X_MCP_USER_CONFIRMATION_REQUIRED`. Full contract:
+[`packages/mcp/README.md`](https://github.com/developerz-ai/ultimate/blob/main/packages/mcp/README.md).
+
+**What the model reads is compact.** A tool's text block is one-line JSON (the 2-space form until
+22.10 spent a third of a large answer on indentation). `list_resources` answers **plain text**, one
+line per resource and one per action, each fact said once where it is cheapest (`As of 2026-10`):
+
+| Rule | Rendered |
+|---|---|
+| a scope every action of a resource shares | once, on the resource line: `(scope posts)` |
+| a scope every action of one KIND shares | once, per kind: `(query scope posts:read; action scope posts:write)` |
+| any other scope, or an unscoped action among scoped ones | on the action itself: `(scope …)` |
+| kind | `(action)` only, from `destructive`; untagged = read-only query — never inferred from the name |
+| `confirms: true` | `(confirms)`, always on the action |
+| params | every required field, then optional ones to 4 fields / 100 chars, then `…`; an enum past 8 literals ends `\|…` |
+
+```text
+2 resource(s). Run one with manage_resource({resource, action, params}); describe_resource({resources:["<name>"]}) has the full input schemas. Untagged = read-only query; (action) may write; (confirms) waits for a human; … = more in describe_resource.
+
+org (scope app:use) — The organisation
+  transferOrg (action) {} — Transfer the org
+
+posts (query scope app:use; action scope posts:write) — Blog posts
+  listPosts {status_eq?: string|number|boolean, title_cont?: string, sort?: "createdAt"|"-createdAt", …} — Posts, filtered
+  publishPost (action; confirms) {postId: string} — Publish a draft post
+```
+
+The same catalog as data, for a test: `mcp.server.catalog(caller)` — every action keeps its own
+`scope` there. `describe_resource` stays compact JSON with the full schema — a schema is JSON.
+
+**Hold the standing surface to a number.** `tools/list` rides every step and `list_resources` sits
+in the transcript all session:
+
+```ts
+import { assertMcpSurfaceBudget } from '@ultimat3/mcp';
+
+test('the staff surface stays small', async () => {
+  // The derivation beside the number: the size measureMcpSurface reported, × a margin.
+  await assertMcpSurfaceBudget(mcp.server, staffCaller, { toolsList: 4_000, listResources: 12_000 });
+});
+```
+
+`measureMcpSurface(server, caller)` returns `{ toolsList, listResources, instructions }` in
+characters, off the wire; the assert refuses every surface over its ceiling in one
+`X_MCP_SURFACE_OVER_BUDGET`.
+
+**Refusals speak to the caller.** `defineAppMcp` renders an error's `callerFix` — what a remote
+agent can do — where the framework's own `fix` names a command only the app's developer can run:
+`X_FORBIDDEN` says to ask the account owner for the permission, not `x policy explain`; a missing
+scope says to ask for a token that carries it; an invalid argument says to correct the field
+against the published schema. The developer's `fix` stays in the log line, `--json`, and on
+`mcpServer` (the dev server), whose default is `errorAudience: 'developer'`;
+`defineAppMcp({ errorAudience: 'developer' })` restores it for an app. **A 5xx cause is the
+server's own business** (`As of 2026-10`): for the caller audience, a thrown 5xx code core's
+`hasPublicCause` does not list (`X_DB_STATEMENT_FAILED` carries the Postgres message and the
+statement) renders a fixed sentence as its `cause` and `x errors explain <code> --json` (or its
+`callerFix`) as its `fix` — in a tool result and in a `resources/read` error's `data` alike, the
+verdict a production problem document gives. The developer audience keeps both. An app's own error takes the
+same two fields, plus `docs` — declared once with `defineError`:
+
+```ts
+import { defineError } from '@ultimat3/core';
+
+export const RefundWindowClosed = defineError<{ readonly ageDays: number }>({
+  code: 'X_REFUND_WINDOW_CLOSED',
+  title: 'the refund window has closed',
+  status: 403,
+  cause: (p) => `the payment is ${p.ageDays} days old; refunds are limited to 90 days`,
+  fix: 'x policy explain refund:create --json',
+  callerFix: 'issue a credit note instead',
+  docs: 'docs://recipes/issue-a-credit-note',
+});
+```
+
+A `docs` other than the framework's one Error-Codes page is a fourth line of the tool result —
+`  docs:  docs://recipes/issue-a-credit-note` — and is the `docs` member of the problem document
+over HTTP. A production problem document carries `callerFix` as its `fix`; a dev one keeps the
+developer's.
+
+### The app's own endpoint is mounted by the web role
+
+`As of 2026-09-05`. `defineAppMcp({ …, resolveToken })` builds `mcp.route`, and `app.config.ts`
+declares `ai: { mcp: { expose: true } }` **by default** (the path is `defineAppMcp`'s own, `/mcp` by default) — and until this date nothing
+between the two served it: `POST /mcp` was `X_ROUTE_NOT_FOUND` under `x dev` and in every
+container. The contract is one file:
+
+| Piece | Where | Rule |
+|---|---|---|
+| the declaration | `apps/<app>/mcp.ts` | `export const mcp = defineAppMcp({ include: 'exposed', resolveToken })` |
+| the switch and the path | `app.config.ts` → `ai.mcp` | `expose` (default `true`), `path` (default `/mcp`) |
+| the mount | `x dev` **and** `runRole`, through one call | `POST <path>` → `mcp.route.handle(request)`; the boot log says `app mcp mounted`, `x dev`'s summary prints `mcp POST /mcp` |
+| auth | the route's own | `auth: 'public'`, `enforcedBy: 'handler'` on the http route — the bearer token is read by `resolveToken`, never pre-judged by the pipeline |
+| exposed and unmountable | `X_MCP_APP_UNMOUNTED`, logged once | no file exports `mcp`, or it was built without `resolveToken` (no `route`); the fix names the file to write |
+| a new app | `x new` writes it | `apps/web/mcp.ts` over the `@<app>/mcp` catalog, `resolveToken` from `apps/web/app/auth/agent-token.ts`: an API key (`issueApiKey`, `x_api_keys`) is an agent in every environment; `dev-agent.<role>` is one under `x dev` alone, and only from a loopback address with no `Origin` (a page rebound to 127.0.0.1 sends one). No token, or one it does not know: 401 |
+
+**A token from an external issuer** (`As of 2026-11-05`): an app whose agents present an IdP's JWT
+rather than an API key hands `defineAppMcp` `jwtBearerResolver` from `@ultimat3/auth` — the twin of
+`apiKeyResolver`. The signature, issuer, audience and time checks are `verifyWorkloadToken`'s; the
+app keeps only its claim check and its actor:
+
+```ts
+// apps/web/app/auth/agent-token.ts
+import { actorFromService, jwksClient, jwtBearerResolver } from '@ultimat3/auth';
+
+const ISSUER = 'https://login.example.com/tenant-a/v2.0';
+const TENANT = 'tenant-a';
+const ORG_ID = '00000000-0000-4000-8000-000000000001';
+const keys = jwksClient({ provider: ISSUER, jwksUri: `${ISSUER}/discovery/v2.0/keys` });
+
+export const resolveAgentToken = jwtBearerResolver({
+  keys: () => keys,                       // read per token: no network at import
+  issuers: [ISSUER],
+  audience: 'api://ledger',
+  requiredScope: 'ledger.agent',          // optional; the scope name is the app's
+  accept: (raw) => raw['tid'] === TENANT, // the app's own claim, after the signature held
+  actorOf: (token) => actorFromService({ ...token.identity, orgId: ORG_ID }),
+});
+```
+
+Every refusal is `null` — the mount's `401` — and one `auth.jwt_bearer.refused` line naming the
+check (`signature`, `issuer`, `audience`, `expiry`, `not-before`, `scope`, `accept`, `key-set`),
+never a claim's value. A key set that cannot be read and an unknown `kid` both fail closed.
+
+#### Several endpoints, one per population
+
+`As of 22.14.0`. `mcp` may be a non-empty **array** of `defineAppMcp` values — one endpoint per
+population, each with its own catalog, `instructions`, `groups`, `scopes` and `oauth`. Same export
+name, same file; the order is the contract:
+
+```ts
+// apps/web/mcp.ts
+export const mcp = [
+  defineAppMcp({ name: 'acme', include: 'exposed', scopes: CUSTOMER_SCOPES, resolveToken: customerToken, oauth }),
+  defineAppMcp({ name: 'acme-admin', path: '/mcp/admin', tools: staffTools, surface: 'meta', groups, scopes: STAFF_SCOPES, resolveToken: staffToken, oauth }),
+  defineAppMcp({ name: 'acme-afiliados', path: '/mcp/afiliados', tools: affiliateTools, scopes: AFFILIATE_SCOPES, resolveToken: affiliateToken, oauth }),
+];
+```
+
+| Rule | Detail |
+|---|---|
+| endpoint #0 | mounts at its own `defineAppMcp({ path })` (default `/mcp`), exactly as a single export does, and owns the root `/.well-known/oauth-protected-resource` |
+| every other endpoint | mounts at its own `defineAppMcp({ path })` (default `/mcp`), route name `mcp:<path>` |
+| isolation | each endpoint is its own `McpServer`: a tool, resource or prompt of one is absent from another's `tools/list` and answers `-32601` there |
+| two endpoints on one route | `X_MCP_PATH_DUPLICATE`, **thrown** at boot — a second endpoint that forgot `path` collides with the default `/mcp` |
+| one endpoint without `resolveToken` | `X_MCP_APP_UNMOUNTED` names it; the others still mount |
+| boot report | one `app mcp mounted` log line and one `mcp POST <path>` summary line per endpoint; `x dev --json` carries `mcp` (endpoint #0) and `mcpPaths` (all) |
+
+The same discovery serves an app's `RuntimeOverrides`: `apps/<app>/runtime.ts` exporting `runtime`
+reaches `x dev` (its middleware, rate-limit store and plain `routes`) and `runRole` when
+`apps/web/server.ts` passes none — one middleware chain in development and in the container, not two.
+`runtime.routes` (`As of 22.6.0`) is the one escape hatch for a URL no primitive projects to and a
+wire format none speaks — an OAuth token endpoint answering RFC 6749 JSON to a form-encoded POST —
+mounted after the framework's routes and before the pages, through the whole pipeline.
+
+### OAuth discovery for remote connectors
+
+`defineAppMcp({ oauth })` makes the endpoint an OAuth 2.1 protected resource per the MCP
+authorization spec (2025-06-18; unchanged in 2025-11-25) and RFC 9728 (`As of 22.6.0`):
+
+```ts
+export const mcp = defineAppMcp({
+  include: 'exposed',
+  scopes: MCP_SCOPES,
+  resolveToken,
+  oauth: {
+    authorizationServers: ['https://www.example.com'], // issuer(s); https, or http on localhost
+    resourceName: 'Example',
+    // resource: 'https://www.example.com/mcp',   // omitted: the request's PUBLIC origin + path
+    // scopesSupported: [...],                     // omitted: Object.keys(scopes)
+  },
+});
+```
+
+| What | Answer |
+|---|---|
+| 401 (no token) | `WWW-Authenticate: Bearer realm="ultimate-mcp", resource_metadata="<origin>/.well-known/oauth-protected-resource/mcp"` |
+| 401 (token did not resolve) | the same plus `error="invalid_token"` |
+| `GET /.well-known/oauth-protected-resource/mcp` and `GET /.well-known/oauth-protected-resource` | `{ resource, authorization_servers, bearer_methods_supported: ['header'], scopes_supported, resource_name }`, public, `access-control-allow-origin: *` — mounted by both boots beside `POST /mcp` |
+| several endpoints | each serves its own path-inserted document (RFC 9728 §3.1) — `/.well-known/oauth-protected-resource/mcp/admin` for `POST /mcp/admin` — with its own `resource` and `scopes_supported`, and its 401 names that document; the root document stays endpoint #0's |
+| origin | `ctx.https` + host as the pipeline resolved them, so a TLS-terminating ingress yields `https://…` |
+
+The authorization server is `mcpAuthorizationServer` (`As of 2026-11`): `@ultimat3/auth`'s
+`authorizationServer` ([Auth](Auth#authorization-server)) bound to this endpoint. One call, one
+consent page:
+
+```ts
+// apps/web/mcp-oauth.ts
+import { postgresAuthServerStore } from '@ultimat3/auth';
+import type { Actor, PgExecutor } from '@ultimat3/core';
+import type { McpScopes } from '@ultimat3/mcp';
+import { defineAppMcp, mcpAuthorizationServer } from '@ultimat3/mcp';
+
+declare const executor: PgExecutor;
+declare const MCP_SCOPES: McpScopes;             // the scope → tool names map, as above
+declare function agentFor(subject: string): Promise<Actor | null>;
+
+export const as = mcpAuthorizationServer({
+  issuer: 'https://www.example.com',
+  resource: 'https://www.example.com/mcp',
+  store: postgresAuthServerStore({ executor }),
+  consentPath: '/oauth/consent',
+  lifetimes: { accessSeconds: 600, refreshIdleDays: 30 },
+  scopes: Object.keys(MCP_SCOPES),
+  registration: 'dynamic',                    // omitted: metadata-document client ids only
+  actor: (grant) => agentFor(grant.subject),  // per call: a removed member is refused at once
+});
+export const mcp = defineAppMcp({ include: 'exposed', scopes: MCP_SCOPES, oauth: as.oauth, resolveToken: as.resolveToken });
+// apps/web/runtime.ts: routes: [...as.routes]
+```
+
+| What | Answer |
+|---|---|
+| `resource` | every grant binds to it, a request naming none included; `resolveToken` refuses a token bound to anything else, or to nothing |
+| `oauth` | `{ authorizationServers: [issuer], resource, scopesSupported: scopes }` for `defineAppMcp` |
+| audit | each server decision is one `mcp.auth.<event>` line — `issued`, `refused`, `revoked` at `info`; `code-replayed`, `refresh-replayed`, `grant-disabled` at `warn` |
+| the protocol | none of it here: every rule is `authorizationServer`'s, so a hand-written server and plain `oauth:` keep working |
+
+Without `oauth`, the 401 is `Bearer realm="ultimate-mcp"` as before. Bad block:
+`X_MCP_OAUTH_INVALID` at definition.
+
+### Per-caller tools and one token across tenants
+
+`As of 2026-11`. Two `defineAppMcp` options, both resolved per request and both behind the same
+gates as every other tool:
+
+| Option | What it does | Refusals |
+|---|---|---|
+| `catalog: (caller) => ({ tools, groups? })` | tools that exist per caller (a tenant's connectors), each a `runtimeTool({ name, description, inputSchema, permission, run })` borrowing a **declared** permission; asked on every `tools/list` and `tools/call`, never cached | `X_MCP_CATALOG_TOOL_DUPLICATE` (a name taken twice), `X_MCP_CATALOG_UNRESOLVED` (the resolver threw, or answered something unservable) — the request fails, never a smaller list |
+| `tenant: { arg, resolve, list }` | the token resolves to a person; every call names its tenant under `arg`, stripped before validation; `resolve` builds the call's actor, so policy and tenancy guards see one org per call; `list_tenants` lists them | `X_MCP_TENANT_REQUIRED` (no tenant named — fix `list_tenants({})`), `X_MCP_TENANT_UNRESOLVED` (the lookup threw), `X_MCP_TENANT_INVALID` (boot: a tool declares the arg) |
+
+A tenant the person is not in (`resolve` answers `null`) gets exactly the answer a hidden tool gets.
+Another caller's runtime tool is an absent tool. The audit line carries `tenant`.
+
+**`tools/list` is person-scoped, and not a promise.** MCP's `tools/list` names no tenant, so `visibleTo` (function form), `can()` and `catalog` run there as the person; `tools/call` runs them as the tenant actor and is authoritative. A tool visible to the person but not the tenant actor is listed and answers as a hidden tool; one visible only through a tenant role or a tenant's catalog row is unlisted and still runs when called by name.
+
+```ts
+import { agentActor } from '@ultimat3/core';
+import { defineAppMcp, runtimeTool } from '@ultimat3/mcp';
+
+declare function roleIn(personId: string, org: string): Promise<string | undefined>;
+
+defineAppMcp({
+  include: 'exposed',
+  tenant: {
+    arg: 'org',
+    resolve: async ({ person, value }) => {
+      const role = await roleIn(person.id, value);
+      return role === undefined ? null : agentActor({ id: person.id, orgId: value, roles: [role] });
+    },
+    list: () => [],
+  },
+  catalog: (caller) => ({
+    tools: caller.actor.orgId === undefined ? [] : [
+      runtimeTool({
+        name: 'org_ping',
+        description: 'Ping the acting org. Read-only.',
+        inputSchema: { type: 'object', properties: {} },
+        permission: 'org:read',
+        destructive: false,
+        run: ({ ctx }) => ({ org: ctx.actor.orgId }),
+      }),
+    ],
+  }),
+});
+```
+
+Worked example: Postly's `quote_<plan>` tools and its multi-org token —
+`examples/dummy/packages/mcp/src/{quotes,tenants}.ts`, proven over its own endpoint by
+`tenant.contract.test.ts`. Full rules: `packages/mcp/README.md`.
+
+### Long tools: `jobTool()` and `job_status`
+
+A tool whose work outlives a client's timeout. `tools: { reindex: jobTool({ job, policy, description, result, input?, jobInput?, inlineMs? }) }`.
+
+| Fact | Rule |
+|---|---|
+| Durable | enqueues at once (`job.as(caller)`); never an in-process promise |
+| Input | the tool's `input` (default: the job's); `jobInput(input, ctx)` builds the job's from it and the caller — a member id read off the request, never an argument an agent fills in |
+| Inline | polls `introspect.job(id)` every 250 ms up to `inlineMs` (default 5,000, ceiling 25,000), honours `ctx.signal`. Also bounded by the poll count, so a clock that does not move still ends the wait. `inlineMs: 0` answers `running` at once — right for a model run |
+| Answers | `done` + the `result` query's rows, read **as the caller** · `running` + `poll: { tool: 'job_status', arguments: { jobId } }` · `failed` + the code it was thrown with (`X_INTERNAL` for an uncoded throw, as HTTP answers one), never the row's text |
+| `job_status` | projected once beside any job tool; no scope (gated per row); another actor, another org and an unknown id are the same `X_JOB_NOT_FOUND` |
+| Dedupe | the job's own `idempotencyKey`: a repeated call attaches to the live run |
+| Refused | `X_MCP_JOB_TOOL_INVALID`: not a `job()` handle, `inlineMs` out of range, a driver with no `introspect` |
+
+The reference app's `draftReview` (`examples/dummy/apps/web/mcp-tools.ts`) queues `reviewDraftLater` for the caller's own member and reads `postReview` back; `mcp-tools.contract.test.ts` drives it over the wire.
+
+### Recipes: `docs://recipes/<name>` and the `docs` tool
+
+`defineAppMcp({ recipes: [{ name, title, description, steps: [{ tool, why, example? }], notes? }] })`.
+
+| Fact | Rule |
+|---|---|
+| Checked at boot | every `step.tool` is in the final catalog (`X_MCP_RECIPE_TOOL_UNKNOWN`), every `example` passes that tool's input (`X_MCP_RECIPE_EXAMPLE_INVALID`) — all offenders in one boot |
+| Served | an `McpResource` (`text/markdown`, rendered from the structure) and a read-only `docs({ name? })` tool |
+| Visibility | only to a caller who can see every step's tool and holds its scope; hidden = unknown, byte for byte |
+| The prefix | with `recipes:`, `resources:` may not hold `docs://recipes/` (`X_MCP_RECIPE_URI_RESERVED`) |
+| Error links | `defineError({ docs: 'docs://recipes/<name>' })` must name a recipe some endpoint serves — `X_MCP_RECIPE_UNKNOWN` at mount; a throw-site link to none is dropped from the render and logged |
+
+**From a markdown folder.** `recipes: await markdownRecipes({ dir: join(import.meta.dir, 'docs') })` reads one `*.md` per recipe and serves its body verbatim. The file name is the recipe name. `title:` and `description:` come from `---` frontmatter, else from the first `# ` heading and the paragraph under it. A fence tagged `call` holds one step, `tool {"arg": "value"}`, and is checked at boot like any `step.example`. A whole `"<postId>"` value is a placeholder, so any value passes at boot. A `docs://recipes/<name>` link in the text that names no recipe is `X_MCP_RECIPE_LINK_DANGLING`.
+
+**Searching.** `docs({})` lists what this caller can follow. `docs({ name })` and `docs({ uri })` read one recipe. `docs({ q })` inlines the best match, scoring 3 for each word in the name or title, 2 in the description and 1 in the body. On a miss it names the nearest recipes, so a caller with any visible recipe never gets an empty answer.
+
+**Per caller.** `extraRecipes: (caller) => recipes` adds the caller's own recipes, such as a tenant's runbooks. They are listed as `source: 'caller'` on `docs`, and on `resources/list` and `resources/read` to that caller only. They are not checked at boot. A recipe that names a tool the caller cannot call is not shown.
+
+**Replayed.** In a contract test, `await replayRecipes(server, recipes, { caller, placeholders })` calls every step's example through the real server as `caller` and reads every link. Three answers fail it with `X_MCP_RECIPE_EXAMPLE_INVALID`: no such tool, refused arguments, and a policy that refuses that caller. A dangling link fails it with `X_MCP_RECIPE_LINK_DANGLING`. A tool's own refusal (`X_NOT_FOUND` for a sample id) counts as the tool working. The reference app replays its recipes as an org owner: `examples/dummy/packages/mcp/src/recipes-replay.contract.test.ts`.
+
+### Consuming another server: `mcpTools()`
+
+Remote MCP tools as this app's actions — the app's `policy`, rate limit, confirmation gate and manifest row; one tool list for an agent.
+
+| Fact | Rule |
+|---|---|
+| Declared | `mcpTools({ name, url, auth, allow, policy, lock })` → `tools: Map<remoteName, Action>`, action names `<name>_<tool>`. `policy` is an app `resource:verb` permission, as a hand-written tool takes: the input is the remote's schema, so a predicate over it would decide nothing |
+| Reviewed | `allow` is required; `lock` is the committed `tools.lock.json` written by `x mcp lock <name>`; `x mcp lock <name> --check` writes nothing and fails on drift |
+| Drift | a missing, re-described or re-shaped tool is `X_MCP_REMOTE_TOOL_CHANGED` — at declaration against the lock, before the first call against the live server |
+| Wire | `outbound()` (`dial: 'public'` by default, deadline, size cap); modern era first, then legacy `initialize` + `Mcp-Session-Id`; JSON or SSE answers |
+| Untrusted | the answer arrives as `{ untrusted, isError }`, fenced `<untrusted source="mcp:<name>/<tool>">`, middle-cut past `maxResultChars` |
+| Read-only | only when a reviewer sets `readOnly: true` in the lock — the remote's hint is never trusted |
+| Failure | `X_MCP_REMOTE_REFUSED` (502), the cause names the status |
+
+**Five outcomes of a call.** `meta.failure` tells them apart without reading a cause:
+
+| Outcome | What the caller gets |
+|---|---|
+| success, text only | `{ isError: false, text }`. No `structuredContent` and no output schema is still a success |
+| the tool's own `isError: true` | a RESULT `{ isError: true, text }`, handed to the model like any tool result |
+| no answer (DNS, connect, deadline) | `X_MCP_REMOTE_REFUSED`, `meta.failure: 'transport'`. An abort from the caller's own signal passes through as `X_ABORTED` |
+| an HTTP status | `X_MCP_REMOTE_REFUSED`, `meta.failure: 'status'`, `meta.status` |
+| a JSON-RPC `error` | `X_MCP_REMOTE_REFUSED`, `meta.failure: 'rpc'`, `meta.rpcCode` |
+| a body that is no JSON-RPC answer | `X_MCP_REMOTE_REFUSED`, `meta.failure: 'malformed'` |
+| `401` | `X_MCP_REMOTE_AUTH_REQUIRED`, `meta.resourceMetadata` from the `WWW-Authenticate` challenge (RFC 9728) when the server sent one |
+
+**Streaming.** A server may answer any POST as JSON or as `text/event-stream`. The client reads the answer with its own id out of the stream, and acts on any `notifications/tools/list_changed` riding with it. A legacy (2025-era) session also gets one long-lived `GET` stream for the server's own messages. It is opened after `initialize`, detached from the request that opened it, and lives `listenMs` (default 5 minutes) before the next call opens another. A `405` means the server offers no such stream, and the client never asks again. `client.close()` ends it. A modern server is stateless, so its changes ride response streams only. Either way, a change makes the next call re-list and hold the list to the lock again.
+
+**OAuth to the upstream.** `mcpOAuthClient({ clientId, clientName, redirectUri, scopes? })` is the client half of the MCP authorization spec:
+
+1. `begin(serverUrl)` reads the server's RFC 9728 protected-resource document, then its authorization server's RFC 8414 metadata. It answers `{ authorizeUrl, handshake }`, with PKCE S256 from `@ultimat3/auth` and `resource` per RFC 8707.
+2. Redirect the person to `authorizeUrl`. Keep `handshake` server-side, in the person's session.
+3. On the callback, `exchange(handshake, { code, state })` answers `{ accessToken, refreshToken, expiresIn }`. `refresh(handshake, refreshToken)` answers the same shape.
+4. Store the tokens per connection. Return the access token from `mcpTools({ auth })`, which is read on every call, so a rotated token is never cached.
+
+`clientId` is the HTTPS URL at which this app serves `metadata()` (a client ID metadata document), or an id the authorization server issued when the app was registered by hand. A `401` mid-session is `X_MCP_REMOTE_AUTH_REQUIRED`: run the handshake again.
+
+**Proved on itself.** `examples/dummy/packages/mcp/src/remote-self.contract.test.ts` points `mcpTools()` at the reference app's own MCP route. It locks the live list, calls `seatReport`, and meets three refusals: the remote policy's `X_FORBIDDEN` as a result, a stolen token as `X_MCP_REMOTE_AUTH_REQUIRED`, and a schema moved since the lock as `X_MCP_REMOTE_TOOL_CHANGED`.
+
+## Three outcomes, deliberately different
+
+Role, scope and policy refuse in three distinguishable ways. The difference is the security property, not an implementation detail.
+
+| Situation | Response | Wire |
+|---|---|---|
+| The actor's role can never invoke the tool | absent from `tools/list`; a direct call answers ToolNotFound | JSON-RPC `-32601`, message `tool not found: <name> — call tools/list to read the catalog this caller may use`, no `data` at all. The hint is the same sentence on both branches — absent and hidden — so it instructs without saying which |
+| The role could invoke it, but the connection's scope does not include it | explicit refusal naming the missing scope | JSON-RPC `-32600`, `data: { code: 'X_MCP_SCOPE_DENIED', scope, fix }` |
+| The tool was invoked and the policy denied this input | `X_FORBIDDEN` with the denial reason | a normal `result` with `isError: true` — identical to the HTTP answer for the same call |
+| The caller may see and call it, and the arguments fail its published schema | `X_INPUT_INVALID`, each issue addressed by path | a normal `result` with `isError: true` (`As of 22.10`; a `-32602` until then, which clients hide from the model). `-32602` remains for a call that is not one — no `params`, a non-string `name` |
+
+Hidden means hidden: `Forbidden` on a hidden tool is an enumeration oracle — an agent, or an attacker driving one, walks a name list and reads the org's feature set, entity names and internal operations off the difference between "not found" and "forbidden". A scope refusal is the opposite case: a well-behaved client can legitimately fix it, so hiding it would only strand the caller.
+
+| Rule | Detail |
+|---|---|
+| `visibleTo` takes two forms | a **role allowlist**, or a **predicate over the caller**. A tool that declares neither is visible to everyone |
+| Both forms are **fail-closed** | a role list admits only the roles it names, so a caller whose role is not in it — including a caller with no role at all — is refused; a caller with no role sees only tools that declare no `visibleTo` |
+| `tools/list` is per connection | answered per caller, never a static catalog |
+| Visibility is input-independent | the predicate takes the caller and nothing else — it structurally cannot read call arguments, so existence cannot be probed by varying them |
+| Gate order is fixed | visibility → scope → the policy's actor half → arguments → policy. Scope runs before the policy, so a refusal never depends on evaluating a policy against attacker-supplied input; a caller the policy refuses whatever they send gets the policy's `X_FORBIDDEN` (the tool's `admit`) before their arguments are checked, so an issue list never describes a tool they may not call; the rest of the policy — predicates over the arguments — runs after them |
+| Every outcome is audited | one structured log line per `tools/call`: `surface: 'mcp'`, tool name, actor id, outcome. ToolNotFound, scope denials and policy denials log at `warn`, a successful call at `info` — ToolNotFound is `warn` on purpose, because an enumeration attempt is a detectable pattern |
+| Audit lines carry no payload | tool name, outcome and error code only — never call arguments, never row data |
+| No trusted-tool mode | there is no flag that skips policy evaluation, on any MCP surface |
+| The actor cannot exceed the human | the actor is the signed-in user's session; an agent inherits exactly those permissions |
+
+Where the first two outcomes are declared:
+
+| Outcome | Declared | Property |
+|---|---|---|
+| Hidden (role) | `mcp: { visibleTo: [...] }`, on the action or query itself | `readonly string[]` — a role allowlist. A primitive declares the list form only: a declared fact stays static and serialisable. The predicate form of `McpVisibility` is for a surface that builds its catalog programmatically (`@ultimat3/admin` derives visibility from the actor's admin permissions) and hands `@ultimat3/mcp` a tool directly. Both are fail-closed: an unnamed role — including a caller carrying no role at all — gets ToolNotFound, never Forbidden. A catalog audience, not an authz rule; the primitive's `policy` still decides every call |
+| Hidden (prompt) | `visibleTo` on an `McpPrompt` passed to `defineAppMcp({ prompts })` | the same `McpVisibility`, the same fail-closed evaluation: absent from `prompts/list`, and `prompts/get` answers exactly as for a prompt that does not exist (`As of 22.14.0`). Resources take the same field |
+| Scope | `scopes:` on `defineAppMcp` | `Readonly<Record<string, readonly string[]>>` — scope name → tool names. A capability of the connection's token, so it is declared once per app rather than beside every primitive. A name the catalog does not contain, or one claimed by two scope entries, refuses at boot: `X_MCP_SCOPE_UNKNOWN`, `X_MCP_SCOPE_CONFLICT` |
+
+Rationale for each: [`docs/architecture/11-ai-surface.md`](https://github.com/developerz-ai/ultimate/blob/main/docs/architecture/11-ai-surface.md).
+
+## Generated facts, hand-written conventions
+
+| Artifact | Author | Contents | Rule |
+|---|---|---|---|
+| `x.manifest.json` | **generated**, every build | routes, entities, actions, mutators, queries, jobs, tasks, policies, cache tags, MCP tools, budgets, build ID | never hand-edited; drift is a `x verify` failure |
+| `openapi.json` | **generated** | HTTP surface from action/query declarations | contract diff in `x verify` |
+| `AGENTS.md` | **human-authored**, short | project-specific conventions an agent cannot infer | never generated, never auto-appended |
+| `CLAUDE.md` | **human-authored**, short | same, compressed-config style, <600 lines | never generated, never auto-appended |
+
+LLM-generated context files measurably reduce task success. A model writing "here is what this codebase does" produces confident, plausible, partly-wrong prose, and the next agent treats it as ground truth — errors compound and cannot be distinguished from facts. So: facts come from code (structured, verifiable, regenerated every build), conventions come from a human (short, opinionated, stable). Ultimate never generates prose documentation at runtime, and `x new` scaffolds `AGENTS.md` as a terse human-editable stub, not an essay.
+
+## LLM gateway
+
+One typed entry point for model calls — provider-agnostic, observable, cached, evaluated.
+
+```ts
+export const summarize = llm({
+  model: 'claude-sonnet-5-5',
+  input:  t.object({ postId: t.uuid }),
+  output: t.object({ summary: t.string, tags: t.string.array() }),
+  prompt: summarizePrompt,                       // versioned artifact
+  cache:  { semantic: { threshold: 0.97, ttl: '7d' } },
+  budget: { tokensIn: 8000, costPerCall: { minor: 5, currency: 'USD' } },
+  policy: can('post:read'),
+});
+```
+
+| Feature | Behavior |
+|---|---|
+| Structured output | `output` schema drives tool-use/JSON mode; a parse failure retries once, then throws `X_LLM_OUTPUT_INVALID` |
+| Streaming | `.stream()` on the returned action; policy, budget, cache and span unchanged; the `done` chunk carries the validated value |
+| Cost + token accounting | per call, per tenant, per prompt version; exceeding `budget` throws before spending |
+| Retries | typed on provider errors; rate limits back off, content refusals do not retry |
+| Breaker | one per provider, on by default (`{ failures: 5, cooldownMs: 30_000 }`): a provider whose retries ran out five passes running is skipped, no call and no sleep, until one probe after the cooldown answers; a 400 never opens it. `providerGateway({ breaker: (provider) => breaker({ name: provider.name, failures: 3, cooldownMs: 10_000 }) })` chooses your own, `breaker: 'off'` declines. [Outbound HTTP](Outbound-HTTP#breaker-stop-paying-for-a-dependency-that-is-down) |
+| Caching | semantic cache from [Caching and invalidation](Caching-And-Invalidation) |
+| Provider prompt caching | the provider keeps a request's stable prefix; on by default for `agent()`, off for `llm()` → [Agents](Agents#provider-prompt-caching) |
+| Tracing | one OTel span per call with model, tokens, cache hit, prompt version |
+| Fallback | ordered provider list per model; the provider that answered is on the span as `llm.provider`, never silent. A refusal names a more capable model for the declaration to adopt — models are never swapped at runtime |
+| Money | `Money = { minor, currency }` — never a float → [Money](Money) |
+
+**A stream is the same action over a different transport, with one difference that is forced rather than chosen: there is no repair turn.** The non-streaming call retries once on a parse failure; a stream cannot, because the tokens are already on the reader's screen and a second answer is two answers to one question. So a stream yields **unvalidated** text increments plus one final `done` carrying the value that did satisfy `output`, and a schema disagreement is `X_LLM_STREAM_INVALID` naming the non-streaming call as the fix. A structured schema wants the non-streaming call.
+
+**Showing what a model wrote is `Markdown` from `@ultimat3/ui`**, never `innerHTML`: the text is
+parsed to a tree and rendered as elements, so a `<script>` or a `javascript:` link in an answer is
+text, and a streaming answer re-parses per token while finished blocks keep their nodes. Tool calls
+and usage records are `JsonView`; a model's proposed edit against the original is `DiffView`
+→ [UI components](UI-Components).
+
+Long or multi-call chains are `job`s with steps, so a model call that fails on step 4 retries step 4 only. See [Jobs and workflows](Jobs-And-Workflows).
+
+## Agents, hives and agent jobs → [Agents](Agents)
+
+`llm()` is one model call. A **tool-using loop** is `agent()`, and it is the same kind of thing — an action factory, not a ninth primitive. Everything the loop needs is a factory over a primitive already in the vocabulary:
+
+| Factory | Over | Gives you |
+|---|---|---|
+| `llm()` | `action()` | one model call, streamable, semantically cacheable |
+| `agent()` | `action()` | the turn loop: real actions as tools, forced structured output, `maxTurns`, a run budget, `ctx.signal`, `onTurn`. **No `.stream()`** |
+| `hive()` | `action()` | one action fanned out over many inputs: bounded concurrency, split-order results, three-way `ok`/`failed`/`skipped` member outcomes |
+| `agentJob()` | `job()` | an agent as durable, retried, cancellable queue work — `As of 2026-08` the only way an agent reaches a queue |
+| `backfill()` | `job()` | the resumable, paced sweep those run over → [Migrations and backfills](Migrations-And-Backfills) |
+
+A tool is one of two things. A real `action()` carrying `mcp: { expose: true }` — the **same** predicate an external MCP client is filtered by, so an in-app agent and an external one are offered exactly the same catalogue of actions and authorize identically. Or an `inProcessTool({ name, description, input, execute })`: a tool that is this process's alone, with no `mcp` flag, in no MCP catalogue, route or `openapi.json`, running as `ctx.actor` under the agent's own policy → [In-process tools](Agents#in-process-tools). An `agent()` returns an action, so an agent is a tool of another agent with no supervisor primitive anywhere.
+
+A tool in `mcpConfirmations({ tools })` waits for a person on **both** surfaces: `X_MCP_CONFIRMATION_PENDING` over `tools/call`, on a server handed `confirmations` or not, and as an agent's `tool_result`. A name no tool answers to is `X_MCP_CONFIRMATION_TOOL_UNKNOWN` at the `manifest` step. One list, no second option → [A tool that waits for a person](Agents#a-tool-that-waits-for-a-person).
+
+**Quorum and signed decisions** (`As of 2026-11`) are two more options on the same factory, never a second gate:
+
+| Option | What it holds |
+|---|---|
+| `approvers: ({ args }) => ({ quorum, from })` | N of M distinct approvers matching `'role:<name>'` / `'actor:<id>'`, resolved per call from the validated arguments and stored on the row. The requesting actor never counts (nor an impersonated decider): `X_FORBIDDEN`. One approver counts once. Short of the quorum, the agent's repeat answers `X_MCP_CONFIRMATION_QUORUM_PENDING`; a bar nobody can meet is `X_MCP_CONFIRMATION_QUORUM_INVALID` before a row opens |
+| `evidence: { verify }` | every vote carries `signature` over the `signingPayload` the view returns (confirmation id, tool, keyed argument digest), checked by the app's `verify` against keys registered to the approver. A missing or failing one stores no vote; the stored row is re-verified before the call runs, so a row written around the decision runs nothing. Both are `X_MCP_CONFIRMATION_EVIDENCE_INVALID` |
+
+`verifyConfirmationEvidence(row, { evidence, quorum, from, arguments })` is exported so a separate process that executes the action re-checks the stored evidence with public keys alone. `ttlMs` expires a partly approved row like any other; one rejection rejects. The full table: [`@ultimat3/mcp` README](https://github.com/developerz-ai/ultimate/blob/main/packages/mcp/README.md#quorum-and-signed-decisions).
+
+Full reference, including the at-least-once trap that makes every mutating tool's idempotency your obligation: [Agents](Agents).
+
+## Prompts as versioned artifacts
+
+```
+apps/web/app/posts/prompts/summarize.v3.md      # the prompt, plain markdown + typed slots
+apps/web/app/posts/prompts/summarize.evals.ts   # evals attached to it
+```
+
+| Rule | Why |
+|---|---|
+| A prompt is a file with a version, not a string literal | diffable, reviewable, attributable in traces |
+| Editing a prompt requires a version bump | invalidates the semantic cache; keeps A/B honest |
+| Slots are typed | a missing variable is a compile error, not a `{{undefined}}` in production |
+| **Every prompt has an evals file** — no evals is a `x verify` failure | an unevaluated prompt is untested code |
+| Old versions retained | traces stay interpretable; rollback is a config line |
+
+## Vectors and hybrid search
+
+pgvector in the same Postgres. No second datastore.
+
+| Piece | Detail |
+|---|---|
+| Embeddings | an `Embedder` value, never an entity option: `remoteEmbedder({ name, dimension, baseUrl? })` speaks the `/v1/embeddings` shape (`EMBEDDINGS_API_KEY`), `hashEmbedder()` is the offline default |
+| Writing | `indexDocument({ store, embedder, document })` chunks, embeds, upserts and prunes the stale tail; a sweep over many rows is the app's `backfill()` calling it per page |
+| Index | HNSW: `postgresVectorStore({ name, dimension }).ddl()` prints the table and both indexes, pasted into a migration by the app |
+| Hybrid search | one `query` primitive fusing pgvector cosine + Postgres FTS with Reciprocal Rank Fusion; weights are config |
+| Filtering | tenant + policy filters applied **in SQL**, so vector search cannot leak across tenants; a store with no tenant bound, read inside a request acting for an org, is `X_VECTOR_UNSCOPED` — a backfill opts in with `scope: UNSCOPED` |
+| Re-embed | `indexDocument` again: every chunk is embedded on each call, so a job decides when text changed |
+
+## Evals as a test type
+
+`eval` is one of the six test types in [Testing](Testing).
+
+| Aspect | Detail |
+|---|---|
+| Shape | fixture set + assertions: exact, schema, rubric (LLM judge), or regression-vs-baseline |
+| Agents | `defineEval({ agent, cases, tools })` runs the whole loop with each tool a double; an undoubled call is `X_EVAL_TOOL_UNDOUBLED` → [Testing](Testing#evals) |
+| Determinism | temperature 0 where possible; judge model and prompt version pinned |
+| Gate | `x verify` fails on a score drop beyond the declared tolerance, not on absolute score |
+| Cost | reported per run; `x test eval --sample 20` for the fast local loop |
+| Output | `--json` with per-case scores, so an agent iterating on a prompt sees which case it broke |
+
+## Branch environments
+
+The shipped surface is `x db branch`, with three verbs. `x branch` (no `db`) is **planned** and exits `X_NOT_IMPLEMENTED`; the build-id and MCP-socket rows below are what it will add.
+
+```bash
+x db branch create feat-new-billing --json
+x db branch list --json
+x db branch delete feat-new-billing --json
+```
+
+| Property | Detail | `As of 2026-08` |
+|---|---|---|
+| DB | `CREATE DATABASE "<source>_branch_<slug>" TEMPLATE "<source>"` copy-on-write clone — cheap, isolated, disposable. `<slug>` is the name with every character outside `[A-Za-z0-9_]` replaced by `_` — a hyphen is not legal in an unquoted Postgres identifier — so `create feat-new-billing` clones into `myapp_branch_feat_new_billing`. Embedded: a copied `pgdata-<name>` directory, which keeps the name **as typed** | **shipped** |
+| Migrations | `db.migrate` applies here, never to the shared dev DB (`X_MCP_NOT_BRANCH_DB`) | **shipped** |
+| Preview URL | `http://<name>.localhost:<PORT>`, reported on `data.preview` | **the URL is computed**; nothing routes that subdomain for you |
+| Teardown | `x db branch delete <name>` — it may only delete what `list` shows | **shipped** |
+| **Build ID scopes the SW** | a per-branch build id giving the branch its own SW scope and cache namespace, so a preview can never poison prod cache → [PWA and offline](PWA-And-Offline) | **planned**, part of `x branch` |
+| Scoped MCP socket | `ws://localhost:9229/<name>` | **planned**, part of `x branch` |
+| Agent use | an agent can migrate, seed, test, and browse a preview without risking anything shared | |
+
+## `--json` everywhere
+
+Every command and every error has a machine-readable form. Same content, different encoding.
+
+```
+$ x verify --json
+{"ok":false,"checks":[{"name":"budgets","ok":false,"failures":[
+  {"route":"site/pricing","metric":"js","actual":"61kb","limit":"40kb",
+   "cause":"chart.js via shared/ui/button.tsx",
+   "fix":"x fix boundary site/pricing/page.tsx"}]}]}
+```
+
+| Surface | Machine form |
+|---|---|
+| CLI | `--json` on every subcommand |
+| Errors | `UltimateError` serializes to `{ code, cause, fix, docs }` |
+| HTTP errors | same JSON body, same codes |
+| Dev overlay | the identical string a terminal shows |
+| MCP | tool errors carry the same code + fix |
+
+## Errors
+
+| Code | Cause | Fix |
+|---|---|---|
+| `X_MCP_TOOL_UNKNOWN` | no visible tool answers that name (role-hidden and absent are indistinguishable) | `tools/list` to read the catalog this caller may use |
+| `X_INPUT_INVALID` | arguments failed the tool's published `inputSchema` — an `isError` result naming each field (`As of 22.10`) | correct the named fields against `inputSchema` (`tools/list`, or `describe_resource` on the meta surface) and resend |
+| `X_MCP_ARGS_INVALID` | not raised on the wire since 22.10 — invalid arguments answer `X_INPUT_INVALID` as a tool result; `McpArgsInvalidError` stays exported | — |
+| `X_MCP_SURFACE_OVER_BUDGET` | `assertMcpSurfaceBudget` measured `tools/list`, `list_resources` or `instructions` over the budget declared for them | shorten what it names, group tools behind `surface: 'meta'`, or raise the budget with the measured size and the reason in the same diff |
+| `X_MCP_IDEMPOTENCY_KEY_SHADOWED` | an `idempotent: true` action's input declares `idempotencyKey`, the tool argument reserved for the retry key — refused at boot | rename the action's input field |
+| `X_MCP_SCOPE_DENIED` | the connection's token does not carry the tool's scope | reconnect with a token whose scopes include the one `cause` names — the app's `resolveToken(token)` is what returns them — or drop that scope from `defineAppMcp({ scopes })`. Scopes are fixed for the life of a connection, so a grant takes effect on the next one. **Not** `x token grant`: that command is `PLANNED` and exits `X_NOT_IMPLEMENTED` |
+| `X_MCP_SCOPE_UNKNOWN` | `defineAppMcp`'s `scopes:` names a tool the server does not project | spell the name as one of the tools the server actually projects, or drop it from that `scopes` entry |
+| `X_MCP_SCOPE_CONFLICT` | two `scopes:` entries claim the same tool | keep the tool under the single scope a token must hold for it, and remove the other entry |
+| `X_MCP_QUERY_REJECTED` | `db.query` was not given one read-only statement | send exactly one **read-only** `SELECT`/`WITH`/`EXPLAIN`/`SHOW`/`TABLE`/`VALUES` — a data-modifying CTE is not a read |
+| `X_MCP_NOT_BRANCH_DB` | `db.migrate` pointed at a database that is not a branch | `x db branch create <name>   # then retry db.migrate` — point the host at the database the create reported (`DATABASE_URL=…/<source>_branch_<slug>`). The target is read from the database's own name, so a shared one can never pass |
+| `X_MCP_PROTOCOL` | malformed envelope, a JSON-RPC **batch** (an array — this server answers one request per message and never walks one), or an unsupported method — a client bug, not an authz outcome | send a JSON-RPC 2.0 body, one request per `POST /mcp` (one per line over stdio); the `-32600` carries `data: { code, fix }` naming which |
+| `X_MCP_BODY_TOO_LARGE` | one message over `bodyLimitBytes` (HTTP, `413`) or `lineLimitBytes` (stdio) — on a JSON-RPC `-32600` with `data: { code, cause, fix, limit }` | send less in one message, or raise the cap where the route is built: `mcpHttpRoute({ bodyLimitBytes })` / `defineAppMcp({ bodyLimitBytes })` / `serveStdio({ lineLimitBytes })` |
+| `X_MCP_RATE_LIMITED` | this caller spent its per-minute allowance for the request's class ([above](#rate-limits-on-the-http-transport)) | wait out the `Retry-After`, or raise it where the route is built: `mcpHttpRoute({ rateLimits })` / `defineAppMcp({ rateLimits })`. Never `X_RATE_LIMITED`'s buckets — they do not govern this route |
+| `X_FORBIDDEN` | the action's policy refused this actor — identical to the HTTP denial | over an app's MCP: the caller's fix — ask the account owner or an administrator for the permission; a retry is refused the same way. As the app's developer: `policies.list` for the permission this tool enforces, then grant it to the actor's role in `apps/web/shared/policies.ts` |
+| `X_LLM_OUTPUT_INVALID` | model output failed the `output` schema twice | tighten the prompt or widen the schema; bump the prompt version |
+| `X_AGENT_TOOL_UNEXPOSED` | an `agent()` lists an action that is not MCP-exposed and is not an `inProcessTool()` — refused at **declaration** | `inProcessTool({ … })` for a tool only the agent may call, or add `mcp: { expose: true, description }` to the action, or drop it from `tools` |
+| `X_AGENT_MAX_TURNS` | an `agent()` used every turn without answering | tell the template when to stop and answer through the respond tool, then bump its version |
+| `X_HIVE_EMPTY` | a `hive()` split into zero members | return at least one member input, or guard the call site |
+| `X_NOT_IMPLEMENTED` | a remote driver stub was reached | configure the local/PGlite driver, or wait for the release named in `fix` |
+
+Full list: [Error codes](Error-Codes). CLI surface: [CLI reference](CLI-Reference).
+
+## Rules
+
+- One authz system. An MCP call and an HTTP call reach the same `policy` with the same actor resolution.
+- One name. A projected tool is called the primitive's export name, verbatim, in `tools/list`, in `scopes:`, in the LLM tool list, in `openapi.json` and in `describe()`. There is no second spelling to derive.
+- Exposure is opt-in per action; the projection carries no scope of its own — `defineAppMcp`'s `scopes:` map may still attach one, from outside the primitive.
+- Visibility is fail-closed and computed per connection. A hidden tool answers ToolNotFound, never Forbidden.
+- Gate order is visibility → scope → arguments → policy, and every outcome is audited. There is no trusted-tool mode.
+- Write tools are branch-scoped. The dev server is never reachable in `ROLE=web`.
+- Facts are generated every build; conventions are hand-written and short.
+- Never generate prose documentation at runtime.
+- Every command and every error has a `--json` form; budgets throw before spending, in `Money`.
+- Every prompt is a versioned file with an evals file.
+
+Source: [`packages/mcp/src`](https://github.com/developerz-ai/ultimate/blob/main/packages/mcp/src)