@agent-native/core 0.7.51 → 0.7.53

package/docs/content/dispatch.md

@@ -0,0 +1,94 @@
+---
+title: "Dispatch"
+description: "The workspace control plane: secrets vault, integration hub, cross-app delegate, and central inbox for Slack, email, Telegram, WhatsApp."
+---
+
+# Dispatch
+
+Dispatch is the central app that sits in front of every other app in your workspace and handles secrets, integrations, messaging, and cross-app delegation. It is the **workspace control plane** — the single agent your team talks to, the single place credentials live, and the single router that decides which specialist app should handle a given request.
+
+Without Dispatch, every app in a multi-app workspace ends up re-implementing the same plumbing: its own Slack bot, its own secret store, its own scheduled jobs, its own copy of the workspace's instructions. Rotating one API key turns into ten redeployments. Adding a new policy turns into ten copy-pastes. Dispatch centralizes all of that in one app so the others stay focused on their domain.
+
+> Dispatch is shipped as a first-party template. This page covers the **concept** — what it is, why you'd want it, and how it fits into a workspace. For the scaffolded app itself (routes, screens, agent guide), see the [Dispatch template](/templates/dispatch).
+
+## When you want Dispatch {#when}
+
+Reach for Dispatch when any of these are true:
+
+- You're running a [multi-app workspace](/docs/multi-app-workspace) — mail, calendar, analytics, content, recruiting — and you don't want one Slack bot per app.
+- You want **one inbox for "the agent"** so users DM a single bot and the right specialist app picks up the work behind the scenes.
+- You have **workspace-wide secrets** (Stripe key, OpenAI key, third-party API tokens) that several apps need but you'd rather grant per-app than copy into every `.env`.
+- You want a **runtime approval flow** in front of sensitive changes (saved destinations, policy edits) so non-admins can request and admins can sign off without a code deploy.
+- You want **shared skills, instructions, and agent profiles** that every app in the workspace inherits — change once, reach all.
+
+If you're running a single template standalone, you don't need Dispatch — each template can wire its own messaging integrations directly. See [Messaging](/docs/messaging) for the standalone setup.
+
+## What Dispatch does {#what-it-does}
+
+Five capabilities, all sitting on top of the same workspace database the other apps use.
+
+### Central inbox
+
+Slack, email, Telegram, and WhatsApp all flow into Dispatch's agent loop. Connect each platform once in **Settings → Messaging** and every channel reaches the same agent with the same memory and tools. A Slack DM and an email to `agent@yourcompany.com` end up as two surfaces on one conversation history, not two disconnected bots.
+
+See [Messaging](/docs/messaging) for the credentials and webhook URLs for each platform.
+
+### Secret vault
+
+Store credentials once in Dispatch's vault and grant them to the apps that need them. Non-admins can **request** a secret for an app; admins **approve**, which creates the secret + grant in one step. Every read, grant, sync, and rotation is captured in an audit log. `sync-vault-to-app` pushes granted secrets into the target app's env so you don't have to redeploy or re-paste anything.
+
+This is what makes "rotate the OpenAI key" a one-click operation across ten apps instead of ten PRs.
+
+### Cross-app delegation
+
+Dispatch auto-discovers the other apps in your workspace as A2A peers — no manual registration, no per-app config. When a user asks "summarize last week's signups" in Slack, Dispatch recognizes that as an analytics request and calls the analytics app over [A2A](/docs/a2a-protocol). When they ask "draft a reply to Alice", it routes to the mail app. Dispatch posts the final answer back in the originating thread.
+
+The behavioral rule lives in the dispatch agent's instructions: domain work belongs to the domain app. Dispatch is the orchestrator, not the specialist.
+
+### Workspace resources
+
+Skills, agent profiles, and instructions can be authored once in Dispatch and granted out to the rest of the workspace. `sync-workspace-resources-to-all` pushes them to every app's `.agents/` directory so every agent in every app picks them up. This is how a team-wide change ("always use British English in customer-facing replies") propagates without editing ten repos.
+
+### Approval flow
+
+Dispatch can gate sensitive runtime changes behind admin review. Today this covers **saved destinations** (the Slack channels and email addresses the agent can proactively send to) and **dispatch approval policy** itself. When the policy is enabled, the change is queued and the agent surfaces an inline approval preview directly in chat — admins approve or reject without leaving the conversation. Resource-wide approval interception is planned but not yet shipped.
+
+## How a Slack message flows through Dispatch {#flow}
+
+Walk through one example end-to-end. A user DMs the bot: _"summarize last week's signups."_
+
+1. **Slack → webhook.** Slack `POST`s to `/_agent-native/integrations/slack/webhook` on the Dispatch app. The handler verifies the signature and **inserts a row into `integration_pending_tasks`**, then fires a self-targeted `POST` to its own processor and returns `200` immediately so Slack doesn't retry.
+2. **Fresh processor execution.** The processor endpoint runs in a brand-new function execution with its own full timeout. It atomically claims the task and starts the agent loop.
+3. **Dispatch agent decides.** The agent reads the message, recognizes "signups" as an analytics intent, and invokes `call-agent` against the analytics app's [A2A endpoint](/docs/a2a-protocol). The actual SQL work runs over there.
+4. **Reply posted in thread.** The analytics agent returns a result. Dispatch formats it and posts back into the same Slack thread the user wrote in, using the linked identity if there is one (so the agent acts with the requester's permissions, not the workspace owner's).
+5. **Recovery if anything dies.** If the processor crashes mid-flight — A2A timeout, downstream agent error, function freeze — a retry job sweeps stuck tasks every 60 seconds and re-fires the processor. Up to three attempts before the task is marked `failed`. The user still gets a reply on the next sweep instead of the message disappearing into the void.
+
+The same flow applies for email, Telegram, and WhatsApp — only the adapter changes.
+
+## Reliability story {#reliability}
+
+The whole pipeline is built to survive on every serverless host (Netlify, Vercel, Cloudflare Workers) without leaning on platform-specific background-execution APIs.
+
+- **Webhook → SQL queue → fresh-execution processor.** The agent loop never runs inside the webhook handler. The handler's only job is to verify, enqueue, and return 200. A separate fresh execution drains the queue, so a slow agent run can never tie up the inbound webhook or cause the platform to retry.
+- **A2A continuation polling.** When Dispatch delegates to another app, it polls the downstream task with a bounded timeout. If the downstream agent takes too long or crashes, Dispatch records the continuation and the retry job picks it up — the user's Slack reply still arrives.
+- **Auto-signed cross-app A2A.** Hosted multi-app workspaces auto-generate per-app A2A credentials at deploy time, so apps in the same workspace can call each other without you ever pasting a JWT secret. Dispatch's agent-discovery layer reads those creds from the workspace database so newly added apps appear as callable peers automatically.
+
+This is the recently-hardened story — see PRs #439, #441, and #443 for the underlying changes. Conceptually: every step that crosses a network or a process boundary is recoverable.
+
+## Setup {#setup}
+
+Three short steps:
+
+1. **Scaffold a workspace that includes Dispatch.** Run `pnpm dlx @agent-native/core create my-company-platform` and pick `dispatch` alongside whatever domain templates you want. Dispatch lives at `apps/dispatch` and the rest of the apps sit beside it. See [Multi-App Workspace](/docs/multi-app-workspace).
+2. **Connect messaging.** Open **Settings → Messaging** in Dispatch and click connect for Slack, Email, Telegram, or WhatsApp. The form fields match the env vars in the [Messaging](/docs/messaging) doc — refer there for what each platform needs.
+3. **Add other apps.** Run `agent-native add-app` from the workspace root for each domain app. They auto-appear as A2A peers in Dispatch's `list-workspace-apps` — no manual registration, no agent-card editing. Dispatch will start delegating to them as soon as their agent cards are reachable.
+
+Then add credentials to the vault, grant them to the apps that need them, and (optionally) author workspace skills under **Resources** and sync them out.
+
+## See also {#see-also}
+
+- [Dispatch template](/templates/dispatch) — the actual scaffolded app, with its full action catalog and agent guide
+- [Messaging](/docs/messaging) — connecting Slack, email, Telegram, WhatsApp
+- [A2A Protocol](/docs/a2a-protocol) — how cross-app delegation works under the hood
+- [Multi-App Workspace](/docs/multi-app-workspace) — the deployment shape Dispatch is built for
+- [Workspace Management](/docs/workspace-management) — git/GitHub governance that pairs with Dispatch's runtime governance

package/docs/content/faq.md

@@ -67,7 +67,7 @@ The framework ships with production-ready templates you can use as daily drivers
 - **[Video](/templates/video)** — video composition with Remotion
 - **[Analytics](/templates/analytics)** — data platform (like Amplitude/Mixpanel)
 - **[Clips](/templates/clips)** — async screen + camera recording (replaces Loom)
-- **[Design](/templates/design)** — agent-native design tool (like Figma/Canva)
+- **[Design](/templates/design)** — agent-native HTML prototyping studio
 - **[Forms](/templates/forms)** — form builder (like Typeform)
 - **[Dispatch](/templates/dispatch)** — workspace control plane: shared secrets, integrations, jobs
 
@@ -123,7 +123,7 @@ Anywhere. The server runs on Nitro, which compiles to any deployment target: Nod
 
 ### Why polling instead of WebSockets? {#why-polling-not-websockets}
 
-Polling works in every deployment environment — including serverless, edge, and container platforms where persistent connections aren't available. The framework polls every 2 seconds using a lightweight version counter. When changes are detected, React Query caches are invalidated and components re-render. It's simple, reliable, and universal. SSE is also supported as an alternative.
+Polling works in every deployment environment — including serverless, edge, and container platforms where persistent connections aren't available. The framework polls every 2 seconds using a lightweight version counter. When changes are detected, React Query caches are invalidated and components re-render. It's simple, reliable, and universal.
 
 ### Why can't the UI call an LLM directly? {#why-no-inline-llm-calls}
 

package/docs/content/frames.md

@@ -41,7 +41,7 @@ The framework provides type-safe APIs so you never deal with raw messaging:
 
 1. **Agent chat** — use `sendToAgentChat()` to send messages to the agent
 2. **Generation state** — use `useAgentChatGenerating()` to track when the agent is running
-3. **File watching** — SSE endpoint keeps UI in sync when the agent modifies files
+3. **Polling sync** — database-backed sync keeps UI caches fresh when the agent changes state
 4. **Action system** — `pnpm action <name>` dispatches to callable actions
 
 Your app code is identical regardless of how the agent is provided.

package/docs/content/getting-started.md

@@ -51,6 +51,14 @@ Common next steps:
 - **Build a brand-new template from scratch** — see [Creating Templates](/docs/creating-templates) for the full Vite, Tailwind, and TypeScript setup.
 - **Understand the architecture** — see [Key Concepts](/docs/key-concepts) for how SQL, actions, polling sync, and context awareness fit together.
 
+## What's next {#next-steps}
+
+Once your app is running, the most common next steps are:
+
+- **Connect Slack or email** so you can message your agent from anywhere — see [Messaging](/docs/messaging).
+- **Set up Dispatch as your central inbox** to triage messages and orchestrate across multiple apps — see [Dispatch](/docs/dispatch).
+- **Customize via Workspace** — edit instructions, skills, memory, and connect MCP servers per user — see [Workspace](/docs/workspace).
+
 ## Templates {#templates}
 
 Each template is a complete app with UI, agent actions, database schema, and AI instructions ready to go:
@@ -64,7 +72,7 @@ Each template is a complete app with UI, agent actions, database schema, and AI
 | [Video](/templates/video)           | video editing                                    |
 | [Analytics](/templates/analytics)   | Amplitude, Mixpanel, Looker                      |
 | [Clips](/templates/clips)           | Replaces Loom — screen + camera recording        |
-| [Design](/templates/design)         | Figma, Canva                                     |
+| [Design](/templates/design)         | HTML prototyping studios                         |
 | [Forms](/docs/template-forms)       | Typeform                                         |
 | [Dispatch](/docs/template-dispatch) | Workspace control plane — secrets, routing, jobs |
 | [Starter](/docs/template-starter)   | Minimal scaffold — build from scratch            |

package/docs/content/key-concepts.md

@@ -46,7 +46,7 @@ Six rules govern the architecture:
 Adopting the framework is valuable mostly because of what you stop having to build. The moment your app follows the six rules, you inherit:
 
 - **One action = four surfaces.** Every action defined with `defineAction()` is simultaneously an agent tool, a typesafe frontend mutation (`useActionMutation("name")`), an HTTP endpoint at `/_agent-native/actions/:name`, and an MCP tool (when MCP is enabled). External agents can call it over [A2A](/docs/a2a-protocol) too. One implementation, four consumers.
-- **A full workspace per user.** Skills, memory (`learnings.md`), `AGENTS.md`, custom sub-agents, scheduled jobs, connected MCP servers — all SQL-backed, per-user, no dev-box required. See [Workspace](/docs/workspace).
+- **A full workspace per user.** Skills, shared `LEARNINGS.md`, personal `memory/MEMORY.md`, `AGENTS.md`, custom sub-agents, scheduled jobs, connected MCP servers — all SQL-backed, no dev-box required. See [Workspace](/docs/workspace).
 - **Drop-in React components.** `<AgentPanel />` and `<AgentSidebar />` render chat + workspace anywhere in your app. See [Drop-in Agent](/docs/drop-in-agent).
 - **Live sync between agent and UI.** A 2-second poll invalidates React Query caches whenever the agent writes to the DB. No WebSockets, no serverless-unfriendly long-lived connections. See [Polling Sync](#polling-sync) below.
 - **Auth, orgs, RBAC.** Better Auth with orgs/members/roles is wired in for every template. See [Authentication](/docs/authentication).
@@ -262,6 +262,22 @@ Never use Node-specific APIs (`fs`, `child_process`, `path`) in server routes or
 
 Never assume a persistent server process. Serverless and edge environments are stateless — no in-memory caches, no long-lived connections. Use the SQL database for all state.
 
+## Workspace {#workspace}
+
+Every user gets a personal **workspace** — instructions, skills, memory, custom sub-agents, scheduled jobs, and connected MCP servers — all stored in SQL rather than files. That makes Claude-Code-level customization viable inside multi-tenant SaaS without spinning up a container per user. See [Workspace](/docs/workspace).
+
+## Dispatch {#dispatch}
+
+**Dispatch** is the workspace control plane: a central inbox for Slack/email/Telegram, a shared secrets vault, scheduled jobs, and an orchestrator agent that delegates domain work to specialist apps over A2A. Run it alongside your domain apps when you have more than one. See [Dispatch](/docs/dispatch).
+
+## Tools {#tools}
+
+**Tools** are sandboxed mini-apps the agent can create at runtime — Alpine.js HTML rendered inside an iframe, with built-in helpers for persistent storage (`toolData`), calling app actions (`appAction`), and proxied external APIs (`toolFetch`). No source-code changes, no schema migrations. See [Tools](/docs/tools).
+
+## A2A {#a2a}
+
+Agent-to-agent (**A2A**) is how apps in the same workspace discover and call each other. Each app publishes an agent card with skill metadata; other agents can invoke its actions over JSON-RPC. Same-origin deploys skip JWT; cross-origin uses a shared secret. See [A2A Protocol](/docs/a2a-protocol).
+
 ## Deep dives {#deep-dives}
 
 For detailed guidance on specific patterns:

package/docs/content/messaging.md

@@ -7,7 +7,7 @@ description: "Talk to your agent from Slack, email, Telegram, or WhatsApp — sa
 
 Connect your agent to Slack, email, Telegram, or WhatsApp so you can chat with it from the apps you already use. It's the same agent — same memory, same tools, same threads — just reachable from more places.
 
-> **Using the Dispatch template?** All of this is wired up for you in **Settings → Messaging**. Click to connect each platform — you don't need to read the rest of this page unless you're customizing or building your own template. See [Dispatch](/docs/template-dispatch).
+> **Using the Dispatch template?** All of this is wired up for you in **Settings → Messaging**. Click to connect each platform — you don't need to read the rest of this page unless you're customizing or building your own template. See [Dispatch](/docs/dispatch) or the [Dispatch template reference](/docs/template-dispatch).
 
 ## What you can do {#what-you-can-do}
 
@@ -31,11 +31,15 @@ Connect your agent to Slack, email, Telegram, or WhatsApp so you can chat with i
    - `chat:write` — lets the agent send messages
    - `app_mentions:read` — lets the agent see when it's @-mentioned (optional)
    - `im:history` — lets the agent read DMs sent to it
+   - `assistant:write` — optional; lets Slack show native "is thinking..." status in assistant threads
+   - `users:read.email` — optional; helps templates such as Mail verify Slack sender email for draft-queue identity
 3. Click **Install to Workspace** at the top of that page. Slack will give you a **Bot User OAuth Token** that starts with `xoxb-`. Copy it.
 4. Go to **Basic Information** in the sidebar and copy the **Signing Secret**.
 5. Open your app's settings (or your hosting provider's environment variable panel) and paste:
    - `SLACK_BOT_TOKEN` — the `xoxb-…` token
    - `SLACK_SIGNING_SECRET` — the signing secret
+   - `SLACK_ALLOWED_TEAM_IDS` — recommended in production; comma-separated Slack workspace/team IDs allowed to send events
+   - `SLACK_ALLOWED_API_APP_IDS` — recommended for multi-workspace apps; comma-separated Slack app IDs allowed to use this signing secret
 6. Back in Slack, open **Event Subscriptions**, toggle it on, and paste this Request URL:
 
    ```text
@@ -51,6 +55,7 @@ Connect your agent to Slack, email, Telegram, or WhatsApp so you can chat with i
 - **Channel mentions** — the bot only responds in channels when it's @-mentioned, to avoid noise.
 - **DMs** — every DM is treated as a private conversation with the agent.
 - **Same identity, all channels** — if a Slack user has the same email as a registered user in your app, the agent treats them as the same person.
+- **Production allowlists** — set `SLACK_ALLOWED_TEAM_IDS` and, for shared Slack apps, `SLACK_ALLOWED_API_APP_IDS` so a valid signing secret cannot be reused by an unexpected workspace.
 
 ## Set up Telegram {#telegram}
 
@@ -168,7 +173,7 @@ Email is the most powerful integration — your agent gets its own address, repl
 
 ## Use Dispatch as your agent's central inbox {#dispatch}
 
-If you're running multiple agent-native apps (mail, calendar, analytics, etc.), the recommended pattern is to set up messaging on the **[Dispatch template](/docs/template-dispatch)** and let it route work to your domain apps over [A2A](/docs/a2a-protocol).
+If you're running multiple agent-native apps (mail, calendar, analytics, etc.), the recommended pattern is to set up messaging on **[Dispatch](/docs/dispatch)** (see also the [template reference](/docs/template-dispatch)) and let it route work to your domain apps over [A2A](/docs/a2a-protocol).
 
 Why this is nice:
 
@@ -186,19 +191,37 @@ Everything below is the technical reference. If you've finished the setup steps
 
 ### How it works {#how-it-works}
 
-Each platform talks to your app via a standard HTTP webhook:
+Inbound platform webhooks use a cross-platform SQL-queue pattern so they work on every serverless host (Netlify, Vercel, Cloudflare Workers, Fly, Render, Node) without relying on platform-specific background-execution APIs.
 
-1. A user sends a message on Slack/Telegram/WhatsApp/email.
-2. The platform `POST`s to `/_agent-native/integrations/<platform>/webhook`.
-3. The integrations plugin verifies the signature, parses the message, and maps it to an internal conversation thread.
-4. The agent runs the same pipeline as the web chat — same system prompt, same actions, same tools.
-5. The response is posted back to the platform in the same thread.
+1. The platform `POST`s to `/_agent-native/integrations/<platform>/webhook`. The handler verifies the signature, parses the payload into an `IncomingMessage`, and **inserts a row into `integration_pending_tasks`** with `status='pending'`.
+2. The handler fires a fire-and-forget `POST /_agent-native/integrations/process-task` and returns `200` immediately, well inside Slack's 3-second SLA.
+3. The processor endpoint runs in a **fresh function execution** with its own full timeout budget. It atomically claims the task (`pending` → `processing` via `claimPendingTask`), runs the agent loop, posts the reply through the adapter, and marks the task `completed`.
+4. A recurring retry job (`startPendingTasksRetryJob`, every 60s) sweeps tasks stuck in `pending` >90s or `processing` >5min and re-fires the processor. Capped at 3 attempts, then marked `failed`.
 
 ```text
-User → Platform webhook → Verify → Parse → Agent runs → Response posted back
+Platform → /webhook → verify + parse → INSERT pending task ──► return 200
+                                                  │
+                                                  └─ fetch /process-task (fire-and-forget)
+                                                                │
+                                                  fresh exec ──► claim → agent loop → adapter.sendResponse → completed
+
+   (every 60s) retry job: sweep stuck tasks → re-fire /process-task (≤3 attempts)
 ```
 
-No polling, no long-lived connections. Inbound and outbound conversations live in the same SQL thread, so you can continue a Slack DM from the web UI or vice versa.
+Inbound and outbound conversations live in the same SQL thread, so you can continue a Slack DM from the web UI or vice versa.
+
+#### Why this pattern (and not the platform-native shortcuts) {#why-this-pattern}
+
+Serverless functions freeze the moment the response is sent. Anything still running — including a fire-and-forget Promise, a deferred LLM call, or an in-flight tool — gets killed mid-execution. The only way to keep an agent loop alive is to start a **new** function execution for it, which is what the self-fired `/process-task` POST does.
+
+Do NOT use any of these alternatives:
+
+- **Netlify Background Functions** — Netlify-only, requires a `-background.ts` filename suffix, breaks on every other host.
+- **Cloudflare `event.waitUntil()`** — CF Workers only, not portable.
+- **Vercel `after()` / Fluid** — Vercel-only, gated behind specific runtimes.
+- **Naked fire-and-forget Promises after `return`** — silently killed when the function freezes; no error in the logs, the user just never gets a reply.
+
+The SQL-queue + self-webhook + retry-job combination is the only thing that works identically on every supported host. The retry job is the safety net — never assume the initial dispatch flushed before the function froze.
 
 ### The integrations plugin {#plugin}
 
@@ -234,12 +257,12 @@ POST /_agent-native/integrations/telegram/setup
 
 ### Environment variables {#env-vars}
 
-| Platform | Required                                                                     | Optional                       |
-| -------- | ---------------------------------------------------------------------------- | ------------------------------ |
-| Slack    | `SLACK_BOT_TOKEN`, `SLACK_SIGNING_SECRET`                                    | —                              |
-| Telegram | `TELEGRAM_BOT_TOKEN`                                                         | —                              |
-| Email    | `EMAIL_AGENT_ADDRESS`, plus one of `RESEND_API_KEY` or `SENDGRID_API_KEY`    | `EMAIL_INBOUND_WEBHOOK_SECRET` |
-| WhatsApp | `WHATSAPP_ACCESS_TOKEN`, `WHATSAPP_VERIFY_TOKEN`, `WHATSAPP_PHONE_NUMBER_ID` | —                              |
+| Platform | Required                                                                     | Optional                                              |
+| -------- | ---------------------------------------------------------------------------- | ----------------------------------------------------- |
+| Slack    | `SLACK_BOT_TOKEN`, `SLACK_SIGNING_SECRET`                                    | `SLACK_ALLOWED_TEAM_IDS`, `SLACK_ALLOWED_API_APP_IDS` |
+| Telegram | `TELEGRAM_BOT_TOKEN`                                                         | —                                                     |
+| Email    | `EMAIL_AGENT_ADDRESS`, plus one of `RESEND_API_KEY` or `SENDGRID_API_KEY`    | `EMAIL_INBOUND_WEBHOOK_SECRET`                        |
+| WhatsApp | `WHATSAPP_ACCESS_TOKEN`, `WHATSAPP_VERIFY_TOKEN`, `WHATSAPP_PHONE_NUMBER_ID` | —                                                     |
 
 All credentials live in env vars — never the database, never source code. Use the sidebar settings UI or your hosting provider's env panel.
 
@@ -259,7 +282,7 @@ External threads appear in the web UI alongside web-originated threads, tagged w
 
 Every incoming webhook is signature-verified before processing:
 
-- **Slack** — HMAC-SHA256 of the body using `SLACK_SIGNING_SECRET`, checked against the `X-Slack-Signature` header.
+- **Slack** — HMAC-SHA256 of the body using `SLACK_SIGNING_SECRET`, checked against the `X-Slack-Signature` header. The first time you save a Request URL in Slack's Event Subscriptions panel, Slack POSTs a `url_verification` challenge to it; the framework's adapter detects this and replies with the `challenge` value automatically, so the URL flips green in Slack without any extra work on your end.
 - **Telegram** — secret token set when registering the webhook.
 - **WhatsApp** — Meta's verification challenge (using `WHATSAPP_VERIFY_TOKEN`) plus payload signature.
 - **Email** — Svix-style signature verification when `EMAIL_INBOUND_WEBHOOK_SECRET` is set (Resend and SendGrid both use this format). If the secret is unset, the webhook is accepted but a warning is logged.
@@ -317,8 +340,22 @@ export default createIntegrationsPlugin({
 
 Reference implementations live in `packages/core/src/integrations/adapters/` (`slack.ts`, `telegram.ts`, `whatsapp.ts`, `email.ts`) — the email adapter is the most complete example, including signature verification, threading, rate limiting, and HTML rendering.
 
+### Reliability via Dispatch + A2A continuations {#reliability}
+
+When [Dispatch](/docs/dispatch) delegates a request to another app over [A2A](/docs/a2a-protocol#continuations), the continuation-recovery flow guarantees the user gets a Slack/email reply even if the downstream agent crashes mid-execution. The original webhook task stays in `processing` until the continuation either resolves or the retry sweep marks it stuck; either way, the platform thread gets a final reply rather than going silent.
+
+This means a multi-app workspace fronted by Dispatch is more resilient than a single template wired to messaging directly — failures in any one downstream app degrade to a graceful error message instead of a dropped reply. See [A2A continuations](/docs/a2a-protocol#continuations) for the full delivery-guarantee story.
+
+### Common pitfalls {#pitfalls}
+
+- **Don't double-read the request body.** h3 v2's body stream is consume-once: if you call `readBody(event)` after the framework has already parsed `event.node.req.body` (or vice versa), the second read hangs the request indefinitely. This shows up most often with Resend and SendGrid — both stream the inbound payload and the dangling read never resolves, the platform times out, and the webhook gets retried until it dedups. If you wrap the framework's webhook handler in your own middleware, pass the already-parsed `IncomingMessage` via the `incoming` option rather than letting the handler re-parse.
+- **Don't run agent loops inside the webhook handler.** The handler must enqueue and return — the agent loop runs in the processor's fresh execution. Putting it inline guarantees serverless freeze kills the run.
+- **Don't rely on dedup memory across cold starts.** The dedup key lives in the SQL `(platform, external_event_key)` unique index, not an in-process Map. If you replace the queue, keep the SQL-level dedup or duplicate Slack retries will trigger duplicate agent runs.
+- **Keep the self-webhook URL reachable.** The processor URL is built from `APP_URL` / `URL` / `DEPLOY_URL` / `BETTER_AUTH_URL`, falling back to the inbound request headers. On preview deploys with rewritten hostnames, set one of these explicitly or the dispatch will hit a 404.
+
 ### See also {#see-also}
 
-- [Dispatch template](/docs/template-dispatch) — recommended central inbox for multi-app workspaces
-- [A2A Protocol](/docs/a2a-protocol) — how Dispatch delegates work to other agents
+- [Dispatch](/docs/dispatch) — concept overview for using a central inbox across apps
+- [Dispatch template reference](/docs/template-dispatch) — recommended central inbox for multi-app workspaces
+- [A2A Protocol](/docs/a2a-protocol) — how Dispatch delegates work to other agents, including continuation recovery
 - [Agent Mentions](/docs/agent-mentions) — `@`-mentioning agents inside the web chat

package/docs/content/multi-app-workspace.md

@@ -1,9 +1,11 @@
 ---
-title: "Multi-App Workspace"
+title: "Multi-App Workspaces"
 description: "Host many agent-native apps in one monorepo with shared auth, RBAC, instructions, skills, components, and credentials."
 ---
 
-# Multi-App Workspace
+# Multi-App Workspaces
+
+> For what a workspace _is_ — the customization layer, `AGENTS.md`, `LEARNINGS.md`, personal memory, skills, and custom agents — see [Workspace](/docs/workspace). This page is about the **deployment shape**: hosting many agent-native apps in one monorepo with shared auth, components, and a unified deploy.
 
 When vibe-coding an internal tool takes an afternoon, you don't stop at one. A team ends up with a CRM, a support inbox, a dashboard, a recruiting tracker, an ops console — ten small apps, each scaffolded independently. That's great until you need to change something in all of them.
 
@@ -239,3 +241,9 @@ The workspace pattern is intentionally narrow. A few things it deliberately does
 - **Encrypted credential vault.** Shared credentials live in the `settings` table as plain text today. Rotate responsibly.
 - **Publishing shared code to private npm.** The shared package is `workspace:*` only; multi-repo sharing via a private registry is doable but not scaffolded.
 - **Opinionated component library.** `packages/shared` is where _you_ put shared components. The framework doesn't force shadcn/ui or any other system into that slot.
+
+## See also {#see-also}
+
+- [Workspace](/docs/workspace) — the customization layer (`AGENTS.md`, `LEARNINGS.md`, personal memory, skills, custom agents) every app in the workspace shares.
+- [Workspace Governance](/docs/workspace-management) — branching, CODEOWNERS, PR review across many apps in one repo.
+- [Dispatch](/docs/dispatch) — the runtime control plane that typically lives inside a multi-app workspace as the secrets vault, integration catalog, and approvals hub.

package/docs/content/notifications.md

@@ -195,5 +195,5 @@ Automations can chain off this — e.g. _"if a critical notification fires, also
 ## What's next
 
 - [**Automations**](/docs/automations) — the most common caller of `notify()`
-- [**Secrets**](/docs/security) — the `${keys.NAME}` substitution that powers the webhook channel
+- [**Security**](/docs/security) — the `${keys.NAME}` substitution that powers the webhook channel
 - [**Server plugins**](/docs/server) — where custom channels are registered at startup

package/docs/content/observability.md

@@ -0,0 +1,184 @@
+---
+title: "Observability"
+description: "Agent traces, evals, feedback, A/B experiments, and the admin dashboard — all built-in with zero configuration."
+---
+
+# Agent Observability
+
+Every agent-native app gets observability out of the box. Traces, automated evals, user feedback, and A/B experiments work with zero configuration — all data lives in the app's own SQL database.
+
+## What's Captured Automatically
+
+When a user sends a message, the framework automatically records:
+
+- **Token usage** — input, output, cache read, cache write
+- **Cost** — computed from token counts and model pricing
+- **Latency** — total duration and time per tool call
+- **Tool calls** — which actions were invoked, success/error status, duration
+- **Automated evals** — 5 quality scores computed after every run
+
+No code changes needed. The instrumentation hooks into `production-agent.ts` transparently.
+
+## The Dashboard
+
+Add the dashboard to any template with a single route:
+
+```tsx
+// app/routes/observability.tsx
+import { ObservabilityDashboard } from "@agent-native/core/client";
+
+export default function ObservabilityPage() {
+  return (
+    <div className="min-h-screen bg-background p-6">
+      <ObservabilityDashboard />
+    </div>
+  );
+}
+```
+
+The dashboard has 5 tabs:
+
+| Tab               | What it shows                                                                   |
+| ----------------- | ------------------------------------------------------------------------------- |
+| **Overview**      | Key metrics — runs, cost, latency, tool success rate, satisfaction, eval score  |
+| **Conversations** | Trace list with drill-down to individual spans (agent_run, llm_call, tool_call) |
+| **Evals**         | Automated eval scores by criteria, trends over time                             |
+| **Experiments**   | A/B test list with status badges, variant results with confidence intervals     |
+| **Feedback**      | Thumbs up/down stream, category breakdown, frustration scores                   |
+
+## User Feedback
+
+### Explicit Feedback
+
+Thumbs up/down buttons render inline on every agent message in the chat UI. Thumbs down opens a category popover (Inaccurate, Not helpful, Wrong tool, Too slow). This is wired into `AssistantChat.tsx` automatically.
+
+### Implicit Feedback (Frustration Index)
+
+The framework computes a Frustration Index (0-100) from conversation signals:
+
+| Signal         | Weight | What it detects                     |
+| -------------- | ------ | ----------------------------------- |
+| Rephrasing     | 30%    | User repeats similar messages       |
+| Retry patterns | 20%    | "Try again", "no that's wrong"      |
+| Abandonment    | 20%    | Session ends shortly after response |
+| Sentiment      | 15%    | Negative language patterns          |
+| Length trend   | 15%    | Declining message lengths           |
+
+Score interpretation: 0-20 = healthy, 20-40 = friction, 40-60 = dissatisfied, 60+ = broken session.
+
+## Automated Evals
+
+Five deterministic scorers run after every agent run:
+
+| Criteria            | What it measures                                       | Score range |
+| ------------------- | ------------------------------------------------------ | ----------- |
+| `tool_success_rate` | % of tool calls without errors                         | 0-1         |
+| `step_efficiency`   | Penalizes excessive LLM iterations for tool-using runs | 0-1         |
+| `latency_score`     | Normalized against 10s/tool baseline                   | 0-1         |
+| `cost_efficiency`   | Normalized against cost baseline                       | 0-1         |
+| `error_recovery`    | Did the agent recover from tool errors?                | 0 or 1      |
+
+### LLM-as-Judge (Optional)
+
+Enable sampled LLM-based evaluation by setting `evalSampleRate`:
+
+```ts
+import { putSetting } from "@agent-native/core/settings";
+
+await putSetting("observability-config", {
+  enabled: true,
+  evalSampleRate: 0.05, // 5% of runs
+});
+```
+
+Custom criteria use natural language rubrics:
+
+```ts
+const criteria = {
+  name: "helpfulness",
+  description: "Was the response helpful and complete?",
+  rubric: "0.0 = unhelpful, 0.5 = partially helpful, 1.0 = fully resolved",
+};
+```
+
+## A/B Experiments
+
+Test different models, temperatures, or agent configurations:
+
+```ts
+// Create via API
+POST /_agent-native/observability/experiments
+{
+  "name": "sonnet-vs-haiku",
+  "variants": [
+    { "id": "control", "weight": 50, "config": { "model": "claude-sonnet-4-6" } },
+    { "id": "treatment", "weight": 50, "config": { "model": "claude-haiku-4-5-20251001" } }
+  ],
+  "metrics": ["cost", "latency", "satisfaction"]
+}
+
+// Start the experiment
+PUT /_agent-native/observability/experiments/:id
+{ "status": "running" }
+```
+
+The agent loop automatically resolves the user's variant and applies the config override. Assignment uses consistent hashing — same user always gets the same variant.
+
+## Configuration
+
+All settings are stored in the `observability-config` key:
+
+```ts
+{
+  enabled: true,           // Master switch
+  capturePrompts: false,   // Store prompt content in traces
+  captureToolArgs: false,  // Store action input arguments
+  captureToolResults: false, // Store action results
+  evalSampleRate: 0,       // 0-1, fraction of runs to LLM-judge
+  exporters: []            // OTLP export targets
+}
+```
+
+Content is **redacted by default** — only token counts, costs, and timing are stored. Opt in to content capture when needed for debugging.
+
+## API Endpoints
+
+All auto-mounted at `/_agent-native/observability/`:
+
+| Method | Path                       | Purpose                        |
+| ------ | -------------------------- | ------------------------------ |
+| GET    | `/`                        | Overview stats                 |
+| GET    | `/traces`                  | List trace summaries           |
+| GET    | `/traces/:runId`           | Trace detail (summary + spans) |
+| GET    | `/traces/:runId/evals`     | Evals for a run                |
+| POST   | `/feedback`                | Submit feedback                |
+| GET    | `/feedback`                | List feedback                  |
+| GET    | `/feedback/stats`          | Feedback aggregation           |
+| GET    | `/satisfaction`            | Satisfaction scores            |
+| GET    | `/evals/stats`             | Eval statistics                |
+| POST   | `/experiments`             | Create experiment              |
+| GET    | `/experiments`             | List experiments               |
+| PUT    | `/experiments/:id`         | Update experiment              |
+| POST   | `/experiments/:id/results` | Compute results                |
+| GET    | `/experiments/:id/results` | Get results                    |
+
+All endpoints support `?since=N` (ms timestamp) and `?limit=N` query params.
+
+## Export to External Platforms
+
+Send traces to Langfuse, Datadog, Grafana, or any OTel-compatible backend:
+
+```ts
+await putSetting("observability-config", {
+  enabled: true,
+  exporters: [
+    {
+      type: "otlp",
+      endpoint: "https://cloud.langfuse.com/api/public/otel",
+      headers: { Authorization: "Bearer sk-..." },
+    },
+  ],
+});
+```
+
+The framework emits `gen_ai.*` semantic convention spans compatible with the OpenTelemetry GenAI spec.

package/docs/content/onboarding.md

@@ -1,3 +1,8 @@
+---
+title: "Onboarding & API Keys"
+description: "Setup checklist for first-run configuration — API keys, OAuth, and provider connections"
+---
+
 # Onboarding
 
 When you first open an app built on the agent-native framework, you'll see a
@@ -33,7 +38,7 @@ behind a picker or disclosure when a step has several equivalent providers.
   OAuth provider, or email provider, paste the value(s), click **Save**.
   Secret fields use a password input so the value isn't shown on screen. Saved
   values go into your local `.env` (or workspace settings) — see
-  [Secrets](/docs/secrets) for where they live.
+  [Security](/docs/security) for where they live.
 - **Open a link** — some steps point to a sign-in page or docs. Click
   **Continue** and finish the flow in the new tab.
 - **Ask the agent** — a few steps offer a "Let the agent set it up" option.
@@ -174,6 +179,6 @@ function MySidebar() {
 ```
 
 For background on where step values are stored and how secrets are handled,
-see [Secrets](/docs/secrets). For end-user messaging touchpoints (invitations,
+see [Security](/docs/security). For end-user messaging touchpoints (invitations,
 password resets) that depend on the **Email delivery** step, see
 [Messaging](/docs/messaging).