@agent-native/core 0.52.0 → 0.54.0

package/dist/templates/workspace-core/.agents/skills/harness-agents/SKILL.md

@@ -80,6 +80,26 @@ existing run routes as `goalId=agent-harness`.
   Preserve `defineAction` auth, request context, timeouts, truncation, and
   read-only metadata.
 
+## Code Execution Sandbox
+
+- The `run-code` tool executes through a pluggable `SandboxAdapter`
+  (`packages/core/src/coding-tools/sandbox/`). The default
+  `LocalChildProcessAdapter` spawns a locked-down local Node child process;
+  swap it via `AGENT_NATIVE_SANDBOX` or `registerSandboxAdapter()` for a
+  Docker/remote/durable backend (the lever to exceed the hosted ~40s code-exec
+  ceiling). An adapter only runs the already-prepared, non-secret module source
+  — it never sees app secrets. See the Sandbox Adapters doc; `agent-native add
+  sandbox docker` emits a full Docker-adapter recipe.
+
+## Sub-Agent Delegation Depth
+
+- Sub-agent spawning is capped server-side (default depth `2`) so delegation
+  chains can't fan out indefinitely. Override at deploy time with
+  `AGENT_NATIVE_MAX_SUBAGENT_DEPTH` (`0` disables sub-agents; clamped to `16`).
+  Enforcement is ambient via `evaluateSubagentDepth` in
+  `packages/core/src/server/agent-teams.ts` — independent of any tool-level
+  guard. See the Agent Teams doc for the depth model.
+
 ## Don't
 
 - Don't add Claude Code, Codex, Cursor, Mastra, or Pi as an `AgentEngine`.

package/dist/templates/workspace-core/.agents/skills/observability/SKILL.md

@@ -75,6 +75,26 @@ const criteria: EvalCriteria = {
 };
 ```
 
+#### Evals (CI gate)
+
+The three layers above score *real production runs* after the fact. For an active, deterministic gate, use the first-class `*.eval.ts` primitive from `@agent-native/core/eval` (source: `packages/core/src/eval/*`). It runs the actual agent loop against fixed inputs and exits non-zero below threshold, so it gates CI/deploys.
+
+```ts
+// evals/faq.eval.ts
+import { defineEval, contains, llmJudge } from "@agent-native/core/eval";
+
+export default defineEval({
+  name: "answers the FAQ",
+  input: { prompt: "What is your return policy?" },
+  threshold: 0.7,
+  scorers: [contains("30 days"), llmJudge({ criteria: "accuracy" })],
+});
+```
+
+- Built-in scorers: `exactMatch` / `contains` / `usesTool` (pure JS) and `llmJudge` (provider-agnostic judge).
+- Custom scorers: `createScorer` with the 4-step `preprocess → analyze → generateScore → generateReason` pipeline (only `generateScore` is required).
+- Run as a gate: `agent-native eval [pattern] [--json] [--threshold N]` — discovers `**/*.eval.ts` and `evals/*.ts`, runs the agent, and exits non-zero if any eval is below its threshold. An app with no eval files exits `0`. Complements (does not replace) the post-hoc scoring in `evals.ts`. See the Evals doc.
+
 ### 4. Experiments
 
 A/B testing with sticky user-level assignment:
@@ -200,3 +220,14 @@ await putSetting("observability-config", {
 ```
 
 The framework emits `gen_ai.*` semantic convention spans compatible with Langfuse, Datadog, Grafana, New Relic, and any OTel-compatible backend.
+
+## Live OpenTelemetry Spans (Optional)
+
+Separate from the `exporters` config above (which ships the in-house traces to an OTLP endpoint), the agent loop can also emit **live OpenTelemetry spans** for every run, model call, and tool call, so a host that already runs an OTel collector sees agent activity alongside its other distributed traces.
+
+This layer is optional and **no-op by default**:
+
+- `@opentelemetry/api` is an **optional dependency**. If it isn't installed, the span helpers degrade to silent no-ops — they never throw into the agent loop.
+- Even with the api package installed, it ships a default no-op tracer. Spans become real only once the **host registers a `TracerProvider`** (via `@opentelemetry/sdk-node` or similar). The framework deliberately does not depend on the heavy SDK/exporter packages and never registers a provider itself — instrumentation is opt-in by the embedding app.
+
+The loop emits `agent.run` (with `agent.run_id`, `agent.thread_id`, `agent.user_id`, `agent.model`), `tool.call` (`tool.name` + status), and `llm.call` spans, each finished with OK/ERROR status. This is purely additive to the in-house `agent_trace_spans` / `agent_trace_summaries` tables. Source: `packages/core/src/observability/tracing.ts` + `traces.ts`. See the Observability doc for the full table.

package/dist/templates/workspace-core/.agents/skills/security/SKILL.md

@@ -139,6 +139,28 @@ export default defineEventHandler(async (event) => {
 
 - Never create unprotected routes that modify data.
 
+## Human-in-the-Loop Approval for High-Consequence Actions
+
+For a small set of outward-facing, hard-to-undo operations — sending an email, charging a card, deleting an account, posting publicly — auth and access control are necessary but not sufficient: you also do not want the **agent** to perform them autonomously. Set `needsApproval` on the `defineAction` so the agent cannot run the action without a human approving the specific call.
+
+```ts
+export default defineAction({
+  description: "Send an email via Gmail.",
+  schema: z.object({ to: z.string(), subject: z.string(), body: z.string() }),
+  needsApproval: true, // or (args, ctx) => boolean | Promise<boolean>
+  run: async (args) => {
+    /* ...actually send... */
+  },
+});
+```
+
+When the gate is truthy and the call is not yet approved, the loop emits an `approval_required` event and **stops the turn — `run()` never executes**. The human approves via the chat UI's Approve affordance, which re-issues the turn with the call's stable `approvalKey`; only then does the action run. A predicate gates conditionally (e.g. only external recipients) and **fails closed** — a throw is treated as "approval required".
+
+Rules:
+
+- Reach for `needsApproval` only for genuinely high-consequence operations. The default is off, and the framework intentionally keeps approvals rare — over-gating turns the agent into a click-through wizard. The canonical (and intentionally lone) framework example is Mail's `send-email`.
+- `needsApproval` is **not** a substitute for `accessFilter` / `assertAccess` or for hiding sensitive operations from the model with `agentTool: false` / `toolCallable: false`. It is the layer for "a human must explicitly bless this specific outward-facing call," not for scoping data. See the `actions` skill for the full surface.
+
 ## Custom HTTP Routes Must Apply Access Control Themselves
 
 This is the single most-failed rule in the codebase. Auto-mounted action routes (`/_agent-native/actions/...`) get a request context wired up automatically. **Hand-written `/api/*` Nitro routes do not.** If your handler queries an ownable resource (any table with `...ownableColumns()`), you MUST:

package/docs/content/actions.md

@@ -49,6 +49,35 @@ That's it. The framework auto-discovers every file in `actions/` and mounts them
 
 The schema is converted to JSON Schema for the Claude API tool definition, _and_ used at runtime to validate inputs before `run()` fires. Invalid inputs never reach your handler.
 
+### Validating the return value {#output-schema}
+
+`schema` validates _inputs_. To also validate what an action **returns**, pass an `outputSchema` (any Standard Schema-compatible schema — Zod, Valibot, ArkType, same surface as `schema`). The framework validates the result _after_ `run()` resolves, composing with input validation: input validated before `run`, output validated after.
+
+```ts
+export default defineAction({
+  description: "Summarize a thread.",
+  schema: z.object({ threadId: z.string() }),
+  outputSchema: z.object({
+    summary: z.string(),
+    messageCount: z.number(),
+  }),
+  outputErrorStrategy: "warn", // default
+  run: async ({ threadId }) => {
+    /* ...returns { summary, messageCount } ... */
+  },
+});
+```
+
+`outputErrorStrategy` controls what happens on a mismatch:
+
+| Strategy     | Behavior on mismatch                                                                               |
+| ------------ | -------------------------------------------------------------------------------------------------- |
+| `"warn"`     | **Default.** `console.warn` the issues and return the **original** result unchanged. Non-breaking. |
+| `"strict"`   | Throw a clear error so a buggy action surfaces loudly.                                             |
+| `"fallback"` | Return the provided `outputFallback` value in place of the invalid result.                         |
+
+On success, the **validated** value is returned, so any coercion or defaults defined on the `outputSchema` take effect (mirroring the input path). When no `outputSchema` is supplied, behavior is byte-for-byte unchanged — there is no wrapping. This is borrowed from Mastra/Flue structured-output and kept dependency-free on the action layer.
+
 ### HTTP config {#http}
 
 By default every action is exposed as `POST /_agent-native/actions/<name>`. Override with the `http` option:
@@ -221,6 +250,26 @@ export default defineAction({
 
 For list and read actions, use `accessFilter` to scope the query to the current user and org. For actions that update or delete a specific row, use `assertAccess` to confirm the caller is allowed before writing. See [Security](/docs/security#access-guards) and [Sharing](/docs/sharing) for the full helper API.
 
+### Human-in-the-loop approval {#needs-approval}
+
+A handful of actions are too consequential to let the agent run autonomously — sending an email, charging a card, deleting an account. For those, set `needsApproval` to pause the loop and require a human to approve the specific call before `run()` executes:
+
+```ts
+export default defineAction({
+  description: "Send an email via Gmail.",
+  schema: z.object({ to: z.string(), subject: z.string(), body: z.string() }),
+  needsApproval: true, // pause; a human must approve this specific send
+  run: async (args) => {
+    /* ...actually send... */
+  },
+});
+```
+
+`needsApproval` accepts a boolean or a predicate `(args, ctx) => boolean | Promise<boolean>` to gate conditionally (e.g. only external recipients, only above a threshold). The predicate **fails closed**: a throw is treated as "approval required". When the gate is truthy and the call isn't yet approved, the loop emits an `approval_required` event and stops the turn — the side effect never happens — and the action runs only once a human approves via the chat UI's Approve affordance.
+
+> [!WARNING]
+> Keep approvals rare. Each gated action is a hard stop in the agent loop. The default is **off**, and almost every action should leave it off. See [Human-in-the-Loop Approvals](/docs/human-approval) for the full flow.
+
 ## Calling it from the UI {#ui}
 
 Two hooks, both in `@agent-native/core/client`. Types are inferred from your `defineAction` schemas — no manual type declarations.
@@ -390,6 +439,7 @@ const args = parseArgs(["--name", "Steve", "--verbose", "--count=3"]);
 
 ## What's next
 
+- [**Human-in-the-Loop Approvals**](/docs/human-approval) — the `needsApproval` gate in depth
 - [**Drop-in Agent**](/docs/drop-in-agent) — `useActionMutation` / `useActionQuery` in React
 - [**Context Awareness**](/docs/context-awareness) — the `view-screen` + `navigate` pattern in depth
 - [**A2A Protocol**](/docs/a2a-protocol) — how other agents discover and call your actions

package/docs/content/agent-teams.md

@@ -141,6 +141,38 @@ You are a meticulous code reviewer. Be terse and concrete — cite file:line whe
 
 Store at `agents/code-review.md` in the workspace. It appears in the `@mention` dropdown and is available to the main agent as a delegation target. See [Workspace — Custom Agents](/docs/workspace#custom-agents) for the full format including `tools`, `delegate-default`, and model overrides.
 
+## Delegation depth guard {#depth-guard}
+
+Sub-agents can spawn sub-agents, which is a runaway/cost risk: an unbounded chain of delegations could fan out indefinitely. The framework enforces a **hard cap on delegation depth**, server-side, independent of any tool-level guard.
+
+The top-level chat is depth `0`. A sub-agent it spawns is depth `1`; that sub-agent may spawn once more (depth `2`); a spawn that would create a depth-`3` sub-agent is **refused**. The default cap is **2**.
+
+```txt
+depth 0  top-level chat        (may spawn)
+depth 1  sub-agent             (may spawn)
+depth 2  sub-agent's sub-agent (at the cap — may NOT spawn)
+depth 3  refused
+```
+
+Enforcement is ambient: each sub-agent runs inside an `AsyncLocalStorage` that records its own depth, so any `spawnTask` reached transitively from that run reads its parent's depth and refuses once the cap is hit — even if the `agent-teams` tool was handed to a sub-agent that should not have had it. The decision is exposed as a pure, unit-testable `evaluateSubagentDepth(parentDepth)`. A refused spawn returns a clear error: _"Delegation depth limit reached (max N); cannot spawn another sub-agent."_
+
+### Configuring the cap {#depth-guard-config}
+
+Override the default at deploy time with `AGENT_NATIVE_MAX_SUBAGENT_DEPTH`:
+
+| Value           | Effect                                                                                                                                |
+| --------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
+| _(unset)_       | Default cap of `2`.                                                                                                                   |
+| `0`             | **No sub-agents may be spawned** — the top-level agent does all work.                                                                 |
+| `1`…`16`        | That many levels of delegation.                                                                                                       |
+| invalid / `>16` | A non-integer / negative / NaN value falls back to `2`; anything above `16` is clamped to `16` so a typo can never disable the guard. |
+
+```bash
+AGENT_NATIVE_MAX_SUBAGENT_DEPTH=1   # sub-agents allowed, but they can't sub-delegate
+```
+
+When a sub-agent is at or below the cap, the framework injects a line into its runtime context telling it how deep it sits and whether it may delegate further, so the model spends its budget appropriately.
+
 ## What's next
 
 - [**Workspace — Custom Agents**](/docs/workspace#custom-agents) — the profile format

package/docs/content/blueprint-installer.md

@@ -0,0 +1,73 @@
+---
+title: "Blueprint Installer"
+description: "agent-native add prints a curated Markdown integration recipe to stdout — pipe it into your coding agent, which applies the changes against your live repo."
+---
+
+# Blueprint Installer
+
+`agent-native add` is **not** a dumb scaffolder that writes files for you. It emits a curated Markdown _integration blueprint_ to stdout. You pipe that blueprint into your own coding agent (Claude Code, Codex, …), which applies the changes against the live repo with full context.
+
+This fits the agent-applies-changes, filesystem-first house style: the framework supplies the recipe (the canonical files to touch, the rules to honor, the verification step), and the coding agent does the editing.
+
+```bash
+agent-native add provider stripe | claude
+agent-native add channel discord  | codex
+```
+
+## Usage {#usage}
+
+```bash
+agent-native add <kind> <name>            # print a curated blueprint
+agent-native add <kind> <https://docs…>   # research-and-integrate from a URL
+agent-native add --list                   # list available kinds and blueprints
+```
+
+- A bare **name** resolves a curated blueprint from `blueprints/<kind>/<name>.md`.
+- A **URL** instead of a name emits a generic _research-and-integrate_ blueprint for that kind, with the URL embedded as the research starting point (a URL is a research seed, not a known recipe).
+- The blueprint goes to **stdout**; diagnostics go to stderr, so `… | claude` only ever receives the blueprint.
+
+## Seeded blueprints {#seeded}
+
+`agent-native add --list` shows what ships in the box:
+
+| Kind       | Name      | What it sets up                                                                    |
+| ---------- | --------- | ---------------------------------------------------------------------------------- |
+| `provider` | `stripe`  | Wire a provider into the `provider-api` substrate (catalog / docs / request trio). |
+| `channel`  | `discord` | Implement a `PlatformAdapter` inbound webhook channel and register it.             |
+| `sandbox`  | `docker`  | Implement the `SandboxAdapter` seam to run `run-code` in a Docker container.       |
+| `action`   | `crud`    | Add a single multi-surface `defineAction` with a Zod schema (one `update` over N). |
+
+Each blueprint is self-contained: the coding agent reading it gets the files to touch, the framework rules to honor (actions are the single source of truth, never hardcode secrets, scope ownable data, add a changeset for `packages/*` source), and a concrete **Verify** section.
+
+## URL → research blueprint {#url}
+
+When you pass a URL the kind doesn't have a curated recipe for (or want a fresh integration), `add` emits a generic "research-and-integrate" blueprint with the URL as the seed:
+
+```bash
+agent-native add provider https://docs.example.com/api | claude
+```
+
+The generated blueprint tells the coding agent to fetch the URL (and the pages it links to) for the real endpoints, auth model, payload shapes, and signature/verification requirements — _not_ to guess from training data — then implement and verify. It also carries kind-specific guidance (e.g. a `provider` URL is steered toward the `provider-api` substrate; a `channel` URL toward a `PlatformAdapter`).
+
+## Adding your own blueprint {#authoring}
+
+Drop a Markdown file into `packages/core/blueprints/<kind>/<name>.md`. The kind is the subdirectory; the name is the filename without `.md`. It is picked up automatically — `--list`, name resolution, and the catalog all read the directory at runtime. No code change is needed to register it.
+
+Blueprint `.md` files ship in the published package via the `blueprints` entry in `package.json` `files`, so they resolve at `node_modules/@agent-native/core/blueprints/**` for end users.
+
+Write each blueprint as an instruction set for a coding agent with no other context. A good blueprint has:
+
+1. **A one-line goal** and a "you are a coding agent in an agent-native app, apply these as real source changes" framing.
+2. **Read first** — the exact files that _are_ the contract.
+3. **Files to touch** — concrete paths and what each change does.
+4. **Framework rules to honor** — actions-first, no hardcoded secrets, scope ownable data, add a changeset for publishable-package source.
+5. **Verify** — typecheck, a focused `*.spec.ts`, and an end-to-end check.
+
+> [!TIP]
+> A new curated blueprint under an existing kind needs no code — but if you create a brand-new kind directory, that kind shows up in `--list` automatically too.
+
+## What's next
+
+- [**Sandbox Adapters**](/docs/sandbox-adapters) — the seam the `add sandbox docker` blueprint targets
+- [**Actions**](/docs/actions) — the single source of truth every blueprint builds on
+- [**External Agents**](/docs/external-agents) — connecting the coding agent you pipe blueprints into

package/docs/content/durable-resume.md

@@ -0,0 +1,49 @@
+---
+title: "Durable Resume"
+description: "When a hosted agent run is interrupted and resumes, completed side-effecting tool calls are not re-run — a tool-call journal derived from the durable ledger blocks duplicate sends, charges, and tickets."
+---
+
+# Durable Resume
+
+Hosted agent runs get interrupted: a serverless function hits its hard timeout mid-stream, a gateway drops the connection at 45s, a socket hangs up, the platform cold-starts. The framework already recovers from these by saving the conversation prefix and re-running the LLM call ("continue from where you left off"). But recovery alone has a sharp edge: if the interrupted attempt **already sent an email or created a ticket**, a naive resume could do it again.
+
+Durable resume closes that gap. On resume, the framework knows which side-effecting tool calls already completed and refuses to re-run them — at two layers.
+
+## The tool-call journal {#journal}
+
+The journal is a **pure read over the durable run-event ledger** — there is no new recording hook in the hot path. It classifies the tool calls already recorded for the current turn:
+
+- **Completed** — a `tool_start` with a matching `tool_done`. The call ran, its side effect happened, and its result was recorded. **Do not re-run.**
+- **Interrupted** — a `tool_start` with **no** matching `tool_done`. The call began, its side effect may or may not have landed, and the interruption ate the result. Outcome unknown.
+
+Matching mirrors how durable turns are rebuilt elsewhere: a `tool_done` pairs with the oldest still-open `tool_start` for the same tool name (FIFO per tool). A `clear` event (discarded partial output) resets the per-turn tally so abandoned partials don't leave phantom open calls.
+
+## Layer 1: prompt-level journal note {#prompt-note}
+
+When a run resumes (soft timeout, gateway timeout, or any resumable transport error), the framework appends a **structured journal note** to the resume prompt, right after the "continue from where you left off" nudge. The note tells the model, in plain text:
+
+- which tool calls **already completed** (with short results) so it reuses them and does **not** re-run them, and
+- which tool calls were **interrupted with unknown outcome** so it verifies state before assuming success or failure.
+
+When the journal is empty (a turn with no tool activity, or a clean continuation), nothing extra is appended and resume behavior is byte-for-byte what it was before. The note is best-effort: a failed ledger read never blocks a recovery that would otherwise succeed.
+
+## Layer 2: tool-layer hard-block {#hard-block}
+
+The prompt note is advisory — a well-behaved model heeds it, but a model isn't a guarantee. So the loop also enforces it at the tool layer.
+
+Before the loop runs in a resumed chunk, it snapshots the journal once (capturing only **prior** chunks of this logical turn). When the model re-dispatches a **write** tool whose tool name **and input** match a completed journal entry, the loop short-circuits: it returns the journaled result instead of executing the action, with a note that the call already completed in an earlier interrupted attempt and was not re-run to avoid a duplicate side effect.
+
+Key properties:
+
+- **Write tools only.** Read-only (`readOnly` / GET) actions are never blocked — re-reading is safe and idempotent.
+- **Content-addressed.** The match is on tool name + input signature, so a resumed call sitting at a different position in the turn still matches; a _different_ call (different args) is treated as fresh and runs normally.
+- **Consume-once.** Each completed entry is claimed when matched, so two genuinely-distinct identical fresh calls in the same turn don't both short-circuit on one journaled completion.
+- **Fresh calls untouched.** A first-turn call sees an empty journal; nothing changes for normal runs.
+
+Together the two layers mean an interrupted run that already had a real side effect resumes without repeating it — no duplicate emails, charges, or tickets — while genuinely new work still runs.
+
+## Related
+
+- [**Real-Time Sync**](/docs/real-time-collaboration) — how the durable run ledger streams to the client and replays on reconnect.
+- [**Actions**](/docs/actions) — `readOnly` marks reads as safe to re-run; everything else is treated as side-effecting.
+- [**In-Loop Processors**](/docs/processors) — another loop-internal hardening seam.

package/docs/content/evals.md

@@ -0,0 +1,141 @@
+---
+title: "Evals (CI Gate)"
+description: "Write *.eval.ts test cases that run the real agent against fixed inputs, score the output with composable scorers, and gate CI/deploys on a threshold."
+---
+
+# Evals (CI Gate)
+
+Evals are a first-class testing primitive: you declare a prompt plus the behavior you expect, the runner **actually runs the agent loop** against that input, scores the output with composable scorers, and exits non-zero if any case scores below its threshold. That non-zero exit makes `agent-native eval` a drop-in CI deploy gate.
+
+This is complementary to the post-hoc scoring in [Observability](/docs/observability):
+
+- **Observability evals** (`observability/evals.ts`) — _"how did this real run do?"_ Passive, sampled, lives next to traces.
+- **`*.eval.ts` (this primitive)** — _"does the agent do the right thing on this fixed input?"_ Active, deterministic, a CI gate run via the CLI.
+
+The runner resolves a provider-agnostic engine/model from the existing registry — no model is hardcoded — so the same suite runs against whatever engine the app is configured for.
+
+## Writing an eval {#writing}
+
+Drop a `*.eval.ts` file anywhere in the app (or an `evals/*.ts` file). Each file `export default defineEval(...)` (or exports an array of them):
+
+```ts
+// evals/greeting.eval.ts
+import { defineEval, contains, llmJudge } from "@agent-native/core/eval";
+
+export default defineEval({
+  name: "greets the user by name",
+  input: { prompt: "Say hi to Ada." },
+  threshold: 0.7, // per-scorer pass bar; default 0.5
+  scorers: [
+    contains("Ada"),
+    llmJudge({ criteria: "friendliness", rubric: "1.0 = warm greeting" }),
+  ],
+});
+```
+
+An eval passes only when **every** scorer meets the threshold. Key `defineEval` fields:
+
+| Field       | Type                  | Notes                                                         |
+| ----------- | --------------------- | ------------------------------------------------------------- |
+| `name`      | string                | Required. Shown in the report.                                |
+| `input`     | `{ prompt, history }` | Required `prompt`; optional prior `{ role, text }` turns.     |
+| `scorers`   | `Scorer[]`            | Required, at least one.                                       |
+| `threshold` | number `0..1`         | Per-scorer pass bar. Default `0.5`; overridable from the CLI. |
+| `run`       | function              | Optional override for custom setup (seed data, multi-turn).   |
+
+The agent run handed to scorers is small and transport-agnostic:
+
+```ts
+interface AgentRunOutput {
+  text: string; // concatenated assistant text
+  toolCalls: readonly string[]; // tool/action names, in call order
+  ok: boolean; // completed without a terminal error
+  error?: string;
+  runId: string;
+  durationMs: number;
+}
+```
+
+## Built-in scorers {#built-in}
+
+Imported from `@agent-native/core/eval`:
+
+| Scorer                   | Score                                                             | Model? |
+| ------------------------ | ----------------------------------------------------------------- | ------ |
+| `exactMatch(expected)`   | `1.0` if text equals `expected` (trimmed, case-insensitive)       | No     |
+| `contains(needles)`      | Fraction of required substrings present (so partial hits surface) | No     |
+| `usesTool(toolName)`     | `1.0` if the agent invoked that tool/action at least once         | No     |
+| `llmJudge({ criteria })` | LLM-as-judge scored against a natural-language rubric, → `0..1`   | Yes    |
+
+`exactMatch` and `contains` take an optional `{ caseSensitive }`. `llmJudge` takes `{ criteria, rubric?, name?, scoreRange? }` — its output is normalized to `[0, 1]`, and the judge model is whatever the runner resolved (never a hardcoded provider).
+
+## Custom scorers: the 4-step pipeline {#custom}
+
+`createScorer` builds a scorer from a Mastra-style 4-step pipeline. Only `generateScore` is required:
+
+```txt
+preprocess(run)     → x          transform the run/output (optional)
+analyze(x, ctx)     → analysis   plain JS OR an LLM judge (optional)
+generateScore(a)    → 0..1       REQUIRED, normalized
+generateReason(...) → string     human-readable why (optional)
+```
+
+`preprocess` and `analyze` default to identity (the scorer sees the raw `AgentRunOutput`). The `analyze` step receives a `ctx` with a provider-agnostic `judge()` helper for LLM-backed scoring:
+
+```ts
+import { createScorer, clamp01 } from "@agent-native/core/eval";
+
+// A scorer that rewards short, tool-using answers.
+const concise = createScorer({
+  name: "concise_with_tool",
+  analyze(run) {
+    return {
+      words: run.text.trim().split(/\s+/).length,
+      usedTool: run.toolCalls.length > 0,
+    };
+  },
+  generateScore({ words, usedTool }) {
+    if (!usedTool) return 0;
+    return clamp01(1 - Math.max(0, words - 40) / 200);
+  },
+  generateReason({ analysis }) {
+    return `${analysis.words} words, tool used: ${analysis.usedTool}`;
+  },
+});
+```
+
+## Running the gate {#cli}
+
+```bash
+agent-native eval                    # run every *.eval.ts; non-zero exit on failure
+agent-native eval billing            # only files whose path contains "billing"
+agent-native eval --json             # machine-readable report (for CI)
+agent-native eval --threshold 0.8    # override every eval's pass threshold (0..1)
+```
+
+The command discovers `**/*.eval.ts` and `evals/*.ts` under the current app, runs the agent for each input, scores it, prints a readable table (or JSON), and **exits non-zero if any eval scores below its threshold**.
+
+Exit codes:
+
+| Code | Meaning                                                         |
+| ---- | --------------------------------------------------------------- |
+| `0`  | All evals passed — _or_ no eval files were found (CI-friendly). |
+| `1`  | At least one eval scored below threshold, or the suite errored. |
+| `2`  | Bad arguments (e.g. `--threshold` outside `[0, 1]`).            |
+
+### As a CI deploy gate {#ci}
+
+Add it to the pipeline that runs before a deploy:
+
+```yaml
+# .github/workflows/deploy.yml (excerpt)
+- run: npx agent-native eval --json
+```
+
+A regression that drops any scorer below threshold fails the step and blocks the deploy. An app with no eval files exits `0`, so adopting evals is opt-in per app.
+
+## What's next
+
+- [**Observability**](/docs/observability) — post-hoc scoring of real production runs (the complementary layer)
+- [**Actions**](/docs/actions) — the tools/actions that show up in `toolCalls`
+- [**Agent Teams**](/docs/agent-teams) — sub-agents an eval might exercise

package/docs/content/external-agents.md

@@ -386,7 +386,7 @@ Every allow-listed template that produces or lists a navigable resource ships a
 
 ## Authoring: the `link` builder {#link-builder}
 
-This section is for template authors. `defineAction` accepts an optional `link` builder. When set, every MCP/A2A result for that tool auto-appends a markdown `[label →](absoluteUrl)` block and a structured `_meta["agent-native/openLink"] = { label, view, webUrl, desktopUrl }`. `tools/list` adds `annotations["agent-native/producesOpenLink"]` and a description suffix so the external agent knows the tool yields an openable link and should surface it.
+This section is for template authors. `defineAction` accepts an optional `link` builder. When set, every MCP/A2A result for that tool auto-appends a markdown `[label →](absoluteUrl)` block and a structured `_meta["agent-native/openLink"] = { label, view, webUrl, desktopUrl, vscodeUrl }`. `tools/list` adds `annotations["agent-native/producesOpenLink"]` and a description suffix so the external agent knows the tool yields an openable link and should surface it.
 
 Build the URL with `buildDeepLink(...)` — it is the single source of truth for the open-route format. Never hand-format the `/_agent-native/open` URL.
 
@@ -431,7 +431,7 @@ See [MCP Apps](/docs/mcp-apps) for the full authoring guide — `embedRoute` vs
 
 The `link` builder is **pure and synchronous — no I/O, no awaits**. It runs best-effort: a throw, `null`, or `undefined` is swallowed and **never** fails the tool call. It only reads the call's `args` and `result`; it must not query the DB, read app-state, or call other actions. Return `null` when there's nothing to open.
 
-`buildDeepLink({ app, view, params?, to?, compose? })` returns the app-relative path `/_agent-native/open?app=…&view=…&<recordId>=…`. The MCP layer turns that into an absolute web URL (`toAbsoluteOpenUrl`, using the request origin) and a desktop `agentnative://open?…` URL (`toDesktopOpenUrl`); the markdown link uses the desktop URL when the client signals `target: "desktop"`.
+`buildDeepLink({ app, view, params?, to?, compose? })` returns the app-relative path `/_agent-native/open?app=…&view=…&<recordId>=…`. The MCP layer turns that into an absolute web URL (`toAbsoluteOpenUrl`, using the request origin), a desktop `agentnative://open?…` URL (`toDesktopOpenUrl`), and a VS Code extension URL (`toVsCodeOpenUrl`) for `vscode://builderio.agent-native/open?url=…`; the markdown link uses the desktop URL when the client signals `target: "desktop"`.
 
 ### The `/_agent-native/open` route {#open-route}
 

package/docs/content/human-approval.md

@@ -0,0 +1,101 @@
+---
+title: "Human-in-the-Loop Approvals"
+description: "Pause the agent before a high-consequence action runs — defineAction's needsApproval gate emits an approval_required event, the human approves, and only then does the tool execute."
+---
+
+# Human-in-the-Loop Approvals
+
+Most actions should just run. A few — sending an email, charging a card, deleting an account — are outward-facing and hard to undo, and you don't want the agent to do them autonomously. For those, `defineAction` has an opt-in **approval gate**: when the agent tries to call the action, the loop pauses, surfaces an Approve/Deny affordance to the human, and runs the action _only_ after the human approves that specific call.
+
+> [!WARNING]
+> Keep approvals rare. Every gated action is a hard stop in the agent loop — it interrupts the run and demands a human round-trip. Use `needsApproval` only for genuinely high-consequence, hard-to-undo, outward-facing operations. If you find yourself gating reads or routine writes, you're holding it wrong. The default is **off**, and almost every action should leave it off.
+
+## The `needsApproval` gate {#needs-approval}
+
+Set `needsApproval` on a `defineAction`. It accepts a boolean or a predicate:
+
+```ts
+// actions/send-email.ts
+export default defineAction({
+  description: "Send an email via Gmail.",
+  schema: z.object({
+    to: z.string(),
+    subject: z.string(),
+    body: z.string(),
+  }),
+  // Sending is outward-facing and hard to undo, so the agent can never send
+  // without a human approving the specific call. Drafting/queueing is
+  // unaffected — only the real send is gated.
+  needsApproval: true,
+  run: async (args) => {
+    /* ...actually send... */
+  },
+});
+```
+
+- **`needsApproval: true`** — always require approval.
+- **`needsApproval: (args, ctx) => boolean | Promise<boolean>`** — require approval only when the predicate returns true. Gate conditionally, e.g. only for external recipients or only above a dollar threshold:
+
+  ```ts
+  needsApproval: (args) => !args.to.endsWith("@your-company.com"),
+  ```
+
+  Keep the predicate pure and fast. **It fails closed**: if the predicate throws, the framework treats that as "approval required" rather than silently running a high-consequence action.
+
+When `needsApproval` is omitted, behavior is byte-for-byte unchanged — there is no extra cost on the common path.
+
+This works the same for legacy `parameters`-style actions and schema-based actions, and for the in-app agent, sub-agents, A2A, and MCP callers (every agent surface routes through the same loop).
+
+## How the loop pauses {#loop}
+
+When the agent calls a gated action and this specific call has **not** already been approved, the loop does **not** execute `run()`. Instead it:
+
+1. Resolves the gate. For a predicate, it calls `needsApproval(input, ctx)`; a throw is treated as "must approve" (fail closed).
+2. Emits a `tool_start` event (so the UI shows the call) followed immediately by an **`approval_required`** event, then stops the turn. The action's side effect never happens.
+
+The `approval_required` event carries everything the client needs to render an affordance:
+
+| Field         | Type     | Notes                                                               |
+| ------------- | -------- | ------------------------------------------------------------------- |
+| `tool`        | `string` | The action name the agent tried to call.                            |
+| `input`       | object   | The arguments the agent passed.                                     |
+| `approvalKey` | `string` | **Stable key** the client echoes back to approve _this exact call_. |
+| `toolCallId`  | `string` | The model-side tool-call id, when available.                        |
+
+The `approvalKey` is derived deterministically from the tool name plus its input, so the same logical call always produces the same key. The model never sees or sets it — it is purely a handshake between the framework and the human's Approve affordance.
+
+The paused tool returns a result telling the model the turn is paused and not to retry, so the model doesn't spin.
+
+## How the human approves {#approve}
+
+On `approval_required`, the chat UI renders an **Approve / Deny** affordance on the paused tool call. This is wired automatically in `AssistantChat` — you don't build it per template.
+
+- **Approve** re-issues the turn (an ordinary continuation message) carrying the call's key in `approvedToolCalls: [approvalKey]`. On the re-issued turn, the gate sees the key in the approved set and lets that specific call run normally.
+- **Deny** dismisses the affordance locally; nothing is re-issued, so the action never runs.
+
+`approvedToolCalls` is a field on the chat request (`AgentChatRequest.approvedToolCalls`). Keys not present in it stay paused — approving one call never blankly approves others. Because the key is content-addressed, an approval authorizes _that call with those arguments_; if the model later proposes a different send, that's a new key and a fresh approval.
+
+## End-to-end {#flow}
+
+```txt
+agent calls send-email
+   │
+   ▼
+needsApproval truthy, call not yet approved
+   │  loop emits tool_start + approval_required { tool, input, approvalKey }
+   ▼
+turn pauses — run() did NOT execute
+   │
+human clicks Approve in the chat UI
+   │  client re-issues the turn with approvedToolCalls: [approvalKey]
+   ▼
+gate sees the key → run() executes → email sends
+```
+
+The canonical (and intentionally rare) use of this gate in the framework is the Mail template's `send-email` action, which sets `needsApproval: true` so the agent can draft and queue freely but can never actually send a message without a human approving the specific send.
+
+## Related
+
+- [**Actions**](/docs/actions#needs-approval) — the full `defineAction` surface, including `outputSchema` for validating return values.
+- [**Security**](/docs/security) — when to reach for an approval gate vs. hiding an action from the model.
+- [**Mail template**](/docs/template-mail) — `send-email` is the reference example.

package/docs/content/observability.md

@@ -188,6 +188,27 @@ await putSetting("observability-config", {
 
 The framework emits `gen_ai.*` semantic convention spans compatible with the OpenTelemetry GenAI spec.
 
+## OpenTelemetry spans {#otel}
+
+Separate from the `exporters` config above (which ships the in-house traces to an OTLP endpoint), the agent loop can also emit **live OpenTelemetry spans** for every run, model call, and tool call — so a host that already runs an OTel collector sees agent activity alongside the rest of its distributed traces.
+
+This layer is **optional and no-op by default**:
+
+- `@opentelemetry/api` is an **optional dependency**. If it isn't installed, the helpers degrade to silent no-ops — nothing here ever throws into the agent loop.
+- Even when the api package _is_ present, it ships a default no-op tracer. Spans only become real once the **host registers a `TracerProvider`** (via `@opentelemetry/sdk-node` or similar). The framework deliberately does **not** depend on the heavy SDK/exporter packages or register a provider itself — instrumentation is opt-in by the embedding app.
+
+So the cost when you haven't wired OTel is a couple of cached property reads per call. To turn it on, install the api package plus your SDK and register a provider at server startup the same way you would for any other Node service.
+
+The agent loop emits three span kinds:
+
+| Span        | When                       | Attributes                                                        |
+| ----------- | -------------------------- | ----------------------------------------------------------------- |
+| `agent.run` | once per agent run         | `agent.run_id`, `agent.thread_id`, `agent.user_id`, `agent.model` |
+| `tool.call` | once per action invocation | `tool.name`, plus success/error status                            |
+| `llm.call`  | per model call             | timing + OK/error status                                          |
+
+Spans are finished with OK/ERROR status and record the error message on failure. Zero/sentinel attribute values are pruned so spans aren't cluttered with noise. This OTel layer is purely additive to the in-house `agent_trace_spans` / `agent_trace_summaries` tables that power the dashboard above — both are produced from the same run events.
+
 ## Error reporting (Sentry) {#sentry}
 
 Server-side errors that escape Nitro route handlers are reported to Sentry when a DSN is configured. Without it the SDK silently no-ops, so it's safe to leave the env vars unset in dev. Browser and server events can go to the same Sentry project; split them into separate projects only when you want operational separation for ownership, volume, quotas, or alert routing.