@agent-native/core 0.51.15 → 0.53.0

package/docs/content/evals.md

@@ -0,0 +1,141 @@
+---
+title: "Evals (CI Gate)"
+description: "Write *.eval.ts test cases that run the real agent against fixed inputs, score the output with composable scorers, and gate CI/deploys on a threshold."
+---
+
+# Evals (CI Gate)
+
+Evals are a first-class testing primitive: you declare a prompt plus the behavior you expect, the runner **actually runs the agent loop** against that input, scores the output with composable scorers, and exits non-zero if any case scores below its threshold. That non-zero exit makes `agent-native eval` a drop-in CI deploy gate.
+
+This is complementary to the post-hoc scoring in [Observability](/docs/observability):
+
+- **Observability evals** (`observability/evals.ts`) — _"how did this real run do?"_ Passive, sampled, lives next to traces.
+- **`*.eval.ts` (this primitive)** — _"does the agent do the right thing on this fixed input?"_ Active, deterministic, a CI gate run via the CLI.
+
+The runner resolves a provider-agnostic engine/model from the existing registry — no model is hardcoded — so the same suite runs against whatever engine the app is configured for.
+
+## Writing an eval {#writing}
+
+Drop a `*.eval.ts` file anywhere in the app (or an `evals/*.ts` file). Each file `export default defineEval(...)` (or exports an array of them):
+
+```ts
+// evals/greeting.eval.ts
+import { defineEval, contains, llmJudge } from "@agent-native/core/eval";
+
+export default defineEval({
+  name: "greets the user by name",
+  input: { prompt: "Say hi to Ada." },
+  threshold: 0.7, // per-scorer pass bar; default 0.5
+  scorers: [
+    contains("Ada"),
+    llmJudge({ criteria: "friendliness", rubric: "1.0 = warm greeting" }),
+  ],
+});
+```
+
+An eval passes only when **every** scorer meets the threshold. Key `defineEval` fields:
+
+| Field       | Type                  | Notes                                                         |
+| ----------- | --------------------- | ------------------------------------------------------------- |
+| `name`      | string                | Required. Shown in the report.                                |
+| `input`     | `{ prompt, history }` | Required `prompt`; optional prior `{ role, text }` turns.     |
+| `scorers`   | `Scorer[]`            | Required, at least one.                                       |
+| `threshold` | number `0..1`         | Per-scorer pass bar. Default `0.5`; overridable from the CLI. |
+| `run`       | function              | Optional override for custom setup (seed data, multi-turn).   |
+
+The agent run handed to scorers is small and transport-agnostic:
+
+```ts
+interface AgentRunOutput {
+  text: string; // concatenated assistant text
+  toolCalls: readonly string[]; // tool/action names, in call order
+  ok: boolean; // completed without a terminal error
+  error?: string;
+  runId: string;
+  durationMs: number;
+}
+```
+
+## Built-in scorers {#built-in}
+
+Imported from `@agent-native/core/eval`:
+
+| Scorer                   | Score                                                             | Model? |
+| ------------------------ | ----------------------------------------------------------------- | ------ |
+| `exactMatch(expected)`   | `1.0` if text equals `expected` (trimmed, case-insensitive)       | No     |
+| `contains(needles)`      | Fraction of required substrings present (so partial hits surface) | No     |
+| `usesTool(toolName)`     | `1.0` if the agent invoked that tool/action at least once         | No     |
+| `llmJudge({ criteria })` | LLM-as-judge scored against a natural-language rubric, → `0..1`   | Yes    |
+
+`exactMatch` and `contains` take an optional `{ caseSensitive }`. `llmJudge` takes `{ criteria, rubric?, name?, scoreRange? }` — its output is normalized to `[0, 1]`, and the judge model is whatever the runner resolved (never a hardcoded provider).
+
+## Custom scorers: the 4-step pipeline {#custom}
+
+`createScorer` builds a scorer from a Mastra-style 4-step pipeline. Only `generateScore` is required:
+
+```txt
+preprocess(run)     → x          transform the run/output (optional)
+analyze(x, ctx)     → analysis   plain JS OR an LLM judge (optional)
+generateScore(a)    → 0..1       REQUIRED, normalized
+generateReason(...) → string     human-readable why (optional)
+```
+
+`preprocess` and `analyze` default to identity (the scorer sees the raw `AgentRunOutput`). The `analyze` step receives a `ctx` with a provider-agnostic `judge()` helper for LLM-backed scoring:
+
+```ts
+import { createScorer, clamp01 } from "@agent-native/core/eval";
+
+// A scorer that rewards short, tool-using answers.
+const concise = createScorer({
+  name: "concise_with_tool",
+  analyze(run) {
+    return {
+      words: run.text.trim().split(/\s+/).length,
+      usedTool: run.toolCalls.length > 0,
+    };
+  },
+  generateScore({ words, usedTool }) {
+    if (!usedTool) return 0;
+    return clamp01(1 - Math.max(0, words - 40) / 200);
+  },
+  generateReason({ analysis }) {
+    return `${analysis.words} words, tool used: ${analysis.usedTool}`;
+  },
+});
+```
+
+## Running the gate {#cli}
+
+```bash
+agent-native eval                    # run every *.eval.ts; non-zero exit on failure
+agent-native eval billing            # only files whose path contains "billing"
+agent-native eval --json             # machine-readable report (for CI)
+agent-native eval --threshold 0.8    # override every eval's pass threshold (0..1)
+```
+
+The command discovers `**/*.eval.ts` and `evals/*.ts` under the current app, runs the agent for each input, scores it, prints a readable table (or JSON), and **exits non-zero if any eval scores below its threshold**.
+
+Exit codes:
+
+| Code | Meaning                                                         |
+| ---- | --------------------------------------------------------------- |
+| `0`  | All evals passed — _or_ no eval files were found (CI-friendly). |
+| `1`  | At least one eval scored below threshold, or the suite errored. |
+| `2`  | Bad arguments (e.g. `--threshold` outside `[0, 1]`).            |
+
+### As a CI deploy gate {#ci}
+
+Add it to the pipeline that runs before a deploy:
+
+```yaml
+# .github/workflows/deploy.yml (excerpt)
+- run: npx agent-native eval --json
+```
+
+A regression that drops any scorer below threshold fails the step and blocks the deploy. An app with no eval files exits `0`, so adopting evals is opt-in per app.
+
+## What's next
+
+- [**Observability**](/docs/observability) — post-hoc scoring of real production runs (the complementary layer)
+- [**Actions**](/docs/actions) — the tools/actions that show up in `toolCalls`
+- [**Agent Teams**](/docs/agent-teams) — sub-agents an eval might exercise

package/docs/content/pr-visual-recap.md

@@ -216,12 +216,15 @@ the prompt instructs the agent to write `plans/pr-123-visual-recap/plan.mdx`
 plus optional visual files and then run:
 
 ```bash
-npx @agent-native/core@latest plan local preview --dir plans/pr-123-visual-recap --kind recap --open
+npx @agent-native/core@latest plan local serve --dir plans/pr-123-visual-recap --kind recap --open
 ```
 
-The returned `file://` preview, or `/local-plans/pr-123-visual-recap` in a local
-Plan app using the same `PLAN_LOCAL_DIR`, is the review link. This mode disables
-the hosted sticky PR comment, inline screenshot upload, usage attachment, and
+The returned URL opens the hosted Plan UI while the browser reads the recap MDX
+from a localhost bridge. Recap content is not written to the hosted Plan
+database, and the URL only works on the machine running the bridge. If you run
+the Plan app locally with the same `PLAN_LOCAL_DIR`, the
+`/local-plans/pr-123-visual-recap` route is also valid. This mode disables the
+hosted sticky PR comment, inline screenshot upload, usage attachment, and
 browser comments until you explicitly publish.
 
 ## It's informational, not a gate

package/docs/content/sandbox-adapters.md

@@ -0,0 +1,134 @@
+---
+title: "Sandbox Adapters"
+description: "Swap the backend that runs the agent's run-code tool — local child process by default, a remote/durable runner when you need to exceed the hosted code-exec ceiling."
+---
+
+# Sandbox Adapters
+
+The `run-code` tool runs agent-supplied JavaScript in an isolated environment. **Sandbox adapters** factor the _execution_ concern out of that tool so the backend can be swapped — a local child process by default, or a Docker / remote / durable runner — without touching the agent loop, `run-code.ts`, the localhost bridge, the env scrub, or the output formatting.
+
+## Why a seam {#why}
+
+The default backend spawns a locked-down local Node child process. That's bounded by the hosting process: on the hosted platform it shares the agent loop's soft execution ceiling (~40s before timeout/continuation thrash). A remote or durable adapter is the lever to exceed that ceiling — it runs large data jobs to completion independently of the request lifecycle.
+
+Keeping the contract narrow means a remote adapter inherits the same security posture. The parent process keeps ownership of everything secret-bearing: it builds the sandbox module, runs the localhost bridge (which holds the request context and applies host allowlists + SSRF guards), scrubs the env, and formats output. An adapter only receives an already-prepared, **non-secret** module source plus resource limits — it is responsible solely for _running_ it and capturing stdout/stderr/exit status.
+
+## The interface {#interface}
+
+The seam lives in core at `packages/core/src/coding-tools/sandbox/` — `adapter.ts` (the contract), `index.ts` (selection: `getSandboxAdapter()` / `registerSandboxAdapter()`), and `local-child-process-adapter.ts` (the default). It is wired in-package by `run-code.ts`; a host plugs in a different backend through the `index.ts` registration helper (or, for a Docker backend, via the [blueprint](/docs/blueprint-installer) that edits these files directly).
+
+Every backend implements `SandboxAdapter`:
+
+```ts
+interface SandboxAdapter {
+  /** Stable id, surfaced for diagnostics and adapter selection. */
+  readonly id: string;
+  /** Execute one prepared sandbox module and capture its output. */
+  run(request: SandboxRunRequest): Promise<SandboxRunResult>;
+}
+```
+
+The request and result are intentionally small and opaque:
+
+```ts
+interface SandboxRunRequest {
+  /**
+   * The complete ESM module source to execute. Already wraps the user's code
+   * and embeds the loopback bridge URL/token; the adapter does NOT parse or
+   * rewrite it.
+   */
+  moduleSource: string;
+  /**
+   * Scrubbed environment — only safe POSIX vars (PATH/HOME/TMPDIR/…), never app
+   * secrets. Adapters must not augment this with the parent's own environment.
+   */
+  env: Record<string, string>;
+  /** Hard wall-clock timeout in milliseconds. The adapter must enforce it. */
+  timeoutMs: number;
+  /**
+   * Loopback port of the parent's bridge server (reachable over 127.0.0.1). A
+   * remote adapter that can't reach the parent's loopback must tunnel or proxy
+   * this to support bridge-backed globals (`appAction`, `providerFetch`, …).
+   */
+  bridgePort: number;
+}
+
+interface SandboxRunResult {
+  stdout: string;
+  stderr: string;
+  /** `0` on clean exit, non-zero on failure, `null` when killed by a signal. */
+  exitCode: number | null;
+  /** True when the run was killed for exceeding `timeoutMs`. */
+  timedOut: boolean;
+}
+```
+
+## The default: `LocalChildProcessAdapter` {#default}
+
+Out of the box, `getSandboxAdapter()` returns `LocalChildProcessAdapter` (`id: "local-child-process"`). It preserves the historical `run-code` behavior byte-for-byte:
+
+- The prepared module source is written to a fresh temp dir.
+- The child runs with the scrubbed env (no secrets), with `TMPDIR`/`TEMP`/`TMP` pointed inside the sandbox dir.
+- When the Node permission model is available (`--permission`, or `--experimental-permission` on Node 20), the child is denied filesystem access outside its temp dir, plus child processes, workers, and native addons. Outbound network is _not_ blocked by the permission model — but the env scrub means such requests carry no credentials, and all authenticated calls go through the parent's loopback bridge.
+- A timeout sends `SIGTERM`, then `SIGKILL` after a 2s grace period.
+- Temp files are cleaned up best-effort after the run.
+
+> [!WARNING]
+> The default adapter uses `node:child_process`, which does not exist on edge/worker runtimes (Cloudflare Workers, Netlify Edge Functions). Run `run-code` in a standard Node.js environment, or register a remote adapter.
+
+## Selecting an adapter {#selection}
+
+Resolution order — an explicitly registered adapter wins; otherwise the env var selects a built-in; otherwise the local default is used:
+
+```txt
+registerSandboxAdapter(adapter)  →  AGENT_NATIVE_SANDBOX  →  local default
+```
+
+### `AGENT_NATIVE_SANDBOX` env var {#env}
+
+Selects a built-in adapter by id. Currently only `local` (the default) is wired; unknown values fall back to local rather than failing the run.
+
+```bash
+AGENT_NATIVE_SANDBOX=local   # the default — explicit
+```
+
+### `registerSandboxAdapter()` {#register}
+
+A host process overrides the backend for all subsequent `run-code` invocations through the seam's `index.ts` — for example, to run every call in a remote container:
+
+```ts
+import {
+  registerSandboxAdapter,
+  type SandboxAdapter,
+} from "./coding-tools/sandbox/index.js";
+
+class RemoteSandboxAdapter implements SandboxAdapter {
+  readonly id = "remote";
+  async run(request) {
+    // Ship request.moduleSource to the durable runner, enforce request.timeoutMs,
+    // proxy bridge calls back to request.bridgePort, and return stdout/stderr/exitCode.
+  }
+}
+
+registerSandboxAdapter(new RemoteSandboxAdapter());
+// Pass `null` to clear the override and fall back to env-var / default resolution.
+```
+
+## The seam for a durable runner {#durable}
+
+This interface is deliberately the seam for a future remote/durable sandbox. A remote or durable adapter (Docker, a Vercel-Sandbox-style runner, or a queued background worker) would:
+
+1. Implement `SandboxAdapter.run` against an out-of-process runtime.
+2. Tunnel the loopback bridge (or proxy bridge calls back to the parent).
+3. Let large data jobs run to completion independently of the request lifecycle — exceeding the hosted ~40s code-exec ceiling that bounds the local child-process adapter.
+
+Register it under a new `AGENT_NATIVE_SANDBOX` value (e.g. `remote`) and/or via `registerSandboxAdapter()`. The agent loop and `run-code.ts` never change.
+
+> [!TIP]
+> The `agent-native add sandbox docker` blueprint emits a full, self-contained recipe for implementing a Docker adapter against this seam. See [Blueprint Installer](/docs/blueprint-installer).
+
+## What's next
+
+- [**Blueprint Installer**](/docs/blueprint-installer) — `agent-native add sandbox docker` prints a Docker-adapter recipe
+- [**Agent Teams**](/docs/agent-teams) — delegating heavy work to sub-agents
+- [**Security**](/docs/security) — the env scrub and bridge allowlist posture

package/docs/content/template-plan.md

@@ -180,22 +180,34 @@ or set the convention for your agent environment:
 export AGENT_NATIVE_PLANS_MODE=local-files
 ```
 
-In this mode the agent writes a local MDX folder under `plans/<slug>/` and must
-not call the hosted Plan MCP tools. The durable files are:
+In this mode the agent writes a local MDX folder and must not call the hosted
+Plan MCP tools. Use a repo folder such as `plans/<slug>/` when you want the plan
+checked in with the code. Use a temp or ignored folder, such as
+`/tmp/agent-native-plans/<slug>/` or `.agent-native/plans/<slug>/`, when the
+plan should stay out of git. The folder contains:
 
 - `plan.mdx`
 - optional `canvas.mdx`
 - optional `prototype.mdx`
 - optional `.plan-state.json`
 
-After writing the folder, the agent validates and previews it locally:
+After writing the folder, the agent starts a tiny localhost bridge and opens the
+hosted Plan UI against that local-only source:
 
 ```bash
-npx @agent-native/core@latest plan local preview --dir plans/<slug> --kind plan --open
+npx @agent-native/core@latest plan local serve --dir plans/<slug> --kind plan --open
 ```
 
-If you run the Plan app locally with the same `PLAN_LOCAL_DIR`, you can open the
-read-only app route:
+The bridge URL looks like
+`https://plan.agent-native.com/local-plans/<slug>?bridge=http://127.0.0.1:...`.
+The page is the normal Plan viewer, but the browser fetches `plan.mdx`,
+`canvas.mdx`, `prototype.mdx`, `.plan-state.json`, and local image assets from
+the localhost bridge. Plan content is not written to the hosted database and is
+not sent through hosted Plan actions. Keep the bridge process running while you
+review; the URL is local to your machine and is not a shareable team link.
+
+If you run the Plan app locally with the same `PLAN_LOCAL_DIR`, you can also
+open the read-only app route:
 
 ```text
 http://localhost:<port>/local-plans/<slug>
@@ -206,8 +218,8 @@ Plan database. It also disables hosted sharing, browser comments, plan history,
 and publish/export receipts until you explicitly opt into publishing. To move a
 local plan into the hosted database, call `publish-visual-plan` with the local
 MDX folder path; this uploads the plan, assigns it a hosted ID, enables sharing
-and commenting, and returns the hosted URL. It does
-not automatically make your coding agent's LLM local; choose a local or approved
+and commenting, and returns the hosted URL. Local-files mode does not
+automatically make your coding agent's LLM local; choose a local or approved
 model if that privacy boundary matters too.
 
 ## Desktop local file sync {#desktop-local-sync}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agent-native/core",
-  "version": "0.51.15",
+  "version": "0.53.0",
   "type": "module",
   "engines": {
     "node": ">=22"
@@ -74,6 +74,7 @@
     "./agent/context-xray/actions/context-evict": "./dist/agent/context-xray/actions/context-evict.js",
     "./agent/context-xray/actions/context-restore": "./dist/agent/context-xray/actions/context-restore.js",
     "./agent/context-xray/actions/context-report": "./dist/agent/context-xray/actions/context-report.js",
+    "./agent/observational-memory": "./dist/agent/observational-memory/index.js",
     "./resources": "./dist/resources/index.js",
     "./resources/store": "./dist/resources/store.js",
     "./resources/metadata": "./dist/resources/metadata.js",
@@ -135,6 +136,7 @@
     "./styles/agent-native.css": "./dist/styles/agent-native.css",
     "./agent/engine": "./dist/agent/engine/index.js",
     "./agent/harness": "./dist/agent/harness/index.js",
+    "./eval": "./dist/eval/index.js",
     "./tsconfig.base.json": "./tsconfig.base.json"
   },
   "sideEffects": [
@@ -142,6 +144,7 @@
   ],
   "files": [
     "bin",
+    "blueprints",
     "dist",
     "docs",
     "tsconfig.base.json",
@@ -264,6 +267,7 @@
     "ws": ">=8"
   },
   "optionalDependencies": {
+    "@opentelemetry/api": "^1.9.1",
     "playwright": "^1.60.0"
   },
   "peerDependenciesMeta": {

package/src/templates/workspace-core/.agents/skills/external-agents/SKILL.md

@@ -329,16 +329,19 @@ The hosted `connect` flow above is the recommended path. For local dev, run
 the app (`pnpm dev` / `pnpm exec agent-native dev`) then point a local agent at it:
 
 ```bash
-pnpm exec agent-native mcp install --client claude-code|claude-code-cli|codex|cowork \
+pnpm exec agent-native mcp install --client claude-code|claude-code-cli|codex|cowork|cursor|opencode|github-copilot \
   [--app <id>] [--scope user|project]
 ```
 
 It provisions a token (random `ACCESS_TOKEN` into the workspace `.env` for
 local dev, or a `signA2AToken` JWT for a detected hosted origin) and writes an
 idempotent stdio server entry — `.mcp.json` / `~/.claude.json` for Claude Code,
-the `[mcp_servers.*]` block in `~/.codex/config.toml` for Codex, the
-Claude-Code JSON shape for Cowork. The entry runs `pnpm exec agent-native mcp serve
---app <id>`, by default a **thin stdio proxy** to the running local app's
+the `[mcp_servers.*]` block in `~/.codex/config.toml` for Codex,
+`.cursor/mcp.json` / `~/.cursor/mcp.json` for Cursor, `opencode.json` /
+`~/.config/opencode/opencode.json` for OpenCode, `.vscode/mcp.json` / VS Code
+user `mcp.json` for GitHub Copilot / VS Code, and the Claude-Code JSON shape
+for Cowork. The entry runs `pnpm exec agent-native mcp serve --app <id>`, by
+default a **thin stdio proxy** to the running local app's
 `/_agent-native/mcp` (live registry + HMR + correct deep links stay the single
 source of truth; `--standalone` builds the registry in-process). Companion
 subcommands: `mcp uninstall`, `mcp status`, `mcp token [--rotate]`. You can
@@ -413,3 +416,13 @@ before telling the user they are unauthenticated.
 - **a2a-protocol** — the `ask-agent` meta-tool and JSON-RPC peer calls
 - **adding-a-feature** — the four-area checklist (add a `link` builder when a
   feature produces a navigable resource)
+
+## Blueprint installer
+
+To add a whole new integration the agent-native way, `agent-native add <kind>
+<name|url>` prints a curated Markdown blueprint to stdout — pipe it into the
+external coding agent you connected (`agent-native add provider stripe |
+claude`) and it applies the changes against the live repo. A URL emits a
+generic research-and-integrate blueprint instead. Seeded kinds:
+`provider` / `channel` / `sandbox` / `action`. Add your own by dropping a
+`.md` in `packages/core/blueprints/<kind>/`. See the Blueprint Installer doc.

package/src/templates/workspace-core/.agents/skills/harness-agents/SKILL.md

@@ -80,6 +80,26 @@ existing run routes as `goalId=agent-harness`.
   Preserve `defineAction` auth, request context, timeouts, truncation, and
   read-only metadata.
 
+## Code Execution Sandbox
+
+- The `run-code` tool executes through a pluggable `SandboxAdapter`
+  (`packages/core/src/coding-tools/sandbox/`). The default
+  `LocalChildProcessAdapter` spawns a locked-down local Node child process;
+  swap it via `AGENT_NATIVE_SANDBOX` or `registerSandboxAdapter()` for a
+  Docker/remote/durable backend (the lever to exceed the hosted ~40s code-exec
+  ceiling). An adapter only runs the already-prepared, non-secret module source
+  — it never sees app secrets. See the Sandbox Adapters doc; `agent-native add
+  sandbox docker` emits a full Docker-adapter recipe.
+
+## Sub-Agent Delegation Depth
+
+- Sub-agent spawning is capped server-side (default depth `2`) so delegation
+  chains can't fan out indefinitely. Override at deploy time with
+  `AGENT_NATIVE_MAX_SUBAGENT_DEPTH` (`0` disables sub-agents; clamped to `16`).
+  Enforcement is ambient via `evaluateSubagentDepth` in
+  `packages/core/src/server/agent-teams.ts` — independent of any tool-level
+  guard. See the Agent Teams doc for the depth model.
+
 ## Don't
 
 - Don't add Claude Code, Codex, Cursor, Mastra, or Pi as an `AgentEngine`.

package/src/templates/workspace-core/.agents/skills/observability/SKILL.md

@@ -75,6 +75,26 @@ const criteria: EvalCriteria = {
 };
 ```
 
+#### Evals (CI gate)
+
+The three layers above score *real production runs* after the fact. For an active, deterministic gate, use the first-class `*.eval.ts` primitive from `@agent-native/core/eval` (source: `packages/core/src/eval/*`). It runs the actual agent loop against fixed inputs and exits non-zero below threshold, so it gates CI/deploys.
+
+```ts
+// evals/faq.eval.ts
+import { defineEval, contains, llmJudge } from "@agent-native/core/eval";
+
+export default defineEval({
+  name: "answers the FAQ",
+  input: { prompt: "What is your return policy?" },
+  threshold: 0.7,
+  scorers: [contains("30 days"), llmJudge({ criteria: "accuracy" })],
+});
+```
+
+- Built-in scorers: `exactMatch` / `contains` / `usesTool` (pure JS) and `llmJudge` (provider-agnostic judge).
+- Custom scorers: `createScorer` with the 4-step `preprocess → analyze → generateScore → generateReason` pipeline (only `generateScore` is required).
+- Run as a gate: `agent-native eval [pattern] [--json] [--threshold N]` — discovers `**/*.eval.ts` and `evals/*.ts`, runs the agent, and exits non-zero if any eval is below its threshold. An app with no eval files exits `0`. Complements (does not replace) the post-hoc scoring in `evals.ts`. See the Evals doc.
+
 ### 4. Experiments
 
 A/B testing with sticky user-level assignment: