@agent-native/core 0.7.50 → 0.7.52

package/docs/content/observability.md

@@ -0,0 +1,184 @@
+---
+title: "Observability"
+description: "Agent traces, evals, feedback, A/B experiments, and the admin dashboard — all built-in with zero configuration."
+---
+
+# Agent Observability
+
+Every agent-native app gets observability out of the box. Traces, automated evals, user feedback, and A/B experiments work with zero configuration — all data lives in the app's own SQL database.
+
+## What's Captured Automatically
+
+When a user sends a message, the framework automatically records:
+
+- **Token usage** — input, output, cache read, cache write
+- **Cost** — computed from token counts and model pricing
+- **Latency** — total duration and time per tool call
+- **Tool calls** — which actions were invoked, success/error status, duration
+- **Automated evals** — 5 quality scores computed after every run
+
+No code changes needed. The instrumentation hooks into `production-agent.ts` transparently.
+
+## The Dashboard
+
+Add the dashboard to any template with a single route:
+
+```tsx
+// app/routes/observability.tsx
+import { ObservabilityDashboard } from "@agent-native/core/client";
+
+export default function ObservabilityPage() {
+  return (
+    <div className="min-h-screen bg-background p-6">
+      <ObservabilityDashboard />
+    </div>
+  );
+}
+```
+
+The dashboard has 5 tabs:
+
+| Tab               | What it shows                                                                   |
+| ----------------- | ------------------------------------------------------------------------------- |
+| **Overview**      | Key metrics — runs, cost, latency, tool success rate, satisfaction, eval score  |
+| **Conversations** | Trace list with drill-down to individual spans (agent_run, llm_call, tool_call) |
+| **Evals**         | Automated eval scores by criteria, trends over time                             |
+| **Experiments**   | A/B test list with status badges, variant results with confidence intervals     |
+| **Feedback**      | Thumbs up/down stream, category breakdown, frustration scores                   |
+
+## User Feedback
+
+### Explicit Feedback
+
+Thumbs up/down buttons render inline on every agent message in the chat UI. Thumbs down opens a category popover (Inaccurate, Not helpful, Wrong tool, Too slow). This is wired into `AssistantChat.tsx` automatically.
+
+### Implicit Feedback (Frustration Index)
+
+The framework computes a Frustration Index (0-100) from conversation signals:
+
+| Signal         | Weight | What it detects                     |
+| -------------- | ------ | ----------------------------------- |
+| Rephrasing     | 30%    | User repeats similar messages       |
+| Retry patterns | 20%    | "Try again", "no that's wrong"      |
+| Abandonment    | 20%    | Session ends shortly after response |
+| Sentiment      | 15%    | Negative language patterns          |
+| Length trend   | 15%    | Declining message lengths           |
+
+Score interpretation: 0-20 = healthy, 20-40 = friction, 40-60 = dissatisfied, 60+ = broken session.
+
+## Automated Evals
+
+Five deterministic scorers run after every agent run:
+
+| Criteria            | What it measures                                       | Score range |
+| ------------------- | ------------------------------------------------------ | ----------- |
+| `tool_success_rate` | % of tool calls without errors                         | 0-1         |
+| `step_efficiency`   | Penalizes excessive LLM iterations for tool-using runs | 0-1         |
+| `latency_score`     | Normalized against 10s/tool baseline                   | 0-1         |
+| `cost_efficiency`   | Normalized against cost baseline                       | 0-1         |
+| `error_recovery`    | Did the agent recover from tool errors?                | 0 or 1      |
+
+### LLM-as-Judge (Optional)
+
+Enable sampled LLM-based evaluation by setting `evalSampleRate`:
+
+```ts
+import { putSetting } from "@agent-native/core/settings";
+
+await putSetting("observability-config", {
+  enabled: true,
+  evalSampleRate: 0.05, // 5% of runs
+});
+```
+
+Custom criteria use natural language rubrics:
+
+```ts
+const criteria = {
+  name: "helpfulness",
+  description: "Was the response helpful and complete?",
+  rubric: "0.0 = unhelpful, 0.5 = partially helpful, 1.0 = fully resolved",
+};
+```
+
+## A/B Experiments
+
+Test different models, temperatures, or agent configurations:
+
+```ts
+// Create via API
+POST /_agent-native/observability/experiments
+{
+  "name": "sonnet-vs-haiku",
+  "variants": [
+    { "id": "control", "weight": 50, "config": { "model": "claude-sonnet-4-6" } },
+    { "id": "treatment", "weight": 50, "config": { "model": "claude-haiku-4-5-20251001" } }
+  ],
+  "metrics": ["cost", "latency", "satisfaction"]
+}
+
+// Start the experiment
+PUT /_agent-native/observability/experiments/:id
+{ "status": "running" }
+```
+
+The agent loop automatically resolves the user's variant and applies the config override. Assignment uses consistent hashing — same user always gets the same variant.
+
+## Configuration
+
+All settings are stored in the `observability-config` key:
+
+```ts
+{
+  enabled: true,           // Master switch
+  capturePrompts: false,   // Store prompt content in traces
+  captureToolArgs: false,  // Store action input arguments
+  captureToolResults: false, // Store action results
+  evalSampleRate: 0,       // 0-1, fraction of runs to LLM-judge
+  exporters: []            // OTLP export targets
+}
+```
+
+Content is **redacted by default** — only token counts, costs, and timing are stored. Opt in to content capture when needed for debugging.
+
+## API Endpoints
+
+All auto-mounted at `/_agent-native/observability/`:
+
+| Method | Path                       | Purpose                        |
+| ------ | -------------------------- | ------------------------------ |
+| GET    | `/`                        | Overview stats                 |
+| GET    | `/traces`                  | List trace summaries           |
+| GET    | `/traces/:runId`           | Trace detail (summary + spans) |
+| GET    | `/traces/:runId/evals`     | Evals for a run                |
+| POST   | `/feedback`                | Submit feedback                |
+| GET    | `/feedback`                | List feedback                  |
+| GET    | `/feedback/stats`          | Feedback aggregation           |
+| GET    | `/satisfaction`            | Satisfaction scores            |
+| GET    | `/evals/stats`             | Eval statistics                |
+| POST   | `/experiments`             | Create experiment              |
+| GET    | `/experiments`             | List experiments               |
+| PUT    | `/experiments/:id`         | Update experiment              |
+| POST   | `/experiments/:id/results` | Compute results                |
+| GET    | `/experiments/:id/results` | Get results                    |
+
+All endpoints support `?since=N` (ms timestamp) and `?limit=N` query params.
+
+## Export to External Platforms
+
+Send traces to Langfuse, Datadog, Grafana, or any OTel-compatible backend:
+
+```ts
+await putSetting("observability-config", {
+  enabled: true,
+  exporters: [
+    {
+      type: "otlp",
+      endpoint: "https://cloud.langfuse.com/api/public/otel",
+      headers: { Authorization: "Bearer sk-..." },
+    },
+  ],
+});
+```
+
+The framework emits `gen_ai.*` semantic convention spans compatible with the OpenTelemetry GenAI spec.

package/docs/content/onboarding.md

@@ -1,3 +1,8 @@
+---
+title: "Onboarding & API Keys"
+description: "Setup checklist for first-run configuration — API keys, OAuth, and provider connections"
+---
+
 # Onboarding
 
 When you first open an app built on the agent-native framework, you'll see a
@@ -33,7 +38,7 @@ behind a picker or disclosure when a step has several equivalent providers.
   OAuth provider, or email provider, paste the value(s), click **Save**.
   Secret fields use a password input so the value isn't shown on screen. Saved
   values go into your local `.env` (or workspace settings) — see
-  [Secrets](/docs/secrets) for where they live.
+  [Security](/docs/security) for where they live.
 - **Open a link** — some steps point to a sign-in page or docs. Click
   **Continue** and finish the flow in the new tab.
 - **Ask the agent** — a few steps offer a "Let the agent set it up" option.
@@ -174,6 +179,6 @@ function MySidebar() {
 ```
 
 For background on where step values are stored and how secrets are handled,
-see [Secrets](/docs/secrets). For end-user messaging touchpoints (invitations,
+see [Security](/docs/security). For end-user messaging touchpoints (invitations,
 password resets) that depend on the **Email delivery** step, see
 [Messaging](/docs/messaging).

package/docs/content/template-dispatch.md

@@ -5,6 +5,8 @@ description: "Dispatch is the workspace control plane — central inbox, cross-a
 
 # Dispatch
 
+> **See also:** for the conceptual overview of what Dispatch does and when you want it, see [Dispatch](/docs/dispatch). This page is the template-specific reference.
+
 Dispatch is the **workspace control plane**. Where other templates are domain apps (Mail, Calendar, Analytics), Dispatch is the app you run _alongside_ them to coordinate everything: a central inbox, a secrets vault, scheduled jobs, Slack/Telegram integration, and an orchestrator agent that delegates domain work to the right specialist app over [A2A](/docs/a2a-protocol).
 
 If you're running an [multi-app workspace](/docs/multi-app-workspace) with many apps, Dispatch is the glue.
@@ -45,7 +47,7 @@ Day-to-day, Dispatch is the place admins and ops folks open to keep the workspac
 _How it works under the hood (for developers)._
 
 - **Orchestrator agent.** The chat is set up as a router: it reads `AGENTS.md`, `LEARNINGS.md`, and routes to specialist sub-agents or remote A2A agents.
-- **Remote agent registry.** A2A manifests live in `remote-agents/*.json` — one per app. Dispatch calls them using the `call-agent` action.
+- **Remote agent registry.** A2A manifests live in `remote-agents/*.json` — one per app. Dispatch calls them using the `call-agent` action. In a multi-app workspace, sibling apps under `apps/` are auto-discovered as A2A peers — no manual registration needed.
 - **Vault schema.** Drizzle tables for secrets, grants, requests, approvals, and audit logs. See `server/db/schema.ts` in the template.
 - **Slack / Telegram plugins.** Server plugins that register webhooks and forward incoming messages to the orchestrator agent.
 - **MCP hub mode.** Dispatch can act as the workspace's [MCP hub](/docs/mcp-clients#hub) so every other app in the workspace pulls the same org-scope MCP server list.

package/docs/content/tools.md

@@ -26,6 +26,26 @@ Tools are different:
 
 Use app code for features that are core to the product. Use tools for everything else — one-off utilities, personal dashboards, quick integrations, monitors, and things you want to spin up in seconds.
 
+## When to build a tool vs. a template feature {#when-to-build}
+
+A quick decision rubric:
+
+**Build a tool when:**
+
+- It's for one user or one team, not the whole product.
+- It's a quick utility, dashboard, or widget you want now, not next sprint.
+- No new database schema is needed (or per-tool key-value storage is enough).
+- You want to ship it inside a single chat turn, no deploy.
+
+**Add a template feature when:**
+
+- Every user of the template should get it.
+- It needs new SQL tables, migrations, or shared schema changes.
+- The UI is complex enough to warrant React components, routes, and proper testing.
+- It's part of the product surface — something you'd advertise on a landing page.
+
+When in doubt, start as a tool. Promoting a tool to a template feature later is straightforward; rolling back a half-shipped product feature is not.
+
 ## Creating a tool {#creating}
 
 ### From the sidebar
@@ -58,16 +78,56 @@ Tools render with modest canvas padding by default so simple widgets and dashboa
 
 ## Persistent storage {#persistent-storage}
 
-Every tool has access to a built-in key-value store. Data is automatically scoped per tool and per user — your data stays yours.
+Every tool has access to a built-in key-value store via the `toolData` helper. Data is automatically scoped per tool and per user — your data stays yours.
 
 When you ask the agent to "add persistence" or "remember state" in a tool, it uses this built-in storage. No database tables to create, no migrations to run.
 
+### Scopes
+
+All `toolData` methods accept an optional `{ scope }` option:
+
+- `'user'` (default) — private to the current user.
+- `'org'` — shared across the user's organization.
+- `'all'` — list/get only; returns both user-scoped and org-scoped items.
+
+```html
+<script>
+  // Private to me
+  await toolData.set('notes', 'note-1', { title: 'My Note' });
+
+  // Shared with my org
+  await toolData.set('notes', 'team-note', { title: 'Team Note' }, { scope: 'org' });
+
+  // List my notes (default)
+  const mine = await toolData.list('notes');
+
+  // List both mine and the org's
+  const everything = await toolData.list('notes', { scope: 'all' });
+</script>
+```
+
 ## API keys and secrets {#secrets}
 
 When a tool needs an API key (for GitHub, OpenAI, a weather service, etc.), the agent will tell you what's needed and where to get it. You add the key through the Settings UI in the agent sidebar.
 
 Keys are encrypted and stored securely. Each key is restricted to specific domains — a GitHub token can only be sent to `api.github.com`, never anywhere else.
 
+### Secrets in tools {#secrets-in-tools}
+
+Tools reference secrets in `toolFetch()` calls using the `${keys.NAME}` template pattern. The proxy substitutes the encrypted value server-side; the actual key never reaches the browser.
+
+```html
+<script>
+  const res = await toolFetch('https://api.github.com/user', {
+    headers: {
+      Authorization: 'Bearer ${keys.GITHUB_TOKEN}',
+    },
+  });
+</script>
+```
+
+When a tool needs a one-off key, the agent can register an ad-hoc secret via `POST /_agent-native/secrets/adhoc` with a `urlAllowlist` that pins which domains the secret may be sent to. A request to any other host is rejected before the proxy fires. Combined with SSRF and private-network protections, this means a leaked tool can't exfiltrate secrets to an attacker-controlled URL.
+
 ## Sharing {#sharing}
 
 Tools are **private by default** — only you can see and use a tool you create.
@@ -79,6 +139,8 @@ You can share tools with your team:
 
 Shared tools have their own URLs, so you can link to them directly.
 
+Under the hood, tools use the same ownable-resource model as the rest of the framework — `ownableColumns()` on the `tools` table and a standard `createSharesTable()` for grants. That means tools plug into the same share dialog, access checks, and audit surfaces as documents, decks, dashboards, and any other shareable resource. See [Security](/docs/security) for the full access model.
+
 ## Security {#security}
 
 Tools run in a secure sandbox:
@@ -89,6 +151,38 @@ Tools run in a secure sandbox:
 - **Private network protection** — tools can't reach internal/private network addresses.
 - **Authentication required** — only logged-in users can use tools.
 
+## Tool API reference {#api-reference}
+
+Every tool runs inside a sandboxed iframe with the following helpers injected on `window`. They are the complete surface area — anything else a tool needs has to go through one of these.
+
+| Helper                                      | Purpose                | Example                                       |
+| ------------------------------------------- | ---------------------- | --------------------------------------------- |
+| `toolData.set(collection, id, data, opts?)` | Persist data per-tool  | `toolData.set('notes', id, { text: '...' })`  |
+| `toolData.list(collection, opts?)`          | List persisted items   | `toolData.list('notes', { scope: 'all' })`    |
+| `toolData.get(collection, id, opts?)`       | Get a single item      | `toolData.get('notes', 'note-1')`             |
+| `toolData.remove(collection, id, opts?)`    | Delete persisted item  | `toolData.remove('notes', 'note-1')`          |
+| `appAction(name, params)`                   | Call any app action    | `appAction('list-emails', { view: 'inbox' })` |
+| `dbQuery(sql, args)`                        | Read from SQL          | `dbQuery('SELECT * FROM tools')`              |
+| `dbExec(sql, args)`                         | Write to SQL           | `dbExec('INSERT INTO ...')`                   |
+| `appFetch(path, options)`                   | Call any app endpoint  | `appFetch('/api/settings')`                   |
+| `toolFetch(url, options)`                   | External API via proxy | `toolFetch('https://api.github.com/...')`     |
+
+`appAction` is the preferred way to trigger app behavior — it routes through the same actions the agent and the frontend use, so authorization and access scoping happen automatically. Drop down to `dbQuery`/`dbExec` only when there's no action that fits.
+
+### Routes {#routes}
+
+The framework mounts the following endpoints under `/_agent-native/tools/`. Tools themselves rarely call these directly — they're useful when integrating tools with external scripts or custom UI.
+
+| Method | Path                              | Purpose                                      |
+| ------ | --------------------------------- | -------------------------------------------- |
+| GET    | `/_agent-native/tools`            | List tools (filtered by ownership + sharing) |
+| POST   | `/_agent-native/tools`            | Create a tool                                |
+| GET    | `/_agent-native/tools/:id`        | Get a single tool                            |
+| PUT    | `/_agent-native/tools/:id`        | Update (supports `patches` for diffing)      |
+| DELETE | `/_agent-native/tools/:id`        | Delete a tool                                |
+| GET    | `/_agent-native/tools/:id/render` | Render the iframe HTML                       |
+| POST   | `/_agent-native/tools/proxy`      | Authenticated proxy with secret injection    |
+
 ## Examples {#examples}
 
 Here are some things people build as tools:

package/docs/content/tracking.md

@@ -1,5 +1,5 @@
 ---
-title: "Analytics Tracking"
+title: "Tracking & Analytics"
 description: "Server-side analytics with pluggable providers — PostHog, Mixpanel, Amplitude, or custom webhook"
 ---
 

package/docs/content/what-is-agent-native.md

@@ -177,5 +177,7 @@ One action, four surfaces: the agent calls it as a tool, the UI calls it as a ty
 - [**Getting Started**](/docs) — pick a template and run it
 - [**Key Concepts**](/docs/key-concepts) — the architecture: SQL, actions, polling sync, context awareness, portability
 - [**Cloneable SaaS**](/docs/cloneable-saas) — templates as complete products you own
-- [**Workspace**](/docs/workspace) — the per-user customization layer (skills, memory, instructions, MCP)
+- [**Workspace**](/docs/workspace) — the per-user customization layer (skills, memory, instructions, MCP) backed by SQL, not files
+- [**Dispatch**](/docs/dispatch) — the workspace control plane: secrets vault, Slack/email inbox, cross-app delegation
+- [**Tools**](/docs/tools) — sandboxed mini-apps the agent creates instantly without code changes
 - [**Drop-in Agent**](/docs/drop-in-agent) — mount `<AgentPanel>` into any React app

package/docs/content/workspace-management.md

@@ -1,13 +1,13 @@
 ---
-title: "Workspace Management"
-description: "Branching strategies, code ownership, PR review, and governance practices for agent-native workspaces."
+title: "Workspace Governance"
+description: "Branching, CODEOWNERS, PR review, and how Dispatch handles runtime governance alongside git-level governance."
 ---
 
-# Workspace Management
+# Workspace Governance
 
 This guide covers the operational side of running an agent-native workspace — how to branch, who reviews what, how to set up code ownership, and how the dispatch control plane fits into your governance model.
 
-For workspace setup, shared auth, and deployment, see [Multi-App Workspace](/docs/multi-app-workspace).
+For workspace setup, shared auth, and deployment, see [Multi-App Workspaces](/docs/multi-app-workspace).
 
 ## Branching
 
@@ -179,7 +179,7 @@ When reviewing PRs in this environment:
 
 ## Dispatch as Governance
 
-The dispatch app is the workspace's runtime control plane. It complements git-level governance with runtime governance:
+The [Dispatch](/docs/dispatch) app is the workspace's runtime control plane. It complements git-level governance with runtime governance:
 
 | Concern                         | Git / GitHub                  | Dispatch                                   |
 | ------------------------------- | ----------------------------- | ------------------------------------------ |

package/docs/content/workspace.md

@@ -5,6 +5,8 @@ description: "Claude-Code-level customization per user — skills, memory, instr
 
 # Workspace
 
+> **See also:** Deploying multiple apps as one workspace? See [Multi-App Workspaces](/docs/multi-app-workspace). Governance, branching, and CODEOWNERS? See [Workspace Governance](/docs/workspace-management).
+
 Every agent-native app ships with a **workspace**: the customization layer that makes the agent yours. It contains team instructions (`AGENTS.md`), per-user memory (`learnings.md`), skills the agent pulls in on demand, custom sub-agents, scheduled jobs, and connected MCP servers — everything you'd expect from a Claude Code / Codex setup.
 
 The twist: **it's SQL rows, not filesystem files.** Each user gets their own workspace stored in the database. There's no dev-box to spin up, no container per user, no files to mount. A multi-tenant SaaS can give every user a fully-customizable agent for essentially free, because all of it is rows — personal memory, personal MCP servers, personal skills, personal sub-agents — and the shared codebase hosts all of them at once.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agent-native/core",
-  "version": "0.7.50",
+  "version": "0.7.52",
   "type": "module",
   "description": "Framework for agent-native application development — where AI agents and UI share state via files",
   "license": "MIT",