@mindstudio-ai/remy 0.1.262 → 0.1.264

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/dist/headless.js CHANGED
@@ -2110,7 +2110,7 @@ var loadSkillTool = {
2110
2110
  definition: {
2111
2111
  clearable: true,
2112
2112
  name: "loadSkill",
2113
- description: "Load the full reference for a platform capability that isn't in your system prompt \u2014 task agents, agent interfaces, MCP interfaces, data sources. The available skills and the trigger for each are listed in <available_skills>. Load one before writing code in its area, not after: these are APIs where a plausible-looking guess is usually wrong. Calling this is cheap and expected \u2014 if you're unsure whether you need it, load it. Only covers the capabilities listed in the catalog; for backend SDK actions and model IDs use askMindStudioSdk.",
2113
+ description: "Load the full reference for a platform capability that isn't in your system prompt \u2014 task agents, agent interfaces, voice interfaces, MCP interfaces, data sources. The available skills and the trigger for each are listed in <available_skills>. Load one before writing code in its area, not after: these are APIs where a plausible-looking guess is usually wrong. Calling this is cheap and expected \u2014 if you're unsure whether you need it, load it. Only covers the capabilities listed in the catalog; for backend SDK actions and model IDs use askMindStudioSdk.",
2114
2114
  inputSchema: {
2115
2115
  type: "object",
2116
2116
  properties: {
@@ -4327,6 +4327,7 @@ Each interface type invokes the same backend methods. Methods don't know which i
4327
4327
  - Email \u2014 inbound email processing
4328
4328
  - MCP \u2014 tool servers for AI assistants
4329
4329
  - Agent \u2014 conversational LLM interface with tool access to backend methods
4330
+ - Voice \u2014 the app's agent as a realtime voice conversation, with the same tool access
4330
4331
 
4331
4332
  ## Backend
4332
4333
 
package/dist/index.js CHANGED
@@ -2927,7 +2927,7 @@ var init_loadSkill = __esm({
2927
2927
  definition: {
2928
2928
  clearable: true,
2929
2929
  name: "loadSkill",
2930
- description: "Load the full reference for a platform capability that isn't in your system prompt \u2014 task agents, agent interfaces, MCP interfaces, data sources. The available skills and the trigger for each are listed in <available_skills>. Load one before writing code in its area, not after: these are APIs where a plausible-looking guess is usually wrong. Calling this is cheap and expected \u2014 if you're unsure whether you need it, load it. Only covers the capabilities listed in the catalog; for backend SDK actions and model IDs use askMindStudioSdk.",
2930
+ description: "Load the full reference for a platform capability that isn't in your system prompt \u2014 task agents, agent interfaces, voice interfaces, MCP interfaces, data sources. The available skills and the trigger for each are listed in <available_skills>. Load one before writing code in its area, not after: these are APIs where a plausible-looking guess is usually wrong. Calling this is cheap and expected \u2014 if you're unsure whether you need it, load it. Only covers the capabilities listed in the catalog; for backend SDK actions and model IDs use askMindStudioSdk.",
2931
2931
  inputSchema: {
2932
2932
  type: "object",
2933
2933
  properties: {
@@ -5164,6 +5164,7 @@ Each interface type invokes the same backend methods. Methods don't know which i
5164
5164
  - Email \u2014 inbound email processing
5165
5165
  - MCP \u2014 tool servers for AI assistants
5166
5166
  - Agent \u2014 conversational LLM interface with tool access to backend methods
5167
+ - Voice \u2014 the app's agent as a realtime voice conversation, with the same tool access
5167
5168
 
5168
5169
  ## Backend
5169
5170
 
@@ -214,6 +214,8 @@ All auth methods throw on failure with a `code` property:
214
214
  | `invalid_state` | 400 | Sign in with Remy: returned CSRF state didn't match (stale/replayed redirect) |
215
215
  | `popup_blocked` | — | Sign in with Remy (embedded): popup was blocked — prompt to allow popups and retry |
216
216
  | `signin_timeout` | — | Sign in with Remy (embedded): popup didn't complete in time |
217
+ | `auth_required` | 401 | Agent/voice interface requires an authenticated user (interface `auth` block) |
218
+ | `role_required` | 403 | Agent/voice interface requires a role the user doesn't hold |
217
219
 
218
220
  ### Phone Helpers
219
221
 
@@ -380,9 +382,19 @@ Roles are declared in the manifest, stored as an array column on the user table,
380
382
  - Writable from dashboard: Remy dashboard shows app users and their roles
381
383
  - Backend enforcement: `auth.requireRole('admin')` works as before
382
384
 
385
+ ## Interface-Level Auth (Agent + Voice)
386
+
387
+ Agent and voice interfaces additionally declare auth **in their config** (a required `auth` key:
388
+ `{ "requireUser": boolean, "requireRole"?: string[] }`) because those sessions spend money without
389
+ necessarily calling a backend method — the platform gates the lobby itself, before any model or
390
+ media spend. `requireRole` uses the same manifest role ids with OR semantics. Denials reach the
391
+ frontend SDK as `MindStudioInterfaceError` codes `auth_required` (401) and `role_required` (403) —
392
+ route them to the app's login flow. See the `agentInterfaces` / `voiceInterfaces` skills for the
393
+ full contract. Method-level `auth.requireRole(...)` checks still apply to every tool call inside.
394
+
383
395
  ## Apps Without Auth
384
396
 
385
- Apps without `auth` in the manifest use anonymous guest sessions. No login, no user identity, no roles. This is the default and works fine for single-user apps, internal tools, and simple utilities.
397
+ Apps without `auth` in the manifest use anonymous guest sessions. No login, no user identity, no roles. This is the default and works fine for single-user apps, internal tools, and simple utilities. (Agent/voice interfaces on such apps declare `"auth": { "requireUser": false }` explicitly — anonymous callers are scoped by a per-browser visitor identity.)
386
398
 
387
399
  ## Important: Designing Auth in Web Interfaces
388
400
 
@@ -88,6 +88,8 @@ auth.logout() // clears session
88
88
 
89
89
  For apps with an agent interface, the SDK also provides `createAgentChatClient()` for thread management and streaming chat. Load the `agentInterfaces` skill for its usage — thread APIs, streaming callbacks, and attachments are all there.
90
90
 
91
+ For apps with a voice interface, `createVoiceClient()` lives on the `@mindstudio-ai/interface/voice` subpath (deliberately separate so non-voice apps ship none of it). Load the `voiceInterfaces` skill for its usage — session lifecycle, live-caption events, tool status, and the voice-UI patterns are all there.
92
+
91
93
  The project uses `"jsx": "react-jsx"` (automatic JSX transform) — do not `import React from 'react'`. Only import the specific hooks and types you need (e.g., `import { useState, useEffect } from 'react'`).
92
94
 
93
95
  On deploy, the platform runs `npm install && npm run build` in the web directory and hosts the output on CDN.
@@ -180,10 +182,16 @@ It supports the full MCP surface: tools (methods the agent can call), resources
180
182
 
181
183
  ## Agent (Conversational Interface)
182
184
 
183
- A conversational interface where an LLM has access to the app's methods as tools. Unlike MCP (which exposes methods for external agents), the agent interface IS the agent — it has its own personality, system prompt, and model config, and orchestrates tool calls against the app's methods internally. Chat runs as the authenticated user, so every tool call carries that user's roles.
185
+ A conversational interface where an LLM has access to the app's methods as tools. Unlike MCP (which exposes methods for external agents), the agent interface IS the agent — it has its own personality, system prompt, and model config, and orchestrates tool calls against the app's methods internally. Chat runs as the authenticated user, so every tool call carries that user's roles. The config must declare an `auth` block (`{ "requireUser": boolean, "requireRole"?: string[] }`) gating who may chat at all.
184
186
 
185
187
  **Load the `agentInterfaces` skill** before authoring `src/interfaces/agent.md` or building the chat UI — the spec frontmatter, compiled output, `agent.json`, and the entire frontend surface are all there.
186
188
 
189
+ ## Voice (Realtime Conversation)
190
+
191
+ The app's agent as a live voice conversation — the user talks, and the agent answers in sub-second, interruptible speech, calling methods mid-conversation. A sibling of the agent interface, not a mode of it: its own spec, a persona written for the ear rather than the screen, and a smaller toolset where every tool carries a latency class governing how the agent handles the wait out loud. Sessions run as the authenticated user, so tool calls carry that user's roles; the platform handles the realtime media, turn-taking, barge-in, and transcripts. The config must declare an `auth` block (`{ "requireUser": boolean, "requireRole"?: string[] }`) gating who may start a session at all.
192
+
193
+ **Load the `voiceInterfaces` skill** before authoring `src/interfaces/voice.md` or building the voice UI — the spoken-register rules, latency classes, spec format, `interface.json`, and the `createVoiceClient()` frontend surface are all there.
194
+
187
195
  ## Manifest Declaration
188
196
 
189
197
  Each interface is declared in `mindstudio.json`:
@@ -197,7 +205,8 @@ Each interface is declared in `mindstudio.json`:
197
205
  { "type": "webhook", "path": "dist/interfaces/webhook/interface.json" },
198
206
  { "type": "email", "path": "dist/interfaces/email/interface.json" },
199
207
  { "type": "mcp", "path": "dist/interfaces/mcp/interface.json" },
200
- { "type": "agent", "path": "dist/interfaces/agent/agent.json" }
208
+ { "type": "agent", "path": "dist/interfaces/agent/agent.json" },
209
+ { "type": "voice", "path": "dist/interfaces/voice/interface.json" }
201
210
  ]
202
211
  }
203
212
  ```
@@ -111,7 +111,7 @@
111
111
 
112
112
  | Field | Type | Required | Description |
113
113
  |-------|------|----------|-------------|
114
- | `type` | `string` | Yes | One of: `web`, `api`, `cron`, `webhook`, `email`, `mcp`, `agent` |
114
+ | `type` | `string` | Yes | One of: `web`, `api`, `cron`, `webhook`, `email`, `mcp`, `agent`, `voice` |
115
115
  | `path` | `string` | No | Path to the interface config file |
116
116
  | `config` | `object` | No | Inline config (alternative to a file) |
117
117
  | `enabled` | `boolean` | No | Default `true`. Set `false` to skip during build. |
@@ -23,6 +23,7 @@ my-app/
23
23
  web.md web UI spec
24
24
  api.md API conventions
25
25
  agent.md agent personality and behavior spec
26
+ voice.md voice agent persona and toolset spec
26
27
  cron.md scheduled job descriptions
27
28
  roadmap/ feature roadmap (one file per item, type: roadmap)
28
29
 
@@ -53,6 +54,10 @@ my-app/
53
54
  agent.json agent config
54
55
  system.md compiled system prompt
55
56
  tools/ tool descriptions (one .md per method)
57
+ voice/ voice interface
58
+ interface.json voice config
59
+ system.md compiled voice-register system prompt
60
+ tools/ tool descriptions (one .md per method)
56
61
  ```
57
62
 
58
63
  ## What Goes Where
@@ -95,7 +100,7 @@ const { vendor } = await api.approveVendor({ vendorId: '...' });
95
100
 
96
101
  - **Managed databases.** SQLite with typed schemas. Push a schema change and the platform diffs, migrates, and promotes atomically.
97
102
  - **Built-in auth.** Opt-in via manifest. Developer builds login UI, platform handles verification codes (email/SMS), cookie sessions, and role enforcement. Backend methods use `auth.requireRole('admin')` for access control.
98
- - **Multiple interfaces, one codebase.** Web, API, Cron, Webhook, Email, MCP — all invoke the same methods. Methods don't know which interface called them.
103
+ - **Multiple interfaces, one codebase.** Web, API, Cron, Webhook, Email, MCP, Agent, Voice — all invoke the same methods. Methods don't know which interface called them.
99
104
  - **Sandboxed execution.** Each method invocation runs in its own isolated execution context with npm packages pre-installed.
100
105
  - **Git-native deployment.** Push to default branch to deploy. Push to feature branch for preview. Rollback is a git revert.
101
106
  - **Secrets.** Encrypted environment variables with separate dev/prod values. Injected as `process.env` in methods. For third-party service credentials not covered by the SDK.
@@ -257,6 +257,7 @@ dist/interfaces/agent/
257
257
  "temperature": 0.5,
258
258
  "maxTokens": 16000,
259
259
  "systemPrompt": "system.md",
260
+ "auth": { "requireUser": true },
260
261
  "tools": [
261
262
  { "method": "create-todo", "description": "tools/createTodo.md" },
262
263
  { "method": "list-todos", "description": "tools/listTodos.md" }
@@ -276,6 +277,7 @@ across rather than copying the key.
276
277
  | `temperature` | Model temperature |
277
278
  | `maxTokens` | Max response tokens (the spec's `maxResponseTokens`) |
278
279
  | `systemPrompt` | Relative path to the compiled system prompt markdown file |
280
+ | `auth` | **Required.** Who may open the lobby: `{ "requireUser": boolean, "requireRole"?: string[] }`. See the Auth section below |
279
281
  | `tools` | Array of tool entries — `method` references a method `id` from the manifest, `description` is a relative path to a markdown file with rich tool docs (when to use, examples, edge cases, parameter guidance) |
280
282
  | `webInterfacePath` | Optional. If the app has a web interface with a chat page, this path tells the IDE where to show the preview. Otherwise the agent is accessed via API. |
281
283
 
@@ -287,8 +289,29 @@ Declare it in `mindstudio.json`:
287
289
 
288
290
  ## Auth
289
291
 
290
- Agent chat runs as the **authenticated user**, not as a system role tool calls carry that user's
291
- roles, so a method gated with `auth.requireRole` behaves exactly as it would if the user had called it
292
- from the web frontend. That's what makes exposing real methods safe; it's also why role restrictions
293
- belong in the tool descriptions, so the agent can decline gracefully instead of surfacing a rejection.
292
+ **Every agent config declares an `auth` block.** Agent chat spends the owner's money on every
293
+ message without necessarily touching a backend method, so the platform gates the lobby itself
294
+ enforced at thread creation and message send:
295
+
296
+ ```json
297
+ "auth": { "requireUser": true, "requireRole": ["support-agent", "admin"] }
298
+ ```
299
+
300
+ - `requireUser: true` — only authenticated app users may chat; `false` — anyone, including
301
+ anonymous visitors. Most apps want `true`; choose `false` deliberately (a public concierge).
302
+ - `requireRole` (optional) — the user must hold **at least one** of the listed manifest role ids
303
+ (OR semantics, same as the backend `auth.requireRole(...)`). Omit or leave empty for no role
304
+ gate. Requires `requireUser: true`. Unknown role ids fail the build.
305
+ - Denials surface to the frontend SDK as `MindStudioInterfaceError` with code `auth_required`
306
+ (401) or `role_required` (403).
307
+ - Dev preview is exempt — the builder is never locked out while testing.
308
+ - Older compiled apps without the block fall back to the manifest's `auth.enabled` (auth-enabled →
309
+ users only; no auth → public). New configs always declare it explicitly.
310
+
311
+ Once inside, agent chat runs as the **authenticated user**, not as a system role — tool calls
312
+ carry that user's roles, so a method gated with `auth.requireRole` behaves exactly as it would if
313
+ the user had called it from the web frontend. That's what makes exposing real methods safe; it's
314
+ also why role restrictions belong in the tool descriptions, so the agent can decline gracefully
315
+ instead of surfacing a rejection. Anonymous visitors (when allowed) are scoped by a per-browser
316
+ visitor identity: their threads are private to their browser, and gated methods still reject.
294
317
 
@@ -30,7 +30,9 @@ short docs that fit in a prompt.
30
30
  - **Limits apply**: 25 data sources per app, 5,000 documents per source, 10,000 chunks per
31
31
  document, 300 searches/minute. Well clear of normal use — but **source names must be fixed, not
32
32
  computed per user or per request**, since referencing one creates it. Partition inside a source
33
- with a metadata filter instead.
33
+ with document metadata instead: tag at add time
34
+ (`add(bytes, { filename, metadata: { userId } })`), narrow at search time
35
+ (`search(q, { filter: { metadata: { userId } } })`).
34
36
 
35
37
  ## Defining and searching
36
38
 
@@ -51,7 +53,22 @@ beside the answer. Retrieval is approximate; a user who can click through can ju
51
53
  An answer with no citation is an assertion.
52
54
 
53
55
  Created on first use, so searching a source the build hasn't populated returns no results rather than
54
- throwing. `search` options: `topK` (default 5, max 50), `scoreThreshold`, `rerank`, `hybrid`.
56
+ throwing. `search` options: `topK` (default 5, max 50), `scoreThreshold`, `filter`, `mode`,
57
+ `maxPerDocument`, `highlight`, `rerank`, `hybrid`.
58
+
59
+ **Filtering** narrows a search before ranking, and every condition only narrows:
60
+ `filter: { metadata: { department: 'legal', year: [2025, 2026] }, filename, documentIds,
61
+ pages: { min?, max? }, contains: 'all these words', phrase: 'exact adjacent sequence' }`.
62
+ Metadata is tagged at add time (scalars only, ≤16 keys); re-adding the same bytes with different
63
+ metadata updates the tags in place, free. Filters are the right tool for scoping retrieval
64
+ (per-user, per-category); they are NOT a substitute for a `db` query over structured data.
65
+
66
+ **Modes**: `mode: 'hybrid'` (default) fuses semantic and keyword retrieval; `'semantic'` is the
67
+ embedding alone; `'lexical'` is keyword-only with **no query embedding** — cheapest and fastest,
68
+ right when the query is an identifier (an error code, a SKU, a name) rather than a meaning.
69
+ `maxPerDocument: 2` stops one document monopolizing the results when the answer should draw on
70
+ several. `highlight: true` adds `matches` (`{start, end}` offsets into `text`) for rendering
71
+ highlighted excerpts.
55
72
 
56
73
  Search is deterministic for a fixed corpus and configuration, so eval sets and regression checks are
57
74
  meaningful — key them on `(documentId, chunkIndex)` rather than on chunk text.
@@ -69,7 +86,9 @@ set with the CLI, so code and reality can't drift.
69
86
 
70
87
  ```bash
71
88
  mindstudio-prod datasources add --source policies --wait docs/*.pdf
89
+ mindstudio-prod datasources add --source policies --metadata department=legal,year=2026 contract.pdf
72
90
  mindstudio-prod datasources search --source policies "what are the payment terms?" # sanity-check
91
+ mindstudio-prod datasources search --source policies --filter department=legal --mode lexical "ERR-7741X"
73
92
  mindstudio-prod datasources delete --source policies # whole source; --source is required, never defaulted
74
93
  ```
75
94
 
@@ -83,7 +102,11 @@ it needs no guard.
83
102
  Use the SDK's `add()` only when *users* upload documents that must become searchable:
84
103
 
85
104
  ```typescript
86
- await Policies.add(buffer, { filename: 'policy.pdf', contentType: 'application/pdf' });
105
+ await Policies.add(buffer, {
106
+ filename: 'policy.pdf',
107
+ contentType: 'application/pdf',
108
+ metadata: { department: 'legal' }, // filterable at search time
109
+ });
87
110
  const docs = await Policies.documents(); // 'processing' | 'done' | 'error'
88
111
  await Policies.remove(documentId);
89
112
  ```
@@ -268,7 +268,7 @@ Declare it in `mindstudio.json`:
268
268
  `custom_subdomain` host (e.g. `myapp.madewithremy.com`), a custom domain if configured, or the UUID
269
269
  host (`<appId>.madewithremy.com` / `.msagent.ai`).
270
270
  - **Auth is optional.** A `Bearer` key resolves to a user with full RBAC, so the method's own
271
- `auth.requireRole`/`requireUser` checks apply as they would for that user. With no key, calls run
271
+ `auth.requireRole(...)`/`hasRole(...)` checks apply as they would for that user. With no key, calls run
272
272
  anonymously — no user, no roles. The method is the boundary: gate sensitive tools, and understand that
273
273
  a public (keyless) server effectively exposes only the un-gated ones.
274
274
  - Input schemas are derived automatically from each method's input contract.
@@ -141,7 +141,7 @@ Routes are mounted at `/_/api{path}` (e.g. `DELETE /_/api/vendors/abc123`).
141
141
  - **Request body** for POST/PUT/PATCH is the input directly (no `{ input: {...} }` wrapper)
142
142
  - **Response** is the method output directly (no `{ output: {...} }` wrapper)
143
143
  - **Auth** via `Authorization: Bearer sk_...` — an API key resolves to a user with full RBAC, so the
144
- method's own `auth.requireRole`/`requireUser` checks apply exactly as they would for that user
144
+ method's own `auth.requireRole(...)`/`hasRole(...)` checks apply exactly as they would for that user
145
145
  - **Streaming**: `Accept: text/event-stream` header returns SSE chunks
146
146
  - **Raw request context**: Every API method receives `input._request` with `{ method, headers, rawBody }`.
147
147
  `rawBody` is the original unparsed body as a UTF-8 string — needed for signature verification, since
@@ -0,0 +1,517 @@
1
+ ---
2
+ name: Voice Interfaces
3
+ what: Realtime voice conversation as a first-class interface — the user talks to the app and its voice agent talks back in sub-second, interruptible speech, calling the app's methods mid-conversation as the authenticated user. The platform handles the media transport, turn-taking, barge-in, and transcripts, so the work is authorship — a persona written for the ear, a small toolset where every tool carries a latency class, and descriptions that say results out loud. Any app whose methods do something interesting can pick up a voice, and it is often the most impressive surface it has.
4
+ when: Before authoring `src/interfaces/voice.md`, choosing a voice model or pipeline, deciding which methods a voice agent gets, or building the voice UI with `createVoiceClient()`.
5
+ ---
6
+
7
+ # Building Voice Interfaces
8
+
9
+ A voice interface is the app's agent as a live phone-call-quality conversation: the user speaks, the
10
+ agent answers in speech, and the app's methods are its tools. It is a **sibling of the agent
11
+ interface, not a mode of it** — the two share a philosophy (an LLM projecting the backend contract
12
+ into conversation; load the `agentInterfaces` skill for that shared ground), but everything you
13
+ author differs. The persona is written for the ear, not the screen. The toolset is smaller and
14
+ curated for conversational latency. And every tool declares how the agent should handle the wait,
15
+ because in a live call, silence reads as a dropped line.
16
+
17
+ The platform owns the hard parts — realtime audio transport, turn detection, interruption handling,
18
+ transcripts, session limits, per-user auth on every tool call. Your job is the spec
19
+ (`src/interfaces/voice.md`) and its compilation into `dist/interfaces/voice/`.
20
+
21
+ ## Voice Agent Design
22
+
23
+ ### Written for the ear
24
+
25
+ Everything the agent produces gets spoken aloud. That inverts several habits that are correct
26
+ everywhere else, and the compiled system prompt must carry them explicitly:
27
+
28
+ - **No visual formatting, ever.** No markdown, no lists, no tables, no emoji, no URLs read as
29
+ punctuation soup. If a tool returns a link, say what it is and where it will be, don't recite it.
30
+ - **Spoken-form values.** "Forty-two fifty," not "$42.50". "Two fifteen in the afternoon," not
31
+ "14:15". Read email addresses and confirmation codes character by character, and read them *back*
32
+ for confirmation before acting on them — mishearing one digit of a phone number is the classic
33
+ voice failure. Collect one value per turn; two asked together blend when spoken.
34
+ - **Brevity is a hard rule, not a style preference.** One to two sentences per turn, one question at
35
+ a time. A paragraph that reads fine in chat is a monologue on a call.
36
+ - **Vary the phrasing.** Repeated openers and acknowledgments sound convincing once and robotic by
37
+ the third turn — give the prompt an explicit variety rule, and treat any sample phrases as
38
+ anchors, never scripts.
39
+ - **Handle unclear audio explicitly.** Give the prompt a rule for it: respond only to clear audio;
40
+ if it's noisy or ambiguous, ask the user to repeat — never guess, never call a tool on input the
41
+ agent isn't sure it heard, and don't reuse the same clarification line twice in a row.
42
+ - **Pin the language.** State the response language in the prompt; don't let the model infer it from
43
+ an accent. If the app's domain has brand names or terms with non-obvious pronunciations, give
44
+ them a line ("pronounce SQL as 'sequel'").
45
+
46
+ Beyond the mechanics, the persona itself should be *of the ear*: pacing, warmth, how it handles
47
+ being interrupted, what it says when it needs a second. This is the fun part, same as the agent
48
+ interface — a distinct character beats a generic assistant, and voice makes character land harder
49
+ than any other surface.
50
+
51
+ ### The shape of `system.md`
52
+
53
+ Structure the compiled prompt as short **labeled sections** — Role & Objective, Personality & Tone,
54
+ Rules, and (when the app has a real call flow) Conversation Flow — with bullets over paragraphs;
55
+ realtime models find and follow sectioned rules far more reliably than prose. Scope rules
56
+ precisely: "confirm before any tool that changes data," not "always confirm everything" — blanket
57
+ `always`/`never` makes the agent rigid and unable to handle reasonable exceptions. And start
58
+ minimal: state the role, the boundaries, and the voice mechanics above, then add rules only for
59
+ behaviors that actually misfire in test calls (the transcripts in the call log are the feedback
60
+ loop) rather than front-loading a policy manual.
61
+
62
+ ### The latency classes
63
+
64
+ Every tool in the spec declares one of three classes. This is the voice-specific discipline — get it
65
+ right and tool use feels like talking to a competent person; get it wrong and every action is an
66
+ awkward pause.
67
+
68
+ - **`fast`** — sub-second reads: lookups, availability checks, small queries. The agent calls
69
+ silently; announcing a sub-second call adds more delay than the call itself.
70
+ - **`slow`** — a noticeable wait, roughly one to three seconds: writes, searches, anything that does
71
+ real work. The agent speaks a one-line preamble ("Let me get that booked") generated in parallel
72
+ with the call, so the line never goes quiet.
73
+ - **`background`** — long-running work: reports, enrichment, bulk operations. The agent
74
+ acknowledges, keeps conversing, and reports the result when it lands. Background tools are
75
+ cancellable — if the user changes course mid-run, the work stops.
76
+
77
+ Classify by how the method actually behaves, not by what it is named. A "lookup" that fans out to an
78
+ external service is `slow`. When in doubt between `fast` and `slow`, pick `slow` — a needless
79
+ preamble is mildly chatty; an unexplained silence feels broken.
80
+
81
+ ### Tool descriptions say results out loud
82
+
83
+ Follow the agent-interface principles for tool descriptions (when to use and when not, parameter
84
+ guidance, what comes back) — plus one voice-specific layer: **how to speak the result.** A tool that
85
+ returns a booking record needs its description to say what the confirmation sounds like ("You're all
86
+ set for Tuesday at two") and what never gets read aloud (internal ids, timestamps, enum values).
87
+
88
+ Curate harder than you would for chat. A voice agent with four excellent tools outperforms one with
89
+ twelve adequate ones — every tool the model considers is a beat of hesitation. Skip batch
90
+ operations, admin utilities, and anything whose output can't be said in a breath or two. Note role
91
+ restrictions in the description so the agent declines gracefully in character instead of surfacing a
92
+ rejection.
93
+
94
+ ### Confirmation scales with risk
95
+
96
+ Bake the policy into the system prompt: read-only tools — just call them. Writes — summarize what's
97
+ about to happen and get a yes. Anything destructive or financial — read the details back first,
98
+ piece by piece. In voice there is no confirmation dialog to lean on; the conversation *is* the
99
+ confirmation UI.
100
+
101
+ And give failure a script: never speak a raw error. When a lookup misses or a tool fails, read back
102
+ the value it used ("I couldn't find an order ending three-one-two-five — did I get part of that
103
+ wrong?"), offer one retry, then move to an alternate path — in character, without blaming the
104
+ caller.
105
+
106
+ ### Choosing the model
107
+
108
+ Two shapes, one `model` field:
109
+
110
+ - **Native speech-to-speech** (`{"model": ..., "voice": ...}`) — one realtime model hears and
111
+ speaks. Lowest latency, most natural prosody, hears tone and hesitation. The default for
112
+ personality-forward, conversational apps.
113
+ - **Cascaded** (`{"llm": ..., "stt": ..., "tts": ..., "voice": ...}`) — streaming transcription
114
+ into any chat model in the catalog, streaming speech out. Slightly higher latency, but the brain
115
+ can be *any* chat model — the right choice when the app's reasoning demands a specific model, or
116
+ when the agent interface already uses one and the voice should think identically. The blessed
117
+ streaming pairing is `"stt": "deepgram-nova-3", "tts": "cartesia-sonic-3"` — the lowest-latency
118
+ combination the platform wires; prefer it unless there's a reason not to. One nuance: cascaded
119
+ engines speak the `greeting` verbatim (they have a real TTS); speech-to-speech engines have the
120
+ model say it, so it may paraphrase slightly.
121
+
122
+ Ask `askMindStudioSdk` for available ids — realtime, transcription, and speech models are separate
123
+ catalogs, and MindStudio ids don't match vendor ids, so treat ids in this document as illustrative.
124
+ Voice ids are model-specific; query for those too. The user's UI has a picker for changing the model
125
+ later, so validate only when you set it.
126
+
127
+ ### Seeding from an existing agent
128
+
129
+ If the app already has an agent interface, start from it: same character, same values, same
130
+ terminology — then rewrite for the ear (shorter, spoken-form, no formatting) and re-curate the
131
+ toolset for latency. Don't copy `agent.md`'s prose wholesale; a chat persona read aloud sounds like
132
+ someone reading chat aloud.
133
+
134
+ ### Anti-patterns
135
+
136
+ - Prose that would render fine in chat — bullet lists, headers, or markdown anywhere in `system.md`.
137
+ - A tool description that explains what to display instead of what to say.
138
+ - Exposing the whole method surface. Voice is the most curated interface the app has.
139
+ - A generic greeting ("Hello! How can I assist you today?"). The greeting is the first thing anyone
140
+ hears; make it the character's.
141
+ - Writing your own current-user placeholder — the platform appends a `## Current User` block (name,
142
+ roles) to every system prompt at runtime.
143
+
144
+ ## Compiling the Voice Spec
145
+
146
+ When building `dist/interfaces/voice/`, consider the spec, the app, and the `@brand/` guidelines —
147
+ the voice agent should be unmistakably the same product as the web UI, projected into sound. Output:
148
+
149
+ **`system.md`** — the persona compiled for the ear. Character first, then the mandatory carries from
150
+ "Written for the ear" above (spoken-form rules, brevity, unclear-audio handling, language pinning,
151
+ confirmation-by-risk), then any preamble phrasing guidance for `slow` tools so the fillers sound like
152
+ the character too.
153
+
154
+ **`tools/*.md`** — one per tool: when to use, parameter guidance, how to say the result, role
155
+ restrictions.
156
+
157
+ **`interface.json`** — the config tying it together. Full shape in "The wiring" below.
158
+
159
+ ## Voice UI
160
+
161
+ When the app has a web interface, voice arrives as a **layer over it**, not a separate page: a
162
+ persistent affordance (a button, an orb in a corner) that starts a session in place, with the app
163
+ still visible and usable. A dedicated full-screen voice mode is the immersive option for apps where
164
+ the conversation *is* the product — earn it, don't default to it.
165
+
166
+ ### Frontend SDK: `createVoiceClient()`
167
+
168
+ Ships as a subpath of the interface SDK so apps that never use voice pay nothing for it. All voice
169
+ UIs go through it — never hand-roll audio capture or transport.
170
+
171
+ ```ts
172
+ import { createVoiceClient } from '@mindstudio-ai/interface/voice';
173
+
174
+ const voice = createVoiceClient();
175
+
176
+ // Prompts for mic permission, mints a session, connects.
177
+ // Throws MindStudioInterfaceError('microphone_denied') on refusal.
178
+ const session = await voice.startSession();
179
+
180
+ session.state; // 'connecting' | 'listening' | 'thinking' | 'speaking' | 'ended'
181
+ session.on('stateChange', (state) => { }); // on() returns an unsubscribe fn
182
+
183
+ // Live captions, both sides. Each event carries the segment's FULL text so
184
+ // far (never a delta) — render by upserting on segmentId, not appending.
185
+ session.on('transcript', ({ role, segmentId, text, final }) => { });
186
+
187
+ session.on('toolCall', ({ method, status }) => { }); // 'running' | 'done' | 'failed'
188
+ session.on('error', (err) => { });
189
+
190
+ session.mute(); session.unmute(); session.isMuted;
191
+ session.sendText('123 Main Street'); // inject text into the live conversation
192
+ session.end();
193
+ ```
194
+
195
+ Agent audio playback is handled inside the SDK (a hidden autoplaying element) — never create audio
196
+ elements for the agent. `startSession()` throws `MindStudioInterfaceError` with code
197
+ `microphone_denied` when mic access is refused (surface that state gently in the UI),
198
+ `voice_concurrency_limit` / `voice_visitor_limit` when the app's session limits are hit, and
199
+ `auth_required` (401) / `role_required` (403) when the interface's `auth` block denies the caller
200
+ (route those to the app's login flow).
201
+
202
+ Past sessions are call records with transcripts: `voice.listSessions()` /
203
+ `voice.getSession(id)` — the material for a history view if the app wants one.
204
+
205
+ ### The state machine, made visible
206
+
207
+ One audio-reactive element carries the session: idle → connecting → listening → thinking → speaking.
208
+ Always pair it with a **text state label** — never signal state by color or motion alone. Calm at
209
+ idle, responsive to actual audio levels while listening and speaking. Respect
210
+ `prefers-reduced-motion` with a static-but-labeled variant.
211
+
212
+ ### Live captions
213
+
214
+ Stream `transcript` events as captions — both sides of the conversation. Captions make the agent
215
+ feel accurate, catch mishearings early, and are the accessibility story. User-side transcripts
216
+ arrive as recognition output and can lag or differ slightly from what the model heard; render them
217
+ as captions, never treat them as input to app logic.
218
+
219
+ ### Controls that must exist
220
+
221
+ **Mute** and **end call**, always visible, always working. `sendText` earns its place the moment the
222
+ conversation needs an exact string — an address, a code, an email — typing it beats spelling it
223
+ aloud three times. Show tool activity as a compact inline status from `toolCall` events, in the
224
+ app's voice ("Booking your appointment…"), never raw names or JSON.
225
+
226
+ ### Anti-patterns
227
+
228
+ - Blocking the whole UI behind the session — voice is a layer, the app stays usable.
229
+ - An orb with no label, or state changes conveyed only by color.
230
+ - Rendering user-side captions as authoritative ("you said X") — they're recognition output.
231
+ - Auto-starting a session on page load. Microphone access is always a deliberate user action.
232
+
233
+ ## Outbound calls (`voice.call`)
234
+
235
+ The agent can call the user. Backend methods (and crons) place outbound phone calls with the
236
+ agent SDK's `voice` namespace — the platform dials the number and connects the callee to this
237
+ app's voice agent (same persona, engine, and tools as the web sessions):
238
+
239
+ ```ts
240
+ import { voice, auth } from '@mindstudio-ai/agent';
241
+
242
+ export async function callMeAboutMyOrder(input: { phone: string }) {
243
+ auth.requireRole('member');
244
+ const call = await voice.call({ to: input.phone, assumeIdentity: true });
245
+ return { calling: call.to, from: call.from };
246
+ }
247
+ ```
248
+
249
+ - **The method is the authorization gate.** The voice interface's `auth` block does not apply to
250
+ calls the backend places deliberately — gate the *method* with `auth.requireRole(...)` exactly
251
+ as you would any sensitive action.
252
+ - **`assumeIdentity: true`** runs the call as the user who invoked the method: the agent knows
253
+ who it's talking to (Current User block) and every tool call carries their roles — regardless
254
+ of which number was dialed (the user types any number into a field; identity comes from their
255
+ session, not the phone). Omitted/false → anonymous call; role-gated tools decline.
256
+ System/cron invocations have no human identity and always run anonymously.
257
+ - **Production needs a dedicated phone number.** The app owner attaches one ($2/month) via the
258
+ dashboard or `mindstudio-prod voice numbers` (see "Managing the phone side from the CLI"
259
+ below) — it becomes the caller ID for every call, in dev sessions too, so users always see
260
+ the same number. Without one, deployed calls throw `phone_out_requires_dedicated_number`, and
261
+ dev sessions fall back to a shared platform test number that varies per call (tighter limits
262
+ apply on the shared pool).
263
+ - **Outcome is on the call record**, not the return value: `voice.call` returns as soon as
264
+ dialing starts (`{ sessionId, status: 'dialing', from, to }`); answered/busy/no-answer land on
265
+ the session in the app's call log (`voice.listSessions()` / the dashboard).
266
+ - **Limits**: the app's concurrent-session policy, a daily outbound-call cap, a per-call
267
+ duration ceiling, and one active call per callee number (`voice_callee_busy`).
268
+ - **Compliance**: automated calls require prior consent. Call your own users who opted in to
269
+ calls from this app, honor reasonable calling hours, never dial purchased or cold lists —
270
+ design the consent moment into the product (a "call me" button IS consent; a scraped list is
271
+ not).
272
+
273
+ ## Inbound calls
274
+
275
+ Once the app has a dedicated phone number, people can call it — the same voice agent answers
276
+ (same persona, engine, and tools). Nothing extra to author for the basic case; the number in the
277
+ app's settings is the whole switch.
278
+
279
+ How answering works:
280
+
281
+ - **Inbound always runs the live release.** There is no dev inbound — test the agent over the
282
+ normal WebRTC session in the editor; the phone is the same interface with a different
283
+ transport. An app with no live voice interface (or at its concurrency limit) doesn't answer.
284
+ - **Callers are anonymous until verified.** The `auth` block still applies, but a phone call
285
+ can't show a login page — so the platform answers first, and `requireUser` becomes an
286
+ in-call verification flow. The agent can serve whatever anonymous callers are allowed, and
287
+ offers verification when the caller wants something account-bound.
288
+ - **Verification uses the app's own auth methods** (`sms-code` / `email-code` from the
289
+ manifest), existing accounts only — there is no sign-up over the phone:
290
+ - SMS: a code is texted to the number the caller is calling from, if an account has that
291
+ number on file. No other number is possible by design.
292
+ - Email: the caller says their address; the platform matches it against the app's users
293
+ (transcription-tolerant — no letter-by-letter spelling ceremony) and emails the account's
294
+ stored address a code.
295
+ - The flow never confirms or denies that an account exists — a code is "sent if an account
296
+ matches", always phrased that neutrally. The persona should offer verification naturally
297
+ when it unlocks something, never as a robotic gate.
298
+ - **Verified mid-call, upgraded mid-call**: once the code checks out, the session becomes that
299
+ user's — Current User block, roles on every tool call — without redialing.
300
+
301
+ ### `phone.trustCallerId`
302
+
303
+ For apps whose users are known by phone number, the interface config may opt into treating
304
+ caller ID as identity:
305
+
306
+ ```json
307
+ "phone": { "trustCallerId": true }
308
+ ```
309
+
310
+ A caller whose number exactly matches an app user's phone starts the call already verified —
311
+ no code. This is a real security tradeoff: **caller ID can be spoofed**, so a motivated
312
+ attacker who knows a user's phone number can impersonate them to this agent. Before enabling
313
+ it, you MUST surface that risk to the user and get their explicit confirmation — it's the
314
+ right call for convenience-first, low-stakes apps (a family assistant, a status line), and the
315
+ wrong one wherever the agent's tools can move money, reveal sensitive records, or take
316
+ destructive actions. It lives in the interface config deliberately: enabling it is a code
317
+ change, visible in review and auditable via deploys, not a dashboard toggle.
318
+
319
+ ## Managing the phone side from the CLI
320
+
321
+ The `mindstudio-prod voice` family covers numbers, the call log, and voice policy:
322
+
323
+ ```bash
324
+ mindstudio-prod voice numbers search --area-code 310 # available numbers to offer the user
325
+ mindstudio-prod voice numbers buy +13105551234 # buy + attach ($2/month — see below)
326
+ mindstudio-prod voice numbers release +13105551234 # permanent; no refund, ~15-day quarantine
327
+ mindstudio-prod voice sessions list --limit 10 # call log: web / phone-out / phone-in
328
+ mindstudio-prod voice sessions get <sessionId> # full transcript + cost breakdown
329
+ ```
330
+
331
+ Also `voice numbers list`, `voice settings get`/`set` (concurrency, per-visitor,
332
+ max duration). `--help` for flags.
333
+
334
+ **Never buy a number without the user's explicit confirmation** — it starts a recurring
335
+ $2/month workspace charge. Search first, present the options with the price, and only run
336
+ `numbers buy` after they've picked one and said yes.
337
+
338
+ Transcripts are how you iterate on a voice persona: after the user test-calls the agent, read
339
+ `voice sessions get` for what was actually said — misheard input, interruptions, tools declining
340
+ — and fix the spec from evidence rather than guesses. (Dev-session test calls carry a
341
+ `devSessionId` in the list, so you can tell them from live traffic.)
342
+
343
+ ---
344
+
345
+ # The wiring
346
+
347
+ ## Spec: `src/interfaces/voice.md`
348
+
349
+ Frontmatter holds the structured fields; the body is the persona plus an explicit `## Tools`
350
+ section.
351
+
352
+ ```yaml
353
+ ---
354
+ name: Front Desk
355
+ description: Books appointments and answers questions by voice.
356
+ type: interface/voice
357
+ model: {"model": "gpt-realtime-mini", "voice": "marin"}
358
+ turnDetection: {"eagerness": "medium"}
359
+ greeting: Hey! I can help you book, reschedule, or answer questions — what do you need?
360
+ ---
361
+ ```
362
+
363
+ Frontmatter fields:
364
+
365
+ - `name` — display name
366
+ - `description` — one-liner for listings
367
+ - `model` — JSON string, two shapes: native speech-to-speech `{"model": <realtime model id>,
368
+ "voice": <voice id>}`, or cascaded `{"llm": <chat model id>, "stt": <transcription model id>,
369
+ "tts": <speech model id>, "voice": <voice id>}`. Optional `config` for model-specific settings.
370
+ Ids via `askMindStudioSdk`.
371
+ - `turnDetection` — optional; `{"eagerness": "low" | "medium" | "high"}` — how quickly the platform
372
+ decides the user finished speaking. High is snappier; low is more patient (users dictating
373
+ numbers or addresses). Default `medium`.
374
+ - `greeting` — optional spoken opener, delivered on session start. Omit and the agent waits for the
375
+ user to speak first. Verbatim on cascaded engines; model-spoken (may paraphrase) on
376
+ speech-to-speech.
377
+
378
+ Body: persona prose (voice register), then the toolset:
379
+
380
+ ```markdown
381
+ ## Tools
382
+
383
+ ### Book appointment
384
+ method: book-appointment
385
+ latency: slow
386
+ ~~~
387
+ Book an appointment once the caller has confirmed a date, time, and service.
388
+ Read the details back and get a yes before calling. Say the confirmation
389
+ naturally ("You're all set for Tuesday the 4th at 2pm") — never read the
390
+ booking id aloud unless asked.
391
+ ~~~
392
+ ```
393
+
394
+ `latency` is one of `fast` / `slow` / `background` (semantics in "The latency classes" above).
395
+ Don't hand-author input schemas — the platform derives them from the method contract.
396
+
397
+ ## Compiled Output: `dist/interfaces/voice/`
398
+
399
+ ```
400
+ dist/interfaces/voice/
401
+ ├── interface.json ← config the platform reads
402
+ ├── system.md ← compiled voice-register system prompt
403
+ └── tools/
404
+ └── bookAppointment.md ← rich tool description, one per tool
405
+ ```
406
+
407
+ ## Config (`interface.json`)
408
+
409
+ The top-level key must match the interface type (`voice`):
410
+
411
+ ```json
412
+ {
413
+ "voice": {
414
+ "name": "Front Desk",
415
+ "description": "Books appointments and answers questions by voice.",
416
+ "model": "gpt-realtime-mini",
417
+ "voice": "marin",
418
+ "turnDetection": { "eagerness": "medium" },
419
+ "greeting": "Hey! I can help you book, reschedule, or answer questions — what do you need?",
420
+ "systemPrompt": "system.md",
421
+ "auth": { "requireUser": true },
422
+ "tools": [
423
+ { "method": "book-appointment", "latency": "slow", "description": "tools/bookAppointment.md" }
424
+ ],
425
+ "webInterfacePath": "/"
426
+ }
427
+ }
428
+ ```
429
+
430
+ | Field | Description |
431
+ |-------|-------------|
432
+ | `name`, `description` | Display name + listing metadata |
433
+ | `model` | Realtime model id (native speech-to-speech). Mutually exclusive with `llm`/`stt`/`tts` |
434
+ | `llm`, `stt`, `tts` | The cascaded alternative: chat model id + streaming transcription id + streaming speech id |
435
+ | `voice` | Provider voice id (model-specific; query `askMindStudioSdk`) |
436
+ | `turnDetection` | `{ "eagerness": "low" \| "medium" \| "high" }`, optional |
437
+ | `greeting` | Optional spoken opener |
438
+ | `systemPrompt` | Relative path to the compiled system prompt |
439
+ | `auth` | **Required.** Who may start a session: `{ "requireUser": boolean, "requireRole"?: string[] }`. See the Auth section below |
440
+ | `phone` | Optional telephony options: `{ "trustCallerId"?: boolean }` — see "Inbound calls" above. Only add it after the user has confirmed the spoofing tradeoff |
441
+ | `context` | Optional session context: `{ "method": <method id> }` — auto-fired in the background at session start; see "Session context" below |
442
+ | `tools` | `{ method, latency, description }` — method `id` from the manifest, a latency class, and a relative path to the tool's markdown |
443
+ | `webInterfacePath` | Optional. Where the voice layer lives in the web interface, for the editor preview |
444
+
445
+ Declare it in `mindstudio.json`:
446
+
447
+ ```json
448
+ { "type": "voice", "path": "dist/interfaces/voice/interface.json" }
449
+ ```
450
+
451
+ ## Session context (auto-loaded)
452
+
453
+ When the config declares `"context": { "method": "session-context" }`, the platform fires that
454
+ backend method automatically when a session starts — in the background, so the greeting is
455
+ never delayed — and appends its return to the system prompt as a `## Session Context` block.
456
+ Use it for situational state that should color every turn: the caller's open orders, account
457
+ standing, where they left off. Timing: the method runs while the greeting audio plays, so the
458
+ agent has the context by roughly the first exchange and is guaranteed to have it shortly
459
+ after — it is NOT guaranteed for the literal first utterance. On inbound phone calls it
460
+ re-fires after the caller verifies mid-call, so the context recomputes for the now-known user.
461
+
462
+ The method contract:
463
+ - Runs as the session's user (same identity/RBAC as a tool call); anonymous sessions run it
464
+ anonymously — return generic or empty content for them.
465
+ - Return a short markdown **string** (a few lines). Results are capped at 4,000 characters;
466
+ keep it situational context, not documents — deep or on-demand data belongs in tools or
467
+ data sources.
468
+ - It is never a model-visible tool, and failures degrade silently to the generic prompt —
469
+ never make correctness depend on it.
470
+
471
+ Rule of thumb: `context` for always-relevant state the agent should just know; tools for
472
+ anything looked up on demand. Identity itself (name, roles) is already injected via the
473
+ Current User block — don't re-fetch it in the context method.
474
+
475
+ ## Platform Behavior
476
+
477
+ - Input schemas are derived from each method's contract — never hand-written.
478
+ - The platform appends a `## Current User` block (name, roles) to the system prompt at runtime;
479
+ never author a placeholder for it.
480
+ - Turn detection, barge-in (interruption truncates the agent's context to the audio the user
481
+ actually heard), and background-noise handling are platform-managed; `turnDetection.eagerness` is
482
+ the only knob.
483
+ - Sessions have a per-app concurrency limit and a maximum duration, both configurable in the app's
484
+ settings; an idle session is ended gracefully after a prompt. Voice minutes and model usage are
485
+ metered.
486
+ - Every session persists as a call record with a transcript, visible in the dashboard and readable
487
+ from the frontend via `voice.listSessions()` / `voice.getSession(id)`.
488
+
489
+ ## Auth
490
+
491
+ **Every voice config declares an `auth` block.** A voice session spends the owner's money for its
492
+ entire duration without necessarily touching a backend method, so the platform gates session
493
+ creation itself:
494
+
495
+ ```json
496
+ "auth": { "requireUser": true, "requireRole": ["member"] }
497
+ ```
498
+
499
+ - `requireUser: true` — only authenticated app users may start a session; `false` — anyone,
500
+ including anonymous visitors. Most apps want `true`; choose `false` deliberately (a public
501
+ front-desk line).
502
+ - `requireRole` (optional) — the user must hold **at least one** of the listed manifest role ids
503
+ (OR semantics, same as the backend `auth.requireRole(...)`). Omit or leave empty for no role
504
+ gate. Requires `requireUser: true`. Unknown role ids fail the build.
505
+ - Denials reject `startSession()` with code `auth_required` (401) or `role_required` (403).
506
+ - On the phone channel there is no login page to bounce to, so `requireUser` becomes
507
+ answer-then-verify — see "Inbound calls" above.
508
+ - Dev preview is exempt — the builder is never locked out while testing.
509
+ - Older compiled apps without the block fall back to the manifest's `auth.enabled` (auth-enabled →
510
+ users only; no auth → public). New configs always declare it explicitly.
511
+
512
+ Once inside, voice sessions run as the **authenticated user** — every tool call carries that
513
+ user's roles, so a method gated with `auth.requireRole` behaves exactly as it would from the web
514
+ frontend or the agent interface. Anonymous sessions (when allowed) have no user and no roles:
515
+ gated methods reject, and the caller's history is scoped to their browser's visitor identity.
516
+ That's why role restrictions belong in the tool descriptions — the agent should decline in
517
+ character, not relay a rejection.
@@ -19,7 +19,7 @@ The scaffold starts with these spec files that cover the full picture of the app
19
19
  - **`src/interfaces/@brand/voice.md`** — voice and terminology: tone, error messages, word choices
20
20
  - **`src/roadmap/`** — feature roadmap. One file per feature (`type: roadmap`). See "Roadmap" below.
21
21
 
22
- These are starting points, not constraints. Create as many spec files as the project needs — the `src/` folder is your workspace and every `.md` file in it becomes compilation context. If the app has substantial content (presentation slides, copy, lesson plans, menu items, quiz questions), put it in its own file (`src/content.md`, `src/slides.md`, `src/menu.md`, etc.) rather than cramming it into `app.md` or `web.md`. If the domain is complex, split `app.md` into multiple files by area (`src/billing.md`, `src/approvals.md`). Add interface specs for other interface types (`api.md`, `webhook.md`, `cron.md`, `email.md`, `mcp.md`, `agent.md`) if the app uses them. Each of those has a skill carrying its spec format and config — `restApi`, `webhooks`, `scheduledJobs`, `inboundEmail`, `mcpInterfaces`, `agentInterfaces` — and you should load the relevant one before writing the spec rather than after, since the spec is what the config is compiled from. For external HTTP the choice is between two of them: the Webhook interface handles inbound provider webhooks (Stripe, GitHub) via secret-in-URL routing, while the API interface covers bearer-auth sync endpoints, public REST APIs, and batch tools. Organize however serves clarity — the platform reads the entire `src/` folder.
22
+ These are starting points, not constraints. Create as many spec files as the project needs — the `src/` folder is your workspace and every `.md` file in it becomes compilation context. If the app has substantial content (presentation slides, copy, lesson plans, menu items, quiz questions), put it in its own file (`src/content.md`, `src/slides.md`, `src/menu.md`, etc.) rather than cramming it into `app.md` or `web.md`. If the domain is complex, split `app.md` into multiple files by area (`src/billing.md`, `src/approvals.md`). Add interface specs for other interface types (`api.md`, `webhook.md`, `cron.md`, `email.md`, `mcp.md`, `agent.md`, `voice.md`) if the app uses them. Each of those has a skill carrying its spec format and config — `restApi`, `webhooks`, `scheduledJobs`, `inboundEmail`, `mcpInterfaces`, `agentInterfaces`, `voiceInterfaces` — and you should load the relevant one before writing the spec rather than after, since the spec is what the config is compiled from. For external HTTP the choice is between two of them: the Webhook interface handles inbound provider webhooks (Stripe, GitHub) via secret-in-URL routing, while the API interface covers bearer-auth sync endpoints, public REST APIs, and batch tools. Organize however serves clarity — the platform reads the entire `src/` folder.
23
23
 
24
24
  Remember: users care about look and feel as much as (and often more than) underlying data structures. Don't treat the brand and interface specs as an afterthought — for many users, the visual identity and voice are the first things they want to get right.
25
25
 
@@ -81,6 +81,6 @@ You have access to the `mindstudio` CLI, which exposes every SDK action as a com
81
81
  ### Production App Management
82
82
  You have access to `mindstudio-prod`, a CLI for managing the user's production app. Use it via your bash tool. All output is JSON. Run `mindstudio-prod --help` or `mindstudio-prod <command> --help` to discover usage and available options.
83
83
 
84
- Available commands: `requests` (server-side request logs, error rates, latency), `crashes` (frontend browser errors — grouped issues + drill-down to individual events), `analytics` (traffic, top pages/referrers/geo, AI-referral attribution, live counters), `releases` (deploy status, history), `domains` (custom subdomains and fully custom domains), `users` (list, set roles), `db` (query production sql db), `data` (live db operations like lift-from-dev), `methods` (list, invoke), `secrets` (list, get, set, delete), `issues` (reference and manage externally-reported bugs and issues. Do not use this to track work you are doing with the user - only read from it and resolve issues if the user asks for help fixing a bug from the issues tracker).
84
+ Available commands: `requests` (server-side request logs, error rates, latency), `crashes` (frontend browser errors — grouped issues + drill-down to individual events), `analytics` (traffic, top pages/referrers/geo, AI-referral attribution, live counters), `releases` (deploy status, history), `diagnostics` (post-deploy Lighthouse audit), `domains` (custom subdomains and fully custom domains), `users` (list, set roles), `db` (query production sql db), `data` (live db operations like lift-from-dev), `methods` (list, invoke), `secrets` (list, get, set, delete), `files` (upload + manage CDN files), `datasources` (build and query document corpora), `prerender` (bot/crawler snapshots), `voice` (dedicated phone numbers: search/buy/release — buying bills $2/month, so never buy without the user's explicit confirmation; call log + transcripts; voice policy settings), `issues` (reference and manage externally-reported bugs and issues. Do not use this to track work you are doing with the user - only read from it and resolve issues if the user asks for help fixing a bug from the issues tracker).
85
85
 
86
86
  Use when the user asks about production behavior (server errors via `requests`, browser crashes via `crashes`, traffic/engagement via `analytics`), wants to manage their live app (domains, users, roles), needs to seed or query production data, or wants to check release status.
@@ -11,7 +11,7 @@ Remy apps are full-stack TypeScript projects. You have a lot to work with:
11
11
  - **Backend (Methods):** TypeScript in a sandboxed runtime. Any npm package. Managed SQLite database with typed schemas and automatic migrations. Built-in app-managed auth with email/SMS verification, cookie sessions, and role enforcement. None of these are required — use what the app needs.
12
12
  - **Frontend (Web Interface):** Starts as Vite + React, but any TypeScript project with a build command works. Any framework, any library, or no framework at all.
13
13
  - **AI & integrations:** The `@mindstudio-ai/agent` SDK gives access to 200+ AI models (OpenAI, Anthropic, Google, Meta, Mistral, and more) and 1000+ integrations (email, SMS, Slack, HubSpot, Google Workspace, web scraping, image/video generation, media processing) with zero configuration — credentials are handled automatically. No API keys needed. Beyond individual actions, `runTask()` lets you spin up lightweight autonomous task agents that chain these actions together with judgment — e.g., a user types a restaurant name and the backend autonomously researches it in the background, finds the address, generates a custom illustration, and saves the finished record itself. These agents can call the app's own methods too, so they can read existing data to decide what needs doing and write results straight back. Think about where this kind of enrichment would make a feature go from functional to magical.
14
- - **Interfaces:** Web UI, REST API, cron jobs, webhooks, MCP tool servers, email processors, conversational AI agents — all backed by the same methods. An app can use any combination.
14
+ - **Interfaces:** Web UI, REST API, cron jobs, webhooks, MCP tool servers, email processors, conversational AI agents (text chat and realtime voice) — all backed by the same methods. An app can use any combination.
15
15
 
16
16
  This is a capable, stable platform. Build with confidence; you're building production-grade apps, not fragile prototypes.
17
17
 
@@ -23,7 +23,7 @@ Don't recite this list to users. Use it to calibrate your sense of what's possib
23
23
  - **AI-powered apps** — a document processor that extracts structured data from uploaded contracts, an AI image tool that transforms selfies into stylized portraits, a content generator that produces a week of social posts from one brief
24
24
  - **Full-stack web apps** — social platforms, membership sites, marketplaces, booking systems, community hubs — multi-user apps with auth, data, UI
25
25
  - **Automations** — cron jobs that monitor competitors and send alerts, webhook handlers that sync data between services, email processors that triage support requests — no UI needed
26
- - **Conversational AI agents** — custom chat UIs backed by any model, with tool access to the app's methods. Full control over what the agent can do and who can use it
26
+ - **Conversational AI agents** — custom chat UIs backed by any model, with tool access to the app's methods. Full control over what the agent can do and who can use it. The same agents can also answer by realtime voice — a live, interruptible conversation with the app
27
27
  - **Agent tools** — MCP tool servers for AI assistants
28
28
  - **Creative projects** — browser games with p5.js or Three.js, interactive visualizations, 3D things, generative art, portfolio sites with dynamic backends
29
29
  - **Marketing & launch pages** — landing pages, waitlist pages with referral mechanics, product sites with scroll animations — visual polish is a strength here
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@mindstudio-ai/remy",
3
- "version": "0.1.262",
3
+ "version": "0.1.264",
4
4
  "description": "Remy coding agent",
5
5
  "repository": {
6
6
  "type": "git",