@mindstudio-ai/remy 0.1.263 → 0.1.265

package/dist/prompt/skills/dataSources.md

@@ -30,7 +30,9 @@ short docs that fit in a prompt.
 - **Limits apply**: 25 data sources per app, 5,000 documents per source, 10,000 chunks per
   document, 300 searches/minute. Well clear of normal use — but **source names must be fixed, not
   computed per user or per request**, since referencing one creates it. Partition inside a source
-  with a metadata filter instead.
+  with document metadata instead: tag at add time
+  (`add(bytes, { filename, metadata: { userId } })`), narrow at search time
+  (`search(q, { filter: { metadata: { userId } } })`).
 
 ## Defining and searching
 
@@ -51,7 +53,22 @@ beside the answer. Retrieval is approximate; a user who can click through can ju
 An answer with no citation is an assertion.
 
 Created on first use, so searching a source the build hasn't populated returns no results rather than
-throwing. `search` options: `topK` (default 5, max 50), `scoreThreshold`, `rerank`, `hybrid`.
+throwing. `search` options: `topK` (default 5, max 50), `scoreThreshold`, `filter`, `mode`,
+`maxPerDocument`, `highlight`, `rerank`, `hybrid`.
+
+**Filtering** narrows a search before ranking, and every condition only narrows:
+`filter: { metadata: { department: 'legal', year: [2025, 2026] }, filename, documentIds,
+pages: { min?, max? }, contains: 'all these words', phrase: 'exact adjacent sequence' }`.
+Metadata is tagged at add time (scalars only, ≤16 keys); re-adding the same bytes with different
+metadata updates the tags in place, free. Filters are the right tool for scoping retrieval
+(per-user, per-category); they are NOT a substitute for a `db` query over structured data.
+
+**Modes**: `mode: 'hybrid'` (default) fuses semantic and keyword retrieval; `'semantic'` is the
+embedding alone; `'lexical'` is keyword-only with **no query embedding** — cheapest and fastest,
+right when the query is an identifier (an error code, a SKU, a name) rather than a meaning.
+`maxPerDocument: 2` stops one document monopolizing the results when the answer should draw on
+several. `highlight: true` adds `matches` (`{start, end}` offsets into `text`) for rendering
+highlighted excerpts.
 
 Search is deterministic for a fixed corpus and configuration, so eval sets and regression checks are
 meaningful — key them on `(documentId, chunkIndex)` rather than on chunk text.
@@ -69,7 +86,9 @@ set with the CLI, so code and reality can't drift.
 
 ```bash
 mindstudio-prod datasources add --source policies --wait docs/*.pdf
+mindstudio-prod datasources add --source policies --metadata department=legal,year=2026 contract.pdf
 mindstudio-prod datasources search --source policies "what are the payment terms?"   # sanity-check
+mindstudio-prod datasources search --source policies --filter department=legal --mode lexical "ERR-7741X"
 mindstudio-prod datasources delete --source policies   # whole source; --source is required, never defaulted
 ```
 
@@ -83,7 +102,11 @@ it needs no guard.
 Use the SDK's `add()` only when *users* upload documents that must become searchable:
 
 ```typescript
-await Policies.add(buffer, { filename: 'policy.pdf', contentType: 'application/pdf' });
+await Policies.add(buffer, {
+  filename: 'policy.pdf',
+  contentType: 'application/pdf',
+  metadata: { department: 'legal' },   // filterable at search time
+});
 const docs = await Policies.documents();   // 'processing' | 'done' | 'error'
 await Policies.remove(documentId);
 ```

package/dist/prompt/skills/voiceInterfaces.md

@@ -30,20 +30,35 @@ everywhere else, and the compiled system prompt must carry them explicitly:
 - **Spoken-form values.** "Forty-two fifty," not "$42.50". "Two fifteen in the afternoon," not
   "14:15". Read email addresses and confirmation codes character by character, and read them *back*
   for confirmation before acting on them — mishearing one digit of a phone number is the classic
-  voice failure.
+  voice failure. Collect one value per turn; two asked together blend when spoken.
 - **Brevity is a hard rule, not a style preference.** One to two sentences per turn, one question at
   a time. A paragraph that reads fine in chat is a monologue on a call.
+- **Vary the phrasing.** Repeated openers and acknowledgments sound convincing once and robotic by
+  the third turn — give the prompt an explicit variety rule, and treat any sample phrases as
+  anchors, never scripts.
 - **Handle unclear audio explicitly.** Give the prompt a rule for it: respond only to clear audio;
-  if it's noisy or ambiguous, ask the user to repeat — never guess, and never call a tool on input
-  the agent isn't sure it heard.
+  if it's noisy or ambiguous, ask the user to repeat — never guess, never call a tool on input the
+  agent isn't sure it heard, and don't reuse the same clarification line twice in a row.
 - **Pin the language.** State the response language in the prompt; don't let the model infer it from
-  an accent.
+  an accent. If the app's domain has brand names or terms with non-obvious pronunciations, give
+  them a line ("pronounce SQL as 'sequel'").
 
 Beyond the mechanics, the persona itself should be *of the ear*: pacing, warmth, how it handles
 being interrupted, what it says when it needs a second. This is the fun part, same as the agent
 interface — a distinct character beats a generic assistant, and voice makes character land harder
 than any other surface.
 
+### The shape of `system.md`
+
+Structure the compiled prompt as short **labeled sections** — Role & Objective, Personality & Tone,
+Rules, and (when the app has a real call flow) Conversation Flow — with bullets over paragraphs;
+realtime models find and follow sectioned rules far more reliably than prose. Scope rules
+precisely: "confirm before any tool that changes data," not "always confirm everything" — blanket
+`always`/`never` makes the agent rigid and unable to handle reasonable exceptions. And start
+minimal: state the role, the boundaries, and the voice mechanics above, then add rules only for
+behaviors that actually misfire in test calls (the transcripts in the call log are the feedback
+loop) rather than front-loading a policy manual.
+
 ### The latency classes
 
 Every tool in the spec declares one of three classes. This is the voice-specific discipline — get it
@@ -63,6 +78,25 @@ Classify by how the method actually behaves, not by what it is named. A "lookup"
 external service is `slow`. When in doubt between `fast` and `slow`, pick `slow` — a needless
 preamble is mildly chatty; an unexplained silence feels broken.
 
+### Forwarding results to the screen (`forwardResult`)
+
+A tool block may declare `forwardResult: true`. On completion, the platform then delivers the
+tool's raw return value to the session's browser on the SDK's `toolCall` event (`result` field) —
+so the UI can render what the agent just did (the citation it found, the record it pulled up, the
+booking it made) in lockstep with the spoken answer. No polling, no key-threading, no model
+involvement: the correlation is platform-guaranteed and scoped to that one session's client.
+
+Opt in deliberately, per tool. The forwarded payload is the method's raw return — the same data
+the model sees — so only enable it on tools whose returns are safe to render for the user in the
+call (no internal fields you wouldn't show on screen). Payloads over ~32KB serialized arrive as
+`resultTruncated: true` with no data — keep forwarded returns compact, or have the UI fetch big
+data itself. Failed calls never forward anything.
+
+For backend-side correlation (writing results to a table keyed by the call, custom channels), the
+method itself can read `session.voiceSessionId` / `session.visitorId` from the agent SDK
+(`import { session } from '@mindstudio-ai/agent'`) — the same id the browser holds as
+`session.sessionId`, guaranteed by the platform rather than echoed by the model.
+
 ### Tool descriptions say results out loud
 
 Follow the agent-interface principles for tool descriptions (when to use and when not, parameter
@@ -83,6 +117,11 @@ about to happen and get a yes. Anything destructive or financial — read the de
 piece by piece. In voice there is no confirmation dialog to lean on; the conversation *is* the
 confirmation UI.
 
+And give failure a script: never speak a raw error. When a lookup misses or a tool fails, read back
+the value it used ("I couldn't find an order ending three-one-two-five — did I get part of that
+wrong?"), offer one retry, then move to an alternate path — in character, without blaming the
+caller.
+
 ### Choosing the model
 
 Two shapes, one `model` field:
@@ -95,7 +134,8 @@ Two shapes, one `model` field:
   can be *any* chat model — the right choice when the app's reasoning demands a specific model, or
   when the agent interface already uses one and the voice should think identically. The blessed
   streaming pairing is `"stt": "deepgram-nova-3", "tts": "cartesia-sonic-3"` — the lowest-latency
-  combination the platform wires; prefer it unless there's a reason not to. One nuance: cascaded
+  combination the platform wires; prefer it unless there's a reason not to (ElevenLabs TTS,
+  `"tts": "elevenlabs-tts"`, is also wired when its voice library fits better). One nuance: cascaded
   engines speak the `greeting` verbatim (they have a real TTS); speech-to-speech engines have the
   model say it, so it may paraphrase slightly.
 
@@ -118,8 +158,8 @@ someone reading chat aloud.
 - Exposing the whole method surface. Voice is the most curated interface the app has.
 - A generic greeting ("Hello! How can I assist you today?"). The greeting is the first thing anyone
   hears; make it the character's.
-- Writing your own current-user placeholder — the platform appends a `## Current User` block (name,
-  roles) to every system prompt at runtime.
+- Writing your own current-user placeholder — the platform appends a `## Current User` block
+  (email, phone, roles) to every system prompt at runtime.
 
 ## Compiling the Voice Spec
 
@@ -164,7 +204,9 @@ session.on('stateChange', (state) => { });          // on() returns an unsubscri
 // far (never a delta) — render by upserting on segmentId, not appending.
 session.on('transcript', ({ role, segmentId, text, final }) => { });
 
-session.on('toolCall', ({ method, status }) => { }); // 'running' | 'done' | 'failed'
+// status: 'running' | 'done' | 'failed'. Tools declared with `forwardResult: true`
+// carry their return value in `result` on 'done' (or `resultTruncated: true` if >~32KB).
+session.on('toolCall', ({ method, status, result }) => { });
 session.on('error', (err) => { });
 
 session.mute(); session.unmute(); session.isMuted;
@@ -210,6 +252,117 @@ app's voice ("Booking your appointment…"), never raw names or JSON.
 - Rendering user-side captions as authoritative ("you said X") — they're recognition output.
 - Auto-starting a session on page load. Microphone access is always a deliberate user action.
 
+## Outbound calls (`voice.call`)
+
+The agent can call the user. Backend methods (and crons) place outbound phone calls with the
+agent SDK's `voice` namespace — the platform dials the number and connects the callee to this
+app's voice agent (same persona, engine, and tools as the web sessions):
+
+```ts
+import { voice, auth } from '@mindstudio-ai/agent';
+
+export async function callMeAboutMyOrder(input: { phone: string }) {
+  auth.requireRole('member');
+  const call = await voice.call({ to: input.phone, assumeIdentity: true });
+  return { calling: call.to, from: call.from };
+}
+```
+
+- **The method is the authorization gate.** The voice interface's `auth` block does not apply to
+  calls the backend places deliberately — gate the *method* with `auth.requireRole(...)` exactly
+  as you would any sensitive action.
+- **`assumeIdentity: true`** runs the call as the user who invoked the method: the agent knows
+  who it's talking to (Current User block) and every tool call carries their roles — regardless
+  of which number was dialed (the user types any number into a field; identity comes from their
+  session, not the phone). Omitted/false → anonymous call; role-gated tools decline.
+  System/cron invocations have no human identity and always run anonymously.
+- **Production needs a dedicated phone number.** The app owner attaches one ($1/month) via the
+  dashboard or `mindstudio-prod voice numbers` (see "Managing the phone side from the CLI"
+  below) — it becomes the caller ID for every call, in dev sessions too, so users always see
+  the same number. Without one, deployed calls throw `phone_out_requires_dedicated_number`, and
+  dev sessions fall back to a shared platform test number that varies per call (tighter limits
+  apply on the shared pool).
+- **Outcome is on the call record**, not the return value: `voice.call` returns as soon as
+  dialing starts (`{ sessionId, status: 'dialing', from, to }`); answered/busy/no-answer land on
+  the session in the app's call log (`voice.listSessions()` / the dashboard).
+- **Limits**: the app's concurrent-session policy, a daily outbound-call cap, a per-call
+  duration ceiling, and one active call per callee number (`voice_callee_busy`).
+- **Compliance**: automated calls require prior consent. Call your own users who opted in to
+  calls from this app, honor reasonable calling hours, never dial purchased or cold lists —
+  design the consent moment into the product (a "call me" button IS consent; a scraped list is
+  not).
+
+## Inbound calls
+
+Once the app has a dedicated phone number, people can call it — the same voice agent answers
+(same persona, engine, and tools). Nothing extra to author for the basic case; the number in the
+app's settings is the whole switch.
+
+How answering works:
+
+- **Inbound always runs the live release.** There is no dev inbound — test the agent over the
+  normal WebRTC session in the editor; the phone is the same interface with a different
+  transport. An app with no live voice interface (or at its concurrency limit) doesn't answer.
+- **Callers are anonymous until verified.** The `auth` block still applies, but a phone call
+  can't show a login page — so the platform answers first, and `requireUser` becomes an
+  in-call verification flow. The agent can serve whatever anonymous callers are allowed, and
+  offers verification when the caller wants something account-bound.
+- **Verification uses the app's own auth methods** (`sms-code` / `email-code` from the
+  manifest), existing accounts only — there is no sign-up over the phone:
+  - SMS: a code is texted to the number the caller is calling from, if an account has that
+    number on file. No other number is possible by design.
+  - Email: the caller says their address; the platform matches it against the app's users
+    (transcription-tolerant — no letter-by-letter spelling ceremony) and emails the account's
+    stored address a code.
+  - The flow never confirms or denies that an account exists — a code is "sent if an account
+    matches", always phrased that neutrally. The persona should offer verification naturally
+    when it unlocks something, never as a robotic gate.
+- **Verified mid-call, upgraded mid-call**: once the code checks out, the session becomes that
+  user's — Current User block, roles on every tool call — without redialing.
+
+### `phone.trustCallerId`
+
+For apps whose users are known by phone number, the interface config may opt into treating
+caller ID as identity:
+
+```json
+"phone": { "trustCallerId": true }
+```
+
+A caller whose number exactly matches an app user's phone starts the call already verified —
+no code. This is a real security tradeoff: **caller ID can be spoofed**, so a motivated
+attacker who knows a user's phone number can impersonate them to this agent. Before enabling
+it, you MUST surface that risk to the user and get their explicit confirmation — it's the
+right call for convenience-first, low-stakes apps (a family assistant, a status line), and the
+wrong one wherever the agent's tools can move money, reveal sensitive records, or take
+destructive actions. It lives in the interface config deliberately: enabling it is a code
+change, visible in review and auditable via deploys, not a dashboard toggle.
+
+## Managing the phone side from the CLI
+
+The `mindstudio-prod voice` family covers numbers, the call log, and voice policy:
+
+```bash
+mindstudio-prod voice numbers search --area-code 310   # available numbers to offer the user
+mindstudio-prod voice numbers buy +13105551234         # buy + attach ($1/month — see below)
+mindstudio-prod voice numbers release +13105551234     # permanent; no refund, ~15-day quarantine
+mindstudio-prod voice sessions list --limit 10         # call log: web / phone-out / phone-in
+mindstudio-prod voice sessions get <sessionId>         # full transcript + cost breakdown
+```
+
+Also `voice numbers list`, `voice numbers set-name` (outbound caller-ID display
+name; 12-72h carrier propagation), `voice settings get`/`set` (concurrency, per-visitor,
+max duration — `set` merges: only the settings you pass change). `--help` for flags.
+
+**Never buy a number without the user's explicit confirmation** — it starts a recurring
+$1/month workspace charge. Search first, present the options with the price, and only run
+`numbers buy` after they've picked one and said yes.
+
+Transcripts are how you iterate on a voice persona: after the user test-calls the agent, read
+`voice sessions get` for what was actually said — misheard input, interruptions, tools declining
+— and fix the spec from evidence rather than guesses. (Dev-session test calls carry a
+`devSessionId` in the list, so you can tell them from live traffic.)
+
 ---
 
 # The wiring
@@ -236,8 +389,7 @@ Frontmatter fields:
 - `description` — one-liner for listings
 - `model` — JSON string, two shapes: native speech-to-speech `{"model": <realtime model id>,
   "voice": <voice id>}`, or cascaded `{"llm": <chat model id>, "stt": <transcription model id>,
-  "tts": <speech model id>, "voice": <voice id>}`. Optional `config` for model-specific settings.
-  Ids via `askMindStudioSdk`.
+  "tts": <speech model id>, "voice": <voice id>}`. Ids via `askMindStudioSdk`.
 - `turnDetection` — optional; `{"eagerness": "low" | "medium" | "high"}` — how quickly the platform
   decides the user finished speaking. High is snappier; low is more patient (users dictating
   numbers or addresses). Default `medium`.
@@ -307,6 +459,8 @@ The top-level key must match the interface type (`voice`):
 | `greeting` | Optional spoken opener |
 | `systemPrompt` | Relative path to the compiled system prompt |
 | `auth` | **Required.** Who may start a session: `{ "requireUser": boolean, "requireRole"?: string[] }`. See the Auth section below |
+| `phone` | Optional telephony options: `{ "trustCallerId"?: boolean }` — see "Inbound calls" above. Only add it after the user has confirmed the spoofing tradeoff |
+| `context` | Optional session context: `{ "method": <method id> }` — auto-fired in the background at session start; see "Session context" below |
 | `tools` | `{ method, latency, description }` — method `id` from the manifest, a latency class, and a relative path to the tool's markdown |
 | `webInterfacePath` | Optional. Where the voice layer lives in the web interface, for the editor preview |
 
@@ -316,14 +470,38 @@ Declare it in `mindstudio.json`:
 { "type": "voice", "path": "dist/interfaces/voice/interface.json" }
 ```
 
+## Session context (auto-loaded)
+
+When the config declares `"context": { "method": "session-context" }`, the platform fires that
+backend method automatically when a session starts — in the background, so the greeting is
+never delayed — and appends its return to the system prompt as a `## Session Context` block.
+Use it for situational state that should color every turn: the caller's open orders, account
+standing, where they left off. Timing: the method runs while the greeting audio plays, so the
+agent has the context by roughly the first exchange and is guaranteed to have it shortly
+after — it is NOT guaranteed for the literal first utterance. On inbound phone calls it
+re-fires after the caller verifies mid-call, so the context recomputes for the now-known user.
+
+The method contract:
+- Runs as the session's user (same identity/RBAC as a tool call); anonymous sessions run it
+  anonymously — return generic or empty content for them.
+- Return a short markdown **string** (a few lines). Results are capped at 4,000 characters;
+  keep it situational context, not documents — deep or on-demand data belongs in tools or
+  data sources.
+- It is never a model-visible tool, and failures degrade silently to the generic prompt —
+  never make correctness depend on it.
+
+Rule of thumb: `context` for always-relevant state the agent should just know; tools for
+anything looked up on demand. Identity itself (name, roles) is already injected via the
+Current User block — don't re-fetch it in the context method.
+
 ## Platform Behavior
 
 - Input schemas are derived from each method's contract — never hand-written.
-- The platform appends a `## Current User` block (name, roles) to the system prompt at runtime;
-  never author a placeholder for it.
+- The platform appends a `## Current User` block (email, phone, roles) to the system prompt at
+  runtime; never author a placeholder for it.
 - Turn detection, barge-in (interruption truncates the agent's context to the audio the user
   actually heard), and background-noise handling are platform-managed; `turnDetection.eagerness` is
-  the only knob.
+  the only knob (not yet wired on Gemini realtime engines — it's a no-op there).
 - Sessions have a per-app concurrency limit and a maximum duration, both configurable in the app's
   settings; an idle session is ended gracefully after a prompt. Voice minutes and model usage are
   metered.
@@ -347,6 +525,8 @@ creation itself:
   (OR semantics, same as the backend `auth.requireRole(...)`). Omit or leave empty for no role
   gate. Requires `requireUser: true`. Unknown role ids fail the build.
 - Denials reject `startSession()` with code `auth_required` (401) or `role_required` (403).
+- On the phone channel there is no login page to bounce to, so `requireUser` becomes
+  answer-then-verify — see "Inbound calls" above.
 - Dev preview is exempt — the builder is never locked out while testing.
 - Older compiled apps without the block fall back to the manifest's `auth.enabled` (auth-enabled →
   users only; no auth → public). New configs always declare it explicitly.

package/dist/prompt/static/coding.md

@@ -81,6 +81,6 @@ You have access to the `mindstudio` CLI, which exposes every SDK action as a com
 ### Production App Management
 You have access to `mindstudio-prod`, a CLI for managing the user's production app. Use it via your bash tool. All output is JSON. Run `mindstudio-prod --help` or `mindstudio-prod <command> --help` to discover usage and available options.
 
-Available commands: `requests` (server-side request logs, error rates, latency), `crashes` (frontend browser errors — grouped issues + drill-down to individual events), `analytics` (traffic, top pages/referrers/geo, AI-referral attribution, live counters), `releases` (deploy status, history), `domains` (custom subdomains and fully custom domains), `users` (list, set roles), `db` (query production sql db), `data` (live db operations like lift-from-dev), `methods` (list, invoke), `secrets` (list, get, set, delete), `issues` (reference and manage externally-reported bugs and issues. Do not use this to track work you are doing with the user - only read from it and resolve issues if the user asks for help fixing a bug from the issues tracker).
+Available commands: `requests` (server-side request logs, error rates, latency), `crashes` (frontend browser errors — grouped issues + drill-down to individual events), `analytics` (traffic, top pages/referrers/geo, AI-referral attribution, live counters), `releases` (deploy status, history), `diagnostics` (post-deploy Lighthouse audit), `domains` (custom subdomains and fully custom domains), `users` (list, set roles), `db` (query production sql db), `data` (live db operations like lift-from-dev), `methods` (list, invoke), `secrets` (list, get, set, delete), `files` (upload + manage CDN files), `datasources` (build and query document corpora), `prerender` (bot/crawler snapshots), `voice` (dedicated phone numbers: search/buy/release — buying bills $1/month, so never buy without the user's explicit confirmation; call log + transcripts; voice policy settings), `issues` (reference and manage externally-reported bugs and issues. Do not use this to track work you are doing with the user - only read from it and resolve issues if the user asks for help fixing a bug from the issues tracker).
 
 Use when the user asks about production behavior (server errors via `requests`, browser crashes via `crashes`, traffic/engagement via `analytics`), wants to manage their live app (domains, users, roles), needs to seed or query production data, or wants to check release status.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mindstudio-ai/remy",
-  "version": "0.1.263",
+  "version": "0.1.265",
   "description": "Remy coding agent",
   "repository": {
     "type": "git",