@mindstudio-ai/remy 0.1.256 → 0.1.258

package/dist/prompt/compiled/scenarios.md

@@ -104,6 +104,22 @@ This is deterministic — same scenario always produces the same state.
 
 Scenarios are useful for seeding initial app state after build for testing, as well as to give the user a first impression of an app that is already filled with data and looks and feels usable. The user can choose to run further scenarios after initial build by clicking the Scenarios tab and selecting a scenario to run.
 
+## What scenarios don't touch
+
+**Scenarios seed database tables and nothing else.** They do not touch file stores or data sources —
+deliberately: both are durable and shared across dev and prod, with no per-release copy to reset, so
+there is nothing to truncate.
+
+Don't try to seed documents into a data source from a scenario, and don't write `clear()`-style reset
+helpers for one. Load a test corpus once from the CLI instead:
+
+```bash
+mindstudio-prod datasources add --source policies --wait fixtures/*.pdf
+```
+
+Re-running it is free (documents are content-addressed), so it's safe to keep in a setup script
+beside your scenarios.
+
 ## Scenario Data
 
 Align scenario data to the vibe of the app - construct data that feels like it fits.

package/dist/prompt/compiled/sdk-actions.md

@@ -83,7 +83,6 @@ result.$billingCost;         // cost in credits (if applicable)
 
 | Action | What it does |
 |--------|-------------|
-| `uploadFile` | Upload a file to CDN |
 | `downloadVideo` | Download a video URL |
 | `getMediaMetadata` | Get dimensions, duration, etc. |
 | `convertPdfToImages` | PDF pages to PNG images |
@@ -154,4 +153,4 @@ Consider the ways in which AI can be incorporated into backend methods to solve
 
 ### Task Agents
 
-For multi-step tasks where the model needs to autonomously compose actions (research + scrape + generate, enrichment pipelines, content creation), use `runTask()` instead of chaining actions manually. It runs an agent loop and returns structured JSON. Its tools can include SDK actions as well as your app's own methods, so the agent can read your data to decide what to do next and write results back itself. See the task agents reference for full details.
+For multi-step tasks where the model needs to autonomously compose actions (research + scrape + generate, enrichment pipelines, content creation), use `runTask()` instead of chaining actions manually. It runs an agent loop and returns structured JSON. Its tools can include SDK actions as well as your app's own methods, so the agent can read your data to decide what to do next and write results back itself. Load the `taskAgents` skill before writing one — it is the full reference.

package/dist/prompt/{compiled/agent-interfaces.md → skills/agentInterfaces.md}

@@ -1,3 +1,9 @@
+---
+name: Agent Interfaces
+what: Conversational AI as a first-class interface to the app — an LLM with authenticated, per-user access to the app's methods as tools, paired with a streaming chat UI. The platform handles auth, tool dispatch, threads, and streaming, so the work is authorship: who the agent is, which methods it can reach, and how each one is described to it. Any app whose methods do something interesting can be projected into a conversation this way, often as its most compelling surface. This reference covers the whole feature — writing the agent spec, compiling it, and building the chat frontend.
+when: Before authoring `src/interfaces/agent.md`, compiling `dist/interfaces/agent/`, or building an agent's chat UI.
+---
+
 # Building Agent Interfaces
 
 Guidance for designing conversational AI agents and their frontends. An agent interface pairs an LLM (with per-user-scoped/authenticated access to app methods as tools, handled by platform automatically) with a chat UI. The developer authors the agent's character in MSFM (`src/interfaces/agent.md`); you compile it into a system prompt and tool descriptions (`dist/interfaces/agent/`).
@@ -10,21 +16,32 @@ A good system prompt establishes who the agent is — personality, tone, judgmen
 
 Short and opinionated beats long and comprehensive. "Sounds like a sharp, organized friend — brief by default" gives the model more to work with than a page of behavioral rules. Define constraints through character, not checklists. Let the model's judgment work.
 
-#### System Prompt Specifics
-Always include a note like "## Tool Usage
-- When multiple tool calls are independent, make them all in a single turn. Searching for three different products, or fetching two reference sites: batch them instead of doing one per turn." to help the model know it can run tools in parallel
-- The user's name and current role(s) at the time of message, if any, will be automatically appended to the end of every system prompt at runtime like:
+Three things every compiled system prompt should carry, on top of the character:
+
+**Parallel tool use.** The model won't batch independent calls unless told it can. Include a section like:
 
+```markdown
+## Tool Usage
+
+When multiple tool calls are independent, make them all in a single turn.
+Searching for three different products, or fetching two reference sites:
+batch them instead of doing one per turn.
 ```
+
+**Markdown and house style.** Unless the user says otherwise, tell the agent it can use markdown (the chat UI renders it) and to avoid em dashes and emojis.
+
+**The current user is appended for you.** At runtime the platform appends the user's name and roles to the end of every system prompt, so don't write your own placeholder for it:
+
+```markdown
 ## Current User
+
 Name: Jane Smith
 Roles: editor
 ```
-- Unless the user specifies otherwise, always include a note that the agent can use markdown in responses (since the chat UI renders it) and should avoid using em dashes and emojis in its responses.
 
 ### Tool descriptions are the most important artifact
 
-The system prompt says *who* the agent is. The tool descriptions say *what it can do*. A great tool description means the agent uses the tool correctly without explicit instruction. Do not be overly precise or micromanage. Your goal with tool descriptions is to provide context and faming- trust that the model is intelligent enough to fill in the gaps.. Each `tools/*.md` file should cover:
+The system prompt says *who* the agent is. The tool descriptions say *what it can do*. A great tool description means the agent uses the tool correctly without explicit instruction. Do not be overly precise or micromanage. Your goal with tool descriptions is to provide context and framing — trust that the model is intelligent enough to fill in the gaps. Each `tools/*.md` file should cover:
 
 - **When to use** this tool (and when NOT to — e.g. "NOT for marking complete, use toggle-todo")
 - **Parameter guidance** beyond the schema — what makes a good value, when to include optional fields, what to skip
@@ -64,6 +81,8 @@ When building the `dist/interfaces/agent/`, consider the agent spec, as well as
 
 **`agent.json`** — ties it together. Model config from frontmatter, paths to system prompt and tool files, optional `webInterfacePath`.
 
+The exact shape of all three, and of the spec frontmatter they compile from, is in "The wiring" at the end of this document.
+
 ## Chat UI Design
 
 When the agent has a web frontend (via `webInterfacePath`), the chat UI is a page within the web interface.
@@ -116,10 +135,12 @@ response.abort();
 
 **Attachments:**
 
-Send images or documents alongside a message. Upload via `platform.uploadFile()` first, then pass CDN URLs as the 4th argument:
+Send images or documents alongside a message. Upload to the app's file store first (see Files & Storage), then pass the returned URLs as the 4th argument:
 
 ```ts
-const url = await platform.uploadFile(file);
+// backend method mints a token; the browser uploads straight to storage
+const token = await api.getUploadSlot({ filename: file.name, contentType: file.type });
+const { url } = await platform.upload(token, file);
 
 chat.sendMessage(threadId, "What's in this document?", {
   onText: (delta) => setText((prev) => prev + delta),
@@ -128,7 +149,7 @@ chat.sendMessage(threadId, "What's in this document?", {
 });
 ```
 
-Images (`i.mscdn.ai`) are sent as vision input. Documents (`f.mscdn.ai`) have text extracted server-side and included in context. Attachments are preserved in thread history.
+Images are sent as vision input; documents have their text extracted server-side and included in context. Attachments are preserved in thread history.
 
 **Key points:**
 - `onText` and `onThinking` receive deltas (append to state, don't replace)
@@ -174,7 +195,7 @@ The first screen should invite conversation. A greeting from the agent, a few su
 
 ### Mobile
 
-Chat is inherently mobile-friendly — lean into it. Pay attention to viewport sizing on mobile as the virtual keyboard changes the available height. 
+Chat is inherently mobile-friendly — lean into it. Pay attention to viewport sizing on mobile as the virtual keyboard changes the available height.
 
 ### Respect the brand
 
@@ -184,3 +205,90 @@ The chat UI uses the app's design system — colors, typography, voice from `@br
 
 - Avoid designs that look like dated messaging apps from 2015
 - Avoid robotic empty states ("Hello! I'm your AI assistant. How can I help you today?")
+
+---
+
+# The wiring
+
+## Spec: `src/interfaces/agent.md`
+
+The human-readable spec. Frontmatter contains structured fields; the prose body is the behavioral spec —
+voice, personality, capabilities, rules — written in MSFM.
+
+```yaml
+---
+name: Todo Assistant
+model: {"model": "claude-4-5-haiku", "temperature": 0.5, "maxResponseTokens": 16000}
+description: Conversational agent that helps users manage their to-do list.
+---
+```
+
+Frontmatter fields:
+
+- `name` — agent display name
+- `model` — JSON string with `model` (MindStudio model ID), `temperature`, `maxResponseTokens`, and
+  optional `config` (model-specific settings like `reasoning`, `tools`, etc.). Ask `askMindStudioSdk`
+  for available model IDs and their config options — MindStudio's ids don't match vendor ids, so treat
+  any id in this document's examples as illustrative rather than current. The user's UI has a visual picker for changing it later, so only validate
+  the model when you're setting it; if the value changes afterwards, assume it's correct.
+- `description` — one-liner for agent card/listing
+
+The prose body contains sections like Voice & Personality, Capabilities, Behavior — whatever structure
+serves the agent's character. This is compiled into the system prompt and tool descriptions.
+
+## Compiled Output: `dist/interfaces/agent/`
+
+```
+dist/interfaces/agent/
+├── agent.json          ← config the platform reads
+├── system.md           ← compiled system prompt
+└── tools/
+    ├── createTodo.md   ← rich tool description per method
+    ├── listTodos.md
+    └── ...
+```
+
+## Config (`agent.json`)
+
+```json
+{
+  "agent": {
+    "model": "claude-4-5-haiku",
+    "temperature": 0.5,
+    "maxTokens": 16000,
+    "systemPrompt": "system.md",
+    "tools": [
+      { "method": "create-todo", "description": "tools/createTodo.md" },
+      { "method": "list-todos", "description": "tools/listTodos.md" }
+    ],
+    "webInterfacePath": "/chat"
+  }
+}
+```
+
+**The token-limit field is renamed during compilation.** The spec frontmatter calls it
+`maxResponseTokens`; the compiled `agent.json` calls it `maxTokens`. Same value, two names — carry it
+across rather than copying the key.
+
+| Field | Description |
+|-------|-------------|
+| `model` | MindStudio model ID. Comes from the spec frontmatter; look it up with `askMindStudioSdk` rather than guessing |
+| `temperature` | Model temperature |
+| `maxTokens` | Max response tokens (the spec's `maxResponseTokens`) |
+| `systemPrompt` | Relative path to the compiled system prompt markdown file |
+| `tools` | Array of tool entries — `method` references a method `id` from the manifest, `description` is a relative path to a markdown file with rich tool docs (when to use, examples, edge cases, parameter guidance) |
+| `webInterfacePath` | Optional. If the app has a web interface with a chat page, this path tells the IDE where to show the preview. Otherwise the agent is accessed via API. |
+
+Declare it in `mindstudio.json`:
+
+```json
+{ "type": "agent", "path": "dist/interfaces/agent/agent.json" }
+```
+
+## Auth
+
+Agent chat runs as the **authenticated user**, not as a system role — tool calls carry that user's
+roles, so a method gated with `auth.requireRole` behaves exactly as it would if the user had called it
+from the web frontend. That's what makes exposing real methods safe; it's also why role restrictions
+belong in the tool descriptions, so the agent can decline gracefully instead of surfacing a rejection.
+

package/dist/prompt/skills/dataSources.md

@@ -0,0 +1,131 @@
+---
+name: Data Sources
+what: A managed retrieval system for document corpora, not a bolt-on keyword search. Documents are chunked and embedded, candidate hits are re-scored by a reranking model, semantic search runs alongside exact keyword matching so part numbers and error codes still land, images inside documents are described by a vision model and made searchable, and every hit returns a citation that links to its source page. Chunking and embedding settings are versioned — a rebuilt corpus can be compared against the live one and promoted without downtime. All of that applies to unstructured documents queried by meaning, and to nothing else.
+when: Only for unstructured documents queried by meaning — "find the clause about early termination". Structured data belongs in `db`: if the question can be expressed as a filter, it is not a search problem, and a `WHERE` clause is faster, cheaper and exact. Load before defining or querying a data source.
+---
+
+# Data Sources (Search Over Documents)
+
+Per-app searchable document corpora: upload documents, ask in plain language, get back the passages
+that answer it with a citation to the source.
+
+**Most apps should not use one — check this before reaching for it.** Structured data (rows with
+fields you filter on) belongs in `db`; a `WHERE` clause is faster, cheaper and exact. A data source
+earns its cost only for **unstructured documents queried by meaning**. "Find the clause about early
+termination" is a data source. "Find contracts signed after March" is a `db` query. If the question
+can be expressed as a filter, it is not a search problem.
+
+Don't use one when: the data is structured (`db`), you only need to store files (`files` — nobody is
+searching the contents), the requirement is exact lookup by identifier, or the corpus is a handful of
+short docs that fit in a prompt.
+
+## Behaviour (read before the API)
+
+- **One corpus shared across dev and prod** — like a file store, not a table. No dev copy, no
+  per-release isolation. A document added while building is already live.
+- **Scenarios never reset a data source.** Don't write `clear()`-style reset helpers.
+- **Re-adding the same bytes is free** — content-addressed, so ingest scripts are safe to re-run.
+- **Ingest is async.** `add()` returns once queued; poll `documents()`, or use `--wait` from the CLI.
+- **Reprocessing costs real money**, so changing how a corpus is built is always explicit.
+- **Limits apply**: 25 data sources per app, 5,000 documents per source, 10,000 chunks per
+  document, 300 searches/minute. Well clear of normal use — but **source names must be fixed, not
+  computed per user or per request**, since referencing one creates it. Partition inside a source
+  with a metadata filter instead.
+
+## Defining and searching
+
+```typescript
+import { dataSources } from '@mindstudio-ai/agent';
+export const Policies = dataSources.defineDataSource('policies');   // lowercase [a-z0-9_-], ≤64
+
+const { results } = await Policies.search('what are the payment terms?', { topK: 5 });
+const context = results.map((r) => r.text).join('\n\n');
+```
+
+Hits are `{ score, text, citation }` with
+`citation: { documentId, filename, pageNumber, chunkIndex, headingPath, boundingBox?, url }`, plus
+`retrievalRank`/`retrievalScore` — the position before reranking, so you can show what reranking did.
+
+**Always render the citation.** `citation.url` is a stable on-domain link — put it in an `<a href>`
+beside the answer. Retrieval is approximate; a user who can click through can judge for themselves.
+An answer with no citation is an assertion.
+
+Created on first use, so searching a source the build hasn't populated returns no results rather than
+throwing. `search` options: `topK` (default 5, max 50), `scoreThreshold`, `rerank`, `hybrid`.
+
+Search is deterministic for a fixed corpus and configuration, so eval sets and regression checks are
+meaningful — key them on `(documentId, chunkIndex)` rather than on chunk text.
+
+**Debugging retrieval.** Two opt-in options, neither of which changes the results or their order:
+`explain: true` adds `explain.{dense, lexical, matchedVia}` (which half of hybrid found each hit;
+costs two extra round trips), and `expand: 1` adds `neighbors.{before, after}` for surrounding
+context. When a document never comes back at all, `Policies.stats()` reports the config actually in
+effect and `Policies.chunks(documentId)` shows exactly how it was split.
+
+**Configuration is not declared in code** — chunking and embedding settings live on the corpus and are
+set with the CLI, so code and reality can't drift.
+
+## Loading documents — normally at build time, from the CLI
+
+```bash
+mindstudio-prod datasources add --source policies --wait docs/*.pdf
+mindstudio-prod datasources search --source policies "what are the payment terms?"   # sanity-check
+mindstudio-prod datasources delete --source policies   # whole source; --source is required, never defaulted
+```
+
+`--wait` blocks until processing finishes and exits non-zero on failure. Also `datasources list`,
+`status` (per-document state + ingest errors), `rm --document <id>`. `--help` for flags.
+
+**Seeding a test corpus:** scenarios don't touch data sources, so load fixtures with the same command
+in a setup script — `datasources add --source <slug> --wait fixtures/*.pdf`. Re-running is free, so
+it needs no guard.
+
+Use the SDK's `add()` only when *users* upload documents that must become searchable:
+
+```typescript
+await Policies.add(buffer, { filename: 'policy.pdf', contentType: 'application/pdf' });
+const docs = await Policies.documents();   // 'processing' | 'done' | 'error'
+await Policies.remove(documentId);
+```
+
+Formats: pdf, docx, pptx, xlsx, odt, rtf, epub, images, txt, md, json, csv, tsv, log, html.
+
+## Answering from results
+
+Retrieve → join passages as context → have a model answer *from that context* → render citations.
+Never paste raw chunks at the user; they're fragments. For agentic flows, give the model `search` as a
+tool so it can query repeatedly and refine, rather than retrieving once up front.
+
+## Tuning — two kinds of setting
+
+| Kind | Settings | Cost |
+|---|---|---|
+| **Free** (ranking) | `--rerank`, `--rerank-model`, `--hybrid`, `--top-k` | none, next search |
+| **Rebuild** (how docs become vectors) | `--max-chars`, `--min-chars`, `--drop-blocks`, `--contextual`, `--describe-images`, `--embedding-model`, `--extraction-model` | every document reprocessed |
+
+Images inside documents are described by a vision model and the description substituted into the
+searchable text (`--describe-images`, on by default) — without it a chart contributes nothing to
+search at all. Documents with no images cost nothing.
+
+`rerank` and `hybrid` default on and are usually right — reranking is the biggest quality lever, and
+hybrid is what finds part numbers, error codes and proper nouns a semantic model never learned. Both
+are also per-query (`search(q, { rerank: false })`) for a latency-sensitive path.
+
+```bash
+mindstudio-prod datasources config --source policies              # show
+mindstudio-prod datasources config --source policies --top-k 8    # free, immediate
+```
+
+**A rebuild-class change on a populated corpus is rejected** — you're told what it would invalidate
+and what it costs. To make it, build a new version alongside the live one:
+
+```bash
+mindstudio-prod datasources revectorize --source policies --max-chars 900 --wait
+mindstudio-prod datasources search --source policies --candidate "payment terms"   # compare
+mindstudio-prod datasources promote --source policies                              # go live
+```
+
+Search serves the current version throughout, so nothing degrades while the new one builds.
+`datasources drop` discards an unwanted candidate.
+
+For anything deeper on the SDK, ask `askMindStudioSdk` rather than guessing at an API.

package/dist/prompt/skills/inboundEmail.md

@@ -0,0 +1,116 @@
+---
+name: Inbound Email
+what: The app has its own email address, and mail sent to it runs a method. Every address on the app's subdomain routes to one handler, so `support@`, `receipts@` and `anything@` all arrive without registering anything — branch on the recipient in code. Attachments arrive as CDN URLs the platform has already uploaded, threading headers come through intact so replies land in the same conversation, and a verified custom domain both receives and sends under the app's own brand. "Forward a receipt and the app files it" is a real feature that costs one method.
+when: Before writing an email-handler method, adding an `email` interface, or promising anything about what happens when a user emails the app.
+---
+
+# Inbound Email Interfaces
+
+Inbound email triggers. Each app has **one** email-handler method; the platform routes all inbound
+mail destined for the app — across any of its address tiers — to that method.
+
+The addresses themselves are configured at the project level by the user through the Remy platform.
+Your job is the `interface.json` and the method that handles the mail, not domain registration or MX
+records.
+
+## Address tiers
+
+Three tiers, all delivered to the same handler method. The new tiers are catchall (no localpart
+registration); the legacy tier is specific-localpart and frozen for new apps.
+
+| Tier | Address | How it's set up |
+|---|---|---|
+| Platform subdomain (default) | `*@<custom_subdomain>.madewithremy.com` | Automatic the moment the app has a `custom_subdomain` set. Every address on that subdomain delivers to the handler. |
+| Custom domain | `*@<their-domain>` | The user adds a domain in the dashboard's email-domains settings and points one MX record at `mx.msagent.ai`. Not something the agent provisions. |
+| Legacy `mindstudio-hooks.com` | `<name>@mindstudio-hooks.com` | Existing apps only — frozen for new apps. Don't recommend it; treat as read-only history. |
+
+Because the new tiers are catchall, `to` carries an arbitrary localpart. Methods that need to branch on
+it should read `input.to` (e.g. `if (input.to.startsWith('support@')) ...`). This is what makes
+per-purpose addresses free: you don't register `support@` anywhere, you just check for it.
+
+A verified custom domain (and the app's `madewithremy.com` subdomain) also **sends** outbound mail, not
+just receives — `sendEmail` picks the app's own-brand sender automatically, configured in the
+dashboard's **Email** settings.
+
+## Config (`interface.json`)
+
+The top-level key must match the interface type (`email`):
+
+```json
+{
+  "email": {
+    "method": "handle-inbound-email",
+    "approvedSenders": ["billing@vendor.com", "*@trusted-partner.com"]
+  }
+}
+```
+
+`approvedSenders` is optional. When set, only senders matching an exact address or `*@domain.com`
+wildcard reach the method; everything else is rejected by the platform with `400 invalid_sender` before
+the method runs (silently — the sender isn't bounced). Matching is case-insensitive. The same list
+applies uniformly across all three address tiers.
+
+Declare it in `mindstudio.json`:
+
+```json
+{ "type": "email", "path": "dist/interfaces/email/interface.json" }
+```
+
+## Input shape
+
+```ts
+{
+  to: string;               // full recipient address; localpart is arbitrary on catchall tiers
+  from: string;             // bare sender address, extracted from "Name <a@b>" form
+  fromName: string | null;  // sender display name, or null
+  subject: string;          // 'No Subject' if missing
+  message: string;          // plain-text body, falls back to HTML if text is missing; 'No Body' if neither was sent
+  html: string;             // HTML body, or '' when text-only
+  attachments: string[];    // CDN URLs — already uploaded by the platform
+  messageId: string | null; // this email's Message-ID, angle-bracketed (<id@host>)
+  inReplyTo: string | null; // Message-ID this email is replying to, if any
+  references: string[];     // prior Message-IDs in the thread (angle-bracketed); [] if none
+  replyTo: string | null;   // Reply-To address — reply here, not `from`, when set
+  cc: string[];             // Cc recipient addresses
+  date: string | null;      // original send time, ISO-8601
+}
+```
+
+## Replying in thread
+
+Replies go out through the SDK's `sendEmail` action (its full parameter list is in the SDK actions
+reference in your system prompt). Set `inReplyTo` to the incoming `messageId` and `references` to
+`[...references, messageId]`. Send to `replyTo` when it's set, otherwise `from`.
+
+```typescript
+await mindstudio.sendEmail({
+  to: input.replyTo ?? input.from,
+  subject: `Re: ${input.subject}`,
+  body: reply,
+  inReplyTo: input.messageId ?? undefined,
+  references: input.messageId
+    ? [...input.references, input.messageId]
+    : input.references,
+  cc: input.cc, // reply-all
+});
+```
+
+`sendEmail` returns `{ recipients, cc, bcc, from }` — who it sent to and the sender used. It does
+**not** return the sent message's own `Message-ID`, so thread off *inbound* mail, never off messages
+you sent.
+
+## Attachments and size limits
+
+`attachments[]` is an array of CDN URLs — the platform has already received and uploaded the files.
+Fetch them server-side via the URL when you need the bytes; pass them through as URLs to UI or
+downstream services.
+
+Max inbound message size is 25 MB total (including all attachments). Oversized messages are rejected by
+the platform before the method runs.
+
+## Auth
+
+Methods invoked through this interface run with `auth.roles: ['system']` — the platform is calling, not
+a user session, so there's no user to impersonate. Use `auth.requireRole('system')` to gate methods that
+should only be reachable via email. The auth reference in your system prompt covers the system role in
+full.