@mindstudio-ai/remy 0.1.265 → 0.1.266

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -265,6 +265,8 @@ The headless IPC protocol uses request correlation and a unified response patter
265
265
  - System events (lifecycle, shutdown) never have a `requestId`
266
266
  - Every command ends with exactly one `completed` event: `{event:"completed", requestId, success, error?}`
267
267
  - Messages sent while a turn is running are queued. When the turn ends, all contiguous queued user messages and background results are delivered together as **one merged turn**: the first queued message's `requestId` becomes the turn's primary id (stamped on `turn_started` and all streaming events), each absorbed message echoes its own `user_message` with its original `requestId` and `queued: true`, and at turn end the primary `completed` is emitted first, followed immediately by one `completed {…same outcome, absorbed: true}` per other absorbed `requestId`. Automated-action (`@@automated::…@@`) messages and chain steps never merge — they always run one turn each.
268
+ - A queued **user** message can be promoted to ASAP delivery via `setQueuedDelivery`. ASAP items are pulled into the **running** turn at its next tool boundary (injected as plain user messages — no abort, no restart), echo `user_message` with `queued: true` and their own `requestId`, and get a `{…, absorbed: true}` completed with the turn's outcome at turn end. Promotion deliberately jumps ahead of anything else in the queue, including chain steps. If the turn ends before injection, the tag is ignored and the item drains in normal FIFO order.
269
+ - Background tool completions have two delivery classes (per-tool `backgroundNotify` on the tool definition). `wake` (default): the result is queued as a `background_results` message and may start a turn when the agent is idle. `passive` (e.g. `specSync`): the result never enters the queue and never wakes the agent — it parks in a persisted holding pen and rides the next real turn as a hidden `background_results` entry (so it never appears in `queuedMessages`, never affects queue-derived busy state, and never triggers resume-on-restart). Both classes emit `tool_background_complete` immediately.
268
270
  - The caller distinguishes command responses from system events with a single check: `if (msg.requestId)`
269
271
 
270
272
  This enables a simple promise-based RPC layer: send a command with a unique ID, store a pending promise keyed by that ID, resolve it when you see `completed` with the matching ID.
@@ -327,6 +329,14 @@ Cancel pending **queued** messages without touching the in-flight turn. Omit `id
327
329
  {"action": "cancelQueued", "requestId": "r6", "id": "r2"}
328
330
  ```
329
331
 
332
+ #### `setQueuedDelivery`
333
+
334
+ Change a queued message's delivery semantics: `"asap"` (inject into the running turn at its next tool boundary) or `"afterTurn"` (default — wait for the turn to end). `id` is the `requestId` of a queued message. Only plain queued **user** messages qualify; automated-action messages and chain/background items respond `completed(success:false, error:"message not found or not promotable")` — as does an item already consumed by the running turn. On success a `queue_changed` event carries the updated snapshot (each item's `delivery` field rides on it).
335
+
336
+ ```json
337
+ {"action": "setQueuedDelivery", "requestId": "r7", "id": "r2", "delivery": "asap"}
338
+ ```
339
+
330
340
  #### `clear`
331
341
 
332
342
  Clear conversation history and delete the session file.
@@ -353,7 +363,7 @@ Events are emitted as newline-delimited JSON. Command responses include `request
353
363
  |-------|--------|-------------|
354
364
  | `ready` | | Headless mode initialized, ready for input |
355
365
  | `session_restored` | `messageCount` | Previous session loaded |
356
- | `queue_changed` | `queuedMessages` | Queue contents changed (any push/shift/drain/cancel). Carries the full current snapshot — an empty array when the queue drains. The live queue-state signal; for the initial snapshot on connect/reconnect, read `queuedMessages` from `get_history`. |
366
+ | `queue_changed` | `queuedMessages` | Queue contents changed (any push/shift/drain/cancel/retag). Carries the full current snapshot — an empty array when the queue drains. Items carry `delivery: "asap"` when promoted (absent = after-turn). The live queue-state signal; for the initial snapshot on connect/reconnect, read `queuedMessages` from `get_history`. |
357
367
  | `stopping` | | Shutdown initiated |
358
368
  | `stopped` | | Shutdown complete |
359
369
 
@@ -365,7 +375,7 @@ All command responses include the `requestId` from the originating command.
365
375
  |-------|--------|-------------|
366
376
  | `text` | `text`, `parentToolId?` | Streaming text chunk |
367
377
  | `thinking` | `text`, `parentToolId?` | Agent's internal reasoning |
368
- | `user_message` | `text`, `attachments?`, `queued?` | Echo of a user message entering the turn. Queue-delivered messages carry `queued: true` and their own original `requestId` (a merged turn emits one per absorbed message); idle sends echo with the turn's requestId and no `queued` flag. |
378
+ | `user_message` | `text`, `attachments?`, `queued?`, `hidden?` | Echo of a user message entering the turn. Queue-delivered messages (including ASAP items injected mid-turn) carry `queued: true` and their own original `requestId` (a merged turn emits one per absorbed message); idle sends echo with the turn's requestId and no `queued` flag. `hidden: true` marks internal entries (e.g. passive background results) that should not render. |
369
379
  | `tool_start` | `id`, `name`, `input`, `partial?`, `parentToolId?` | Tool execution started. `partial: true` means more `tool_start` events will follow for this id (progressive input streaming). |
370
380
  | `tool_input_delta` | `id`, `name`, `result`, `parentToolId?` | Progressive tool content (streaming tools only) |
371
381
  | `tool_done` | `id`, `name`, `result`, `isError`, `parentToolId?` | Tool execution completed |
@@ -70,6 +70,15 @@ declare class HeadlessSession {
70
70
  * to .remy-stats.json so queued work survives process restarts.
71
71
  */
72
72
  private queue;
73
+ /**
74
+ * Holding pen for passive background results (tools with
75
+ * `backgroundNotify: 'passive'`, e.g. specSync). Deliberately outside the
76
+ * message queue: pen contents never initiate a turn, never latch the
77
+ * sandbox's queue-derived busy state, and never trigger resume-on-restart.
78
+ * Swept into the next real turn as a hidden background_results entry.
79
+ * Persisted to .remy-stats.json alongside the queue.
80
+ */
81
+ private passivePen;
73
82
  private pendingTools;
74
83
  private earlyResults;
75
84
  private pendingBlockUpdates;
@@ -87,7 +96,7 @@ declare class HeadlessSession {
87
96
  private primaryOutcome;
88
97
  /** Dispatch a simple (non-streaming) command: call handler, emit response + completed. */
89
98
  private dispatchSimple;
90
- /** Persist sessionStats + queue snapshot to .remy-stats.json. */
99
+ /** Persist sessionStats + queue snapshot + passive pen to .remy-stats.json. */
91
100
  private persistStats;
92
101
  /** Apply queued tool block updates to state.messages. Safe to call any time. */
93
102
  private applyPendingBlockUpdates;
package/dist/headless.js CHANGED
@@ -4099,7 +4099,7 @@ var BROWSER_TOOLS = [
4099
4099
  "screenshotViewport",
4100
4100
  "setViewport"
4101
4101
  ],
4102
- description: 'snapshot: accessibility tree of the page (waits for network to settle). click: click an element (animated cursor, full event sequence). type: type text into input (one char at a time, works with React/Vue/Svelte). select: select a dropdown option by text. wait: wait for an element to appear (polls 100ms, waits for network). navigate: navigate to a URL within the app (waits for load, subsequent steps run on new page). evaluate: run JS in the page. styles: read computed CSS styles from elements (pass properties array with camelCase names, or omit for defaults). screenshotFullPage: screenshot of the whole page top-to-bottom (returns a CDN url with dimensions and a written analysis). screenshotViewport: screenshot of just the visible viewport \u2014 pass `scrollToSelector` (or `scrollY`) on this step to scroll a section into view and capture it in one atomic step (no separate scroll needed). setViewport: switch the browser between desktop and mobile rendering (pass `mode`: "desktop" or "mobile"). Reloads the page so responsive layouts, media queries, and matchMedia re-evaluate \u2014 use it to QA mobile/responsive views.'
4102
+ description: 'snapshot: accessibility tree of the page (waits for network to settle). click: click an element (animated cursor, full event sequence). type: type text into input (one char at a time, works with React/Vue/Svelte). select: select a dropdown option by text. wait: wait for an element to appear (polls 100ms, waits for network). navigate: navigate to a URL within the app (waits for the route, subsequent steps run on the new page; soft in-app route change by default \u2014 pass `fresh: true` for a real full page load; the result reports the URL actually landed on, so app redirects are visible). evaluate: run JS in the page. styles: read computed CSS styles from elements (pass properties array with camelCase names, or omit for defaults). screenshotFullPage: screenshot of the whole page top-to-bottom (returns a CDN url with dimensions and a written analysis). screenshotViewport: screenshot of just the visible viewport \u2014 pass `scrollToSelector` (or `scrollY`) on this step to scroll a section into view and capture it in one atomic step (no separate scroll needed). setViewport: switch the browser between desktop and mobile rendering (pass `mode`: "desktop" or "mobile"). Reloads the page so responsive layouts, media queries, and matchMedia re-evaluate \u2014 use it to QA mobile/responsive views.'
4103
4103
  },
4104
4104
  ref: {
4105
4105
  type: "string",
@@ -4141,6 +4141,10 @@ var BROWSER_TOOLS = [
4141
4141
  type: "string",
4142
4142
  description: 'For navigate: the URL to navigate to (e.g., "/quiz", "/settings").'
4143
4143
  },
4144
+ fresh: {
4145
+ type: "boolean",
4146
+ description: "For navigate: force a real full page load (fresh document) instead of a soft in-app route change. Use when testing what a user sees on entry \u2014 landing pages, join/invite links, signed-out views \u2014 where reusing the SPA\u2019s in-memory state would test the wrong thing."
4147
+ },
4144
4148
  properties: {
4145
4149
  type: "array",
4146
4150
  items: { type: "string" },
@@ -5955,10 +5959,13 @@ ${readAsset(
5955
5959
  </mindstudio_flavored_markdown_spec_docs>`;
5956
5960
  var specSyncTool = {
5957
5961
  backgroundOnly: true,
5962
+ // Fire-and-forget: completion never wakes the agent. The outcome rides the
5963
+ // next real turn as a hidden background_results note instead.
5964
+ backgroundNotify: "passive",
5958
5965
  definition: {
5959
5966
  clearable: false,
5960
5967
  name: "specSync",
5961
- description: "Reconcile the spec to bring it in line with code changes you have made. Provide a brief, bulleted list of what changed and why; it finds the affected spec sections and updates them to match. Always runs in the background \u2014 it returns immediately and reports back when done.",
5968
+ description: "Reconcile the spec to bring it in line with code changes you have made. Provide a brief, bulleted list of what changed and why; it finds the affected spec sections and updates them to match. Always runs in the background and completes silently \u2014 do not wait for it; its outcome appears as an automated note at the start of a later turn.",
5962
5969
  inputSchema: {
5963
5970
  type: "object",
5964
5971
  properties: {
@@ -7627,6 +7634,7 @@ async function runTurn(params) {
7627
7634
  onboardingState,
7628
7635
  signal,
7629
7636
  onEvent,
7637
+ takeSteering,
7630
7638
  resolveExternalTool,
7631
7639
  requestId,
7632
7640
  toolRegistry,
@@ -7661,7 +7669,7 @@ async function runTurn(params) {
7661
7669
  onEvent({ type: "error", error: "Empty message" });
7662
7670
  return;
7663
7671
  }
7664
- for (const entry of keptEntries) {
7672
+ const appendEntry = (entry) => {
7665
7673
  const hasAttachments = (entry.attachments?.length ?? 0) > 0;
7666
7674
  const userMsg = { role: "user", content: entry.text };
7667
7675
  if (entry.hidden) {
@@ -7684,6 +7692,9 @@ async function runTurn(params) {
7684
7692
  ...entry.requestId && { requestId: entry.requestId },
7685
7693
  ...entry.queued && { queued: true }
7686
7694
  });
7695
+ };
7696
+ for (const entry of keptEntries) {
7697
+ appendEntry(entry);
7687
7698
  }
7688
7699
  const isFirstMessage = state.messages.filter((m) => m.role === "user").length === 1;
7689
7700
  const STATUS_EXCLUDED_TOOLS = /* @__PURE__ */ new Set([
@@ -8200,6 +8211,18 @@ async function runTurn(params) {
8200
8211
  isToolError: r.isError
8201
8212
  });
8202
8213
  }
8214
+ if (takeSteering && !signal?.aborted) {
8215
+ const injected = (await takeSteering()).filter(
8216
+ (e) => e.text.trim().length > 0 || (e.attachments?.length ?? 0) > 0
8217
+ );
8218
+ if (injected.length > 0) {
8219
+ for (const entry of injected) {
8220
+ appendEntry(entry);
8221
+ keptEntries.push(entry);
8222
+ }
8223
+ saveSession(state);
8224
+ }
8225
+ }
8203
8226
  if (signal?.aborted) {
8204
8227
  onEvent({ type: "turn_cancelled" });
8205
8228
  saveSession(state);
@@ -8375,13 +8398,24 @@ function loadQueue() {
8375
8398
  }
8376
8399
  return [];
8377
8400
  }
8378
- function writeStats(stats, queue) {
8401
+ function loadPassiveResults() {
8402
+ try {
8403
+ const stats = JSON.parse(readFileSync2(STATS_FILE, "utf-8"));
8404
+ if (Array.isArray(stats.passiveResults)) {
8405
+ return stats.passiveResults;
8406
+ }
8407
+ } catch {
8408
+ }
8409
+ return [];
8410
+ }
8411
+ function writeStats(stats, queue, passiveResults) {
8379
8412
  try {
8380
8413
  writeFileSync2(
8381
8414
  STATS_FILE,
8382
8415
  JSON.stringify({
8383
8416
  ...stats,
8384
- queue
8417
+ queue,
8418
+ passiveResults
8385
8419
  })
8386
8420
  );
8387
8421
  } catch {
@@ -8445,6 +8479,21 @@ var MessageQueue = class {
8445
8479
  }
8446
8480
  return removed;
8447
8481
  }
8482
+ /**
8483
+ * Change a queued item's delivery semantics, keyed by its command
8484
+ * requestId. Fires onChange (→ persist + queue_changed) on success.
8485
+ * Returns the item, or undefined if no queued item matches (e.g. it was
8486
+ * already consumed).
8487
+ */
8488
+ setDelivery(id, delivery) {
8489
+ const item = this.items.find((it) => it.command.requestId === id);
8490
+ if (!item) {
8491
+ return void 0;
8492
+ }
8493
+ item.delivery = delivery;
8494
+ this.onChange?.();
8495
+ return item;
8496
+ }
8448
8497
  /** Copy of current queue contents (for surfacing on events). */
8449
8498
  snapshot() {
8450
8499
  return [...this.items];
@@ -8559,6 +8608,15 @@ var HeadlessSession = class {
8559
8608
  * to .remy-stats.json so queued work survives process restarts.
8560
8609
  */
8561
8610
  queue;
8611
+ /**
8612
+ * Holding pen for passive background results (tools with
8613
+ * `backgroundNotify: 'passive'`, e.g. specSync). Deliberately outside the
8614
+ * message queue: pen contents never initiate a turn, never latch the
8615
+ * sandbox's queue-derived busy state, and never trigger resume-on-restart.
8616
+ * Swept into the next real turn as a hidden background_results entry.
8617
+ * Persisted to .remy-stats.json alongside the queue.
8618
+ */
8619
+ passivePen = [];
8562
8620
  // External tool bridge
8563
8621
  pendingTools = /* @__PURE__ */ new Map();
8564
8622
  earlyResults = /* @__PURE__ */ new Map();
@@ -8594,6 +8652,7 @@ var HeadlessSession = class {
8594
8652
  this.persistStats();
8595
8653
  this.emit("queue_changed", { queuedMessages: this.queue.snapshot() });
8596
8654
  });
8655
+ this.passivePen = loadPassiveResults();
8597
8656
  if (resumed) {
8598
8657
  this.emit("session_restored", {
8599
8658
  messageCount: this.state.messages.length,
@@ -8705,10 +8764,10 @@ var HeadlessSession = class {
8705
8764
  //////////////////////////////////////////////////////////////////////////////
8706
8765
  // Stats + queue persistence
8707
8766
  //////////////////////////////////////////////////////////////////////////////
8708
- /** Persist sessionStats + queue snapshot to .remy-stats.json. */
8767
+ /** Persist sessionStats + queue snapshot + passive pen to .remy-stats.json. */
8709
8768
  persistStats() {
8710
8769
  this.sessionStats.updatedAt = Date.now();
8711
- writeStats(this.sessionStats, this.queue.snapshot());
8770
+ writeStats(this.sessionStats, this.queue.snapshot(), this.passivePen);
8712
8771
  }
8713
8772
  //////////////////////////////////////////////////////////////////////////////
8714
8773
  // Background completions (tool-block mutation; message delivery via queue)
@@ -8770,10 +8829,12 @@ var HeadlessSession = class {
8770
8829
  }
8771
8830
  }
8772
8831
  onBackgroundComplete = (toolCallId, name, result, subAgentMessages) => {
8832
+ const notify = getToolByName(name)?.backgroundNotify ?? "wake";
8773
8833
  this.pendingBlockUpdates.push({ toolCallId, result, subAgentMessages });
8774
8834
  log16.info("Background complete", {
8775
8835
  toolCallId,
8776
8836
  name,
8837
+ notify,
8777
8838
  requestId: this.currentRequestId
8778
8839
  });
8779
8840
  this.onEvent({
@@ -8782,6 +8843,14 @@ var HeadlessSession = class {
8782
8843
  name,
8783
8844
  result
8784
8845
  });
8846
+ if (notify === "passive") {
8847
+ this.passivePen.push({ toolCallId, name, result });
8848
+ this.persistStats();
8849
+ if (!this.running) {
8850
+ this.applyPendingBlockUpdates();
8851
+ }
8852
+ return;
8853
+ }
8785
8854
  this.queue.push({
8786
8855
  command: {
8787
8856
  action: "message",
@@ -8850,7 +8919,10 @@ var HeadlessSession = class {
8850
8919
  ...e.attachments && { attachments: e.attachments },
8851
8920
  // Queue-delivered entries are flagged so the frontend renders the
8852
8921
  // echo (idle sends are rendered optimistically instead).
8853
- ...e.queued && { queued: true }
8922
+ ...e.queued && { queued: true },
8923
+ // Hidden entries (e.g. the passive background_results sweep) are
8924
+ // flagged so the sandbox/frontend suppress the bubble.
8925
+ ...e.hidden && { hidden: true }
8854
8926
  },
8855
8927
  // A merged turn emits one user_message per absorbed entry — each
8856
8928
  // carries its own original requestId, not the turn's.
@@ -9160,6 +9232,37 @@ var HeadlessSession = class {
9160
9232
  this.completedEmitted = false;
9161
9233
  this.lastCompleted = null;
9162
9234
  this.turnStart = Date.now();
9235
+ if (this.passivePen.length > 0) {
9236
+ const swept = this.passivePen.splice(0);
9237
+ this.persistStats();
9238
+ entries.unshift({
9239
+ text: buildBackgroundResultsMessage(swept),
9240
+ hidden: true
9241
+ });
9242
+ }
9243
+ const consumedRids = [];
9244
+ const takeSteering = async () => {
9245
+ const items = this.queue.removeWhere(
9246
+ (it) => it.source === "user" && it.delivery === "asap" && !this.isDrainBarrier(it)
9247
+ );
9248
+ const steered = [];
9249
+ for (const it of items) {
9250
+ const attachments = it.command.attachments;
9251
+ const attachmentHeader = await this.persistEntryAttachments(attachments);
9252
+ const rid = it.command.requestId;
9253
+ if (rid) {
9254
+ consumedRids.push(rid);
9255
+ }
9256
+ steered.push({
9257
+ text: it.command.text ?? "",
9258
+ attachments,
9259
+ attachmentHeader,
9260
+ requestId: rid,
9261
+ queued: true
9262
+ });
9263
+ }
9264
+ return steered;
9265
+ };
9163
9266
  await this.runForcedCompactionIfNeeded(requestId);
9164
9267
  try {
9165
9268
  await runTurn({
@@ -9173,6 +9276,7 @@ var HeadlessSession = class {
9173
9276
  requestId,
9174
9277
  signal: this.currentAbort.signal,
9175
9278
  onEvent: this.onEvent,
9279
+ takeSteering,
9176
9280
  resolveExternalTool: this.resolveExternalTool,
9177
9281
  toolRegistry: this.toolRegistry,
9178
9282
  onBackgroundComplete: this.onBackgroundComplete
@@ -9202,7 +9306,7 @@ var HeadlessSession = class {
9202
9306
  });
9203
9307
  }
9204
9308
  const outcome = this.primaryOutcome();
9205
- for (const rid of absorbedRids) {
9309
+ for (const rid of [...absorbedRids, ...consumedRids]) {
9206
9310
  this.emit(
9207
9311
  "completed",
9208
9312
  {
@@ -9497,6 +9601,33 @@ var HeadlessSession = class {
9497
9601
  );
9498
9602
  return;
9499
9603
  }
9604
+ if (action === "setQueuedDelivery") {
9605
+ const id = parsed.id;
9606
+ const delivery = parsed.delivery;
9607
+ if (!id || delivery !== "asap" && delivery !== "afterTurn") {
9608
+ this.emit(
9609
+ "completed",
9610
+ {
9611
+ success: false,
9612
+ error: 'setQueuedDelivery requires id and delivery ("asap" | "afterTurn")'
9613
+ },
9614
+ requestId
9615
+ );
9616
+ return;
9617
+ }
9618
+ const target = this.queue.snapshot().find((it) => it.command.requestId === id);
9619
+ if (!target || target.source !== "user" || this.isDrainBarrier(target)) {
9620
+ this.emit(
9621
+ "completed",
9622
+ { success: false, error: "message not found or not promotable" },
9623
+ requestId
9624
+ );
9625
+ return;
9626
+ }
9627
+ this.queue.setDelivery(id, delivery);
9628
+ this.emit("completed", { success: true }, requestId);
9629
+ return;
9630
+ }
9500
9631
  if (action === "stop_tool") {
9501
9632
  const id = parsed.id;
9502
9633
  const mode = parsed.mode ?? "hard";
package/dist/index.js CHANGED
@@ -4934,7 +4934,7 @@ var init_tools2 = __esm({
4934
4934
  "screenshotViewport",
4935
4935
  "setViewport"
4936
4936
  ],
4937
- description: 'snapshot: accessibility tree of the page (waits for network to settle). click: click an element (animated cursor, full event sequence). type: type text into input (one char at a time, works with React/Vue/Svelte). select: select a dropdown option by text. wait: wait for an element to appear (polls 100ms, waits for network). navigate: navigate to a URL within the app (waits for load, subsequent steps run on new page). evaluate: run JS in the page. styles: read computed CSS styles from elements (pass properties array with camelCase names, or omit for defaults). screenshotFullPage: screenshot of the whole page top-to-bottom (returns a CDN url with dimensions and a written analysis). screenshotViewport: screenshot of just the visible viewport \u2014 pass `scrollToSelector` (or `scrollY`) on this step to scroll a section into view and capture it in one atomic step (no separate scroll needed). setViewport: switch the browser between desktop and mobile rendering (pass `mode`: "desktop" or "mobile"). Reloads the page so responsive layouts, media queries, and matchMedia re-evaluate \u2014 use it to QA mobile/responsive views.'
4937
+ description: 'snapshot: accessibility tree of the page (waits for network to settle). click: click an element (animated cursor, full event sequence). type: type text into input (one char at a time, works with React/Vue/Svelte). select: select a dropdown option by text. wait: wait for an element to appear (polls 100ms, waits for network). navigate: navigate to a URL within the app (waits for the route, subsequent steps run on the new page; soft in-app route change by default \u2014 pass `fresh: true` for a real full page load; the result reports the URL actually landed on, so app redirects are visible). evaluate: run JS in the page. styles: read computed CSS styles from elements (pass properties array with camelCase names, or omit for defaults). screenshotFullPage: screenshot of the whole page top-to-bottom (returns a CDN url with dimensions and a written analysis). screenshotViewport: screenshot of just the visible viewport \u2014 pass `scrollToSelector` (or `scrollY`) on this step to scroll a section into view and capture it in one atomic step (no separate scroll needed). setViewport: switch the browser between desktop and mobile rendering (pass `mode`: "desktop" or "mobile"). Reloads the page so responsive layouts, media queries, and matchMedia re-evaluate \u2014 use it to QA mobile/responsive views.'
4938
4938
  },
4939
4939
  ref: {
4940
4940
  type: "string",
@@ -4976,6 +4976,10 @@ var init_tools2 = __esm({
4976
4976
  type: "string",
4977
4977
  description: 'For navigate: the URL to navigate to (e.g., "/quiz", "/settings").'
4978
4978
  },
4979
+ fresh: {
4980
+ type: "boolean",
4981
+ description: "For navigate: force a real full page load (fresh document) instead of a soft in-app route change. Use when testing what a user sees on entry \u2014 landing pages, join/invite links, signed-out views \u2014 where reusing the SPA\u2019s in-memory state would test the wrong thing."
4982
+ },
4979
4983
  properties: {
4980
4984
  type: "array",
4981
4985
  items: { type: "string" },
@@ -7149,10 +7153,13 @@ ${readAsset(
7149
7153
  </mindstudio_flavored_markdown_spec_docs>`;
7150
7154
  specSyncTool = {
7151
7155
  backgroundOnly: true,
7156
+ // Fire-and-forget: completion never wakes the agent. The outcome rides the
7157
+ // next real turn as a hidden background_results note instead.
7158
+ backgroundNotify: "passive",
7152
7159
  definition: {
7153
7160
  clearable: false,
7154
7161
  name: "specSync",
7155
- description: "Reconcile the spec to bring it in line with code changes you have made. Provide a brief, bulleted list of what changed and why; it finds the affected spec sections and updates them to match. Always runs in the background \u2014 it returns immediately and reports back when done.",
7162
+ description: "Reconcile the spec to bring it in line with code changes you have made. Provide a brief, bulleted list of what changed and why; it finds the affected spec sections and updates them to match. Always runs in the background and completes silently \u2014 do not wait for it; its outcome appears as an automated note at the start of a later turn.",
7156
7163
  inputSchema: {
7157
7164
  type: "object",
7158
7165
  properties: {
@@ -8104,6 +8111,7 @@ async function runTurn(params) {
8104
8111
  onboardingState,
8105
8112
  signal,
8106
8113
  onEvent,
8114
+ takeSteering,
8107
8115
  resolveExternalTool,
8108
8116
  requestId,
8109
8117
  toolRegistry,
@@ -8138,7 +8146,7 @@ async function runTurn(params) {
8138
8146
  onEvent({ type: "error", error: "Empty message" });
8139
8147
  return;
8140
8148
  }
8141
- for (const entry of keptEntries) {
8149
+ const appendEntry = (entry) => {
8142
8150
  const hasAttachments = (entry.attachments?.length ?? 0) > 0;
8143
8151
  const userMsg = { role: "user", content: entry.text };
8144
8152
  if (entry.hidden) {
@@ -8161,6 +8169,9 @@ async function runTurn(params) {
8161
8169
  ...entry.requestId && { requestId: entry.requestId },
8162
8170
  ...entry.queued && { queued: true }
8163
8171
  });
8172
+ };
8173
+ for (const entry of keptEntries) {
8174
+ appendEntry(entry);
8164
8175
  }
8165
8176
  const isFirstMessage = state.messages.filter((m) => m.role === "user").length === 1;
8166
8177
  const STATUS_EXCLUDED_TOOLS = /* @__PURE__ */ new Set([
@@ -8677,6 +8688,18 @@ async function runTurn(params) {
8677
8688
  isToolError: r.isError
8678
8689
  });
8679
8690
  }
8691
+ if (takeSteering && !signal?.aborted) {
8692
+ const injected = (await takeSteering()).filter(
8693
+ (e) => e.text.trim().length > 0 || (e.attachments?.length ?? 0) > 0
8694
+ );
8695
+ if (injected.length > 0) {
8696
+ for (const entry of injected) {
8697
+ appendEntry(entry);
8698
+ keptEntries.push(entry);
8699
+ }
8700
+ saveSession(state);
8701
+ }
8702
+ }
8680
8703
  if (signal?.aborted) {
8681
8704
  onEvent({ type: "turn_cancelled" });
8682
8705
  saveSession(state);
@@ -9244,13 +9267,24 @@ function loadQueue() {
9244
9267
  }
9245
9268
  return [];
9246
9269
  }
9247
- function writeStats(stats, queue) {
9270
+ function loadPassiveResults() {
9271
+ try {
9272
+ const stats = JSON.parse(readFileSync2(STATS_FILE, "utf-8"));
9273
+ if (Array.isArray(stats.passiveResults)) {
9274
+ return stats.passiveResults;
9275
+ }
9276
+ } catch {
9277
+ }
9278
+ return [];
9279
+ }
9280
+ function writeStats(stats, queue, passiveResults) {
9248
9281
  try {
9249
9282
  writeFileSync2(
9250
9283
  STATS_FILE,
9251
9284
  JSON.stringify({
9252
9285
  ...stats,
9253
- queue
9286
+ queue,
9287
+ passiveResults
9254
9288
  })
9255
9289
  );
9256
9290
  } catch {
@@ -9325,6 +9359,21 @@ var init_messageQueue = __esm({
9325
9359
  }
9326
9360
  return removed;
9327
9361
  }
9362
+ /**
9363
+ * Change a queued item's delivery semantics, keyed by its command
9364
+ * requestId. Fires onChange (→ persist + queue_changed) on success.
9365
+ * Returns the item, or undefined if no queued item matches (e.g. it was
9366
+ * already consumed).
9367
+ */
9368
+ setDelivery(id, delivery) {
9369
+ const item = this.items.find((it) => it.command.requestId === id);
9370
+ if (!item) {
9371
+ return void 0;
9372
+ }
9373
+ item.delivery = delivery;
9374
+ this.onChange?.();
9375
+ return item;
9376
+ }
9328
9377
  /** Copy of current queue contents (for surfacing on events). */
9329
9378
  snapshot() {
9330
9379
  return [...this.items];
@@ -9431,6 +9480,7 @@ var init_headless = __esm({
9431
9480
  init_attachments();
9432
9481
  init_planFile();
9433
9482
  init_stats();
9483
+ init_tools8();
9434
9484
  init_messageQueue();
9435
9485
  init_resolve();
9436
9486
  init_sentinel();
@@ -9474,6 +9524,15 @@ var init_headless = __esm({
9474
9524
  * to .remy-stats.json so queued work survives process restarts.
9475
9525
  */
9476
9526
  queue;
9527
+ /**
9528
+ * Holding pen for passive background results (tools with
9529
+ * `backgroundNotify: 'passive'`, e.g. specSync). Deliberately outside the
9530
+ * message queue: pen contents never initiate a turn, never latch the
9531
+ * sandbox's queue-derived busy state, and never trigger resume-on-restart.
9532
+ * Swept into the next real turn as a hidden background_results entry.
9533
+ * Persisted to .remy-stats.json alongside the queue.
9534
+ */
9535
+ passivePen = [];
9477
9536
  // External tool bridge
9478
9537
  pendingTools = /* @__PURE__ */ new Map();
9479
9538
  earlyResults = /* @__PURE__ */ new Map();
@@ -9509,6 +9568,7 @@ var init_headless = __esm({
9509
9568
  this.persistStats();
9510
9569
  this.emit("queue_changed", { queuedMessages: this.queue.snapshot() });
9511
9570
  });
9571
+ this.passivePen = loadPassiveResults();
9512
9572
  if (resumed) {
9513
9573
  this.emit("session_restored", {
9514
9574
  messageCount: this.state.messages.length,
@@ -9620,10 +9680,10 @@ var init_headless = __esm({
9620
9680
  //////////////////////////////////////////////////////////////////////////////
9621
9681
  // Stats + queue persistence
9622
9682
  //////////////////////////////////////////////////////////////////////////////
9623
- /** Persist sessionStats + queue snapshot to .remy-stats.json. */
9683
+ /** Persist sessionStats + queue snapshot + passive pen to .remy-stats.json. */
9624
9684
  persistStats() {
9625
9685
  this.sessionStats.updatedAt = Date.now();
9626
- writeStats(this.sessionStats, this.queue.snapshot());
9686
+ writeStats(this.sessionStats, this.queue.snapshot(), this.passivePen);
9627
9687
  }
9628
9688
  //////////////////////////////////////////////////////////////////////////////
9629
9689
  // Background completions (tool-block mutation; message delivery via queue)
@@ -9685,10 +9745,12 @@ var init_headless = __esm({
9685
9745
  }
9686
9746
  }
9687
9747
  onBackgroundComplete = (toolCallId, name, result, subAgentMessages) => {
9748
+ const notify = getToolByName(name)?.backgroundNotify ?? "wake";
9688
9749
  this.pendingBlockUpdates.push({ toolCallId, result, subAgentMessages });
9689
9750
  log16.info("Background complete", {
9690
9751
  toolCallId,
9691
9752
  name,
9753
+ notify,
9692
9754
  requestId: this.currentRequestId
9693
9755
  });
9694
9756
  this.onEvent({
@@ -9697,6 +9759,14 @@ var init_headless = __esm({
9697
9759
  name,
9698
9760
  result
9699
9761
  });
9762
+ if (notify === "passive") {
9763
+ this.passivePen.push({ toolCallId, name, result });
9764
+ this.persistStats();
9765
+ if (!this.running) {
9766
+ this.applyPendingBlockUpdates();
9767
+ }
9768
+ return;
9769
+ }
9700
9770
  this.queue.push({
9701
9771
  command: {
9702
9772
  action: "message",
@@ -9765,7 +9835,10 @@ var init_headless = __esm({
9765
9835
  ...e.attachments && { attachments: e.attachments },
9766
9836
  // Queue-delivered entries are flagged so the frontend renders the
9767
9837
  // echo (idle sends are rendered optimistically instead).
9768
- ...e.queued && { queued: true }
9838
+ ...e.queued && { queued: true },
9839
+ // Hidden entries (e.g. the passive background_results sweep) are
9840
+ // flagged so the sandbox/frontend suppress the bubble.
9841
+ ...e.hidden && { hidden: true }
9769
9842
  },
9770
9843
  // A merged turn emits one user_message per absorbed entry — each
9771
9844
  // carries its own original requestId, not the turn's.
@@ -10075,6 +10148,37 @@ var init_headless = __esm({
10075
10148
  this.completedEmitted = false;
10076
10149
  this.lastCompleted = null;
10077
10150
  this.turnStart = Date.now();
10151
+ if (this.passivePen.length > 0) {
10152
+ const swept = this.passivePen.splice(0);
10153
+ this.persistStats();
10154
+ entries.unshift({
10155
+ text: buildBackgroundResultsMessage(swept),
10156
+ hidden: true
10157
+ });
10158
+ }
10159
+ const consumedRids = [];
10160
+ const takeSteering = async () => {
10161
+ const items = this.queue.removeWhere(
10162
+ (it) => it.source === "user" && it.delivery === "asap" && !this.isDrainBarrier(it)
10163
+ );
10164
+ const steered = [];
10165
+ for (const it of items) {
10166
+ const attachments = it.command.attachments;
10167
+ const attachmentHeader = await this.persistEntryAttachments(attachments);
10168
+ const rid = it.command.requestId;
10169
+ if (rid) {
10170
+ consumedRids.push(rid);
10171
+ }
10172
+ steered.push({
10173
+ text: it.command.text ?? "",
10174
+ attachments,
10175
+ attachmentHeader,
10176
+ requestId: rid,
10177
+ queued: true
10178
+ });
10179
+ }
10180
+ return steered;
10181
+ };
10078
10182
  await this.runForcedCompactionIfNeeded(requestId);
10079
10183
  try {
10080
10184
  await runTurn({
@@ -10088,6 +10192,7 @@ var init_headless = __esm({
10088
10192
  requestId,
10089
10193
  signal: this.currentAbort.signal,
10090
10194
  onEvent: this.onEvent,
10195
+ takeSteering,
10091
10196
  resolveExternalTool: this.resolveExternalTool,
10092
10197
  toolRegistry: this.toolRegistry,
10093
10198
  onBackgroundComplete: this.onBackgroundComplete
@@ -10117,7 +10222,7 @@ var init_headless = __esm({
10117
10222
  });
10118
10223
  }
10119
10224
  const outcome = this.primaryOutcome();
10120
- for (const rid of absorbedRids) {
10225
+ for (const rid of [...absorbedRids, ...consumedRids]) {
10121
10226
  this.emit(
10122
10227
  "completed",
10123
10228
  {
@@ -10412,6 +10517,33 @@ var init_headless = __esm({
10412
10517
  );
10413
10518
  return;
10414
10519
  }
10520
+ if (action === "setQueuedDelivery") {
10521
+ const id = parsed.id;
10522
+ const delivery = parsed.delivery;
10523
+ if (!id || delivery !== "asap" && delivery !== "afterTurn") {
10524
+ this.emit(
10525
+ "completed",
10526
+ {
10527
+ success: false,
10528
+ error: 'setQueuedDelivery requires id and delivery ("asap" | "afterTurn")'
10529
+ },
10530
+ requestId
10531
+ );
10532
+ return;
10533
+ }
10534
+ const target = this.queue.snapshot().find((it) => it.command.requestId === id);
10535
+ if (!target || target.source !== "user" || this.isDrainBarrier(target)) {
10536
+ this.emit(
10537
+ "completed",
10538
+ { success: false, error: "message not found or not promotable" },
10539
+ requestId
10540
+ );
10541
+ return;
10542
+ }
10543
+ this.queue.setDelivery(id, delivery);
10544
+ this.emit("completed", { success: true }, requestId);
10545
+ return;
10546
+ }
10415
10547
  if (action === "stop_tool") {
10416
10548
  const id = parsed.id;
10417
10549
  const mode = parsed.mode ?? "hard";
@@ -103,6 +103,30 @@ const { threads, nextCursor } = await chat.listThreads();
103
103
  const full = await chat.getThread(thread.id);
104
104
  await chat.updateThread(thread.id, 'New title');
105
105
  await chat.deleteThread(thread.id);
106
+
107
+ // Progressive auth: threads started anonymously become unreachable after the
108
+ // user signs in (login replaces the session and its visitor identity). Claim
109
+ // them right after your verification/login succeeds so the conversation
110
+ // survives — the client remembers each thread's pre-login token automatically
111
+ // for threads touched this page session.
112
+ await chat.claimThread(thread.id);
113
+ ```
114
+
115
+ **Client tools** — a tool whose effect happens in the browser (open a sheet, navigate, highlight)
116
+ is declared with `target: "client"` and a `name` + inline `inputSchema` instead of a `method`
117
+ (names must not collide with method ids; the schema is authored — there's no method contract to
118
+ derive it from). The agent's invocation arrives as the `client_tool_call` stream event / the
119
+ `onClientToolCall` callback on `sendMessage`; run the action there. Fire-and-forget on this
120
+ surface: the agent is told the action was displayed and keeps going — the user's next message
121
+ closes the loop.
122
+
123
+ ```js
124
+ await chat.sendMessage(thread.id, text, {
125
+ onText: (delta) => append(delta),
126
+ onClientToolCall: (name, input) => {
127
+ if (name === 'showVerification') openVerifySheet(input);
128
+ },
129
+ });
106
130
  ```
107
131
 
108
132
  **Sending messages (streaming):**
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  name: Voice Interfaces
3
3
  what: Realtime voice conversation as a first-class interface — the user talks to the app and its voice agent talks back in sub-second, interruptible speech, calling the app's methods mid-conversation as the authenticated user. The platform handles the media transport, turn-taking, barge-in, and transcripts, so the work is authorship — a persona written for the ear, a small toolset where every tool carries a latency class, and descriptions that say results out loud. Any app whose methods do something interesting can pick up a voice, and it is often the most impressive surface it has.
4
- when: Before authoring `src/interfaces/voice.md`, choosing a voice model or pipeline, deciding which methods a voice agent gets, or building the voice UI with `createVoiceClient()`.
4
+ when: Before authoring `src/interfaces/voice.md`, choosing a voice model or pipeline, deciding which methods a voice agent gets, building the voice UI with `createVoiceClient()`, or working out why a voice agent behaved the way it did on a call.
5
5
  ---
6
6
 
7
7
  # Building Voice Interfaces
@@ -57,7 +57,8 @@ precisely: "confirm before any tool that changes data," not "always confirm ever
57
57
  `always`/`never` makes the agent rigid and unable to handle reasonable exceptions. And start
58
58
  minimal: state the role, the boundaries, and the voice mechanics above, then add rules only for
59
59
  behaviors that actually misfire in test calls (the transcripts in the call log are the feedback
60
- loop) rather than front-loading a policy manual.
60
+ loop — `mindstudio-prod voice sessions get` reads a call verbatim) rather than front-loading a
61
+ policy manual.
61
62
 
62
63
  ### The latency classes
63
64
 
@@ -78,25 +79,65 @@ Classify by how the method actually behaves, not by what it is named. A "lookup"
78
79
  external service is `slow`. When in doubt between `fast` and `slow`, pick `slow` — a needless
79
80
  preamble is mildly chatty; an unexplained silence feels broken.
80
81
 
81
- ### Forwarding results to the screen (`forwardResult`)
82
+ ### Tool results reach the screen
82
83
 
83
- A tool block may declare `forwardResult: true`. On completion, the platform then delivers the
84
- tool's raw return value to the session's browser on the SDK's `toolCall` event (`result` field) —
85
- so the UI can render what the agent just did (the citation it found, the record it pulled up, the
86
- booking it made) in lockstep with the spoken answer. No polling, no key-threading, no model
87
- involvement: the correlation is platform-guaranteed and scoped to that one session's client.
84
+ Every successful tool call delivers its raw return value to the session's browser on the SDK's
85
+ `toolCall` event (`result` field, on `done`) — so the UI can render what the agent just did (the
86
+ citation it found, the record it pulled up, the booking it made) in lockstep with the spoken
87
+ answer. No flag, no polling, no key-threading, no model involvement: delivery to the invoking
88
+ session's own client is the same security context as the invocation itself (an RPC response), and
89
+ it's scoped to that one session.
88
90
 
89
- Opt in deliberately, per tool. The forwarded payload is the method's raw return — the same data
90
- the model seesso only enable it on tools whose returns are safe to render for the user in the
91
- call (no internal fields you wouldn't show on screen). Payloads over ~32KB serialized arrive as
92
- `resultTruncated: true` with no data — keep forwarded returns compact, or have the UI fetch big
93
- data itself. Failed calls never forward anything.
91
+ Consequence for authoring: **a tool's return value is user-visible by definition.** Return what
92
+ the user may see no internal fields, keys, or diagnostics you wouldn't put on screen (the same
93
+ discipline as agent-interface tools, whose results render in chat). Payloads over ~32KB serialized
94
+ arrive as `resultTruncated: true` with no data — keep returns compact, or have the UI fetch big
95
+ data itself. Failed calls deliver nothing to the client (the model gets the `{ error }` and speaks
96
+ a decline).
94
97
 
95
98
  For backend-side correlation (writing results to a table keyed by the call, custom channels), the
96
99
  method itself can read `session.voiceSessionId` / `session.visitorId` from the agent SDK
97
100
  (`import { session } from '@mindstudio-ai/agent'`) — the same id the browser holds as
98
101
  `session.sessionId`, guaranteed by the platform rather than echoed by the model.
99
102
 
103
+ ### Client tools: actions that happen on screen (`target: "client"`)
104
+
105
+ A tool whose effect belongs in the browser — open the verification sheet, navigate to a page,
106
+ highlight a record — is declared with `target: "client"` instead of a `method`:
107
+
108
+ ```json
109
+ {
110
+ "target": "client",
111
+ "name": "showVerification",
112
+ "description": "tools/showVerification.md",
113
+ "inputSchema": { "type": "object", "properties": { "reason": { "type": "string" } } }
114
+ }
115
+ ```
116
+
117
+ The platform never touches the backend for these: the agent's invocation is delivered to the
118
+ session's browser, the app's registered handler runs, and the handler's **return value goes back
119
+ to the agent as the tool result** — a real request/response, so the agent knows the sheet
120
+ actually opened (or that the user dismissed it) and speaks accordingly. Rules:
121
+
122
+ - `name` instead of `method`; must not collide with any backend method id. No latency class —
123
+ the agent holds the turn while the browser responds (up to ~30s, then a timeout error).
124
+ - `inputSchema` is authored inline (an object schema) — there's no method contract to derive
125
+ it from. Keep it small; these are UI directives, not data payloads.
126
+ - The frontend must register a handler, or invocations fail as `unhandled_client_tool`:
127
+
128
+ ```js
129
+ session.registerClientTool('showVerification', async ({ reason }) => {
130
+ openVerifySheet(reason);
131
+ return { opened: true }; // what the agent hears back
132
+ });
133
+ ```
134
+
135
+ - One client tool runs at a time per session; the description should tell the agent when to use
136
+ it and what to say while it's on screen. Throwing from the handler (or returning nothing)
137
+ becomes an error/ack the agent can speak around.
138
+ - The progressive-auth pattern above is the canonical use: make the verification sheet a client
139
+ tool and the agent opens it deliberately instead of the frontend inferring it from tool events.
140
+
100
141
  ### Tool descriptions say results out loud
101
142
 
102
143
  Follow the agent-interface principles for tool descriptions (when to use and when not, parameter
@@ -124,24 +165,45 @@ caller.
124
165
 
125
166
  ### Choosing the model
126
167
 
127
- Two shapes, one `model` field:
128
-
129
- - **Native speech-to-speech** (`{"model": ..., "voice": ...}`) — one realtime model hears and
130
- speaks. Lowest latency, most natural prosody, hears tone and hesitation. The default for
131
- personality-forward, conversational apps.
132
- - **Cascaded** (`{"llm": ..., "stt": ..., "tts": ..., "voice": ...}`) — streaming transcription
133
- into any chat model in the catalog, streaming speech out. Slightly higher latency, but the brain
134
- can be *any* chat model — the right choice when the app's reasoning demands a specific model, or
135
- when the agent interface already uses one and the voice should think identically. The blessed
136
- streaming pairing is `"stt": "deepgram-nova-3", "tts": "cartesia-sonic-3"` — the lowest-latency
137
- combination the platform wires; prefer it unless there's a reason not to (ElevenLabs TTS,
138
- `"tts": "elevenlabs-tts"`, is also wired when its voice library fits better). One nuance: cascaded
139
- engines speak the `greeting` verbatim (they have a real TTS); speech-to-speech engines have the
140
- model say it, so it may paraphrase slightly.
141
-
142
- Ask `askMindStudioSdk` for available ids realtime, transcription, and speech models are separate
143
- catalogs, and MindStudio ids don't match vendor ids, so treat ids in this document as illustrative.
144
- Voice ids are model-specific; query for those too. The user's UI has a picker for changing the model
168
+ Two shapes, one `model` field. **Use native speech-to-speech unless the user specifically asks
169
+ for a cascaded pipeline** — one realtime model hears and speaks: lowest latency, most natural
170
+ prosody, hears tone and hesitation.
171
+
172
+ **Native** (`{"model": ..., "voice": ...}`) — **default to `gpt-realtime-2.1` with voice
173
+ `marin`.**
174
+
175
+ - `gpt-realtime-2.1` — the default. Voices: `marin` (default), `cedar`, `alloy`, `ash`,
176
+ `ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`.
177
+ - `gpt-realtime-2.1-mini` — the same family, lighter; same voices.
178
+ - `gemini-2.5-flash-native-audio-preview-12-2025` the Gemini pick, with a large expressive
179
+ roster: `Puck` (default, upbeat), `Zephyr` (bright), `Charon` (informative), `Kore` (firm),
180
+ `Fenrir` (excitable), `Leda` (youthful), `Orus` (firm), `Aoede` (breezy), `Callirrhoe`
181
+ (easy-going), `Autonoe` (bright), `Enceladus` (breathy), `Iapetus` (clear), `Umbriel`
182
+ (easy-going), `Algieba` (smooth), `Despina` (smooth), `Erinome` (clear), `Algenib` (gravelly),
183
+ `Rasalgethi` (informative), `Laomedeia` (upbeat), `Achernar` (soft), `Alnilam` (firm),
184
+ `Schedar` (even), `Gacrux` (mature), `Pulcherrima` (forward), `Achird` (friendly),
185
+ `Zubenelgenubi` (casual), `Vindemiatrix` (gentle), `Sadachbia` (lively), `Sadaltager`
186
+ (knowledgeable), `Sulafat` (warm).
187
+ - `gemini-3.1-flash-live-preview` — newer Gemini, same voices as 2.5, but currently can't speak
188
+ an opening greeting or take mid-call prompt updates (a plugin limitation expected to resolve
189
+ upstream) — prefer 2.5 until then.
190
+ - `grok-voice-think-fast-2.0` — a distinct personality register. Voices: `eve` (default),
191
+ `altair`, `ara`, `atlas`, `aurora`, `carina`, `castor`, `celeste`, `cosmo`, `helios`, `helix`,
192
+ `iris`, `kepler`, `leo`, `liora`, `lumen`, `luna`, `lux`, `naksh`, `orion`, `perseus`, `rex`,
193
+ `rigel`, `sal`, `sirius`, `ursa`, `zagan`, `zenith`.
194
+
195
+ **Cascaded** (`{"llm": ..., "stt": ..., "tts": ..., "voice": ...}`) — streaming transcription
196
+ into any chat model in the catalog, streaming speech out. Slightly higher latency; reach for it
197
+ only when the user wants it or the app's reasoning demands a specific chat model (e.g. the agent
198
+ interface already uses one and the voice should think identically). Slots: `stt` is
199
+ `deepgram-nova-3`; `tts` is `cartesia-sonic-3` (voices are per-account Cartesia UUIDs — see
200
+ play.cartesia.ai) or `elevenlabs-tts` (the account's ElevenLabs voice library); `llm` is any chat
201
+ model — ask `askMindStudioSdk` for chat model ids. One nuance: cascaded engines speak the
202
+ `greeting` verbatim (they have a real TTS); speech-to-speech engines have the model say it, so it
203
+ may paraphrase slightly.
204
+
205
+ The model and voice ids above are current and maintained with the platform — use them as written
206
+ (they are MindStudio ids, not vendor ids). The user's UI has a picker for changing the model
145
207
  later, so validate only when you set it.
146
208
 
147
209
  ### Seeding from an existing agent
@@ -204,13 +266,15 @@ session.on('stateChange', (state) => { }); // on() returns an unsubscri
204
266
  // far (never a delta) — render by upserting on segmentId, not appending.
205
267
  session.on('transcript', ({ role, segmentId, text, final }) => { });
206
268
 
207
- // status: 'running' | 'done' | 'failed'. Tools declared with `forwardResult: true`
208
- // carry their return value in `result` on 'done' (or `resultTruncated: true` if >~32KB).
269
+ // status: 'running' | 'done' | 'failed'. Every 'done' carries the tool's raw
270
+ // return value in `result` (or `resultTruncated: true` if >~32KB serialized).
209
271
  session.on('toolCall', ({ method, status, result }) => { });
210
272
  session.on('error', (err) => { });
211
273
 
212
274
  session.mute(); session.unmute(); session.isMuted;
213
275
  session.sendText('123 Main Street'); // inject text into the live conversation
276
+ await session.refreshIdentity(); // after in-app verification — upgrade the
277
+ // live session anonymous → signed-in in place
214
278
  session.end();
215
279
  ```
216
280
 
@@ -277,7 +341,7 @@ export async function callMeAboutMyOrder(input: { phone: string }) {
277
341
  session, not the phone). Omitted/false → anonymous call; role-gated tools decline.
278
342
  System/cron invocations have no human identity and always run anonymously.
279
343
  - **Production needs a dedicated phone number.** The app owner attaches one ($1/month) via the
280
- dashboard or `mindstudio-prod voice numbers` (see "Managing the phone side from the CLI"
344
+ dashboard or `mindstudio-prod voice numbers` (see "The voice CLI"
281
345
  below) — it becomes the caller ID for every call, in dev sessions too, so users always see
282
346
  the same number. Without one, deployed calls throw `phone_out_requires_dedicated_number`, and
283
347
  dev sessions fall back to a shared platform test number that varies per call (tighter limits
@@ -338,7 +402,7 @@ wrong one wherever the agent's tools can move money, reveal sensitive records, o
338
402
  destructive actions. It lives in the interface config deliberately: enabling it is a code
339
403
  change, visible in review and auditable via deploys, not a dashboard toggle.
340
404
 
341
- ## Managing the phone side from the CLI
405
+ ## The voice CLI
342
406
 
343
407
  The `mindstudio-prod voice` family covers numbers, the call log, and voice policy:
344
408
 
@@ -377,7 +441,7 @@ section.
377
441
  name: Front Desk
378
442
  description: Books appointments and answers questions by voice.
379
443
  type: interface/voice
380
- model: {"model": "gpt-realtime-mini", "voice": "marin"}
444
+ model: {"model": "gpt-realtime-2.1", "voice": "marin"}
381
445
  turnDetection: {"eagerness": "medium"}
382
446
  greeting: Hey! I can help you book, reschedule, or answer questions — what do you need?
383
447
  ---
@@ -435,7 +499,7 @@ The top-level key must match the interface type (`voice`):
435
499
  "voice": {
436
500
  "name": "Front Desk",
437
501
  "description": "Books appointments and answers questions by voice.",
438
- "model": "gpt-realtime-mini",
502
+ "model": "gpt-realtime-2.1",
439
503
  "voice": "marin",
440
504
  "turnDetection": { "eagerness": "medium" },
441
505
  "greeting": "Hey! I can help you book, reschedule, or answer questions — what do you need?",
@@ -537,3 +601,30 @@ frontend or the agent interface. Anonymous sessions (when allowed) have no user
537
601
  gated methods reject, and the caller's history is scoped to their browser's visitor identity.
538
602
  That's why role restrictions belong in the tool descriptions — the agent should decline in
539
603
  character, not relay a rejection.
604
+
605
+ ### Progressive auth: verify mid-call without dropping the conversation
606
+
607
+ The best pattern for apps that allow anonymous sessions (`requireUser: false`): let visitors
608
+ explore by voice, and verify only when they hit an account-bound action — without killing the
609
+ live call. Four pieces, all platform rails:
610
+
611
+ 1. **Account-gated tools return a standard not-verified shape** instead of doing the work:
612
+ `{ verified: false, message: 'The caller is not verified. Offer to verify them before sharing
613
+ account details.' }`. The agent speaks the offer in character (reinforce tone in the system
614
+ prompt's verification section). Check with the agent SDK's `auth.userId` inside the method.
615
+ 2. **The frontend opens its verification sheet off the same signal.** It already receives every
616
+ tool's `toolCall` event (and the tool's return in `result`) — when an account tool fires (or
617
+ returns `verified: false`) while the app has no signed-in user, open the sheet.
618
+ 3. **The sheet runs the platform's auth rails** — `auth.sendSmsCode()` / `auth.verifySmsCode()`
619
+ (or the email pair) from `@mindstudio-ai/interface`. On success the app's session becomes the
620
+ verified user.
621
+ 4. **Hand the verified session back to the live call**: `await session.refreshIdentity()`. The
622
+ platform upgrades the running voice session in place — subsequent tool calls carry the user's
623
+ identity and roles, and the agent's Current User context refreshes — no teardown, no lost
624
+ conversation. (Phone calls don't need this: they verify through the agent's built-in flow.)
625
+
626
+ `refreshIdentity()` is upgrade-only (anonymous → signed-in; an already-identified session rejects
627
+ with `already_identified`) and requires the session to have been started by this same browser. If
628
+ it fails, ending and restarting the session is the graceful fallback. The chat sibling for agent
629
+ interfaces is `claimThread(threadId)` — anonymous threads become unreachable after login until
630
+ claimed.
@@ -13,6 +13,8 @@ When the content you need to test is behind authentication, use the `setupBrowse
13
13
 
14
14
  If you need to test the login/signup flow itself (e.g., verifying the UI, error states, or the verification code input), navigate it manually: use `remy@mindstudio.ai` for email and `+15551234567` for phone. In the dev environment, verification codes are bypassed for this email and any 555-prefixed phone number — enter any 6-digit code (e.g., `123456`).
15
15
 
16
+ To test as a **signed-out visitor** (public pages, landing/join links), call `setupBrowser` with NO `auth` — it clears the auth cookie and reloads at the given path, giving you a clean unauthenticated session. Combine with `navigate` + `fresh: true` when you need a fresh-document view of an entry page mid-run.
17
+
16
18
  ## Browser Commands
17
19
 
18
20
  Your session always starts on the app root / in a logged out/unauthenticated state. Use `setupBrowser` to authenticate before testing protected pages.
@@ -40,13 +42,43 @@ Note: the snapshot concatenates inline text and strips whitespace. If you need t
40
42
  - `type`: Type text into an input. Characters appear one at a time. Set `clear: true` to clear the field first.
41
43
  - `select`: Select a dropdown option by text. Target the `<select>` element, set `option` to the option text.
42
44
  - `wait`: Wait for an element to appear (polls every 100ms, default 5s timeout). Also waits for network to settle after the element is found.
43
- - `navigate`: Navigate to a new URL within the app. Waits for the new page to load before continuing with subsequent steps. Use this instead of evaluate with `window.location.href` when you need to navigate and then continue interacting with the new page. Steps after navigate execute on the new page automatically.
45
+ - `navigate`: Navigate to a new URL within the app. Waits for the route to load before continuing with subsequent steps. Use this instead of evaluate with `window.location.href` when you need to navigate and then continue interacting with the new page. Steps after navigate execute on the new page automatically. Same-origin navigation is a soft in-app route change (like clicking a link in an SPA — in-memory app state survives); set `fresh: true` to force a real full page load with a fresh document instead. Use `fresh: true` when the test is about what a user sees on *entry* — landing pages, join/invite links, "what does a signed-out visitor see" — where reusing the SPA's in-memory state would test the wrong thing. The result reports the URL the page actually landed on, so if the app redirected you (e.g. an auth wall bounced you off a public page), you'll see the real destination — check it instead of assuming the navigation stuck.
44
46
  - `evaluate`: Run arbitrary JavaScript in the page and return the result.
45
47
  - `styles`: Read computed CSS styles from page elements. Pass a `properties` array with camelCase CSS property names (e.g., `["backgroundColor", "borderRadius", "fontSize"]`). Omit `properties` for a default set covering colors, typography, spacing, borders, shadows, dimensions, and layout. Uses the same targeting as click/type (ref, text, role, label, selector). Omit the target to get styles for all elements from the last snapshot.
46
48
  - `screenshotFullPage`: Take a screenshot of the whole page, top to bottom. Returns CDN url with full text analysis and dimensions. Use for overall composition or content past the fold.
47
49
  - `screenshotViewport`: Take a screenshot of the visible viewport. Returns CDN url with full text analysis and dimensions. To capture a specific section, set `scrollToSelector` (a CSS selector) — or `scrollY` (an absolute offset) — on this same step; it scrolls the target into view and captures it atomically, so you do NOT need a separate scroll step. Do not use if you can get what you need with other tools - only use when you need to visually see the viewport.
48
50
  - `setViewport`: Switch the browser between desktop and mobile rendering. Set `mode` to `"desktop"` or `"mobile"`. Mobile emulates a phone (390-wide, touch, device pixel ratio 2); desktop is the standard wide viewport. This reloads the page so media queries, responsive layouts, and `matchMedia` re-evaluate — the reload clears in-page state, so switch before you set up the state you want to inspect. The mode persists across navigations within a run. Each run starts in the app's default mode, so only use this when you need to check the other one.
49
51
 
52
+ ### Voice interfaces
53
+
54
+ Apps with a voice interface are testable end to end — the UI layer included. The sandbox browser
55
+ auto-grants a (silent) microphone, and while a session is live the SDK publishes a handle at
56
+ `window.__MS_VOICE__` so you can converse by text: the agent treats injected text exactly like
57
+ user speech (interrupts and replies), backend tools run for real, and client tools render their
58
+ real UI (cards, sheets) in the page.
59
+
60
+ The loop:
61
+
62
+ 1. Start a session through the app's real UI — `click` its voice affordance (orb/button). No mic
63
+ prompt appears. Then `wait` briefly and confirm the session is live:
64
+ `evaluate: window.__MS_VOICE__?.state` (undefined means no session started — report that,
65
+ don't improvise).
66
+ 2. Speak by injection: `evaluate: window.__MS_VOICE__.sendText("I'd like to book Tuesday at 2")`.
67
+ 3. Give the agent a few seconds to respond (replies are generated speech — slower than chat).
68
+ `wait` for the UI you expect (client-tool cards appear via the app's real handlers), and read
69
+ the conversation: `evaluate: window.__MS_VOICE__.transcript` (one entry per utterance, both
70
+ sides, `final` marks settled ones) and `window.__MS_VOICE__.toolCalls` (which tools ran;
71
+ `done` entries carry the tool's return value).
72
+ 4. Verify visuals with `screenshotViewport` like any other flow.
73
+ 5. Read `transcript`/`toolCalls` BEFORE ending — then `evaluate: window.__MS_VOICE__.end()` (the
74
+ handle is removed when the session ends).
75
+
76
+ Voice sessions are the most expensive thing you can run — real voice-model minutes are metered,
77
+ and the agent speaks its replies out loud even when you type at it. Keep voice tests short and
78
+ purposeful: a handful of turns that exercise the target behavior, then end the session. What you
79
+ cannot test is the audio layer itself (mishearing, interruptions, pronunciation) — never attempt
80
+ to simulate audio; report that scope limit instead.
81
+
50
82
  ### Element targeting (tried in order)
51
83
 
52
84
  1. `ref`: From the last snapshot. Most reliable.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@mindstudio-ai/remy",
3
- "version": "0.1.265",
3
+ "version": "0.1.266",
4
4
  "description": "Remy coding agent",
5
5
  "repository": {
6
6
  "type": "git",