@mindstudio-ai/remy 0.1.265 → 0.1.266
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +12 -2
- package/dist/headless.d.ts +10 -1
- package/dist/headless.js +140 -9
- package/dist/index.js +141 -9
- package/dist/prompt/skills/agentInterfaces.md +24 -0
- package/dist/prompt/skills/voiceInterfaces.md +128 -37
- package/dist/subagents/browserAutomation/prompt.md +33 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -265,6 +265,8 @@ The headless IPC protocol uses request correlation and a unified response patter
|
|
|
265
265
|
- System events (lifecycle, shutdown) never have a `requestId`
|
|
266
266
|
- Every command ends with exactly one `completed` event: `{event:"completed", requestId, success, error?}`
|
|
267
267
|
- Messages sent while a turn is running are queued. When the turn ends, all contiguous queued user messages and background results are delivered together as **one merged turn**: the first queued message's `requestId` becomes the turn's primary id (stamped on `turn_started` and all streaming events), each absorbed message echoes its own `user_message` with its original `requestId` and `queued: true`, and at turn end the primary `completed` is emitted first, followed immediately by one `completed {…same outcome, absorbed: true}` per other absorbed `requestId`. Automated-action (`@@automated::…@@`) messages and chain steps never merge — they always run one turn each.
|
|
268
|
+
- A queued **user** message can be promoted to ASAP delivery via `setQueuedDelivery`. ASAP items are pulled into the **running** turn at its next tool boundary (injected as plain user messages — no abort, no restart), echo `user_message` with `queued: true` and their own `requestId`, and get a `{…, absorbed: true}` completed with the turn's outcome at turn end. Promotion deliberately jumps ahead of anything else in the queue, including chain steps. If the turn ends before injection, the tag is ignored and the item drains in normal FIFO order.
|
|
269
|
+
- Background tool completions have two delivery classes (per-tool `backgroundNotify` on the tool definition). `wake` (default): the result is queued as a `background_results` message and may start a turn when the agent is idle. `passive` (e.g. `specSync`): the result never enters the queue and never wakes the agent — it parks in a persisted holding pen and rides the next real turn as a hidden `background_results` entry (so it never appears in `queuedMessages`, never affects queue-derived busy state, and never triggers resume-on-restart). Both classes emit `tool_background_complete` immediately.
|
|
268
270
|
- The caller distinguishes command responses from system events with a single check: `if (msg.requestId)`
|
|
269
271
|
|
|
270
272
|
This enables a simple promise-based RPC layer: send a command with a unique ID, store a pending promise keyed by that ID, resolve it when you see `completed` with the matching ID.
|
|
@@ -327,6 +329,14 @@ Cancel pending **queued** messages without touching the in-flight turn. Omit `id
|
|
|
327
329
|
{"action": "cancelQueued", "requestId": "r6", "id": "r2"}
|
|
328
330
|
```
|
|
329
331
|
|
|
332
|
+
#### `setQueuedDelivery`
|
|
333
|
+
|
|
334
|
+
Change a queued message's delivery semantics: `"asap"` (inject into the running turn at its next tool boundary) or `"afterTurn"` (default — wait for the turn to end). `id` is the `requestId` of a queued message. Only plain queued **user** messages qualify; automated-action messages and chain/background items respond `completed(success:false, error:"message not found or not promotable")` — as does an item already consumed by the running turn. On success a `queue_changed` event carries the updated snapshot (each item's `delivery` field rides on it).
|
|
335
|
+
|
|
336
|
+
```json
|
|
337
|
+
{"action": "setQueuedDelivery", "requestId": "r7", "id": "r2", "delivery": "asap"}
|
|
338
|
+
```
|
|
339
|
+
|
|
330
340
|
#### `clear`
|
|
331
341
|
|
|
332
342
|
Clear conversation history and delete the session file.
|
|
@@ -353,7 +363,7 @@ Events are emitted as newline-delimited JSON. Command responses include `request
|
|
|
353
363
|
|-------|--------|-------------|
|
|
354
364
|
| `ready` | | Headless mode initialized, ready for input |
|
|
355
365
|
| `session_restored` | `messageCount` | Previous session loaded |
|
|
356
|
-
| `queue_changed` | `queuedMessages` | Queue contents changed (any push/shift/drain/cancel). Carries the full current snapshot — an empty array when the queue drains. The live queue-state signal; for the initial snapshot on connect/reconnect, read `queuedMessages` from `get_history`. |
|
|
366
|
+
| `queue_changed` | `queuedMessages` | Queue contents changed (any push/shift/drain/cancel/retag). Carries the full current snapshot — an empty array when the queue drains. Items carry `delivery: "asap"` when promoted (absent = after-turn). The live queue-state signal; for the initial snapshot on connect/reconnect, read `queuedMessages` from `get_history`. |
|
|
357
367
|
| `stopping` | | Shutdown initiated |
|
|
358
368
|
| `stopped` | | Shutdown complete |
|
|
359
369
|
|
|
@@ -365,7 +375,7 @@ All command responses include the `requestId` from the originating command.
|
|
|
365
375
|
|-------|--------|-------------|
|
|
366
376
|
| `text` | `text`, `parentToolId?` | Streaming text chunk |
|
|
367
377
|
| `thinking` | `text`, `parentToolId?` | Agent's internal reasoning |
|
|
368
|
-
| `user_message` | `text`, `attachments?`, `queued?` | Echo of a user message entering the turn. Queue-delivered messages carry `queued: true` and their own original `requestId` (a merged turn emits one per absorbed message); idle sends echo with the turn's requestId and no `queued` flag. |
|
|
378
|
+
| `user_message` | `text`, `attachments?`, `queued?`, `hidden?` | Echo of a user message entering the turn. Queue-delivered messages (including ASAP items injected mid-turn) carry `queued: true` and their own original `requestId` (a merged turn emits one per absorbed message); idle sends echo with the turn's requestId and no `queued` flag. `hidden: true` marks internal entries (e.g. passive background results) that should not render. |
|
|
369
379
|
| `tool_start` | `id`, `name`, `input`, `partial?`, `parentToolId?` | Tool execution started. `partial: true` means more `tool_start` events will follow for this id (progressive input streaming). |
|
|
370
380
|
| `tool_input_delta` | `id`, `name`, `result`, `parentToolId?` | Progressive tool content (streaming tools only) |
|
|
371
381
|
| `tool_done` | `id`, `name`, `result`, `isError`, `parentToolId?` | Tool execution completed |
|
package/dist/headless.d.ts
CHANGED
|
@@ -70,6 +70,15 @@ declare class HeadlessSession {
|
|
|
70
70
|
* to .remy-stats.json so queued work survives process restarts.
|
|
71
71
|
*/
|
|
72
72
|
private queue;
|
|
73
|
+
/**
|
|
74
|
+
* Holding pen for passive background results (tools with
|
|
75
|
+
* `backgroundNotify: 'passive'`, e.g. specSync). Deliberately outside the
|
|
76
|
+
* message queue: pen contents never initiate a turn, never latch the
|
|
77
|
+
* sandbox's queue-derived busy state, and never trigger resume-on-restart.
|
|
78
|
+
* Swept into the next real turn as a hidden background_results entry.
|
|
79
|
+
* Persisted to .remy-stats.json alongside the queue.
|
|
80
|
+
*/
|
|
81
|
+
private passivePen;
|
|
73
82
|
private pendingTools;
|
|
74
83
|
private earlyResults;
|
|
75
84
|
private pendingBlockUpdates;
|
|
@@ -87,7 +96,7 @@ declare class HeadlessSession {
|
|
|
87
96
|
private primaryOutcome;
|
|
88
97
|
/** Dispatch a simple (non-streaming) command: call handler, emit response + completed. */
|
|
89
98
|
private dispatchSimple;
|
|
90
|
-
/** Persist sessionStats + queue snapshot to .remy-stats.json. */
|
|
99
|
+
/** Persist sessionStats + queue snapshot + passive pen to .remy-stats.json. */
|
|
91
100
|
private persistStats;
|
|
92
101
|
/** Apply queued tool block updates to state.messages. Safe to call any time. */
|
|
93
102
|
private applyPendingBlockUpdates;
|
package/dist/headless.js
CHANGED
|
@@ -4099,7 +4099,7 @@ var BROWSER_TOOLS = [
|
|
|
4099
4099
|
"screenshotViewport",
|
|
4100
4100
|
"setViewport"
|
|
4101
4101
|
],
|
|
4102
|
-
description: 'snapshot: accessibility tree of the page (waits for network to settle). click: click an element (animated cursor, full event sequence). type: type text into input (one char at a time, works with React/Vue/Svelte). select: select a dropdown option by text. wait: wait for an element to appear (polls 100ms, waits for network). navigate: navigate to a URL within the app (waits for
|
|
4102
|
+
description: 'snapshot: accessibility tree of the page (waits for network to settle). click: click an element (animated cursor, full event sequence). type: type text into input (one char at a time, works with React/Vue/Svelte). select: select a dropdown option by text. wait: wait for an element to appear (polls 100ms, waits for network). navigate: navigate to a URL within the app (waits for the route, subsequent steps run on the new page; soft in-app route change by default \u2014 pass `fresh: true` for a real full page load; the result reports the URL actually landed on, so app redirects are visible). evaluate: run JS in the page. styles: read computed CSS styles from elements (pass properties array with camelCase names, or omit for defaults). screenshotFullPage: screenshot of the whole page top-to-bottom (returns a CDN url with dimensions and a written analysis). screenshotViewport: screenshot of just the visible viewport \u2014 pass `scrollToSelector` (or `scrollY`) on this step to scroll a section into view and capture it in one atomic step (no separate scroll needed). setViewport: switch the browser between desktop and mobile rendering (pass `mode`: "desktop" or "mobile"). Reloads the page so responsive layouts, media queries, and matchMedia re-evaluate \u2014 use it to QA mobile/responsive views.'
|
|
4103
4103
|
},
|
|
4104
4104
|
ref: {
|
|
4105
4105
|
type: "string",
|
|
@@ -4141,6 +4141,10 @@ var BROWSER_TOOLS = [
|
|
|
4141
4141
|
type: "string",
|
|
4142
4142
|
description: 'For navigate: the URL to navigate to (e.g., "/quiz", "/settings").'
|
|
4143
4143
|
},
|
|
4144
|
+
fresh: {
|
|
4145
|
+
type: "boolean",
|
|
4146
|
+
description: "For navigate: force a real full page load (fresh document) instead of a soft in-app route change. Use when testing what a user sees on entry \u2014 landing pages, join/invite links, signed-out views \u2014 where reusing the SPA\u2019s in-memory state would test the wrong thing."
|
|
4147
|
+
},
|
|
4144
4148
|
properties: {
|
|
4145
4149
|
type: "array",
|
|
4146
4150
|
items: { type: "string" },
|
|
@@ -5955,10 +5959,13 @@ ${readAsset(
|
|
|
5955
5959
|
</mindstudio_flavored_markdown_spec_docs>`;
|
|
5956
5960
|
var specSyncTool = {
|
|
5957
5961
|
backgroundOnly: true,
|
|
5962
|
+
// Fire-and-forget: completion never wakes the agent. The outcome rides the
|
|
5963
|
+
// next real turn as a hidden background_results note instead.
|
|
5964
|
+
backgroundNotify: "passive",
|
|
5958
5965
|
definition: {
|
|
5959
5966
|
clearable: false,
|
|
5960
5967
|
name: "specSync",
|
|
5961
|
-
description: "Reconcile the spec to bring it in line with code changes you have made. Provide a brief, bulleted list of what changed and why; it finds the affected spec sections and updates them to match. Always runs in the background \u2014 it
|
|
5968
|
+
description: "Reconcile the spec to bring it in line with code changes you have made. Provide a brief, bulleted list of what changed and why; it finds the affected spec sections and updates them to match. Always runs in the background and completes silently \u2014 do not wait for it; its outcome appears as an automated note at the start of a later turn.",
|
|
5962
5969
|
inputSchema: {
|
|
5963
5970
|
type: "object",
|
|
5964
5971
|
properties: {
|
|
@@ -7627,6 +7634,7 @@ async function runTurn(params) {
|
|
|
7627
7634
|
onboardingState,
|
|
7628
7635
|
signal,
|
|
7629
7636
|
onEvent,
|
|
7637
|
+
takeSteering,
|
|
7630
7638
|
resolveExternalTool,
|
|
7631
7639
|
requestId,
|
|
7632
7640
|
toolRegistry,
|
|
@@ -7661,7 +7669,7 @@ async function runTurn(params) {
|
|
|
7661
7669
|
onEvent({ type: "error", error: "Empty message" });
|
|
7662
7670
|
return;
|
|
7663
7671
|
}
|
|
7664
|
-
|
|
7672
|
+
const appendEntry = (entry) => {
|
|
7665
7673
|
const hasAttachments = (entry.attachments?.length ?? 0) > 0;
|
|
7666
7674
|
const userMsg = { role: "user", content: entry.text };
|
|
7667
7675
|
if (entry.hidden) {
|
|
@@ -7684,6 +7692,9 @@ async function runTurn(params) {
|
|
|
7684
7692
|
...entry.requestId && { requestId: entry.requestId },
|
|
7685
7693
|
...entry.queued && { queued: true }
|
|
7686
7694
|
});
|
|
7695
|
+
};
|
|
7696
|
+
for (const entry of keptEntries) {
|
|
7697
|
+
appendEntry(entry);
|
|
7687
7698
|
}
|
|
7688
7699
|
const isFirstMessage = state.messages.filter((m) => m.role === "user").length === 1;
|
|
7689
7700
|
const STATUS_EXCLUDED_TOOLS = /* @__PURE__ */ new Set([
|
|
@@ -8200,6 +8211,18 @@ async function runTurn(params) {
|
|
|
8200
8211
|
isToolError: r.isError
|
|
8201
8212
|
});
|
|
8202
8213
|
}
|
|
8214
|
+
if (takeSteering && !signal?.aborted) {
|
|
8215
|
+
const injected = (await takeSteering()).filter(
|
|
8216
|
+
(e) => e.text.trim().length > 0 || (e.attachments?.length ?? 0) > 0
|
|
8217
|
+
);
|
|
8218
|
+
if (injected.length > 0) {
|
|
8219
|
+
for (const entry of injected) {
|
|
8220
|
+
appendEntry(entry);
|
|
8221
|
+
keptEntries.push(entry);
|
|
8222
|
+
}
|
|
8223
|
+
saveSession(state);
|
|
8224
|
+
}
|
|
8225
|
+
}
|
|
8203
8226
|
if (signal?.aborted) {
|
|
8204
8227
|
onEvent({ type: "turn_cancelled" });
|
|
8205
8228
|
saveSession(state);
|
|
@@ -8375,13 +8398,24 @@ function loadQueue() {
|
|
|
8375
8398
|
}
|
|
8376
8399
|
return [];
|
|
8377
8400
|
}
|
|
8378
|
-
function
|
|
8401
|
+
function loadPassiveResults() {
|
|
8402
|
+
try {
|
|
8403
|
+
const stats = JSON.parse(readFileSync2(STATS_FILE, "utf-8"));
|
|
8404
|
+
if (Array.isArray(stats.passiveResults)) {
|
|
8405
|
+
return stats.passiveResults;
|
|
8406
|
+
}
|
|
8407
|
+
} catch {
|
|
8408
|
+
}
|
|
8409
|
+
return [];
|
|
8410
|
+
}
|
|
8411
|
+
function writeStats(stats, queue, passiveResults) {
|
|
8379
8412
|
try {
|
|
8380
8413
|
writeFileSync2(
|
|
8381
8414
|
STATS_FILE,
|
|
8382
8415
|
JSON.stringify({
|
|
8383
8416
|
...stats,
|
|
8384
|
-
queue
|
|
8417
|
+
queue,
|
|
8418
|
+
passiveResults
|
|
8385
8419
|
})
|
|
8386
8420
|
);
|
|
8387
8421
|
} catch {
|
|
@@ -8445,6 +8479,21 @@ var MessageQueue = class {
|
|
|
8445
8479
|
}
|
|
8446
8480
|
return removed;
|
|
8447
8481
|
}
|
|
8482
|
+
/**
|
|
8483
|
+
* Change a queued item's delivery semantics, keyed by its command
|
|
8484
|
+
* requestId. Fires onChange (→ persist + queue_changed) on success.
|
|
8485
|
+
* Returns the item, or undefined if no queued item matches (e.g. it was
|
|
8486
|
+
* already consumed).
|
|
8487
|
+
*/
|
|
8488
|
+
setDelivery(id, delivery) {
|
|
8489
|
+
const item = this.items.find((it) => it.command.requestId === id);
|
|
8490
|
+
if (!item) {
|
|
8491
|
+
return void 0;
|
|
8492
|
+
}
|
|
8493
|
+
item.delivery = delivery;
|
|
8494
|
+
this.onChange?.();
|
|
8495
|
+
return item;
|
|
8496
|
+
}
|
|
8448
8497
|
/** Copy of current queue contents (for surfacing on events). */
|
|
8449
8498
|
snapshot() {
|
|
8450
8499
|
return [...this.items];
|
|
@@ -8559,6 +8608,15 @@ var HeadlessSession = class {
|
|
|
8559
8608
|
* to .remy-stats.json so queued work survives process restarts.
|
|
8560
8609
|
*/
|
|
8561
8610
|
queue;
|
|
8611
|
+
/**
|
|
8612
|
+
* Holding pen for passive background results (tools with
|
|
8613
|
+
* `backgroundNotify: 'passive'`, e.g. specSync). Deliberately outside the
|
|
8614
|
+
* message queue: pen contents never initiate a turn, never latch the
|
|
8615
|
+
* sandbox's queue-derived busy state, and never trigger resume-on-restart.
|
|
8616
|
+
* Swept into the next real turn as a hidden background_results entry.
|
|
8617
|
+
* Persisted to .remy-stats.json alongside the queue.
|
|
8618
|
+
*/
|
|
8619
|
+
passivePen = [];
|
|
8562
8620
|
// External tool bridge
|
|
8563
8621
|
pendingTools = /* @__PURE__ */ new Map();
|
|
8564
8622
|
earlyResults = /* @__PURE__ */ new Map();
|
|
@@ -8594,6 +8652,7 @@ var HeadlessSession = class {
|
|
|
8594
8652
|
this.persistStats();
|
|
8595
8653
|
this.emit("queue_changed", { queuedMessages: this.queue.snapshot() });
|
|
8596
8654
|
});
|
|
8655
|
+
this.passivePen = loadPassiveResults();
|
|
8597
8656
|
if (resumed) {
|
|
8598
8657
|
this.emit("session_restored", {
|
|
8599
8658
|
messageCount: this.state.messages.length,
|
|
@@ -8705,10 +8764,10 @@ var HeadlessSession = class {
|
|
|
8705
8764
|
//////////////////////////////////////////////////////////////////////////////
|
|
8706
8765
|
// Stats + queue persistence
|
|
8707
8766
|
//////////////////////////////////////////////////////////////////////////////
|
|
8708
|
-
/** Persist sessionStats + queue snapshot to .remy-stats.json. */
|
|
8767
|
+
/** Persist sessionStats + queue snapshot + passive pen to .remy-stats.json. */
|
|
8709
8768
|
persistStats() {
|
|
8710
8769
|
this.sessionStats.updatedAt = Date.now();
|
|
8711
|
-
writeStats(this.sessionStats, this.queue.snapshot());
|
|
8770
|
+
writeStats(this.sessionStats, this.queue.snapshot(), this.passivePen);
|
|
8712
8771
|
}
|
|
8713
8772
|
//////////////////////////////////////////////////////////////////////////////
|
|
8714
8773
|
// Background completions (tool-block mutation; message delivery via queue)
|
|
@@ -8770,10 +8829,12 @@ var HeadlessSession = class {
|
|
|
8770
8829
|
}
|
|
8771
8830
|
}
|
|
8772
8831
|
onBackgroundComplete = (toolCallId, name, result, subAgentMessages) => {
|
|
8832
|
+
const notify = getToolByName(name)?.backgroundNotify ?? "wake";
|
|
8773
8833
|
this.pendingBlockUpdates.push({ toolCallId, result, subAgentMessages });
|
|
8774
8834
|
log16.info("Background complete", {
|
|
8775
8835
|
toolCallId,
|
|
8776
8836
|
name,
|
|
8837
|
+
notify,
|
|
8777
8838
|
requestId: this.currentRequestId
|
|
8778
8839
|
});
|
|
8779
8840
|
this.onEvent({
|
|
@@ -8782,6 +8843,14 @@ var HeadlessSession = class {
|
|
|
8782
8843
|
name,
|
|
8783
8844
|
result
|
|
8784
8845
|
});
|
|
8846
|
+
if (notify === "passive") {
|
|
8847
|
+
this.passivePen.push({ toolCallId, name, result });
|
|
8848
|
+
this.persistStats();
|
|
8849
|
+
if (!this.running) {
|
|
8850
|
+
this.applyPendingBlockUpdates();
|
|
8851
|
+
}
|
|
8852
|
+
return;
|
|
8853
|
+
}
|
|
8785
8854
|
this.queue.push({
|
|
8786
8855
|
command: {
|
|
8787
8856
|
action: "message",
|
|
@@ -8850,7 +8919,10 @@ var HeadlessSession = class {
|
|
|
8850
8919
|
...e.attachments && { attachments: e.attachments },
|
|
8851
8920
|
// Queue-delivered entries are flagged so the frontend renders the
|
|
8852
8921
|
// echo (idle sends are rendered optimistically instead).
|
|
8853
|
-
...e.queued && { queued: true }
|
|
8922
|
+
...e.queued && { queued: true },
|
|
8923
|
+
// Hidden entries (e.g. the passive background_results sweep) are
|
|
8924
|
+
// flagged so the sandbox/frontend suppress the bubble.
|
|
8925
|
+
...e.hidden && { hidden: true }
|
|
8854
8926
|
},
|
|
8855
8927
|
// A merged turn emits one user_message per absorbed entry — each
|
|
8856
8928
|
// carries its own original requestId, not the turn's.
|
|
@@ -9160,6 +9232,37 @@ var HeadlessSession = class {
|
|
|
9160
9232
|
this.completedEmitted = false;
|
|
9161
9233
|
this.lastCompleted = null;
|
|
9162
9234
|
this.turnStart = Date.now();
|
|
9235
|
+
if (this.passivePen.length > 0) {
|
|
9236
|
+
const swept = this.passivePen.splice(0);
|
|
9237
|
+
this.persistStats();
|
|
9238
|
+
entries.unshift({
|
|
9239
|
+
text: buildBackgroundResultsMessage(swept),
|
|
9240
|
+
hidden: true
|
|
9241
|
+
});
|
|
9242
|
+
}
|
|
9243
|
+
const consumedRids = [];
|
|
9244
|
+
const takeSteering = async () => {
|
|
9245
|
+
const items = this.queue.removeWhere(
|
|
9246
|
+
(it) => it.source === "user" && it.delivery === "asap" && !this.isDrainBarrier(it)
|
|
9247
|
+
);
|
|
9248
|
+
const steered = [];
|
|
9249
|
+
for (const it of items) {
|
|
9250
|
+
const attachments = it.command.attachments;
|
|
9251
|
+
const attachmentHeader = await this.persistEntryAttachments(attachments);
|
|
9252
|
+
const rid = it.command.requestId;
|
|
9253
|
+
if (rid) {
|
|
9254
|
+
consumedRids.push(rid);
|
|
9255
|
+
}
|
|
9256
|
+
steered.push({
|
|
9257
|
+
text: it.command.text ?? "",
|
|
9258
|
+
attachments,
|
|
9259
|
+
attachmentHeader,
|
|
9260
|
+
requestId: rid,
|
|
9261
|
+
queued: true
|
|
9262
|
+
});
|
|
9263
|
+
}
|
|
9264
|
+
return steered;
|
|
9265
|
+
};
|
|
9163
9266
|
await this.runForcedCompactionIfNeeded(requestId);
|
|
9164
9267
|
try {
|
|
9165
9268
|
await runTurn({
|
|
@@ -9173,6 +9276,7 @@ var HeadlessSession = class {
|
|
|
9173
9276
|
requestId,
|
|
9174
9277
|
signal: this.currentAbort.signal,
|
|
9175
9278
|
onEvent: this.onEvent,
|
|
9279
|
+
takeSteering,
|
|
9176
9280
|
resolveExternalTool: this.resolveExternalTool,
|
|
9177
9281
|
toolRegistry: this.toolRegistry,
|
|
9178
9282
|
onBackgroundComplete: this.onBackgroundComplete
|
|
@@ -9202,7 +9306,7 @@ var HeadlessSession = class {
|
|
|
9202
9306
|
});
|
|
9203
9307
|
}
|
|
9204
9308
|
const outcome = this.primaryOutcome();
|
|
9205
|
-
for (const rid of absorbedRids) {
|
|
9309
|
+
for (const rid of [...absorbedRids, ...consumedRids]) {
|
|
9206
9310
|
this.emit(
|
|
9207
9311
|
"completed",
|
|
9208
9312
|
{
|
|
@@ -9497,6 +9601,33 @@ var HeadlessSession = class {
|
|
|
9497
9601
|
);
|
|
9498
9602
|
return;
|
|
9499
9603
|
}
|
|
9604
|
+
if (action === "setQueuedDelivery") {
|
|
9605
|
+
const id = parsed.id;
|
|
9606
|
+
const delivery = parsed.delivery;
|
|
9607
|
+
if (!id || delivery !== "asap" && delivery !== "afterTurn") {
|
|
9608
|
+
this.emit(
|
|
9609
|
+
"completed",
|
|
9610
|
+
{
|
|
9611
|
+
success: false,
|
|
9612
|
+
error: 'setQueuedDelivery requires id and delivery ("asap" | "afterTurn")'
|
|
9613
|
+
},
|
|
9614
|
+
requestId
|
|
9615
|
+
);
|
|
9616
|
+
return;
|
|
9617
|
+
}
|
|
9618
|
+
const target = this.queue.snapshot().find((it) => it.command.requestId === id);
|
|
9619
|
+
if (!target || target.source !== "user" || this.isDrainBarrier(target)) {
|
|
9620
|
+
this.emit(
|
|
9621
|
+
"completed",
|
|
9622
|
+
{ success: false, error: "message not found or not promotable" },
|
|
9623
|
+
requestId
|
|
9624
|
+
);
|
|
9625
|
+
return;
|
|
9626
|
+
}
|
|
9627
|
+
this.queue.setDelivery(id, delivery);
|
|
9628
|
+
this.emit("completed", { success: true }, requestId);
|
|
9629
|
+
return;
|
|
9630
|
+
}
|
|
9500
9631
|
if (action === "stop_tool") {
|
|
9501
9632
|
const id = parsed.id;
|
|
9502
9633
|
const mode = parsed.mode ?? "hard";
|
package/dist/index.js
CHANGED
|
@@ -4934,7 +4934,7 @@ var init_tools2 = __esm({
|
|
|
4934
4934
|
"screenshotViewport",
|
|
4935
4935
|
"setViewport"
|
|
4936
4936
|
],
|
|
4937
|
-
description: 'snapshot: accessibility tree of the page (waits for network to settle). click: click an element (animated cursor, full event sequence). type: type text into input (one char at a time, works with React/Vue/Svelte). select: select a dropdown option by text. wait: wait for an element to appear (polls 100ms, waits for network). navigate: navigate to a URL within the app (waits for
|
|
4937
|
+
description: 'snapshot: accessibility tree of the page (waits for network to settle). click: click an element (animated cursor, full event sequence). type: type text into input (one char at a time, works with React/Vue/Svelte). select: select a dropdown option by text. wait: wait for an element to appear (polls 100ms, waits for network). navigate: navigate to a URL within the app (waits for the route, subsequent steps run on the new page; soft in-app route change by default \u2014 pass `fresh: true` for a real full page load; the result reports the URL actually landed on, so app redirects are visible). evaluate: run JS in the page. styles: read computed CSS styles from elements (pass properties array with camelCase names, or omit for defaults). screenshotFullPage: screenshot of the whole page top-to-bottom (returns a CDN url with dimensions and a written analysis). screenshotViewport: screenshot of just the visible viewport \u2014 pass `scrollToSelector` (or `scrollY`) on this step to scroll a section into view and capture it in one atomic step (no separate scroll needed). setViewport: switch the browser between desktop and mobile rendering (pass `mode`: "desktop" or "mobile"). Reloads the page so responsive layouts, media queries, and matchMedia re-evaluate \u2014 use it to QA mobile/responsive views.'
|
|
4938
4938
|
},
|
|
4939
4939
|
ref: {
|
|
4940
4940
|
type: "string",
|
|
@@ -4976,6 +4976,10 @@ var init_tools2 = __esm({
|
|
|
4976
4976
|
type: "string",
|
|
4977
4977
|
description: 'For navigate: the URL to navigate to (e.g., "/quiz", "/settings").'
|
|
4978
4978
|
},
|
|
4979
|
+
fresh: {
|
|
4980
|
+
type: "boolean",
|
|
4981
|
+
description: "For navigate: force a real full page load (fresh document) instead of a soft in-app route change. Use when testing what a user sees on entry \u2014 landing pages, join/invite links, signed-out views \u2014 where reusing the SPA\u2019s in-memory state would test the wrong thing."
|
|
4982
|
+
},
|
|
4979
4983
|
properties: {
|
|
4980
4984
|
type: "array",
|
|
4981
4985
|
items: { type: "string" },
|
|
@@ -7149,10 +7153,13 @@ ${readAsset(
|
|
|
7149
7153
|
</mindstudio_flavored_markdown_spec_docs>`;
|
|
7150
7154
|
specSyncTool = {
|
|
7151
7155
|
backgroundOnly: true,
|
|
7156
|
+
// Fire-and-forget: completion never wakes the agent. The outcome rides the
|
|
7157
|
+
// next real turn as a hidden background_results note instead.
|
|
7158
|
+
backgroundNotify: "passive",
|
|
7152
7159
|
definition: {
|
|
7153
7160
|
clearable: false,
|
|
7154
7161
|
name: "specSync",
|
|
7155
|
-
description: "Reconcile the spec to bring it in line with code changes you have made. Provide a brief, bulleted list of what changed and why; it finds the affected spec sections and updates them to match. Always runs in the background \u2014 it
|
|
7162
|
+
description: "Reconcile the spec to bring it in line with code changes you have made. Provide a brief, bulleted list of what changed and why; it finds the affected spec sections and updates them to match. Always runs in the background and completes silently \u2014 do not wait for it; its outcome appears as an automated note at the start of a later turn.",
|
|
7156
7163
|
inputSchema: {
|
|
7157
7164
|
type: "object",
|
|
7158
7165
|
properties: {
|
|
@@ -8104,6 +8111,7 @@ async function runTurn(params) {
|
|
|
8104
8111
|
onboardingState,
|
|
8105
8112
|
signal,
|
|
8106
8113
|
onEvent,
|
|
8114
|
+
takeSteering,
|
|
8107
8115
|
resolveExternalTool,
|
|
8108
8116
|
requestId,
|
|
8109
8117
|
toolRegistry,
|
|
@@ -8138,7 +8146,7 @@ async function runTurn(params) {
|
|
|
8138
8146
|
onEvent({ type: "error", error: "Empty message" });
|
|
8139
8147
|
return;
|
|
8140
8148
|
}
|
|
8141
|
-
|
|
8149
|
+
const appendEntry = (entry) => {
|
|
8142
8150
|
const hasAttachments = (entry.attachments?.length ?? 0) > 0;
|
|
8143
8151
|
const userMsg = { role: "user", content: entry.text };
|
|
8144
8152
|
if (entry.hidden) {
|
|
@@ -8161,6 +8169,9 @@ async function runTurn(params) {
|
|
|
8161
8169
|
...entry.requestId && { requestId: entry.requestId },
|
|
8162
8170
|
...entry.queued && { queued: true }
|
|
8163
8171
|
});
|
|
8172
|
+
};
|
|
8173
|
+
for (const entry of keptEntries) {
|
|
8174
|
+
appendEntry(entry);
|
|
8164
8175
|
}
|
|
8165
8176
|
const isFirstMessage = state.messages.filter((m) => m.role === "user").length === 1;
|
|
8166
8177
|
const STATUS_EXCLUDED_TOOLS = /* @__PURE__ */ new Set([
|
|
@@ -8677,6 +8688,18 @@ async function runTurn(params) {
|
|
|
8677
8688
|
isToolError: r.isError
|
|
8678
8689
|
});
|
|
8679
8690
|
}
|
|
8691
|
+
if (takeSteering && !signal?.aborted) {
|
|
8692
|
+
const injected = (await takeSteering()).filter(
|
|
8693
|
+
(e) => e.text.trim().length > 0 || (e.attachments?.length ?? 0) > 0
|
|
8694
|
+
);
|
|
8695
|
+
if (injected.length > 0) {
|
|
8696
|
+
for (const entry of injected) {
|
|
8697
|
+
appendEntry(entry);
|
|
8698
|
+
keptEntries.push(entry);
|
|
8699
|
+
}
|
|
8700
|
+
saveSession(state);
|
|
8701
|
+
}
|
|
8702
|
+
}
|
|
8680
8703
|
if (signal?.aborted) {
|
|
8681
8704
|
onEvent({ type: "turn_cancelled" });
|
|
8682
8705
|
saveSession(state);
|
|
@@ -9244,13 +9267,24 @@ function loadQueue() {
|
|
|
9244
9267
|
}
|
|
9245
9268
|
return [];
|
|
9246
9269
|
}
|
|
9247
|
-
function
|
|
9270
|
+
function loadPassiveResults() {
|
|
9271
|
+
try {
|
|
9272
|
+
const stats = JSON.parse(readFileSync2(STATS_FILE, "utf-8"));
|
|
9273
|
+
if (Array.isArray(stats.passiveResults)) {
|
|
9274
|
+
return stats.passiveResults;
|
|
9275
|
+
}
|
|
9276
|
+
} catch {
|
|
9277
|
+
}
|
|
9278
|
+
return [];
|
|
9279
|
+
}
|
|
9280
|
+
function writeStats(stats, queue, passiveResults) {
|
|
9248
9281
|
try {
|
|
9249
9282
|
writeFileSync2(
|
|
9250
9283
|
STATS_FILE,
|
|
9251
9284
|
JSON.stringify({
|
|
9252
9285
|
...stats,
|
|
9253
|
-
queue
|
|
9286
|
+
queue,
|
|
9287
|
+
passiveResults
|
|
9254
9288
|
})
|
|
9255
9289
|
);
|
|
9256
9290
|
} catch {
|
|
@@ -9325,6 +9359,21 @@ var init_messageQueue = __esm({
|
|
|
9325
9359
|
}
|
|
9326
9360
|
return removed;
|
|
9327
9361
|
}
|
|
9362
|
+
/**
|
|
9363
|
+
* Change a queued item's delivery semantics, keyed by its command
|
|
9364
|
+
* requestId. Fires onChange (→ persist + queue_changed) on success.
|
|
9365
|
+
* Returns the item, or undefined if no queued item matches (e.g. it was
|
|
9366
|
+
* already consumed).
|
|
9367
|
+
*/
|
|
9368
|
+
setDelivery(id, delivery) {
|
|
9369
|
+
const item = this.items.find((it) => it.command.requestId === id);
|
|
9370
|
+
if (!item) {
|
|
9371
|
+
return void 0;
|
|
9372
|
+
}
|
|
9373
|
+
item.delivery = delivery;
|
|
9374
|
+
this.onChange?.();
|
|
9375
|
+
return item;
|
|
9376
|
+
}
|
|
9328
9377
|
/** Copy of current queue contents (for surfacing on events). */
|
|
9329
9378
|
snapshot() {
|
|
9330
9379
|
return [...this.items];
|
|
@@ -9431,6 +9480,7 @@ var init_headless = __esm({
|
|
|
9431
9480
|
init_attachments();
|
|
9432
9481
|
init_planFile();
|
|
9433
9482
|
init_stats();
|
|
9483
|
+
init_tools8();
|
|
9434
9484
|
init_messageQueue();
|
|
9435
9485
|
init_resolve();
|
|
9436
9486
|
init_sentinel();
|
|
@@ -9474,6 +9524,15 @@ var init_headless = __esm({
|
|
|
9474
9524
|
* to .remy-stats.json so queued work survives process restarts.
|
|
9475
9525
|
*/
|
|
9476
9526
|
queue;
|
|
9527
|
+
/**
|
|
9528
|
+
* Holding pen for passive background results (tools with
|
|
9529
|
+
* `backgroundNotify: 'passive'`, e.g. specSync). Deliberately outside the
|
|
9530
|
+
* message queue: pen contents never initiate a turn, never latch the
|
|
9531
|
+
* sandbox's queue-derived busy state, and never trigger resume-on-restart.
|
|
9532
|
+
* Swept into the next real turn as a hidden background_results entry.
|
|
9533
|
+
* Persisted to .remy-stats.json alongside the queue.
|
|
9534
|
+
*/
|
|
9535
|
+
passivePen = [];
|
|
9477
9536
|
// External tool bridge
|
|
9478
9537
|
pendingTools = /* @__PURE__ */ new Map();
|
|
9479
9538
|
earlyResults = /* @__PURE__ */ new Map();
|
|
@@ -9509,6 +9568,7 @@ var init_headless = __esm({
|
|
|
9509
9568
|
this.persistStats();
|
|
9510
9569
|
this.emit("queue_changed", { queuedMessages: this.queue.snapshot() });
|
|
9511
9570
|
});
|
|
9571
|
+
this.passivePen = loadPassiveResults();
|
|
9512
9572
|
if (resumed) {
|
|
9513
9573
|
this.emit("session_restored", {
|
|
9514
9574
|
messageCount: this.state.messages.length,
|
|
@@ -9620,10 +9680,10 @@ var init_headless = __esm({
|
|
|
9620
9680
|
//////////////////////////////////////////////////////////////////////////////
|
|
9621
9681
|
// Stats + queue persistence
|
|
9622
9682
|
//////////////////////////////////////////////////////////////////////////////
|
|
9623
|
-
/** Persist sessionStats + queue snapshot to .remy-stats.json. */
|
|
9683
|
+
/** Persist sessionStats + queue snapshot + passive pen to .remy-stats.json. */
|
|
9624
9684
|
persistStats() {
|
|
9625
9685
|
this.sessionStats.updatedAt = Date.now();
|
|
9626
|
-
writeStats(this.sessionStats, this.queue.snapshot());
|
|
9686
|
+
writeStats(this.sessionStats, this.queue.snapshot(), this.passivePen);
|
|
9627
9687
|
}
|
|
9628
9688
|
//////////////////////////////////////////////////////////////////////////////
|
|
9629
9689
|
// Background completions (tool-block mutation; message delivery via queue)
|
|
@@ -9685,10 +9745,12 @@ var init_headless = __esm({
|
|
|
9685
9745
|
}
|
|
9686
9746
|
}
|
|
9687
9747
|
onBackgroundComplete = (toolCallId, name, result, subAgentMessages) => {
|
|
9748
|
+
const notify = getToolByName(name)?.backgroundNotify ?? "wake";
|
|
9688
9749
|
this.pendingBlockUpdates.push({ toolCallId, result, subAgentMessages });
|
|
9689
9750
|
log16.info("Background complete", {
|
|
9690
9751
|
toolCallId,
|
|
9691
9752
|
name,
|
|
9753
|
+
notify,
|
|
9692
9754
|
requestId: this.currentRequestId
|
|
9693
9755
|
});
|
|
9694
9756
|
this.onEvent({
|
|
@@ -9697,6 +9759,14 @@ var init_headless = __esm({
|
|
|
9697
9759
|
name,
|
|
9698
9760
|
result
|
|
9699
9761
|
});
|
|
9762
|
+
if (notify === "passive") {
|
|
9763
|
+
this.passivePen.push({ toolCallId, name, result });
|
|
9764
|
+
this.persistStats();
|
|
9765
|
+
if (!this.running) {
|
|
9766
|
+
this.applyPendingBlockUpdates();
|
|
9767
|
+
}
|
|
9768
|
+
return;
|
|
9769
|
+
}
|
|
9700
9770
|
this.queue.push({
|
|
9701
9771
|
command: {
|
|
9702
9772
|
action: "message",
|
|
@@ -9765,7 +9835,10 @@ var init_headless = __esm({
|
|
|
9765
9835
|
...e.attachments && { attachments: e.attachments },
|
|
9766
9836
|
// Queue-delivered entries are flagged so the frontend renders the
|
|
9767
9837
|
// echo (idle sends are rendered optimistically instead).
|
|
9768
|
-
...e.queued && { queued: true }
|
|
9838
|
+
...e.queued && { queued: true },
|
|
9839
|
+
// Hidden entries (e.g. the passive background_results sweep) are
|
|
9840
|
+
// flagged so the sandbox/frontend suppress the bubble.
|
|
9841
|
+
...e.hidden && { hidden: true }
|
|
9769
9842
|
},
|
|
9770
9843
|
// A merged turn emits one user_message per absorbed entry — each
|
|
9771
9844
|
// carries its own original requestId, not the turn's.
|
|
@@ -10075,6 +10148,37 @@ var init_headless = __esm({
|
|
|
10075
10148
|
this.completedEmitted = false;
|
|
10076
10149
|
this.lastCompleted = null;
|
|
10077
10150
|
this.turnStart = Date.now();
|
|
10151
|
+
if (this.passivePen.length > 0) {
|
|
10152
|
+
const swept = this.passivePen.splice(0);
|
|
10153
|
+
this.persistStats();
|
|
10154
|
+
entries.unshift({
|
|
10155
|
+
text: buildBackgroundResultsMessage(swept),
|
|
10156
|
+
hidden: true
|
|
10157
|
+
});
|
|
10158
|
+
}
|
|
10159
|
+
const consumedRids = [];
|
|
10160
|
+
const takeSteering = async () => {
|
|
10161
|
+
const items = this.queue.removeWhere(
|
|
10162
|
+
(it) => it.source === "user" && it.delivery === "asap" && !this.isDrainBarrier(it)
|
|
10163
|
+
);
|
|
10164
|
+
const steered = [];
|
|
10165
|
+
for (const it of items) {
|
|
10166
|
+
const attachments = it.command.attachments;
|
|
10167
|
+
const attachmentHeader = await this.persistEntryAttachments(attachments);
|
|
10168
|
+
const rid = it.command.requestId;
|
|
10169
|
+
if (rid) {
|
|
10170
|
+
consumedRids.push(rid);
|
|
10171
|
+
}
|
|
10172
|
+
steered.push({
|
|
10173
|
+
text: it.command.text ?? "",
|
|
10174
|
+
attachments,
|
|
10175
|
+
attachmentHeader,
|
|
10176
|
+
requestId: rid,
|
|
10177
|
+
queued: true
|
|
10178
|
+
});
|
|
10179
|
+
}
|
|
10180
|
+
return steered;
|
|
10181
|
+
};
|
|
10078
10182
|
await this.runForcedCompactionIfNeeded(requestId);
|
|
10079
10183
|
try {
|
|
10080
10184
|
await runTurn({
|
|
@@ -10088,6 +10192,7 @@ var init_headless = __esm({
|
|
|
10088
10192
|
requestId,
|
|
10089
10193
|
signal: this.currentAbort.signal,
|
|
10090
10194
|
onEvent: this.onEvent,
|
|
10195
|
+
takeSteering,
|
|
10091
10196
|
resolveExternalTool: this.resolveExternalTool,
|
|
10092
10197
|
toolRegistry: this.toolRegistry,
|
|
10093
10198
|
onBackgroundComplete: this.onBackgroundComplete
|
|
@@ -10117,7 +10222,7 @@ var init_headless = __esm({
|
|
|
10117
10222
|
});
|
|
10118
10223
|
}
|
|
10119
10224
|
const outcome = this.primaryOutcome();
|
|
10120
|
-
for (const rid of absorbedRids) {
|
|
10225
|
+
for (const rid of [...absorbedRids, ...consumedRids]) {
|
|
10121
10226
|
this.emit(
|
|
10122
10227
|
"completed",
|
|
10123
10228
|
{
|
|
@@ -10412,6 +10517,33 @@ var init_headless = __esm({
|
|
|
10412
10517
|
);
|
|
10413
10518
|
return;
|
|
10414
10519
|
}
|
|
10520
|
+
if (action === "setQueuedDelivery") {
|
|
10521
|
+
const id = parsed.id;
|
|
10522
|
+
const delivery = parsed.delivery;
|
|
10523
|
+
if (!id || delivery !== "asap" && delivery !== "afterTurn") {
|
|
10524
|
+
this.emit(
|
|
10525
|
+
"completed",
|
|
10526
|
+
{
|
|
10527
|
+
success: false,
|
|
10528
|
+
error: 'setQueuedDelivery requires id and delivery ("asap" | "afterTurn")'
|
|
10529
|
+
},
|
|
10530
|
+
requestId
|
|
10531
|
+
);
|
|
10532
|
+
return;
|
|
10533
|
+
}
|
|
10534
|
+
const target = this.queue.snapshot().find((it) => it.command.requestId === id);
|
|
10535
|
+
if (!target || target.source !== "user" || this.isDrainBarrier(target)) {
|
|
10536
|
+
this.emit(
|
|
10537
|
+
"completed",
|
|
10538
|
+
{ success: false, error: "message not found or not promotable" },
|
|
10539
|
+
requestId
|
|
10540
|
+
);
|
|
10541
|
+
return;
|
|
10542
|
+
}
|
|
10543
|
+
this.queue.setDelivery(id, delivery);
|
|
10544
|
+
this.emit("completed", { success: true }, requestId);
|
|
10545
|
+
return;
|
|
10546
|
+
}
|
|
10415
10547
|
if (action === "stop_tool") {
|
|
10416
10548
|
const id = parsed.id;
|
|
10417
10549
|
const mode = parsed.mode ?? "hard";
|
|
@@ -103,6 +103,30 @@ const { threads, nextCursor } = await chat.listThreads();
|
|
|
103
103
|
const full = await chat.getThread(thread.id);
|
|
104
104
|
await chat.updateThread(thread.id, 'New title');
|
|
105
105
|
await chat.deleteThread(thread.id);
|
|
106
|
+
|
|
107
|
+
// Progressive auth: threads started anonymously become unreachable after the
|
|
108
|
+
// user signs in (login replaces the session and its visitor identity). Claim
|
|
109
|
+
// them right after your verification/login succeeds so the conversation
|
|
110
|
+
// survives — the client remembers each thread's pre-login token automatically
|
|
111
|
+
// for threads touched this page session.
|
|
112
|
+
await chat.claimThread(thread.id);
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
**Client tools** — a tool whose effect happens in the browser (open a sheet, navigate, highlight)
|
|
116
|
+
is declared with `target: "client"` and a `name` + inline `inputSchema` instead of a `method`
|
|
117
|
+
(names must not collide with method ids; the schema is authored — there's no method contract to
|
|
118
|
+
derive it from). The agent's invocation arrives as the `client_tool_call` stream event / the
|
|
119
|
+
`onClientToolCall` callback on `sendMessage`; run the action there. Fire-and-forget on this
|
|
120
|
+
surface: the agent is told the action was displayed and keeps going — the user's next message
|
|
121
|
+
closes the loop.
|
|
122
|
+
|
|
123
|
+
```js
|
|
124
|
+
await chat.sendMessage(thread.id, text, {
|
|
125
|
+
onText: (delta) => append(delta),
|
|
126
|
+
onClientToolCall: (name, input) => {
|
|
127
|
+
if (name === 'showVerification') openVerifySheet(input);
|
|
128
|
+
},
|
|
129
|
+
});
|
|
106
130
|
```
|
|
107
131
|
|
|
108
132
|
**Sending messages (streaming):**
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: Voice Interfaces
|
|
3
3
|
what: Realtime voice conversation as a first-class interface — the user talks to the app and its voice agent talks back in sub-second, interruptible speech, calling the app's methods mid-conversation as the authenticated user. The platform handles the media transport, turn-taking, barge-in, and transcripts, so the work is authorship — a persona written for the ear, a small toolset where every tool carries a latency class, and descriptions that say results out loud. Any app whose methods do something interesting can pick up a voice, and it is often the most impressive surface it has.
|
|
4
|
-
when: Before authoring `src/interfaces/voice.md`, choosing a voice model or pipeline, deciding which methods a voice agent gets,
|
|
4
|
+
when: Before authoring `src/interfaces/voice.md`, choosing a voice model or pipeline, deciding which methods a voice agent gets, building the voice UI with `createVoiceClient()`, or working out why a voice agent behaved the way it did on a call.
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
# Building Voice Interfaces
|
|
@@ -57,7 +57,8 @@ precisely: "confirm before any tool that changes data," not "always confirm ever
|
|
|
57
57
|
`always`/`never` makes the agent rigid and unable to handle reasonable exceptions. And start
|
|
58
58
|
minimal: state the role, the boundaries, and the voice mechanics above, then add rules only for
|
|
59
59
|
behaviors that actually misfire in test calls (the transcripts in the call log are the feedback
|
|
60
|
-
loop) rather than front-loading a
|
|
60
|
+
loop — `mindstudio-prod voice sessions get` reads a call verbatim) rather than front-loading a
|
|
61
|
+
policy manual.
|
|
61
62
|
|
|
62
63
|
### The latency classes
|
|
63
64
|
|
|
@@ -78,25 +79,65 @@ Classify by how the method actually behaves, not by what it is named. A "lookup"
|
|
|
78
79
|
external service is `slow`. When in doubt between `fast` and `slow`, pick `slow` — a needless
|
|
79
80
|
preamble is mildly chatty; an unexplained silence feels broken.
|
|
80
81
|
|
|
81
|
-
###
|
|
82
|
+
### Tool results reach the screen
|
|
82
83
|
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
84
|
+
Every successful tool call delivers its raw return value to the session's browser on the SDK's
|
|
85
|
+
`toolCall` event (`result` field, on `done`) — so the UI can render what the agent just did (the
|
|
86
|
+
citation it found, the record it pulled up, the booking it made) in lockstep with the spoken
|
|
87
|
+
answer. No flag, no polling, no key-threading, no model involvement: delivery to the invoking
|
|
88
|
+
session's own client is the same security context as the invocation itself (an RPC response), and
|
|
89
|
+
it's scoped to that one session.
|
|
88
90
|
|
|
89
|
-
|
|
90
|
-
the
|
|
91
|
-
|
|
92
|
-
`resultTruncated: true` with no data — keep
|
|
93
|
-
data itself. Failed calls
|
|
91
|
+
Consequence for authoring: **a tool's return value is user-visible by definition.** Return what
|
|
92
|
+
the user may see — no internal fields, keys, or diagnostics you wouldn't put on screen (the same
|
|
93
|
+
discipline as agent-interface tools, whose results render in chat). Payloads over ~32KB serialized
|
|
94
|
+
arrive as `resultTruncated: true` with no data — keep returns compact, or have the UI fetch big
|
|
95
|
+
data itself. Failed calls deliver nothing to the client (the model gets the `{ error }` and speaks
|
|
96
|
+
a decline).
|
|
94
97
|
|
|
95
98
|
For backend-side correlation (writing results to a table keyed by the call, custom channels), the
|
|
96
99
|
method itself can read `session.voiceSessionId` / `session.visitorId` from the agent SDK
|
|
97
100
|
(`import { session } from '@mindstudio-ai/agent'`) — the same id the browser holds as
|
|
98
101
|
`session.sessionId`, guaranteed by the platform rather than echoed by the model.
|
|
99
102
|
|
|
103
|
+
### Client tools: actions that happen on screen (`target: "client"`)
|
|
104
|
+
|
|
105
|
+
A tool whose effect belongs in the browser — open the verification sheet, navigate to a page,
|
|
106
|
+
highlight a record — is declared with `target: "client"` instead of a `method`:
|
|
107
|
+
|
|
108
|
+
```json
|
|
109
|
+
{
|
|
110
|
+
"target": "client",
|
|
111
|
+
"name": "showVerification",
|
|
112
|
+
"description": "tools/showVerification.md",
|
|
113
|
+
"inputSchema": { "type": "object", "properties": { "reason": { "type": "string" } } }
|
|
114
|
+
}
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
The platform never touches the backend for these: the agent's invocation is delivered to the
|
|
118
|
+
session's browser, the app's registered handler runs, and the handler's **return value goes back
|
|
119
|
+
to the agent as the tool result** — a real request/response, so the agent knows the sheet
|
|
120
|
+
actually opened (or that the user dismissed it) and speaks accordingly. Rules:
|
|
121
|
+
|
|
122
|
+
- `name` instead of `method`; must not collide with any backend method id. No latency class —
|
|
123
|
+
the agent holds the turn while the browser responds (up to ~30s, then a timeout error).
|
|
124
|
+
- `inputSchema` is authored inline (an object schema) — there's no method contract to derive
|
|
125
|
+
it from. Keep it small; these are UI directives, not data payloads.
|
|
126
|
+
- The frontend must register a handler, or invocations fail as `unhandled_client_tool`:
|
|
127
|
+
|
|
128
|
+
```js
|
|
129
|
+
session.registerClientTool('showVerification', async ({ reason }) => {
|
|
130
|
+
openVerifySheet(reason);
|
|
131
|
+
return { opened: true }; // what the agent hears back
|
|
132
|
+
});
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
- One client tool runs at a time per session; the description should tell the agent when to use
|
|
136
|
+
it and what to say while it's on screen. Throwing from the handler (or returning nothing)
|
|
137
|
+
becomes an error/ack the agent can speak around.
|
|
138
|
+
- The progressive-auth pattern above is the canonical use: make the verification sheet a client
|
|
139
|
+
tool and the agent opens it deliberately instead of the frontend inferring it from tool events.
|
|
140
|
+
|
|
100
141
|
### Tool descriptions say results out loud
|
|
101
142
|
|
|
102
143
|
Follow the agent-interface principles for tool descriptions (when to use and when not, parameter
|
|
@@ -124,24 +165,45 @@ caller.
|
|
|
124
165
|
|
|
125
166
|
### Choosing the model
|
|
126
167
|
|
|
127
|
-
Two shapes, one `model` field
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
168
|
+
Two shapes, one `model` field. **Use native speech-to-speech unless the user specifically asks
|
|
169
|
+
for a cascaded pipeline** — one realtime model hears and speaks: lowest latency, most natural
|
|
170
|
+
prosody, hears tone and hesitation.
|
|
171
|
+
|
|
172
|
+
**Native** (`{"model": ..., "voice": ...}`) — **default to `gpt-realtime-2.1` with voice
|
|
173
|
+
`marin`.**
|
|
174
|
+
|
|
175
|
+
- `gpt-realtime-2.1` — the default. Voices: `marin` (default), `cedar`, `alloy`, `ash`,
|
|
176
|
+
`ballad`, `coral`, `echo`, `sage`, `shimmer`, `verse`.
|
|
177
|
+
- `gpt-realtime-2.1-mini` — the same family, lighter; same voices.
|
|
178
|
+
- `gemini-2.5-flash-native-audio-preview-12-2025` — the Gemini pick, with a large expressive
|
|
179
|
+
roster: `Puck` (default, upbeat), `Zephyr` (bright), `Charon` (informative), `Kore` (firm),
|
|
180
|
+
`Fenrir` (excitable), `Leda` (youthful), `Orus` (firm), `Aoede` (breezy), `Callirrhoe`
|
|
181
|
+
(easy-going), `Autonoe` (bright), `Enceladus` (breathy), `Iapetus` (clear), `Umbriel`
|
|
182
|
+
(easy-going), `Algieba` (smooth), `Despina` (smooth), `Erinome` (clear), `Algenib` (gravelly),
|
|
183
|
+
`Rasalgethi` (informative), `Laomedeia` (upbeat), `Achernar` (soft), `Alnilam` (firm),
|
|
184
|
+
`Schedar` (even), `Gacrux` (mature), `Pulcherrima` (forward), `Achird` (friendly),
|
|
185
|
+
`Zubenelgenubi` (casual), `Vindemiatrix` (gentle), `Sadachbia` (lively), `Sadaltager`
|
|
186
|
+
(knowledgeable), `Sulafat` (warm).
|
|
187
|
+
- `gemini-3.1-flash-live-preview` — newer Gemini, same voices as 2.5, but currently can't speak
|
|
188
|
+
an opening greeting or take mid-call prompt updates (a plugin limitation expected to resolve
|
|
189
|
+
upstream) — prefer 2.5 until then.
|
|
190
|
+
- `grok-voice-think-fast-2.0` — a distinct personality register. Voices: `eve` (default),
|
|
191
|
+
`altair`, `ara`, `atlas`, `aurora`, `carina`, `castor`, `celeste`, `cosmo`, `helios`, `helix`,
|
|
192
|
+
`iris`, `kepler`, `leo`, `liora`, `lumen`, `luna`, `lux`, `naksh`, `orion`, `perseus`, `rex`,
|
|
193
|
+
`rigel`, `sal`, `sirius`, `ursa`, `zagan`, `zenith`.
|
|
194
|
+
|
|
195
|
+
**Cascaded** (`{"llm": ..., "stt": ..., "tts": ..., "voice": ...}`) — streaming transcription
|
|
196
|
+
into any chat model in the catalog, streaming speech out. Slightly higher latency; reach for it
|
|
197
|
+
only when the user wants it or the app's reasoning demands a specific chat model (e.g. the agent
|
|
198
|
+
interface already uses one and the voice should think identically). Slots: `stt` is
|
|
199
|
+
`deepgram-nova-3`; `tts` is `cartesia-sonic-3` (voices are per-account Cartesia UUIDs — see
|
|
200
|
+
play.cartesia.ai) or `elevenlabs-tts` (the account's ElevenLabs voice library); `llm` is any chat
|
|
201
|
+
model — ask `askMindStudioSdk` for chat model ids. One nuance: cascaded engines speak the
|
|
202
|
+
`greeting` verbatim (they have a real TTS); speech-to-speech engines have the model say it, so it
|
|
203
|
+
may paraphrase slightly.
|
|
204
|
+
|
|
205
|
+
The model and voice ids above are current and maintained with the platform — use them as written
|
|
206
|
+
(they are MindStudio ids, not vendor ids). The user's UI has a picker for changing the model
|
|
145
207
|
later, so validate only when you set it.
|
|
146
208
|
|
|
147
209
|
### Seeding from an existing agent
|
|
@@ -204,13 +266,15 @@ session.on('stateChange', (state) => { }); // on() returns an unsubscri
|
|
|
204
266
|
// far (never a delta) — render by upserting on segmentId, not appending.
|
|
205
267
|
session.on('transcript', ({ role, segmentId, text, final }) => { });
|
|
206
268
|
|
|
207
|
-
// status: 'running' | 'done' | 'failed'.
|
|
208
|
-
//
|
|
269
|
+
// status: 'running' | 'done' | 'failed'. Every 'done' carries the tool's raw
|
|
270
|
+
// return value in `result` (or `resultTruncated: true` if >~32KB serialized).
|
|
209
271
|
session.on('toolCall', ({ method, status, result }) => { });
|
|
210
272
|
session.on('error', (err) => { });
|
|
211
273
|
|
|
212
274
|
session.mute(); session.unmute(); session.isMuted;
|
|
213
275
|
session.sendText('123 Main Street'); // inject text into the live conversation
|
|
276
|
+
await session.refreshIdentity(); // after in-app verification — upgrade the
|
|
277
|
+
// live session anonymous → signed-in in place
|
|
214
278
|
session.end();
|
|
215
279
|
```
|
|
216
280
|
|
|
@@ -277,7 +341,7 @@ export async function callMeAboutMyOrder(input: { phone: string }) {
|
|
|
277
341
|
session, not the phone). Omitted/false → anonymous call; role-gated tools decline.
|
|
278
342
|
System/cron invocations have no human identity and always run anonymously.
|
|
279
343
|
- **Production needs a dedicated phone number.** The app owner attaches one ($1/month) via the
|
|
280
|
-
dashboard or `mindstudio-prod voice numbers` (see "
|
|
344
|
+
dashboard or `mindstudio-prod voice numbers` (see "The voice CLI"
|
|
281
345
|
below) — it becomes the caller ID for every call, in dev sessions too, so users always see
|
|
282
346
|
the same number. Without one, deployed calls throw `phone_out_requires_dedicated_number`, and
|
|
283
347
|
dev sessions fall back to a shared platform test number that varies per call (tighter limits
|
|
@@ -338,7 +402,7 @@ wrong one wherever the agent's tools can move money, reveal sensitive records, o
|
|
|
338
402
|
destructive actions. It lives in the interface config deliberately: enabling it is a code
|
|
339
403
|
change, visible in review and auditable via deploys, not a dashboard toggle.
|
|
340
404
|
|
|
341
|
-
##
|
|
405
|
+
## The voice CLI
|
|
342
406
|
|
|
343
407
|
The `mindstudio-prod voice` family covers numbers, the call log, and voice policy:
|
|
344
408
|
|
|
@@ -377,7 +441,7 @@ section.
|
|
|
377
441
|
name: Front Desk
|
|
378
442
|
description: Books appointments and answers questions by voice.
|
|
379
443
|
type: interface/voice
|
|
380
|
-
model: {"model": "gpt-realtime-
|
|
444
|
+
model: {"model": "gpt-realtime-2.1", "voice": "marin"}
|
|
381
445
|
turnDetection: {"eagerness": "medium"}
|
|
382
446
|
greeting: Hey! I can help you book, reschedule, or answer questions — what do you need?
|
|
383
447
|
---
|
|
@@ -435,7 +499,7 @@ The top-level key must match the interface type (`voice`):
|
|
|
435
499
|
"voice": {
|
|
436
500
|
"name": "Front Desk",
|
|
437
501
|
"description": "Books appointments and answers questions by voice.",
|
|
438
|
-
"model": "gpt-realtime-
|
|
502
|
+
"model": "gpt-realtime-2.1",
|
|
439
503
|
"voice": "marin",
|
|
440
504
|
"turnDetection": { "eagerness": "medium" },
|
|
441
505
|
"greeting": "Hey! I can help you book, reschedule, or answer questions — what do you need?",
|
|
@@ -537,3 +601,30 @@ frontend or the agent interface. Anonymous sessions (when allowed) have no user
|
|
|
537
601
|
gated methods reject, and the caller's history is scoped to their browser's visitor identity.
|
|
538
602
|
That's why role restrictions belong in the tool descriptions — the agent should decline in
|
|
539
603
|
character, not relay a rejection.
|
|
604
|
+
|
|
605
|
+
### Progressive auth: verify mid-call without dropping the conversation
|
|
606
|
+
|
|
607
|
+
The best pattern for apps that allow anonymous sessions (`requireUser: false`): let visitors
|
|
608
|
+
explore by voice, and verify only when they hit an account-bound action — without killing the
|
|
609
|
+
live call. Four pieces, all platform rails:
|
|
610
|
+
|
|
611
|
+
1. **Account-gated tools return a standard not-verified shape** instead of doing the work:
|
|
612
|
+
`{ verified: false, message: 'The caller is not verified. Offer to verify them before sharing
|
|
613
|
+
account details.' }`. The agent speaks the offer in character (reinforce tone in the system
|
|
614
|
+
prompt's verification section). Check with the agent SDK's `auth.userId` inside the method.
|
|
615
|
+
2. **The frontend opens its verification sheet off the same signal.** It already receives every
|
|
616
|
+
tool's `toolCall` event (and the tool's return in `result`) — when an account tool fires (or
|
|
617
|
+
returns `verified: false`) while the app has no signed-in user, open the sheet.
|
|
618
|
+
3. **The sheet runs the platform's auth rails** — `auth.sendSmsCode()` / `auth.verifySmsCode()`
|
|
619
|
+
(or the email pair) from `@mindstudio-ai/interface`. On success the app's session becomes the
|
|
620
|
+
verified user.
|
|
621
|
+
4. **Hand the verified session back to the live call**: `await session.refreshIdentity()`. The
|
|
622
|
+
platform upgrades the running voice session in place — subsequent tool calls carry the user's
|
|
623
|
+
identity and roles, and the agent's Current User context refreshes — no teardown, no lost
|
|
624
|
+
conversation. (Phone calls don't need this: they verify through the agent's built-in flow.)
|
|
625
|
+
|
|
626
|
+
`refreshIdentity()` is upgrade-only (anonymous → signed-in; an already-identified session rejects
|
|
627
|
+
with `already_identified`) and requires the session to have been started by this same browser. If
|
|
628
|
+
it fails, ending and restarting the session is the graceful fallback. The chat sibling for agent
|
|
629
|
+
interfaces is `claimThread(threadId)` — anonymous threads become unreachable after login until
|
|
630
|
+
claimed.
|
|
@@ -13,6 +13,8 @@ When the content you need to test is behind authentication, use the `setupBrowse
|
|
|
13
13
|
|
|
14
14
|
If you need to test the login/signup flow itself (e.g., verifying the UI, error states, or the verification code input), navigate it manually: use `remy@mindstudio.ai` for email and `+15551234567` for phone. In the dev environment, verification codes are bypassed for this email and any 555-prefixed phone number — enter any 6-digit code (e.g., `123456`).
|
|
15
15
|
|
|
16
|
+
To test as a **signed-out visitor** (public pages, landing/join links), call `setupBrowser` with NO `auth` — it clears the auth cookie and reloads at the given path, giving you a clean unauthenticated session. Combine with `navigate` + `fresh: true` when you need a fresh-document view of an entry page mid-run.
|
|
17
|
+
|
|
16
18
|
## Browser Commands
|
|
17
19
|
|
|
18
20
|
Your session always starts on the app root / in a logged out/unauthenticated state. Use `setupBrowser` to authenticate before testing protected pages.
|
|
@@ -40,13 +42,43 @@ Note: the snapshot concatenates inline text and strips whitespace. If you need t
|
|
|
40
42
|
- `type`: Type text into an input. Characters appear one at a time. Set `clear: true` to clear the field first.
|
|
41
43
|
- `select`: Select a dropdown option by text. Target the `<select>` element, set `option` to the option text.
|
|
42
44
|
- `wait`: Wait for an element to appear (polls every 100ms, default 5s timeout). Also waits for network to settle after the element is found.
|
|
43
|
-
- `navigate`: Navigate to a new URL within the app. Waits for the
|
|
45
|
+
- `navigate`: Navigate to a new URL within the app. Waits for the route to load before continuing with subsequent steps. Use this instead of evaluate with `window.location.href` when you need to navigate and then continue interacting with the new page. Steps after navigate execute on the new page automatically. Same-origin navigation is a soft in-app route change (like clicking a link in an SPA — in-memory app state survives); set `fresh: true` to force a real full page load with a fresh document instead. Use `fresh: true` when the test is about what a user sees on *entry* — landing pages, join/invite links, "what does a signed-out visitor see" — where reusing the SPA's in-memory state would test the wrong thing. The result reports the URL the page actually landed on, so if the app redirected you (e.g. an auth wall bounced you off a public page), you'll see the real destination — check it instead of assuming the navigation stuck.
|
|
44
46
|
- `evaluate`: Run arbitrary JavaScript in the page and return the result.
|
|
45
47
|
- `styles`: Read computed CSS styles from page elements. Pass a `properties` array with camelCase CSS property names (e.g., `["backgroundColor", "borderRadius", "fontSize"]`). Omit `properties` for a default set covering colors, typography, spacing, borders, shadows, dimensions, and layout. Uses the same targeting as click/type (ref, text, role, label, selector). Omit the target to get styles for all elements from the last snapshot.
|
|
46
48
|
- `screenshotFullPage`: Take a screenshot of the whole page, top to bottom. Returns CDN url with full text analysis and dimensions. Use for overall composition or content past the fold.
|
|
47
49
|
- `screenshotViewport`: Take a screenshot of the visible viewport. Returns CDN url with full text analysis and dimensions. To capture a specific section, set `scrollToSelector` (a CSS selector) — or `scrollY` (an absolute offset) — on this same step; it scrolls the target into view and captures it atomically, so you do NOT need a separate scroll step. Do not use if you can get what you need with other tools - only use when you need to visually see the viewport.
|
|
48
50
|
- `setViewport`: Switch the browser between desktop and mobile rendering. Set `mode` to `"desktop"` or `"mobile"`. Mobile emulates a phone (390-wide, touch, device pixel ratio 2); desktop is the standard wide viewport. This reloads the page so media queries, responsive layouts, and `matchMedia` re-evaluate — the reload clears in-page state, so switch before you set up the state you want to inspect. The mode persists across navigations within a run. Each run starts in the app's default mode, so only use this when you need to check the other one.
|
|
49
51
|
|
|
52
|
+
### Voice interfaces
|
|
53
|
+
|
|
54
|
+
Apps with a voice interface are testable end to end — the UI layer included. The sandbox browser
|
|
55
|
+
auto-grants a (silent) microphone, and while a session is live the SDK publishes a handle at
|
|
56
|
+
`window.__MS_VOICE__` so you can converse by text: the agent treats injected text exactly like
|
|
57
|
+
user speech (interrupts and replies), backend tools run for real, and client tools render their
|
|
58
|
+
real UI (cards, sheets) in the page.
|
|
59
|
+
|
|
60
|
+
The loop:
|
|
61
|
+
|
|
62
|
+
1. Start a session through the app's real UI — `click` its voice affordance (orb/button). No mic
|
|
63
|
+
prompt appears. Then `wait` briefly and confirm the session is live:
|
|
64
|
+
`evaluate: window.__MS_VOICE__?.state` (undefined means no session started — report that,
|
|
65
|
+
don't improvise).
|
|
66
|
+
2. Speak by injection: `evaluate: window.__MS_VOICE__.sendText("I'd like to book Tuesday at 2")`.
|
|
67
|
+
3. Give the agent a few seconds to respond (replies are generated speech — slower than chat).
|
|
68
|
+
`wait` for the UI you expect (client-tool cards appear via the app's real handlers), and read
|
|
69
|
+
the conversation: `evaluate: window.__MS_VOICE__.transcript` (one entry per utterance, both
|
|
70
|
+
sides, `final` marks settled ones) and `window.__MS_VOICE__.toolCalls` (which tools ran;
|
|
71
|
+
`done` entries carry the tool's return value).
|
|
72
|
+
4. Verify visuals with `screenshotViewport` like any other flow.
|
|
73
|
+
5. Read `transcript`/`toolCalls` BEFORE ending — then `evaluate: window.__MS_VOICE__.end()` (the
|
|
74
|
+
handle is removed when the session ends).
|
|
75
|
+
|
|
76
|
+
Voice sessions are the most expensive thing you can run — real voice-model minutes are metered,
|
|
77
|
+
and the agent speaks its replies out loud even when you type at it. Keep voice tests short and
|
|
78
|
+
purposeful: a handful of turns that exercise the target behavior, then end the session. What you
|
|
79
|
+
cannot test is the audio layer itself (mishearing, interruptions, pronunciation) — never attempt
|
|
80
|
+
to simulate audio; report that scope limit instead.
|
|
81
|
+
|
|
50
82
|
### Element targeting (tried in order)
|
|
51
83
|
|
|
52
84
|
1. `ref`: From the last snapshot. Most reliable.
|