@mastra/mcp-docs-server 1.1.42-alpha.2 → 1.1.42-alpha.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.docs/docs/agent-builder/memory.md +1 -1
- package/.docs/docs/agents/adding-voice.md +31 -0
- package/.docs/docs/agents/agent-approval.md +14 -0
- package/.docs/docs/agents/channels.md +2 -0
- package/.docs/docs/agents/code-mode.md +163 -0
- package/.docs/docs/agents/signals.md +132 -71
- package/.docs/docs/getting-started/manual-install.md +1 -1
- package/.docs/docs/memory/observational-memory.md +19 -0
- package/.docs/docs/memory/semantic-recall.md +1 -1
- package/.docs/docs/observability/metrics/overview.md +1 -0
- package/.docs/docs/observability/metrics/querying.md +292 -0
- package/.docs/docs/server/auth/fga.md +2 -0
- package/.docs/docs/voice/overview.md +62 -0
- package/.docs/docs/voice/speech-to-speech.md +52 -1
- package/.docs/docs/workspace/sandbox.md +4 -2
- package/.docs/guides/deployment/inngest.md +69 -0
- package/.docs/guides/guide/firecrawl.md +5 -5
- package/.docs/models/embeddings.md +2 -2
- package/.docs/reference/agents/agent.md +46 -17
- package/.docs/reference/agents/channels.md +1 -1
- package/.docs/reference/evals/create-scorer.md +2 -0
- package/.docs/reference/index.md +3 -0
- package/.docs/reference/memory/observational-memory.md +2 -0
- package/.docs/reference/observability/metrics/automatic-metrics.md +7 -1
- package/.docs/reference/processors/tool-search-processor.md +31 -0
- package/.docs/reference/server/routes.md +81 -9
- package/.docs/reference/templates/overview.md +2 -2
- package/.docs/reference/tools/mcp-client.md +51 -0
- package/.docs/reference/vectors/pg.md +2 -0
- package/.docs/reference/voice/inworld-realtime.md +353 -0
- package/.docs/reference/voice/inworld.md +2 -0
- package/.docs/reference/workspace/agentcore-runtime-sandbox.md +202 -0
- package/.docs/reference/workspace/blaxel-sandbox.md +3 -0
- package/.docs/reference/workspace/docker-sandbox.md +3 -1
- package/.docs/reference/workspace/vercel-microvm-sandbox.md +199 -0
- package/.docs/reference/workspace/vercel.md +2 -0
- package/CHANGELOG.md +15 -0
- package/package.json +7 -5
|
@@ -0,0 +1,353 @@
|
|
|
1
|
+
# Inworld Realtime voice
|
|
2
|
+
|
|
3
|
+
The `InworldRealtimeVoice` class provides real-time, full-duplex voice interaction using [Inworld AI's Realtime API](https://docs.inworld.ai/realtime/quickstart-websocket) over WebSockets. It supports speech-to-speech, tool calling, and Inworld-specific session knobs such as semantic voice activity detection, MCP tool routing, and playback speed.
|
|
4
|
+
|
|
5
|
+
Inworld's wire protocol is the OpenAI Realtime GA spec, so client and server event names match `@mastra/voice-openai-realtime`. The provider-level differences are the endpoint (which uses a client-generated session key in the URL), the `Authorization: Basic <key>` header, the typed `session` constructor field for Inworld-specific knobs, and a typed `providerData` object for Inworld extensions (STT, TTS, memory, back-channel, responsiveness) sent under `session.providerData`.
|
|
6
|
+
|
|
7
|
+
For batch text-to-speech and speech-to-text, see [`@mastra/voice-inworld`](https://mastra.ai/reference/voice/inworld).
|
|
8
|
+
|
|
9
|
+
## Usage example
|
|
10
|
+
|
|
11
|
+
```typescript
|
|
12
|
+
import { InworldRealtimeVoice } from '@mastra/voice-inworld'
|
|
13
|
+
import { playAudio, getMicrophoneStream } from '@mastra/node-audio'
|
|
14
|
+
|
|
15
|
+
// Initialize with INWORLD_API_KEY from the environment
|
|
16
|
+
const voice = new InworldRealtimeVoice()
|
|
17
|
+
|
|
18
|
+
// Or initialize with explicit configuration
|
|
19
|
+
const voiceWithConfig = new InworldRealtimeVoice({
|
|
20
|
+
apiKey: 'your-inworld-api-key',
|
|
21
|
+
model: 'inworld/models/gemma-4-26b-a4b-it',
|
|
22
|
+
speaker: 'Sarah',
|
|
23
|
+
instructions: 'You are a helpful voice assistant.',
|
|
24
|
+
session: {
|
|
25
|
+
audio: {
|
|
26
|
+
output: { speed: 1.1 },
|
|
27
|
+
input: { turn_detection: { type: 'semantic_vad', eagerness: 'high' } },
|
|
28
|
+
},
|
|
29
|
+
},
|
|
30
|
+
})
|
|
31
|
+
|
|
32
|
+
// Establish connection
|
|
33
|
+
await voice.connect()
|
|
34
|
+
|
|
35
|
+
// Listen for audio output (PCM16 @ 24 kHz by default)
|
|
36
|
+
voice.on('speaker', stream => {
|
|
37
|
+
playAudio(stream)
|
|
38
|
+
})
|
|
39
|
+
|
|
40
|
+
voice.on('writing', ({ text, role }) => {
|
|
41
|
+
console.log(`${role}: ${text}`)
|
|
42
|
+
})
|
|
43
|
+
|
|
44
|
+
// Convert text to speech
|
|
45
|
+
await voice.speak('Hello, how can I help you today?', {
|
|
46
|
+
speaker: 'Hades',
|
|
47
|
+
})
|
|
48
|
+
|
|
49
|
+
// Stream microphone audio to the model
|
|
50
|
+
const microphoneStream = getMicrophoneStream()
|
|
51
|
+
await voice.send(microphoneStream)
|
|
52
|
+
|
|
53
|
+
// Clean up
|
|
54
|
+
voice.close()
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
> Inworld API keys ship pre-Basic-encoded. Paste them verbatim into `INWORLD_API_KEY`; the package does not re-encode them.
|
|
58
|
+
|
|
59
|
+
## Constructor parameters
|
|
60
|
+
|
|
61
|
+
**apiKey** (`string`): Inworld API key. Falls back to the INWORLD\_API\_KEY environment variable. Keys are Basic-encoded and passed verbatim in the Authorization header.
|
|
62
|
+
|
|
63
|
+
**url** (`string`): Realtime WebSocket endpoint. A client-generated session key and protocol parameter are appended automatically. (Default: `'wss://api.inworld.ai/api/v1/realtime/session'`)
|
|
64
|
+
|
|
65
|
+
**model** (`string`): LLM Router model ID. Sent via the initial session.update, not in the URL. Any model supported by Inworld's router is accepted. (Default: `'inworld/models/gemma-4-26b-a4b-it'`)
|
|
66
|
+
|
|
67
|
+
**speaker** (`string`): Default voice ID for speech synthesis. Any voice from Inworld's catalog is accepted. (Default: `'Sarah'`)
|
|
68
|
+
|
|
69
|
+
**sessionId** (`string`): Client-generated session key surfaced as the URL \`key\` parameter. A timestamp-based key is generated automatically when omitted. (Default: `'voice-{Date.now()}'`)
|
|
70
|
+
|
|
71
|
+
**instructions** (`string`): System prompt sent with the initial session.update.
|
|
72
|
+
|
|
73
|
+
**session** (`Partial<InworldSessionConfig>`): Typed first-class session options (audio, tool\_choice, output\_modalities, temperature, ...). Deep-merged into every session.update so nested fields like audio.output.voice and audio.output.speed compose rather than overwrite each other. See the session field below.
|
|
74
|
+
|
|
75
|
+
**debug** (`boolean`): Log raw server events. (Default: `false`)
|
|
76
|
+
|
|
77
|
+
**providerData** (`InworldProviderData`): Typed Inworld extension config (stt, tts, memory, backchannel, responsiveness, plus user\_id and metadata). Sent under session.providerData on every session.update. Composes with any session.providerData set via the \`session\` field; the constructor option wins on key collisions.
|
|
78
|
+
|
|
79
|
+
**connectTimeoutMs** (`number`): Max time \`connect()\` will wait for both the WebSocket handshake and the initial \`session.updated\` round-trip. A pre-open error or close on the WebSocket — or this timeout expiring — surfaces as a rejected promise instead of an uncaught socket error. (Default: `15000`)
|
|
80
|
+
|
|
81
|
+
### `session` (typed knobs)
|
|
82
|
+
|
|
83
|
+
Use the typed `session` field for documented Inworld realtime options. Fields compose with the connect-time defaults (e.g. `audio.output.voice` set from `speaker`):
|
|
84
|
+
|
|
85
|
+
**output\_modalities** (`Array<"text" | "audio">`): Modalities the model should produce.
|
|
86
|
+
|
|
87
|
+
**audio.output.voice** (`string`): Voice catalog ID. When omitted, the constructor \`speaker\` is used.
|
|
88
|
+
|
|
89
|
+
**audio.output.speed** (`number`): Playback speed multiplier for synthesized audio (0.25 to 1.5).
|
|
90
|
+
|
|
91
|
+
**audio.output.model** (`string`): Inworld TTS model (e.g. "inworld-tts-2").
|
|
92
|
+
|
|
93
|
+
**audio.output.format** (`InworldAudioFormat`): Output audio encoding. A codec string (e.g. "audio/pcm", "audio/pcmu", "audio/pcma", "audio/float32") or an object \`{ type, rate? }\`. \`rate\` (Hz) applies to audio/pcm and audio/float32 (default 24000); audio/pcmu and audio/pcma are fixed at 8 kHz.
|
|
94
|
+
|
|
95
|
+
**audio.input.format** (`InworldAudioFormat`): Input audio encoding sent to the server. Same shape as \`audio.output.format\` — a codec string or \`{ type, rate? }\` object.
|
|
96
|
+
|
|
97
|
+
**audio.input.noise\_reduction** (`{ type: "near_field" | "far_field" }`): Input noise-reduction mode applied before transcription and VAD.
|
|
98
|
+
|
|
99
|
+
**audio.input.transcription** (`{ model?: string; language?: string; prompt?: string }`): Server-side transcription for incoming user audio. Defaults to \`{ model: "inworld/inworld-stt-1" }\`. \`prompt\` biases transcription with vocabulary, spelling, or style hints. Supply your own object to override; set to \`null\` to disable user-side transcription.
|
|
100
|
+
|
|
101
|
+
**audio.input.turn\_detection** (`InworldTurnDetection | null`): Voice activity / turn detection. Defaults to \`{ type: "semantic\_vad", eagerness: "medium", create\_response: true, interrupt\_response: true }\`. Supply your own object to override; set to \`null\` to disable turn detection entirely. The \`eagerness\` field controls how quickly semantic VAD ends a user turn — \`low\` waits for clearer pauses (more interruption-resistant), \`high\` ends turns sooner (snappier, more prone to cutting users off). Default \`medium\` balances both. \`idle\_timeout\_ms\` (server\_vad only) sets the idle window before the server commits a turn.
|
|
102
|
+
|
|
103
|
+
**tool\_choice** (`string | { type: "function"; name: string } | { type: "mcp"; server_label: string }`): Tool selection strategy. Use the mcp variant to route tool calls through a configured Inworld MCP server.
|
|
104
|
+
|
|
105
|
+
**temperature** (`number`): Sampling temperature for the model.
|
|
106
|
+
|
|
107
|
+
**max\_output\_tokens** (`number | "inf"`): Maximum tokens generated per response.
|
|
108
|
+
|
|
109
|
+
**truncation** (`"auto" | "disabled" | { type: "retention_ratio"; retention_ratio: number }`): Conversation truncation strategy.
|
|
110
|
+
|
|
111
|
+
**tracing** (`"auto" | { workflow_name?: string; group_id?: string; metadata?: Record<string, unknown> }`): Distributed-tracing config. Use "auto" for server defaults, or name the workflow/group explicitly.
|
|
112
|
+
|
|
113
|
+
**include** (`Array<"item.input_audio_transcription.logprobs">`): Opt-in extra fields the server should include on emitted events.
|
|
114
|
+
|
|
115
|
+
**prompt** (`string | null`): Reference to a server-side prompt template. Pass null to clear it.
|
|
116
|
+
|
|
117
|
+
### `providerData` (Inworld extensions)
|
|
118
|
+
|
|
119
|
+
`providerData` is a typed object for Inworld-specific realtime extensions. It is sent under `session.providerData` on every `session.update`, and composes with any `session.providerData` you set via the `session` field — the constructor `providerData` wins on key collisions.
|
|
120
|
+
|
|
121
|
+
It has five branches plus two session-level fields:
|
|
122
|
+
|
|
123
|
+
- `stt`: STT tuning, such as `prompt`, `voice_profile`, `language_hints`, and VAD or end-of-turn thresholds.
|
|
124
|
+
- `tts`: TTS segmentation and delivery, such as `segmenter_strategy`, `steering_handling`, `delivery_mode`, `conversational`, and `user_turn_mode`.
|
|
125
|
+
- `memory`: automatic rolling memory, such as `enabled`, `turn_interval`, and `max_facts`. Inworld echoes its state back through the `memory` event.
|
|
126
|
+
- `backchannel`: short acknowledgements ("uh-huh") while the user speaks. Audio arrives on the `backchannel` event.
|
|
127
|
+
- `responsiveness`: early filler audio while the main response generates. Filler audio reuses the normal `speaker` and `speaking` events, so there are no distinct events.
|
|
128
|
+
- `user_id` and `metadata`: session-level identifiers passed through to Inworld.
|
|
129
|
+
|
|
130
|
+
```typescript
|
|
131
|
+
const voice = new InworldRealtimeVoice({
|
|
132
|
+
providerData: {
|
|
133
|
+
stt: { voice_profile: true, language_hints: ['en-US'] },
|
|
134
|
+
tts: { delivery_mode: 'CREATIVE', segmenter_strategy: 'balanced' },
|
|
135
|
+
memory: { enabled: true, turn_interval: 4 },
|
|
136
|
+
backchannel: { enabled: true, max_per_turn: 1 },
|
|
137
|
+
user_id: 'user-123',
|
|
138
|
+
},
|
|
139
|
+
})
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
## Methods
|
|
143
|
+
|
|
144
|
+
### `connect()`
|
|
145
|
+
|
|
146
|
+
Opens the WebSocket connection, sends the initial `session.update`, and resolves once the server acknowledges with `session.updated`. Must be called before `speak()`, `listen()`, or `send()`.
|
|
147
|
+
|
|
148
|
+
A pre-open `error` or `close` on the WebSocket — or a handshake that exceeds `connectTimeoutMs` (15s default) — surfaces as a rejected promise instead of an uncaught socket error. On reject, the half-open socket is closed.
|
|
149
|
+
|
|
150
|
+
```typescript
|
|
151
|
+
await voice.connect()
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
Returns: `Promise<void>`
|
|
155
|
+
|
|
156
|
+
### `speak()`
|
|
157
|
+
|
|
158
|
+
Sends a text message to the model and triggers an audio response. The returned promise resolves only after the full response lifecycle completes (`response.done` for the response this call triggered), and rejects if the response is interrupted by user speech or if a transport error occurs.
|
|
159
|
+
|
|
160
|
+
Serial `speak()` calls are the supported pattern. Concurrent calls share the same listener pool and have undefined response-pinning order.
|
|
161
|
+
|
|
162
|
+
**input** (`string | NodeJS.ReadableStream`): Text or text stream to convert to speech.
|
|
163
|
+
|
|
164
|
+
**options** (`Options`): Per-call configuration.
|
|
165
|
+
|
|
166
|
+
**options.speaker** (`string`): Voice ID to use for this specific request.
|
|
167
|
+
|
|
168
|
+
Returns: `Promise<void>`
|
|
169
|
+
|
|
170
|
+
### `listen()`
|
|
171
|
+
|
|
172
|
+
Sends a single audio buffer as a user turn and asks the model to respond with text only.
|
|
173
|
+
|
|
174
|
+
**audioData** (`NodeJS.ReadableStream`): Audio stream to transcribe.
|
|
175
|
+
|
|
176
|
+
Returns: `Promise<void>`
|
|
177
|
+
|
|
178
|
+
### `send()`
|
|
179
|
+
|
|
180
|
+
Streams audio data to the server in real time. Useful for continuous microphone input.
|
|
181
|
+
|
|
182
|
+
**audioData** (`NodeJS.ReadableStream | Int16Array`): Audio data to stream. Int16Array is sent as a single base64 chunk; a readable stream is forwarded chunk by chunk.
|
|
183
|
+
|
|
184
|
+
**eventId** (`string`): Optional event ID forwarded to the server with each audio chunk.
|
|
185
|
+
|
|
186
|
+
Returns: `Promise<void>`
|
|
187
|
+
|
|
188
|
+
### `updateConfig()`
|
|
189
|
+
|
|
190
|
+
Sends a `session.update` to the server. The typed `session` field is deep-merged into the payload, and any constructor `providerData` is nested under `session.providerData`.
|
|
191
|
+
|
|
192
|
+
**sessionConfig** (`InworldSessionConfig | Record<string, unknown>`): Partial session configuration to apply.
|
|
193
|
+
|
|
194
|
+
Returns: `void`
|
|
195
|
+
|
|
196
|
+
### `addInstructions()`
|
|
197
|
+
|
|
198
|
+
Sets the system instructions used on the next `connect()` or `updateConfig()` call.
|
|
199
|
+
|
|
200
|
+
**instructions** (`string`): System prompt for the model.
|
|
201
|
+
|
|
202
|
+
Returns: `void`
|
|
203
|
+
|
|
204
|
+
### `addTools()`
|
|
205
|
+
|
|
206
|
+
Registers tools that the model can call during the session. When `InworldRealtimeVoice` is attached to an Agent, tools configured for the Agent are made available automatically.
|
|
207
|
+
|
|
208
|
+
**tools** (`ToolsInput`): Tools configuration to equip.
|
|
209
|
+
|
|
210
|
+
Returns: `void`
|
|
211
|
+
|
|
212
|
+
### `answer()`
|
|
213
|
+
|
|
214
|
+
Sends a `response.create` event to trigger a model response, optionally with per-response options.
|
|
215
|
+
|
|
216
|
+
**options** (`Record<string, unknown>`): Response options forwarded to the server.
|
|
217
|
+
|
|
218
|
+
Returns: `Promise<void>`
|
|
219
|
+
|
|
220
|
+
### Turn-taking
|
|
221
|
+
|
|
222
|
+
#### `commitInput()`
|
|
223
|
+
|
|
224
|
+
Manually commits buffered input audio as a user turn. Use this for push-to-talk or manual turn-taking when `turn_detection` is set to `null`.
|
|
225
|
+
|
|
226
|
+
```typescript
|
|
227
|
+
voice.commitInput()
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
Returns: `void`
|
|
231
|
+
|
|
232
|
+
#### `clearInput()`
|
|
233
|
+
|
|
234
|
+
Discards buffered input audio without committing it as a user turn.
|
|
235
|
+
|
|
236
|
+
```typescript
|
|
237
|
+
voice.clearInput()
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
Returns: `void`
|
|
241
|
+
|
|
242
|
+
#### `clearOutput()`
|
|
243
|
+
|
|
244
|
+
Clears the server's entire output audio buffer, stopping playback. This also stops any in-flight back-channel audio. The default barge-in path (`response.cancel` on `interrupted`) is back-channel-safe; prefer it. Use `clearOutput()` only when you want to flush everything.
|
|
245
|
+
|
|
246
|
+
```typescript
|
|
247
|
+
voice.clearOutput()
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
Returns: `void`
|
|
251
|
+
|
|
252
|
+
### `close()` and `disconnect()`
|
|
253
|
+
|
|
254
|
+
Both methods close the WebSocket and mark the instance as disconnected.
|
|
255
|
+
|
|
256
|
+
Returns: `void`
|
|
257
|
+
|
|
258
|
+
### `getSpeakers()`
|
|
259
|
+
|
|
260
|
+
Returns the curated voice list bundled with the package. Inworld's catalog is larger than this list; any voice ID can be passed to `speaker` at runtime.
|
|
261
|
+
|
|
262
|
+
Returns: `Promise<Array<{ voiceId: string }>>`
|
|
263
|
+
|
|
264
|
+
### `on()` and `off()`
|
|
265
|
+
|
|
266
|
+
Register and remove event listeners. See [Events](#events) below.
|
|
267
|
+
|
|
268
|
+
## Events
|
|
269
|
+
|
|
270
|
+
The `InworldRealtimeVoice` class emits the following events:
|
|
271
|
+
|
|
272
|
+
**speaker** (`event`): Emitted once per response with a PassThrough stream of PCM audio. Use this when piping audio to a player.
|
|
273
|
+
|
|
274
|
+
**speaking** (`event`): Emitted for each audio delta. Callback receives { audio: Buffer, response\_id: string }.
|
|
275
|
+
|
|
276
|
+
**speaking.done** (`event`): Emitted when audio output for a response is complete. Callback receives { response\_id: string }.
|
|
277
|
+
|
|
278
|
+
**writing** (`event`): Emitted as transcribed text becomes available. Callback receives { text: string, response\_id: string, role: "assistant" | "user", voiceProfile? }. Deduplicated across audio-transcript and text deltas in the same response so a single response only emits one stream. On user events, voiceProfile is present when providerData.stt.voice\_profile is enabled.
|
|
279
|
+
|
|
280
|
+
**speech-started** (`event`): Raw \`input\_audio\_buffer.speech\_started\` VAD edge from the server.
|
|
281
|
+
|
|
282
|
+
**speech-stopped** (`event`): Raw \`input\_audio\_buffer.speech\_stopped\` VAD edge from the server.
|
|
283
|
+
|
|
284
|
+
**interrupted** (`event`): Synthetic client-side signal: emitted once per in-flight \`response\_id\` when the user starts speaking. Use this to stop main response playback on barge-in. Callback receives \`{ response\_id: string }\`. Only carries main-response ids — never back-channel ids — so stopping the matching \`speaker\` stream leaves \`backchannel\` streams playing (back-channels are meant to overlap user speech and are never cancelled by barge-in).
|
|
285
|
+
|
|
286
|
+
**turn-suggestion** (`event`): Smart-turn endpointing hint for a buffered user utterance. Callback receives { item\_id, utterance\_index, probability, trailing\_silence\_ms?, audio\_duration\_ms?, inference\_ms? }.
|
|
287
|
+
|
|
288
|
+
**turn-suggestion-revoked** (`event`): A previously emitted turn suggestion was retracted. Callback receives { item\_id, utterance\_index }.
|
|
289
|
+
|
|
290
|
+
**input-committed** (`event`): Buffered input audio was committed as a user turn (via commitInput() or auto-VAD). Callback receives { item\_id, previous\_item\_id? } where previous\_item\_id may be null.
|
|
291
|
+
|
|
292
|
+
**input-cleared** (`event`): Buffered input audio was discarded (via clearInput()). Callback receives {}.
|
|
293
|
+
|
|
294
|
+
**input-timeout** (`event`): A server-VAD idle timeout committed a user turn. Callback receives { audio\_start\_ms, audio\_end\_ms, item\_id }.
|
|
295
|
+
|
|
296
|
+
**output-audio-started** (`event`): Server began emitting output audio. Callback receives {}.
|
|
297
|
+
|
|
298
|
+
**output-audio-stopped** (`event`): Server stopped emitting output audio for the current response. Callback receives {}.
|
|
299
|
+
|
|
300
|
+
**output-audio-cleared** (`event`): Server output audio buffer was flushed, stopping playback (via clearOutput()). Callback receives {}.
|
|
301
|
+
|
|
302
|
+
**memory** (`event`): Emitted with Inworld's rolling summary and facts state, deduplicated by version. Requires providerData.memory.enabled. Callback receives InworldMemoryState.
|
|
303
|
+
|
|
304
|
+
**backchannel** (`event`): Emitted with a PassThrough stream of back-channel PCM audio (short acknowledgements while the user speaks). Each stream's \`.id\` is a \`backchannel\_id\` that never appears in \`interrupted\`, so play these on a separate track that barge-in does not stop. Requires providerData.backchannel.enabled.
|
|
305
|
+
|
|
306
|
+
**backchannel.done** (`event`): Emitted when a back-channel finishes. Callback receives { backchannel\_id: string, phrase? }.
|
|
307
|
+
|
|
308
|
+
**backchannel.skipped** (`event`): Emitted when the decider skips a back-channel before any audio is produced. Callback receives { reason: string }.
|
|
309
|
+
|
|
310
|
+
**response.created** (`event`): Emitted when a new response begins. Callback receives the full server event.
|
|
311
|
+
|
|
312
|
+
**response.done** (`event`): Emitted when a response completes. Callback receives the full server event.
|
|
313
|
+
|
|
314
|
+
**conversation.item.added** (`event`): Emitted when a new conversation item is appended.
|
|
315
|
+
|
|
316
|
+
**conversation.item.done** (`event`): Emitted when a conversation item finishes.
|
|
317
|
+
|
|
318
|
+
**function\_call.arguments** (`event`): Emitted with complete tool call arguments. Callback receives { call\_id, name, arguments }.
|
|
319
|
+
|
|
320
|
+
**tool-call-start** (`event`): Emitted before a registered tool is executed.
|
|
321
|
+
|
|
322
|
+
**tool-call-result** (`event`): Emitted after a registered tool returns.
|
|
323
|
+
|
|
324
|
+
**error** (`event`): Emitted on transport or server errors.
|
|
325
|
+
|
|
326
|
+
## Voices
|
|
327
|
+
|
|
328
|
+
The package ships with a curated set of voice IDs returned from `getSpeakers()`:
|
|
329
|
+
|
|
330
|
+
- `Dennis`
|
|
331
|
+
- `Hades`
|
|
332
|
+
- `Wendy`
|
|
333
|
+
- `Edward`
|
|
334
|
+
- `Olivia`
|
|
335
|
+
- `Sarah`
|
|
336
|
+
- `Timothy`
|
|
337
|
+
- `Priya`
|
|
338
|
+
- `Ronald`
|
|
339
|
+
- `Deborah`
|
|
340
|
+
|
|
341
|
+
Any voice ID from [Inworld's voice catalog](https://docs.inworld.ai/quickstart-tts) can be passed to `speaker` at runtime.
|
|
342
|
+
|
|
343
|
+
## Notes
|
|
344
|
+
|
|
345
|
+
- API keys can be provided via constructor options or the `INWORLD_API_KEY` environment variable. Keys are pre-Basic-encoded; do not re-encode them.
|
|
346
|
+
- The WebSocket URL appends `?key=<sessionId>&protocol=realtime`. The model is configured via the initial `session.update`, not the URL.
|
|
347
|
+
- Per-call `speak(input, { speaker })` scopes the voice override to a single response (via the flat `response.voice` field) and does NOT mutate the session.
|
|
348
|
+
- Audio output defaults to PCM16 at 24 kHz. Telephony `audio/pcmu` and `audio/pcma` at 8 kHz, and `audio/float32`, are also supported via `session.audio.output.format`.
|
|
349
|
+
- Use `connect()` before any send, speak, or listen call. Events sent before the WebSocket is open are queued and flushed once the server acknowledges `session.updated`.
|
|
350
|
+
- The voice instance must be closed with `close()` or `disconnect()` to release the WebSocket.
|
|
351
|
+
- `audio.input.turn_detection` defaults to semantic VAD when `session` does not supply it. Override with your own object, or pass `null` to disable turn detection entirely.
|
|
352
|
+
- `audio.input.transcription` defaults to `{ model: 'inworld/inworld-stt-1' }`, so user-side `writing` events fire out of the box. Override with your own object, or pass `null` to disable user-side transcription.
|
|
353
|
+
- `on()` and `off()` are typed against `InworldVoiceEventMap` — known event names yield a typed callback payload, unknown names fall back to `unknown`.
|
|
@@ -2,6 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
The Inworld voice implementation in Mastra provides streaming text-to-speech (TTS) and batch speech-to-text (STT) capabilities using Inworld AI's API. It supports multiple TTS and STT models, configurable audio encodings, and progressive audio streaming.
|
|
4
4
|
|
|
5
|
+
For real-time, full-duplex speech-to-speech, the same package exports [`InworldRealtimeVoice`](https://mastra.ai/reference/voice/inworld-realtime).
|
|
6
|
+
|
|
5
7
|
## Usage example
|
|
6
8
|
|
|
7
9
|
```typescript
|
|
@@ -0,0 +1,202 @@
|
|
|
1
|
+
# AgentCoreRuntimeSandbox
|
|
2
|
+
|
|
3
|
+
Executes shell commands in an [AWS Bedrock AgentCore Runtime](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-execute-command.html) session by using `InvokeAgentRuntimeCommand`.
|
|
4
|
+
|
|
5
|
+
Use `AgentCoreRuntimeSandbox` when your agent already runs in AgentCore Runtime and you want Mastra workspace command execution to use the same runtime session.
|
|
6
|
+
|
|
7
|
+
> **Info:** For interface details, see [WorkspaceSandbox interface](https://mastra.ai/reference/workspace/sandbox).
|
|
8
|
+
|
|
9
|
+
> **Warning:** `AgentCoreRuntimeSandbox` only supports one-shot command execution. It does not support background process management, stdin, or filesystem mounts. AgentCore Code Interpreter is a separate AWS service and is not part of this provider.
|
|
10
|
+
|
|
11
|
+
## Installation
|
|
12
|
+
|
|
13
|
+
**npm**:
|
|
14
|
+
|
|
15
|
+
```bash
|
|
16
|
+
npm install @mastra/agentcore
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
**pnpm**:
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
pnpm add @mastra/agentcore
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
**Yarn**:
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
yarn add @mastra/agentcore
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
**Bun**:
|
|
32
|
+
|
|
33
|
+
```bash
|
|
34
|
+
bun add @mastra/agentcore
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
## Usage
|
|
38
|
+
|
|
39
|
+
Add an `AgentCoreRuntimeSandbox` to a workspace and assign it to an agent:
|
|
40
|
+
|
|
41
|
+
```typescript
|
|
42
|
+
import { Agent } from '@mastra/core/agent'
|
|
43
|
+
import { Workspace } from '@mastra/core/workspace'
|
|
44
|
+
import { AgentCoreRuntimeSandbox } from '@mastra/agentcore'
|
|
45
|
+
|
|
46
|
+
const workspace = new Workspace({
|
|
47
|
+
sandbox: new AgentCoreRuntimeSandbox({
|
|
48
|
+
region: 'us-west-2',
|
|
49
|
+
agentRuntimeArn: process.env.AGENTCORE_RUNTIME_ARN!,
|
|
50
|
+
runtimeSessionId: '12345678-1234-1234-1234-123456789012',
|
|
51
|
+
}),
|
|
52
|
+
})
|
|
53
|
+
|
|
54
|
+
const agent = new Agent({
|
|
55
|
+
name: 'dev-agent',
|
|
56
|
+
model: 'anthropic/claude-sonnet-4-6',
|
|
57
|
+
instructions: 'You are a helpful development assistant.',
|
|
58
|
+
workspace,
|
|
59
|
+
})
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
Run commands programmatically through the sandbox:
|
|
63
|
+
|
|
64
|
+
```typescript
|
|
65
|
+
const result = await workspace.sandbox?.executeCommand?.('npm', ['test'], {
|
|
66
|
+
cwd: '/workspace',
|
|
67
|
+
env: {
|
|
68
|
+
NODE_ENV: 'test',
|
|
69
|
+
},
|
|
70
|
+
timeout: 300_000,
|
|
71
|
+
})
|
|
72
|
+
|
|
73
|
+
if (!result?.success) {
|
|
74
|
+
console.error(result?.stderr)
|
|
75
|
+
}
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
## Constructor parameters
|
|
79
|
+
|
|
80
|
+
**agentRuntimeArn** (`string`): AgentCore Runtime ARN where commands execute.
|
|
81
|
+
|
|
82
|
+
**region** (`string`): AWS region for the Bedrock AgentCore client. Falls back to the AWS SDK default region chain.
|
|
83
|
+
|
|
84
|
+
**runtimeSessionId** (`string`): AgentCore Runtime session ID. Defaults to a generated UUID, which satisfies AgentCore Runtime session ID length requirements. (Default: `Generated UUID`)
|
|
85
|
+
|
|
86
|
+
**qualifier** (`string`): Agent runtime qualifier or endpoint. (Default: `DEFAULT`)
|
|
87
|
+
|
|
88
|
+
**contentType** (`string`): MIME type sent for command requests. (Default: `application/json`)
|
|
89
|
+
|
|
90
|
+
**accept** (`string`): Accept header used for command event streams. (Default: `application/vnd.amazon.eventstream`)
|
|
91
|
+
|
|
92
|
+
**commandTimeout** (`number`): Default command timeout in milliseconds. (Default: `300000`)
|
|
93
|
+
|
|
94
|
+
**stopSessionOnLifecycle** (`boolean`): Whether \`stop()\` and \`destroy()\` should call \`StopRuntimeSession\`. Defaults to false because AgentCore Runtime sessions are often shared with agent invocations outside the sandbox instance. (Default: `false`)
|
|
95
|
+
|
|
96
|
+
**stopClientToken** (`string`): Client token used when \`StopRuntimeSession\` is called. (Default: `Generated UUID`)
|
|
97
|
+
|
|
98
|
+
**client** (`BedrockAgentCoreClient`): Preconfigured AWS SDK client. Use this for custom credentials, retry behavior, or tests.
|
|
99
|
+
|
|
100
|
+
**instructions** (`string | ((opts) => string)`): Custom instructions that override the default instructions returned by \`getInstructions()\`. Pass a string to replace the defaults, or a function to extend them.
|
|
101
|
+
|
|
102
|
+
## Properties
|
|
103
|
+
|
|
104
|
+
**id** (`string`): Runtime session ID used by this sandbox instance.
|
|
105
|
+
|
|
106
|
+
**name** (`'AgentCoreRuntimeSandbox'`): Human-readable name.
|
|
107
|
+
|
|
108
|
+
**provider** (`'agentcore'`): Provider type identifier.
|
|
109
|
+
|
|
110
|
+
**status** (`ProviderStatus`): Current lifecycle status: \`'pending'\`, \`'starting'\`, \`'running'\`, \`'stopping'\`, \`'stopped'\`, \`'destroying'\`, \`'destroyed'\`, or \`'error'\`.
|
|
111
|
+
|
|
112
|
+
**runtimeSessionId** (`string`): AgentCore Runtime session ID used for command execution.
|
|
113
|
+
|
|
114
|
+
**agentRuntimeArn** (`string`): AgentCore Runtime ARN where commands execute.
|
|
115
|
+
|
|
116
|
+
## Methods
|
|
117
|
+
|
|
118
|
+
### Command execution
|
|
119
|
+
|
|
120
|
+
#### `executeCommand(command, args?, options?)`
|
|
121
|
+
|
|
122
|
+
Runs a one-shot shell command in the AgentCore Runtime session and returns stdout, stderr, exit code, and timeout status.
|
|
123
|
+
|
|
124
|
+
```typescript
|
|
125
|
+
const result = await sandbox.executeCommand('npm', ['test'], {
|
|
126
|
+
cwd: '/workspace',
|
|
127
|
+
env: {
|
|
128
|
+
NODE_ENV: 'test',
|
|
129
|
+
},
|
|
130
|
+
timeout: 300_000,
|
|
131
|
+
})
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
Returns: `Promise<CommandResult>`.
|
|
135
|
+
|
|
136
|
+
`options.timeout` is specified in milliseconds. AgentCore Runtime accepts command timeouts from 1 to 3600 seconds. The provider converts milliseconds to seconds before sending the request.
|
|
137
|
+
|
|
138
|
+
### Lifecycle
|
|
139
|
+
|
|
140
|
+
#### `start()`
|
|
141
|
+
|
|
142
|
+
Runs the sandbox lifecycle start hook. This provider does not create an AgentCore Runtime session during `start()`.
|
|
143
|
+
|
|
144
|
+
```typescript
|
|
145
|
+
await sandbox.start()
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
#### `stop()`
|
|
149
|
+
|
|
150
|
+
Stops the AgentCore Runtime session only when `stopSessionOnLifecycle` is `true`.
|
|
151
|
+
|
|
152
|
+
```typescript
|
|
153
|
+
await sandbox.stop()
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
#### `stopRuntimeSession()`
|
|
157
|
+
|
|
158
|
+
Explicitly stops the AgentCore Runtime session used by this sandbox.
|
|
159
|
+
|
|
160
|
+
Use this when the sandbox owns the runtime session and you want to clean it up directly. `destroy()` does not call this method unless `stopSessionOnLifecycle` is `true`, because AgentCore Runtime sessions can be shared with agent invocations outside the Workspace sandbox lifecycle.
|
|
161
|
+
|
|
162
|
+
```typescript
|
|
163
|
+
await sandbox.stopRuntimeSession()
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
#### `destroy()`
|
|
167
|
+
|
|
168
|
+
Destroys the sandbox instance. If this instance owns its AWS SDK client, `destroy()` also destroys the client. If `stopSessionOnLifecycle` is `true`, it calls `StopRuntimeSession`.
|
|
169
|
+
|
|
170
|
+
```typescript
|
|
171
|
+
await sandbox.destroy()
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
### Metadata
|
|
175
|
+
|
|
176
|
+
#### `getInfo()`
|
|
177
|
+
|
|
178
|
+
Returns sandbox status and AgentCore Runtime metadata.
|
|
179
|
+
|
|
180
|
+
```typescript
|
|
181
|
+
const info = await sandbox.getInfo()
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
Returns: `Promise<SandboxInfo>`.
|
|
185
|
+
|
|
186
|
+
## Limitations
|
|
187
|
+
|
|
188
|
+
`AgentCoreRuntimeSandbox` follows AgentCore Runtime command execution semantics:
|
|
189
|
+
|
|
190
|
+
- **One-shot commands**: Each command runs to completion or timeout.
|
|
191
|
+
- **No persistent shell**: Shell state does not carry over between commands. Encode state in each command, for example `cd /workspace && npm test`.
|
|
192
|
+
- **Background processes are not supported**: The provider does not expose a `processes` manager.
|
|
193
|
+
- **Interactive stdin is unavailable**: Runtime command execution does not provide an interactive stdin stream through this provider.
|
|
194
|
+
- **Workspace filesystem mounts are unsupported**: Workspace filesystem mounting is not supported by this provider.
|
|
195
|
+
- **Container-dependent tools**: Commands can only use tools installed in the AgentCore Runtime container image.
|
|
196
|
+
|
|
197
|
+
## Related
|
|
198
|
+
|
|
199
|
+
- [WorkspaceSandbox interface](https://mastra.ai/reference/workspace/sandbox)
|
|
200
|
+
- [Workspace class](https://mastra.ai/reference/workspace/workspace-class)
|
|
201
|
+
- [Sandbox overview](https://mastra.ai/docs/workspace/sandbox)
|
|
202
|
+
- [AWS Bedrock AgentCore Runtime command execution](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-execute-command.html)
|
|
@@ -61,6 +61,7 @@ const workspace = new Workspace({
|
|
|
61
61
|
sandbox: new BlaxelSandbox({
|
|
62
62
|
image: 'node:20-slim',
|
|
63
63
|
memory: 8192,
|
|
64
|
+
region: 'auto',
|
|
64
65
|
timeout: '10m',
|
|
65
66
|
}),
|
|
66
67
|
})
|
|
@@ -76,6 +77,8 @@ const workspace = new Workspace({
|
|
|
76
77
|
|
|
77
78
|
**timeout** (`string`): Execution timeout as a duration string (e.g. '5m', '1h'). Maps to the Blaxel sandbox TTL.
|
|
78
79
|
|
|
80
|
+
**region** (`string`): Blaxel region where the sandbox should be created. Use 'auto' to choose a region automatically, or set a concrete region like 'us-pdx-1'. (Default: `process.env.BL_REGION || 'auto'`)
|
|
81
|
+
|
|
79
82
|
**env** (`Record<string, string>`): Environment variables to set in the sandbox.
|
|
80
83
|
|
|
81
84
|
**labels** (`Record<string, string>`): Custom labels for the sandbox.
|
|
@@ -56,7 +56,9 @@ const agent = new Agent({
|
|
|
56
56
|
|
|
57
57
|
## Constructor parameters
|
|
58
58
|
|
|
59
|
-
**id** (`string`): Unique identifier for this sandbox instance. Used for
|
|
59
|
+
**id** (`string`): Unique identifier for this sandbox instance. Used for label-based reconnection. (Default: `Auto-generated`)
|
|
60
|
+
|
|
61
|
+
**name** (`string`): Container display name passed to Docker as \`--name\`. Characters outside \`\[a-zA-Z0-9\_.-]\` are replaced with \`-\` and the result is prefixed if it would not start with an alphanumeric character. (Default: `` The sandbox `id` ``)
|
|
60
62
|
|
|
61
63
|
**image** (`string`): Docker image to use for the container. (Default: `'node:22-slim'`)
|
|
62
64
|
|