@mastra/mcp-docs-server 1.1.42-alpha.4 → 1.1.42-alpha.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.docs/docs/agent-builder/memory.md +1 -1
- package/.docs/docs/agents/adding-voice.md +31 -0
- package/.docs/docs/agents/agent-approval.md +14 -0
- package/.docs/docs/agents/code-mode.md +163 -0
- package/.docs/docs/agents/signals.md +132 -71
- package/.docs/docs/getting-started/manual-install.md +1 -1
- package/.docs/docs/memory/semantic-recall.md +1 -1
- package/.docs/docs/observability/metrics/overview.md +1 -0
- package/.docs/docs/observability/metrics/querying.md +292 -0
- package/.docs/docs/server/auth/fga.md +2 -0
- package/.docs/docs/voice/overview.md +62 -0
- package/.docs/docs/voice/speech-to-speech.md +52 -1
- package/.docs/docs/workspace/sandbox.md +4 -2
- package/.docs/guides/guide/firecrawl.md +5 -5
- package/.docs/models/embeddings.md +2 -2
- package/.docs/reference/agents/agent.md +46 -17
- package/.docs/reference/index.md +3 -0
- package/.docs/reference/observability/metrics/automatic-metrics.md +7 -1
- package/.docs/reference/processors/tool-search-processor.md +31 -0
- package/.docs/reference/server/routes.md +81 -9
- package/.docs/reference/templates/overview.md +2 -2
- package/.docs/reference/tools/mcp-client.md +51 -0
- package/.docs/reference/vectors/pg.md +2 -0
- package/.docs/reference/voice/inworld-realtime.md +353 -0
- package/.docs/reference/voice/inworld.md +2 -0
- package/.docs/reference/workspace/agentcore-runtime-sandbox.md +202 -0
- package/.docs/reference/workspace/blaxel-sandbox.md +3 -0
- package/.docs/reference/workspace/docker-sandbox.md +3 -1
- package/.docs/reference/workspace/vercel-microvm-sandbox.md +199 -0
- package/.docs/reference/workspace/vercel.md +2 -0
- package/CHANGELOG.md +8 -0
- package/package.json +8 -6
|
@@ -35,6 +35,8 @@ const toolSearch = new ToolSearchProcessor({
|
|
|
35
35
|
|
|
36
36
|
**options.ttl** (`number`): Time-to-live for thread state in milliseconds. After this duration of inactivity, thread state will be cleaned up. Set to 0 to disable cleanup.
|
|
37
37
|
|
|
38
|
+
**options.filter** (`(args: ToolSearchFilterArgs) => boolean | Promise<boolean>`): Optional request-aware hook for hiding tools from search results, blocking tool loading, or hiding already-loaded tools for the current request.
|
|
39
|
+
|
|
38
40
|
## Returns
|
|
39
41
|
|
|
40
42
|
**id** (`string`): Processor identifier set to 'tool-search'
|
|
@@ -43,6 +45,35 @@ const toolSearch = new ToolSearchProcessor({
|
|
|
43
45
|
|
|
44
46
|
**processInputStep** (`(args: ProcessInputStepArgs) => Promise<ProcessInputStepResult>`): Processes each step to inject search/load meta-tools and any previously loaded tools into the agent's tool set.
|
|
45
47
|
|
|
48
|
+
## Request-aware filtering
|
|
49
|
+
|
|
50
|
+
Use `filter` to apply request-specific policy to dynamic tools. The hook receives the resolved tool ID as `toolName`, the tool, request context, and phase. `toolName` is the ID returned by `search_tools`, which may differ from the key used in the `tools` object.
|
|
51
|
+
|
|
52
|
+
```typescript
|
|
53
|
+
import { ToolSearchProcessor } from '@mastra/core/processors'
|
|
54
|
+
|
|
55
|
+
const toolSearch = new ToolSearchProcessor({
|
|
56
|
+
tools: allTools,
|
|
57
|
+
filter: ({ toolName, requestContext, phase }) => {
|
|
58
|
+
const plan = requestContext?.get('plan')
|
|
59
|
+
|
|
60
|
+
if (phase === 'search') {
|
|
61
|
+
return true
|
|
62
|
+
}
|
|
63
|
+
|
|
64
|
+
return plan === 'pro' || !toolName.startsWith('premium_')
|
|
65
|
+
},
|
|
66
|
+
})
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
The `phase` value describes where the filter is being applied:
|
|
70
|
+
|
|
71
|
+
- `search`: Filters results returned by `search_tools`.
|
|
72
|
+
- `load`: Blocks `load_tool` from loading disallowed tools.
|
|
73
|
+
- `active`: Hides already-loaded tools from the current request if they are no longer allowed.
|
|
74
|
+
|
|
75
|
+
If the hook throws or rejects, `ToolSearchProcessor` treats the tool as disallowed for that request. The hook may run for every matching search candidate, so keep async policy checks cheap or cached. The meta-tools `search_tools` and `load_tool` are always available. Tools passed directly through the agent or `processInputStep` remain available unless you filter them outside `ToolSearchProcessor`.
|
|
76
|
+
|
|
46
77
|
## Extended usage example
|
|
47
78
|
|
|
48
79
|
```typescript
|
|
@@ -4,15 +4,19 @@ Server adapters register these routes when you call `server.init()`. All routes
|
|
|
4
4
|
|
|
5
5
|
## Agents
|
|
6
6
|
|
|
7
|
-
| Method | Path | Description
|
|
8
|
-
| ------ | -------------------------------------------- |
|
|
9
|
-
| `GET` | `/api/agents` | List all agents
|
|
10
|
-
| `GET` | `/api/agents/:agentId` | Get agent by ID (supports version query params)
|
|
11
|
-
| `POST` | `/api/agents/:agentId/generate` | Generate agent response
|
|
12
|
-
| `POST` | `/api/agents/:agentId/stream` | Stream agent response
|
|
13
|
-
| `POST` | `/api/agents/:agentId/
|
|
14
|
-
| `
|
|
15
|
-
| `POST` | `/api/agents/:agentId/
|
|
7
|
+
| Method | Path | Description |
|
|
8
|
+
| ------ | -------------------------------------------- | ----------------------------------------------------- |
|
|
9
|
+
| `GET` | `/api/agents` | List all agents |
|
|
10
|
+
| `GET` | `/api/agents/:agentId` | Get agent by ID (supports version query params) |
|
|
11
|
+
| `POST` | `/api/agents/:agentId/generate` | Generate agent response |
|
|
12
|
+
| `POST` | `/api/agents/:agentId/stream` | Stream agent response |
|
|
13
|
+
| `POST` | `/api/agents/:agentId/send-message` | Send a user message to an active or idle thread |
|
|
14
|
+
| `POST` | `/api/agents/:agentId/queue-message` | Queue a user message for the next thread turn |
|
|
15
|
+
| `POST` | `/api/agents/:agentId/signals` | Send a lower-level signal to an active or idle thread |
|
|
16
|
+
| `POST` | `/api/agents/:agentId/subscribe-thread` | Subscribe to a thread stream |
|
|
17
|
+
| `POST` | `/api/agents/:agentId/resume-stream` | Resume a suspended agent stream with custom data |
|
|
18
|
+
| `GET` | `/api/agents/:agentId/tools` | List agent tools |
|
|
19
|
+
| `POST` | `/api/agents/:agentId/tools/:toolId/execute` | Execute agent tool |
|
|
16
20
|
|
|
17
21
|
### Get agent query parameters
|
|
18
22
|
|
|
@@ -68,6 +72,74 @@ GET /api/agents/my-agent?versionId=abc123
|
|
|
68
72
|
}
|
|
69
73
|
```
|
|
70
74
|
|
|
75
|
+
### Agent message routes
|
|
76
|
+
|
|
77
|
+
Use `POST /api/agents/:agentId/send-message` to send a user message to the active agent loop or wake an idle thread. Use `POST /api/agents/:agentId/queue-message` when the active run should finish before Mastra starts a follow-up run.
|
|
78
|
+
|
|
79
|
+
Both routes accept the same request body:
|
|
80
|
+
|
|
81
|
+
```typescript
|
|
82
|
+
{
|
|
83
|
+
message: string | Array<TextPart | FilePart> | {
|
|
84
|
+
contents: string | Array<TextPart | FilePart>;
|
|
85
|
+
attributes?: Record<string, JSONValue>;
|
|
86
|
+
metadata?: Record<string, unknown>;
|
|
87
|
+
providerOptions?: ProviderMetadata;
|
|
88
|
+
};
|
|
89
|
+
runId?: string;
|
|
90
|
+
resourceId?: string;
|
|
91
|
+
threadId?: string;
|
|
92
|
+
ifActive?: {
|
|
93
|
+
behavior?: 'deliver' | 'persist' | 'discard';
|
|
94
|
+
attributes?: Record<string, string | number | boolean>;
|
|
95
|
+
};
|
|
96
|
+
ifIdle?: {
|
|
97
|
+
behavior?: 'wake' | 'persist' | 'discard';
|
|
98
|
+
streamOptions?: Omit<AgentExecutionOptions, 'messages'>;
|
|
99
|
+
attributes?: Record<string, string | number | boolean>;
|
|
100
|
+
};
|
|
101
|
+
}
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
When `runId` is omitted, `resourceId` and `threadId` are required. `ifIdle` only applies to thread-targeted requests, not run-targeted requests.
|
|
105
|
+
|
|
106
|
+
#### Send a message
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
curl -X POST http://localhost:4111/api/agents/supportAgent/send-message \
|
|
110
|
+
-H 'Content-Type: application/json' \
|
|
111
|
+
-d '{
|
|
112
|
+
"message": {
|
|
113
|
+
"contents": "Show the shorter version.",
|
|
114
|
+
"attributes": { "sentFrom": "web" }
|
|
115
|
+
},
|
|
116
|
+
"resourceId": "user_123",
|
|
117
|
+
"threadId": "thread_456"
|
|
118
|
+
}'
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
#### Queue a message
|
|
122
|
+
|
|
123
|
+
```bash
|
|
124
|
+
curl -X POST http://localhost:4111/api/agents/supportAgent/queue-message \
|
|
125
|
+
-H 'Content-Type: application/json' \
|
|
126
|
+
-d '{
|
|
127
|
+
"message": "Also check whether the tests need updates.",
|
|
128
|
+
"resourceId": "user_123",
|
|
129
|
+
"threadId": "thread_456"
|
|
130
|
+
}'
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
Both routes return:
|
|
134
|
+
|
|
135
|
+
```typescript
|
|
136
|
+
{
|
|
137
|
+
accepted: true;
|
|
138
|
+
runId: string;
|
|
139
|
+
signal?: CreatedAgentSignal;
|
|
140
|
+
}
|
|
141
|
+
```
|
|
142
|
+
|
|
71
143
|
## Workflows
|
|
72
144
|
|
|
73
145
|
| Method | Path | Description |
|
|
@@ -175,7 +175,7 @@ Include a `.env.example` file with all required environment variables:
|
|
|
175
175
|
# LLM provider API keys (choose one or more)
|
|
176
176
|
OPENAI_API_KEY=your_openai_api_key_here
|
|
177
177
|
ANTHROPIC_API_KEY=your_anthropic_api_key_here
|
|
178
|
-
|
|
178
|
+
GOOGLE_API_KEY=your_google_api_key_here
|
|
179
179
|
|
|
180
180
|
# Other service API keys as needed
|
|
181
181
|
OTHER_SERVICE_API_KEY=your_api_key_here
|
|
@@ -233,7 +233,7 @@ Detailed explanation of the template's functionality and use case.
|
|
|
233
233
|
|
|
234
234
|
- `OPENAI_API_KEY`: Your OpenAI API key. Get one at [OpenAI Platform](https://platform.openai.com/api-keys)
|
|
235
235
|
- `ANTHROPIC_API_KEY`: Your Anthropic API key. Get one at [Anthropic Console](https://console.anthropic.com/settings/keys)
|
|
236
|
-
- `
|
|
236
|
+
- `GOOGLE_API_KEY`: Your Google AI API key. Get one at [Google AI Studio](https://makersuite.google.com/app/apikey)
|
|
237
237
|
- `OTHER_API_KEY`: Description of what this key is for
|
|
238
238
|
|
|
239
239
|
## Usage
|
|
@@ -53,6 +53,10 @@ Each server in the `servers` map is configured using the `MastraMCPServerDefinit
|
|
|
53
53
|
|
|
54
54
|
**enableServerLogs** (`boolean`): Whether to enable logging for this server. (Default: `true`)
|
|
55
55
|
|
|
56
|
+
**forwardInstructions** (`boolean`): Whether to append instructions advertised by this MCP server to an agent's system prompt when the agent uses this server's tools. Disabled by default; enable it only for servers you trust, since the instructions are injected into the agent's system prompt. (Default: `false`)
|
|
57
|
+
|
|
58
|
+
**instructionsMaxLength** (`number`): Maximum number of server instruction characters to append to an agent's system prompt. (Default: `512`)
|
|
59
|
+
|
|
56
60
|
**requireToolApproval** (`boolean | (params: RequireToolApprovalContext) => boolean | Promise<boolean>`): Require human approval before executing tools from this server. When set to \`true\`, all tools require approval. When set to a function, the function is called with the tool name, arguments, request context, and any tool annotations advertised by the server to dynamically decide whether approval is needed.
|
|
57
61
|
|
|
58
62
|
## Tool approval
|
|
@@ -123,6 +127,36 @@ Per the MCP specification, **clients MUST consider tool annotations to be untrus
|
|
|
123
127
|
|
|
124
128
|
The same annotations are also exposed on the tools returned by `listTools()` and `listToolsets()` under `tool.mcp.annotations`, so you can inspect them when wiring tools into an agent.
|
|
125
129
|
|
|
130
|
+
## Server instructions
|
|
131
|
+
|
|
132
|
+
When an MCP server advertises instructions during initialization, `MCPClient` stores them for that server. Forwarding those instructions into an agent's system prompt is **opt-in**: set `forwardInstructions: true` on a server to have agents that use its tools (via `listTools()` or `listToolsets()`) receive its instructions automatically.
|
|
133
|
+
|
|
134
|
+
The guidance is grouped by server name and truncated to `instructionsMaxLength` characters per server.
|
|
135
|
+
|
|
136
|
+
```typescript
|
|
137
|
+
const mcp = new MCPClient({
|
|
138
|
+
servers: {
|
|
139
|
+
db: {
|
|
140
|
+
url: new URL('http://localhost:3000/mcp'),
|
|
141
|
+
forwardInstructions: true,
|
|
142
|
+
instructionsMaxLength: 512,
|
|
143
|
+
},
|
|
144
|
+
},
|
|
145
|
+
})
|
|
146
|
+
|
|
147
|
+
const agent = new Agent({
|
|
148
|
+
id: 'db-agent',
|
|
149
|
+
name: 'DB Agent',
|
|
150
|
+
instructions: 'Help with database changes.',
|
|
151
|
+
model,
|
|
152
|
+
tools: await mcp.listTools(),
|
|
153
|
+
})
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
When `forwardInstructions` is omitted (the default), instructions are still cached and can be inspected via [`getServerInstructions()`](#getserverinstructions), but are not added to any agent's system prompt.
|
|
157
|
+
|
|
158
|
+
> **Security note:** server instructions are forwarded verbatim (subject only to length truncation) into the agent's system prompt. A malicious or compromised MCP server can use them to inject instructions the agent will treat as trusted system guidance. Only enable `forwardInstructions` for servers you trust, and prefer reviewing instructions with `getServerInstructions()` before forwarding instructions from third-party servers.
|
|
159
|
+
|
|
126
160
|
## Methods
|
|
127
161
|
|
|
128
162
|
### `listTools()`
|
|
@@ -143,6 +177,23 @@ const res = await agent.stream(prompt, {
|
|
|
143
177
|
})
|
|
144
178
|
```
|
|
145
179
|
|
|
180
|
+
### `getServerInstructions()`
|
|
181
|
+
|
|
182
|
+
Returns the instructions currently known for each configured MCP server. Servers that have not connected yet, or do not advertise instructions, return `undefined`.
|
|
183
|
+
|
|
184
|
+
```typescript
|
|
185
|
+
getServerInstructions(): Record<string, string | undefined>
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
Example:
|
|
189
|
+
|
|
190
|
+
```typescript
|
|
191
|
+
await mcp.listTools()
|
|
192
|
+
|
|
193
|
+
const instructionsByServer = mcp.getServerInstructions()
|
|
194
|
+
console.log(instructionsByServer.db)
|
|
195
|
+
```
|
|
196
|
+
|
|
146
197
|
### `disconnect()`
|
|
147
198
|
|
|
148
199
|
Disconnects from all MCP servers and cleans up resources.
|
|
@@ -26,6 +26,8 @@ The PgVector class provides vector search using [PostgreSQL](https://www.postgre
|
|
|
26
26
|
|
|
27
27
|
**pgPoolOptions** (`PoolConfig`): Additional pg pool configuration options
|
|
28
28
|
|
|
29
|
+
**disableInit** (`boolean`): When true, automatic DDL (schema, extension, table, and index creation) inside \`createIndex\` is skipped. Useful for CI/CD pipelines where schema and indexes are managed separately and the runtime database role lacks DDL privileges. Can also be enabled with the \`MASTRA\_DISABLE\_STORAGE\_INIT\` environment variable. (Default: `false`)
|
|
30
|
+
|
|
29
31
|
## Constructor examples
|
|
30
32
|
|
|
31
33
|
### Connection String
|
|
@@ -0,0 +1,353 @@
|
|
|
1
|
+
# Inworld Realtime voice
|
|
2
|
+
|
|
3
|
+
The `InworldRealtimeVoice` class provides real-time, full-duplex voice interaction using [Inworld AI's Realtime API](https://docs.inworld.ai/realtime/quickstart-websocket) over WebSockets. It supports speech-to-speech, tool calling, and Inworld-specific session knobs such as semantic voice activity detection, MCP tool routing, and playback speed.
|
|
4
|
+
|
|
5
|
+
Inworld's wire protocol is the OpenAI Realtime GA spec, so client and server event names match `@mastra/voice-openai-realtime`. The provider-level differences are the endpoint (which uses a client-generated session key in the URL), the `Authorization: Basic <key>` header, the typed `session` constructor field for Inworld-specific knobs, and a typed `providerData` object for Inworld extensions (STT, TTS, memory, back-channel, responsiveness) sent under `session.providerData`.
|
|
6
|
+
|
|
7
|
+
For batch text-to-speech and speech-to-text, see [`@mastra/voice-inworld`](https://mastra.ai/reference/voice/inworld).
|
|
8
|
+
|
|
9
|
+
## Usage example
|
|
10
|
+
|
|
11
|
+
```typescript
|
|
12
|
+
import { InworldRealtimeVoice } from '@mastra/voice-inworld'
|
|
13
|
+
import { playAudio, getMicrophoneStream } from '@mastra/node-audio'
|
|
14
|
+
|
|
15
|
+
// Initialize with INWORLD_API_KEY from the environment
|
|
16
|
+
const voice = new InworldRealtimeVoice()
|
|
17
|
+
|
|
18
|
+
// Or initialize with explicit configuration
|
|
19
|
+
const voiceWithConfig = new InworldRealtimeVoice({
|
|
20
|
+
apiKey: 'your-inworld-api-key',
|
|
21
|
+
model: 'inworld/models/gemma-4-26b-a4b-it',
|
|
22
|
+
speaker: 'Sarah',
|
|
23
|
+
instructions: 'You are a helpful voice assistant.',
|
|
24
|
+
session: {
|
|
25
|
+
audio: {
|
|
26
|
+
output: { speed: 1.1 },
|
|
27
|
+
input: { turn_detection: { type: 'semantic_vad', eagerness: 'high' } },
|
|
28
|
+
},
|
|
29
|
+
},
|
|
30
|
+
})
|
|
31
|
+
|
|
32
|
+
// Establish connection
|
|
33
|
+
await voice.connect()
|
|
34
|
+
|
|
35
|
+
// Listen for audio output (PCM16 @ 24 kHz by default)
|
|
36
|
+
voice.on('speaker', stream => {
|
|
37
|
+
playAudio(stream)
|
|
38
|
+
})
|
|
39
|
+
|
|
40
|
+
voice.on('writing', ({ text, role }) => {
|
|
41
|
+
console.log(`${role}: ${text}`)
|
|
42
|
+
})
|
|
43
|
+
|
|
44
|
+
// Convert text to speech
|
|
45
|
+
await voice.speak('Hello, how can I help you today?', {
|
|
46
|
+
speaker: 'Hades',
|
|
47
|
+
})
|
|
48
|
+
|
|
49
|
+
// Stream microphone audio to the model
|
|
50
|
+
const microphoneStream = getMicrophoneStream()
|
|
51
|
+
await voice.send(microphoneStream)
|
|
52
|
+
|
|
53
|
+
// Clean up
|
|
54
|
+
voice.close()
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
> Inworld API keys ship pre-Basic-encoded. Paste them verbatim into `INWORLD_API_KEY`; the package does not re-encode them.
|
|
58
|
+
|
|
59
|
+
## Constructor parameters
|
|
60
|
+
|
|
61
|
+
**apiKey** (`string`): Inworld API key. Falls back to the INWORLD\_API\_KEY environment variable. Keys are Basic-encoded and passed verbatim in the Authorization header.
|
|
62
|
+
|
|
63
|
+
**url** (`string`): Realtime WebSocket endpoint. A client-generated session key and protocol parameter are appended automatically. (Default: `'wss://api.inworld.ai/api/v1/realtime/session'`)
|
|
64
|
+
|
|
65
|
+
**model** (`string`): LLM Router model ID. Sent via the initial session.update, not in the URL. Any model supported by Inworld's router is accepted. (Default: `'inworld/models/gemma-4-26b-a4b-it'`)
|
|
66
|
+
|
|
67
|
+
**speaker** (`string`): Default voice ID for speech synthesis. Any voice from Inworld's catalog is accepted. (Default: `'Sarah'`)
|
|
68
|
+
|
|
69
|
+
**sessionId** (`string`): Client-generated session key surfaced as the URL \`key\` parameter. A timestamp-based key is generated automatically when omitted. (Default: `'voice-{Date.now()}'`)
|
|
70
|
+
|
|
71
|
+
**instructions** (`string`): System prompt sent with the initial session.update.
|
|
72
|
+
|
|
73
|
+
**session** (`Partial<InworldSessionConfig>`): Typed first-class session options (audio, tool\_choice, output\_modalities, temperature, ...). Deep-merged into every session.update so nested fields like audio.output.voice and audio.output.speed compose rather than overwrite each other. See the session field below.
|
|
74
|
+
|
|
75
|
+
**debug** (`boolean`): Log raw server events. (Default: `false`)
|
|
76
|
+
|
|
77
|
+
**providerData** (`InworldProviderData`): Typed Inworld extension config (stt, tts, memory, backchannel, responsiveness, plus user\_id and metadata). Sent under session.providerData on every session.update. Composes with any session.providerData set via the \`session\` field; the constructor option wins on key collisions.
|
|
78
|
+
|
|
79
|
+
**connectTimeoutMs** (`number`): Max time \`connect()\` will wait for both the WebSocket handshake and the initial \`session.updated\` round-trip. A pre-open error or close on the WebSocket — or this timeout expiring — surfaces as a rejected promise instead of an uncaught socket error. (Default: `15000`)
|
|
80
|
+
|
|
81
|
+
### `session` (typed knobs)
|
|
82
|
+
|
|
83
|
+
Use the typed `session` field for documented Inworld realtime options. Fields compose with the connect-time defaults (e.g. `audio.output.voice` set from `speaker`):
|
|
84
|
+
|
|
85
|
+
**output\_modalities** (`Array<"text" | "audio">`): Modalities the model should produce.
|
|
86
|
+
|
|
87
|
+
**audio.output.voice** (`string`): Voice catalog ID. When omitted, the constructor \`speaker\` is used.
|
|
88
|
+
|
|
89
|
+
**audio.output.speed** (`number`): Playback speed multiplier for synthesized audio (0.25 to 1.5).
|
|
90
|
+
|
|
91
|
+
**audio.output.model** (`string`): Inworld TTS model (e.g. "inworld-tts-2").
|
|
92
|
+
|
|
93
|
+
**audio.output.format** (`InworldAudioFormat`): Output audio encoding. A codec string (e.g. "audio/pcm", "audio/pcmu", "audio/pcma", "audio/float32") or an object \`{ type, rate? }\`. \`rate\` (Hz) applies to audio/pcm and audio/float32 (default 24000); audio/pcmu and audio/pcma are fixed at 8 kHz.
|
|
94
|
+
|
|
95
|
+
**audio.input.format** (`InworldAudioFormat`): Input audio encoding sent to the server. Same shape as \`audio.output.format\` — a codec string or \`{ type, rate? }\` object.
|
|
96
|
+
|
|
97
|
+
**audio.input.noise\_reduction** (`{ type: "near_field" | "far_field" }`): Input noise-reduction mode applied before transcription and VAD.
|
|
98
|
+
|
|
99
|
+
**audio.input.transcription** (`{ model?: string; language?: string; prompt?: string }`): Server-side transcription for incoming user audio. Defaults to \`{ model: "inworld/inworld-stt-1" }\`. \`prompt\` biases transcription with vocabulary, spelling, or style hints. Supply your own object to override; set to \`null\` to disable user-side transcription.
|
|
100
|
+
|
|
101
|
+
**audio.input.turn\_detection** (`InworldTurnDetection | null`): Voice activity / turn detection. Defaults to \`{ type: "semantic\_vad", eagerness: "medium", create\_response: true, interrupt\_response: true }\`. Supply your own object to override; set to \`null\` to disable turn detection entirely. The \`eagerness\` field controls how quickly semantic VAD ends a user turn — \`low\` waits for clearer pauses (more interruption-resistant), \`high\` ends turns sooner (snappier, more prone to cutting users off). Default \`medium\` balances both. \`idle\_timeout\_ms\` (server\_vad only) sets the idle window before the server commits a turn.
|
|
102
|
+
|
|
103
|
+
**tool\_choice** (`string | { type: "function"; name: string } | { type: "mcp"; server_label: string }`): Tool selection strategy. Use the mcp variant to route tool calls through a configured Inworld MCP server.
|
|
104
|
+
|
|
105
|
+
**temperature** (`number`): Sampling temperature for the model.
|
|
106
|
+
|
|
107
|
+
**max\_output\_tokens** (`number | "inf"`): Maximum tokens generated per response.
|
|
108
|
+
|
|
109
|
+
**truncation** (`"auto" | "disabled" | { type: "retention_ratio"; retention_ratio: number }`): Conversation truncation strategy.
|
|
110
|
+
|
|
111
|
+
**tracing** (`"auto" | { workflow_name?: string; group_id?: string; metadata?: Record<string, unknown> }`): Distributed-tracing config. Use "auto" for server defaults, or name the workflow/group explicitly.
|
|
112
|
+
|
|
113
|
+
**include** (`Array<"item.input_audio_transcription.logprobs">`): Opt-in extra fields the server should include on emitted events.
|
|
114
|
+
|
|
115
|
+
**prompt** (`string | null`): Reference to a server-side prompt template. Pass null to clear it.
|
|
116
|
+
|
|
117
|
+
### `providerData` (Inworld extensions)
|
|
118
|
+
|
|
119
|
+
`providerData` is a typed object for Inworld-specific realtime extensions. It is sent under `session.providerData` on every `session.update`, and composes with any `session.providerData` you set via the `session` field — the constructor `providerData` wins on key collisions.
|
|
120
|
+
|
|
121
|
+
It has five branches plus two session-level fields:
|
|
122
|
+
|
|
123
|
+
- `stt`: STT tuning, such as `prompt`, `voice_profile`, `language_hints`, and VAD or end-of-turn thresholds.
|
|
124
|
+
- `tts`: TTS segmentation and delivery, such as `segmenter_strategy`, `steering_handling`, `delivery_mode`, `conversational`, and `user_turn_mode`.
|
|
125
|
+
- `memory`: automatic rolling memory, such as `enabled`, `turn_interval`, and `max_facts`. Inworld echoes its state back through the `memory` event.
|
|
126
|
+
- `backchannel`: short acknowledgements ("uh-huh") while the user speaks. Audio arrives on the `backchannel` event.
|
|
127
|
+
- `responsiveness`: early filler audio while the main response generates. Filler audio reuses the normal `speaker` and `speaking` events, so there are no distinct events.
|
|
128
|
+
- `user_id` and `metadata`: session-level identifiers passed through to Inworld.
|
|
129
|
+
|
|
130
|
+
```typescript
|
|
131
|
+
const voice = new InworldRealtimeVoice({
|
|
132
|
+
providerData: {
|
|
133
|
+
stt: { voice_profile: true, language_hints: ['en-US'] },
|
|
134
|
+
tts: { delivery_mode: 'CREATIVE', segmenter_strategy: 'balanced' },
|
|
135
|
+
memory: { enabled: true, turn_interval: 4 },
|
|
136
|
+
backchannel: { enabled: true, max_per_turn: 1 },
|
|
137
|
+
user_id: 'user-123',
|
|
138
|
+
},
|
|
139
|
+
})
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
## Methods
|
|
143
|
+
|
|
144
|
+
### `connect()`
|
|
145
|
+
|
|
146
|
+
Opens the WebSocket connection, sends the initial `session.update`, and resolves once the server acknowledges with `session.updated`. Must be called before `speak()`, `listen()`, or `send()`.
|
|
147
|
+
|
|
148
|
+
A pre-open `error` or `close` on the WebSocket — or a handshake that exceeds `connectTimeoutMs` (15s default) — surfaces as a rejected promise instead of an uncaught socket error. On reject, the half-open socket is closed.
|
|
149
|
+
|
|
150
|
+
```typescript
|
|
151
|
+
await voice.connect()
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
Returns: `Promise<void>`
|
|
155
|
+
|
|
156
|
+
### `speak()`
|
|
157
|
+
|
|
158
|
+
Sends a text message to the model and triggers an audio response. The returned promise resolves only after the full response lifecycle completes (`response.done` for the response this call triggered), and rejects if the response is interrupted by user speech or if a transport error occurs.
|
|
159
|
+
|
|
160
|
+
Serial `speak()` calls are the supported pattern. Concurrent calls share the same listener pool and have undefined response-pinning order.
|
|
161
|
+
|
|
162
|
+
**input** (`string | NodeJS.ReadableStream`): Text or text stream to convert to speech.
|
|
163
|
+
|
|
164
|
+
**options** (`Options`): Per-call configuration.
|
|
165
|
+
|
|
166
|
+
**options.speaker** (`string`): Voice ID to use for this specific request.
|
|
167
|
+
|
|
168
|
+
Returns: `Promise<void>`
|
|
169
|
+
|
|
170
|
+
### `listen()`
|
|
171
|
+
|
|
172
|
+
Sends a single audio buffer as a user turn and asks the model to respond with text only.
|
|
173
|
+
|
|
174
|
+
**audioData** (`NodeJS.ReadableStream`): Audio stream to transcribe.
|
|
175
|
+
|
|
176
|
+
Returns: `Promise<void>`
|
|
177
|
+
|
|
178
|
+
### `send()`
|
|
179
|
+
|
|
180
|
+
Streams audio data to the server in real time. Useful for continuous microphone input.
|
|
181
|
+
|
|
182
|
+
**audioData** (`NodeJS.ReadableStream | Int16Array`): Audio data to stream. Int16Array is sent as a single base64 chunk; a readable stream is forwarded chunk by chunk.
|
|
183
|
+
|
|
184
|
+
**eventId** (`string`): Optional event ID forwarded to the server with each audio chunk.
|
|
185
|
+
|
|
186
|
+
Returns: `Promise<void>`
|
|
187
|
+
|
|
188
|
+
### `updateConfig()`
|
|
189
|
+
|
|
190
|
+
Sends a `session.update` to the server. The typed `session` field is deep-merged into the payload, and any constructor `providerData` is nested under `session.providerData`.
|
|
191
|
+
|
|
192
|
+
**sessionConfig** (`InworldSessionConfig | Record<string, unknown>`): Partial session configuration to apply.
|
|
193
|
+
|
|
194
|
+
Returns: `void`
|
|
195
|
+
|
|
196
|
+
### `addInstructions()`
|
|
197
|
+
|
|
198
|
+
Sets the system instructions used on the next `connect()` or `updateConfig()` call.
|
|
199
|
+
|
|
200
|
+
**instructions** (`string`): System prompt for the model.
|
|
201
|
+
|
|
202
|
+
Returns: `void`
|
|
203
|
+
|
|
204
|
+
### `addTools()`
|
|
205
|
+
|
|
206
|
+
Registers tools that the model can call during the session. When `InworldRealtimeVoice` is attached to an Agent, tools configured for the Agent are made available automatically.
|
|
207
|
+
|
|
208
|
+
**tools** (`ToolsInput`): Tools configuration to equip.
|
|
209
|
+
|
|
210
|
+
Returns: `void`
|
|
211
|
+
|
|
212
|
+
### `answer()`
|
|
213
|
+
|
|
214
|
+
Sends a `response.create` event to trigger a model response, optionally with per-response options.
|
|
215
|
+
|
|
216
|
+
**options** (`Record<string, unknown>`): Response options forwarded to the server.
|
|
217
|
+
|
|
218
|
+
Returns: `Promise<void>`
|
|
219
|
+
|
|
220
|
+
### Turn-taking
|
|
221
|
+
|
|
222
|
+
#### `commitInput()`
|
|
223
|
+
|
|
224
|
+
Manually commits buffered input audio as a user turn. Use this for push-to-talk or manual turn-taking when `turn_detection` is set to `null`.
|
|
225
|
+
|
|
226
|
+
```typescript
|
|
227
|
+
voice.commitInput()
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
Returns: `void`
|
|
231
|
+
|
|
232
|
+
#### `clearInput()`
|
|
233
|
+
|
|
234
|
+
Discards buffered input audio without committing it as a user turn.
|
|
235
|
+
|
|
236
|
+
```typescript
|
|
237
|
+
voice.clearInput()
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
Returns: `void`
|
|
241
|
+
|
|
242
|
+
#### `clearOutput()`
|
|
243
|
+
|
|
244
|
+
Clears the server's entire output audio buffer, stopping playback. This also stops any in-flight back-channel audio. The default barge-in path (`response.cancel` on `interrupted`) is back-channel-safe; prefer it. Use `clearOutput()` only when you want to flush everything.
|
|
245
|
+
|
|
246
|
+
```typescript
|
|
247
|
+
voice.clearOutput()
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
Returns: `void`
|
|
251
|
+
|
|
252
|
+
### `close()` and `disconnect()`
|
|
253
|
+
|
|
254
|
+
Both methods close the WebSocket and mark the instance as disconnected.
|
|
255
|
+
|
|
256
|
+
Returns: `void`
|
|
257
|
+
|
|
258
|
+
### `getSpeakers()`
|
|
259
|
+
|
|
260
|
+
Returns the curated voice list bundled with the package. Inworld's catalog is larger than this list; any voice ID can be passed to `speaker` at runtime.
|
|
261
|
+
|
|
262
|
+
Returns: `Promise<Array<{ voiceId: string }>>`
|
|
263
|
+
|
|
264
|
+
### `on()` and `off()`
|
|
265
|
+
|
|
266
|
+
Register and remove event listeners. See [Events](#events) below.
|
|
267
|
+
|
|
268
|
+
## Events
|
|
269
|
+
|
|
270
|
+
The `InworldRealtimeVoice` class emits the following events:
|
|
271
|
+
|
|
272
|
+
**speaker** (`event`): Emitted once per response with a PassThrough stream of PCM audio. Use this when piping audio to a player.
|
|
273
|
+
|
|
274
|
+
**speaking** (`event`): Emitted for each audio delta. Callback receives { audio: Buffer, response\_id: string }.
|
|
275
|
+
|
|
276
|
+
**speaking.done** (`event`): Emitted when audio output for a response is complete. Callback receives { response\_id: string }.
|
|
277
|
+
|
|
278
|
+
**writing** (`event`): Emitted as transcribed text becomes available. Callback receives { text: string, response\_id: string, role: "assistant" | "user", voiceProfile? }. Deduplicated across audio-transcript and text deltas in the same response so a single response only emits one stream. On user events, voiceProfile is present when providerData.stt.voice\_profile is enabled.
|
|
279
|
+
|
|
280
|
+
**speech-started** (`event`): Raw \`input\_audio\_buffer.speech\_started\` VAD edge from the server.
|
|
281
|
+
|
|
282
|
+
**speech-stopped** (`event`): Raw \`input\_audio\_buffer.speech\_stopped\` VAD edge from the server.
|
|
283
|
+
|
|
284
|
+
**interrupted** (`event`): Synthetic client-side signal: emitted once per in-flight \`response\_id\` when the user starts speaking. Use this to stop main response playback on barge-in. Callback receives \`{ response\_id: string }\`. Only carries main-response ids — never back-channel ids — so stopping the matching \`speaker\` stream leaves \`backchannel\` streams playing (back-channels are meant to overlap user speech and are never cancelled by barge-in).
|
|
285
|
+
|
|
286
|
+
**turn-suggestion** (`event`): Smart-turn endpointing hint for a buffered user utterance. Callback receives { item\_id, utterance\_index, probability, trailing\_silence\_ms?, audio\_duration\_ms?, inference\_ms? }.
|
|
287
|
+
|
|
288
|
+
**turn-suggestion-revoked** (`event`): A previously emitted turn suggestion was retracted. Callback receives { item\_id, utterance\_index }.
|
|
289
|
+
|
|
290
|
+
**input-committed** (`event`): Buffered input audio was committed as a user turn (via commitInput() or auto-VAD). Callback receives { item\_id, previous\_item\_id? } where previous\_item\_id may be null.
|
|
291
|
+
|
|
292
|
+
**input-cleared** (`event`): Buffered input audio was discarded (via clearInput()). Callback receives {}.
|
|
293
|
+
|
|
294
|
+
**input-timeout** (`event`): A server-VAD idle timeout committed a user turn. Callback receives { audio\_start\_ms, audio\_end\_ms, item\_id }.
|
|
295
|
+
|
|
296
|
+
**output-audio-started** (`event`): Server began emitting output audio. Callback receives {}.
|
|
297
|
+
|
|
298
|
+
**output-audio-stopped** (`event`): Server stopped emitting output audio for the current response. Callback receives {}.
|
|
299
|
+
|
|
300
|
+
**output-audio-cleared** (`event`): Server output audio buffer was flushed, stopping playback (via clearOutput()). Callback receives {}.
|
|
301
|
+
|
|
302
|
+
**memory** (`event`): Emitted with Inworld's rolling summary and facts state, deduplicated by version. Requires providerData.memory.enabled. Callback receives InworldMemoryState.
|
|
303
|
+
|
|
304
|
+
**backchannel** (`event`): Emitted with a PassThrough stream of back-channel PCM audio (short acknowledgements while the user speaks). Each stream's \`.id\` is a \`backchannel\_id\` that never appears in \`interrupted\`, so play these on a separate track that barge-in does not stop. Requires providerData.backchannel.enabled.
|
|
305
|
+
|
|
306
|
+
**backchannel.done** (`event`): Emitted when a back-channel finishes. Callback receives { backchannel\_id: string, phrase? }.
|
|
307
|
+
|
|
308
|
+
**backchannel.skipped** (`event`): Emitted when the decider skips a back-channel before any audio is produced. Callback receives { reason: string }.
|
|
309
|
+
|
|
310
|
+
**response.created** (`event`): Emitted when a new response begins. Callback receives the full server event.
|
|
311
|
+
|
|
312
|
+
**response.done** (`event`): Emitted when a response completes. Callback receives the full server event.
|
|
313
|
+
|
|
314
|
+
**conversation.item.added** (`event`): Emitted when a new conversation item is appended.
|
|
315
|
+
|
|
316
|
+
**conversation.item.done** (`event`): Emitted when a conversation item finishes.
|
|
317
|
+
|
|
318
|
+
**function\_call.arguments** (`event`): Emitted with complete tool call arguments. Callback receives { call\_id, name, arguments }.
|
|
319
|
+
|
|
320
|
+
**tool-call-start** (`event`): Emitted before a registered tool is executed.
|
|
321
|
+
|
|
322
|
+
**tool-call-result** (`event`): Emitted after a registered tool returns.
|
|
323
|
+
|
|
324
|
+
**error** (`event`): Emitted on transport or server errors.
|
|
325
|
+
|
|
326
|
+
## Voices
|
|
327
|
+
|
|
328
|
+
The package ships with a curated set of voice IDs returned from `getSpeakers()`:
|
|
329
|
+
|
|
330
|
+
- `Dennis`
|
|
331
|
+
- `Hades`
|
|
332
|
+
- `Wendy`
|
|
333
|
+
- `Edward`
|
|
334
|
+
- `Olivia`
|
|
335
|
+
- `Sarah`
|
|
336
|
+
- `Timothy`
|
|
337
|
+
- `Priya`
|
|
338
|
+
- `Ronald`
|
|
339
|
+
- `Deborah`
|
|
340
|
+
|
|
341
|
+
Any voice ID from [Inworld's voice catalog](https://docs.inworld.ai/quickstart-tts) can be passed to `speaker` at runtime.
|
|
342
|
+
|
|
343
|
+
## Notes
|
|
344
|
+
|
|
345
|
+
- API keys can be provided via constructor options or the `INWORLD_API_KEY` environment variable. Keys are pre-Basic-encoded; do not re-encode them.
|
|
346
|
+
- The WebSocket URL appends `?key=<sessionId>&protocol=realtime`. The model is configured via the initial `session.update`, not the URL.
|
|
347
|
+
- Per-call `speak(input, { speaker })` scopes the voice override to a single response (via the flat `response.voice` field) and does NOT mutate the session.
|
|
348
|
+
- Audio output defaults to PCM16 at 24 kHz. Telephony `audio/pcmu` and `audio/pcma` at 8 kHz, and `audio/float32`, are also supported via `session.audio.output.format`.
|
|
349
|
+
- Use `connect()` before any send, speak, or listen call. Events sent before the WebSocket is open are queued and flushed once the server acknowledges `session.updated`.
|
|
350
|
+
- The voice instance must be closed with `close()` or `disconnect()` to release the WebSocket.
|
|
351
|
+
- `audio.input.turn_detection` defaults to semantic VAD when `session` does not supply it. Override with your own object, or pass `null` to disable turn detection entirely.
|
|
352
|
+
- `audio.input.transcription` defaults to `{ model: 'inworld/inworld-stt-1' }`, so user-side `writing` events fire out of the box. Override with your own object, or pass `null` to disable user-side transcription.
|
|
353
|
+
- `on()` and `off()` are typed against `InworldVoiceEventMap` — known event names yield a typed callback payload, unknown names fall back to `unknown`.
|
|
@@ -2,6 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
The Inworld voice implementation in Mastra provides streaming text-to-speech (TTS) and batch speech-to-text (STT) capabilities using Inworld AI's API. It supports multiple TTS and STT models, configurable audio encodings, and progressive audio streaming.
|
|
4
4
|
|
|
5
|
+
For real-time, full-duplex speech-to-speech, the same package exports [`InworldRealtimeVoice`](https://mastra.ai/reference/voice/inworld-realtime).
|
|
6
|
+
|
|
5
7
|
## Usage example
|
|
6
8
|
|
|
7
9
|
```typescript
|