@mastra/mcp-docs-server 0.0.0-commonjs-20250414101718

package/.docs/raw/voice/voice-to-voice.mdx

@@ -0,0 +1,310 @@
+---
+title: Voice-to-Voice Capabilities in Mastra | Mastra Docs
+description: Overview of voice-to-voice capabilities in Mastra, including real-time interactions and event-driven architecture.
+---
+
+# Voice-to-Voice Capabilities in Mastra
+
+## Introduction
+
+Voice-to-Voice in Mastra provides a standardized interface for real-time speech-to-speech interactions across multiple service providers. This section covers configuration, event-driven architecture, and implementation methods for creating conversational voice experiences. For integrating Voice-to-Voice capabilities with agents, refer to the [Adding Voice to Agents](../agents/adding-voice.mdx) documentation.
+
+## Real-time Voice Interactions
+
+Mastra's real-time voice system enables continuous bidirectional audio communication through an event-driven architecture. Unlike separate TTS and STT operations, real-time voice maintains an open connection that processes speech continuously in both directions.
+
+### Example Implementation
+
+```typescript
+import { Agent } from "@mastra/core/agent";
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+
+const agent = new Agent({
+  name: 'Agent',
+  instructions: `You are a helpful assistant with real-time voice capabilities.`,
+  model: openai('gpt-4o'),
+  voice: new OpenAIRealtimeVoice(),
+});
+
+// Connect to the voice service
+await agent.voice.connect();
+
+// Listen for agent audio responses
+agent.voice.on('speaking', ({ audio }) => {
+  playAudio(audio);
+});
+
+// Initiate the conversation
+await agent.voice.speak('How can I help you today?');
+
+// Send continuous audio from the microphone
+const micStream = getMicrophoneStream();
+await agent.voice.send(micStream);
+```
+
+## Event-Driven Architecture
+
+Mastra's voice-to-voice implementation is built on an event-driven architecture. Developers register event listeners to handle incoming audio progressively, allowing for more responsive interactions than waiting for complete audio responses.
+
+
+## Configuration
+
+When initializing a voice-to-voice provider, you can provide configuration options to customize its behavior:
+
+### Constructor Options
+
+- **`chatModel`**: Configuration for the OpenAI realtime model.
+  - **`apiKey`**: Your OpenAI API key. Falls back to the `OPENAI_API_KEY` environment variable.
+  - **`model`**: The model ID to use for real-time voice interactions (e.g., `gpt-4o-mini-realtime`).
+  - **`options`**: Additional options for the realtime client, such as session configuration.
+
+- **`speaker`**: The default voice ID for speech synthesis. This allows you to specify which voice to use for the speech output.
+
+### Example Configuration
+
+```typescript
+const voice = new OpenAIRealtimeVoice({
+  chatModel: {
+    apiKey: 'your-openai-api-key',
+    model: 'gpt-4o-mini-realtime',
+    options: {
+      sessionConfig: {
+        turn_detection: {
+          type: 'server_vad',
+          threshold: 0.6,
+          silence_duration_ms: 1200,
+        },
+      },
+    },
+  },
+  speaker: 'alloy', // Default voice
+});
+
+// If using default settings the configuration can be simplified to:
+const voice = new OpenAIRealtimeVoice();
+```
+
+## Core Methods
+
+The `OpenAIRealtimeVoice` class provides the following core methods for voice interactions:
+
+### connect()
+
+Establishes a connection to the OpenAI realtime service.
+
+**Usage:**
+```typescript
+await voice.connect();
+```
+
+**Notes:**
+- Must be called before using any other interaction methods
+- Returns a Promise that resolves when the connection is established
+
+### speak(text, options?)
+
+Emits a speaking event using the configured voice model.
+
+**Parameters:**
+- `text`: String content to be spoken
+- `options`: Optional configuration object
+  - `speaker`: Voice ID to use (overrides default)
+  - `properties`: Additional provider-specific properties
+
+**Usage:**
+```typescript
+voice.speak('Hello, how can I help you today?', {
+  speaker: 'alloy'
+});
+```
+
+**Notes:**
+- Emits 'speaker' event rather than returning an audio stream
+
+### listen(audioInput, options?)
+
+Processes audio input for speech recognition.
+
+**Parameters:**
+- `audioInput`: Readable stream of audio data
+- `options`: Optional configuration object
+  - `filetype`: Audio format (default: 'mp3')
+  - Additional provider-specific options
+
+**Usage:**
+```typescript
+const audioData = getMicrophoneStream();
+voice.listen(audioData, {
+  filetype: 'wav'
+});
+```
+
+**Notes:**
+- Emits 'writing' events with transcribed text
+
+### send(audioStream)
+
+Streams audio data in real-time for continuous processing.
+
+**Parameters:**
+- `audioStream`: Readable stream of audio data
+
+**Usage:**
+```typescript
+const micStream = getMicrophoneStream();
+await voice.send(micStream);
+```
+
+**Notes:**
+- Used for continuous audio streaming scenarios like live microphone input
+- Returns a Promise that resolves when the stream is accepted
+
+### answer(params)
+
+Sends a response to the OpenAI Realtime API.
+
+**Parameters:**
+- `params`: The parameters object
+  - `options`: Configuration options for the response
+    - `content`: Text content of the response
+    - `voice`: Voice ID to use for the response
+
+**Usage:**
+```typescript
+await voice.answer({
+  options: {
+    content: "Hello, how can I help you today?",
+    voice: "alloy"
+  }
+});
+```
+
+**Notes:**
+- Triggers a response to the real-time session
+- Returns a Promise that resolves when the response has been sent
+
+## Utility Methods
+
+### updateConfig(config)
+
+Updates the session configuration for the voice instance.
+
+**Parameters:**
+- `config`: New session configuration object
+
+**Usage:**
+```typescript
+voice.updateConfig({
+  turn_detection: {
+    type: 'server_vad',
+    threshold: 0.6,
+    silence_duration_ms: 1200,
+  }
+});
+```
+
+### addTools(tools)
+
+Adds a set of tools to the voice instance.
+
+**Parameters:**
+- `tools`: Array of tool objects that the model can call
+
+**Usage:**
+```typescript
+voice.addTools([
+  createTool({
+    id: "Get Weather Information",
+    inputSchema: z.object({
+        city: z.string(),
+    }),
+    description: `Fetches the current weather information for a given city`,
+    execute: async ({ city }) => {...},
+  })
+]);
+```
+
+### close()
+
+Disconnects from the OpenAI realtime session and cleans up resources.
+
+**Usage:**
+```typescript
+voice.close();
+```
+
+**Notes:**
+- Should be called when you're done with the voice instance to free resources
+
+### on(event, callback)
+
+Registers an event listener for voice events.
+
+**Parameters:**
+- `event`: Event name ('speaker', 'writing', or 'error')
+- `callback`: Function to call when the event occurs
+
+**Usage:**
+```typescript
+voice.on('speaker', (stream) => {
+  stream.pipe(speaker)
+});
+```
+
+### off(event, callback)
+
+Removes a previously registered event listener.
+
+**Parameters:**
+- `event`: Event name
+- `callback`: The callback function to remove
+
+**Usage:**
+```typescript
+voice.off('speaking', callbackFunction);
+```
+
+## Events
+
+The `OpenAIRealtimeVoice` class emits the following events:
+
+### speaker
+
+Emitted when audio data is received from the model.
+
+**Event payload:**
+- `stream`: A stream of audio data as a readable stream
+
+```typescript
+agent.voice.on('speaker', (stream) => {
+  stream.pipe(speaker)
+});
+```
+
+### writing
+
+Emitted when transcribed text is available.
+
+**Event payload:**
+- `text`: The transcribed text
+- `role`: The role of the speaker (user or assistant)
+- `done`: Boolean indicating if this is the final transcription
+
+```typescript
+agent.voice.on('writing', ({ text, role }) => {
+  console.log(`${role}: ${text}`); // Log who said what
+});
+```
+
+### error
+
+Emitted when an error occurs.
+
+**Event payload:**
+- Error object with details about what went wrong
+
+```typescript
+agent.voice.on('error', (error) => {
+  console.error('Voice error:', error);
+});
+```