@mastra/voice-openai-realtime 0.11.12 → 0.12.0-beta.1

package/dist/docs/voice/03-reference.md

@@ -0,0 +1,1096 @@
+# Voice API Reference
+
+> API reference for voice - 12 entries
+
+
+---
+
+## Reference: OpenAI Realtime Voice
+
+> Documentation for the OpenAIRealtimeVoice class, providing real-time text-to-speech and speech-to-text capabilities via WebSockets.
+
+The OpenAIRealtimeVoice class provides real-time voice interaction capabilities using OpenAI's WebSocket-based API. It supports real time speech to speech, voice activity detection, and event-based audio streaming.
+
+## Usage Example
+
+```typescript
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+import { playAudio, getMicrophoneStream } from "@mastra/node-audio";
+
+// Initialize with default configuration using environment variables
+const voice = new OpenAIRealtimeVoice();
+
+// Or initialize with specific configuration
+const voiceWithConfig = new OpenAIRealtimeVoice({
+  apiKey: "your-openai-api-key",
+  model: "gpt-5.1-realtime-preview-2024-12-17",
+  speaker: "alloy", // Default voice
+});
+
+voiceWithConfig.updateSession({
+  turn_detection: {
+    type: "server_vad",
+    threshold: 0.6,
+    silence_duration_ms: 1200,
+  },
+});
+
+// Establish connection
+await voice.connect();
+
+// Set up event listeners
+voice.on("speaker", ({ audio }) => {
+  // Handle audio data (Int16Array) pcm format by default
+  playAudio(audio);
+});
+
+voice.on("writing", ({ text, role }) => {
+  // Handle transcribed text
+  console.log(`${role}: ${text}`);
+});
+
+// Convert text to speech
+await voice.speak("Hello, how can I help you today?", {
+  speaker: "echo", // Override default voice
+});
+
+// Process audio input
+const microphoneStream = getMicrophoneStream();
+await voice.send(microphoneStream);
+
+// When done, disconnect
+voice.connect();
+```
+
+## Configuration
+
+### Constructor Options
+
+### Voice Activity Detection (VAD) Configuration
+
+## Methods
+
+### connect()
+
+Establishes a connection to the OpenAI realtime service. Must be called before using speak, listen, or send functions.
+
+### speak()
+
+Emits a speaking event using the configured voice model. Can accept either a string or a readable stream as input.
+
+Returns: `Promise<void>`
+
+### listen()
+
+Processes audio input for speech recognition. Takes a readable stream of audio data and emits a 'listening' event with the transcribed text.
+
+Returns: `Promise<void>`
+
+### send()
+
+Streams audio data in real-time to the OpenAI service for continuous audio streaming scenarios like live microphone input.
+
+Returns: `Promise<void>`
+
+### updateConfig()
+
+Updates the session configuration for the voice instance. This can be used to modify voice settings, turn detection, and other parameters.
+
+Returns: `void`
+
+### addTools()
+
+Adds a set of tools to the voice instance. Tools allow the model to perform additional actions during conversations. When OpenAIRealtimeVoice is added to an Agent, any tools configured for the Agent will automatically be available to the voice interface.
+
+Returns: `void`
+
+### close()
+
+Disconnects from the OpenAI realtime session and cleans up resources. Should be called when you're done with the voice instance.
+
+Returns: `void`
+
+### getSpeakers()
+
+Returns a list of available voice speakers.
+
+Returns: `Promise<Array<{ voiceId: string; [key: string]: any }>>`
+
+### on()
+
+Registers an event listener for voice events.
+
+Returns: `void`
+
+### off()
+
+Removes a previously registered event listener.
+
+Returns: `void`
+
+## Events
+
+The OpenAIRealtimeVoice class emits the following events:
+
+### OpenAI Realtime Events
+
+You can also listen to [OpenAI Realtime utility events](https://github.com/openai/openai-realtime-api-beta#reference-client-utility-events) by prefixing with 'openAIRealtime:':
+
+## Available Voices
+
+The following voice options are available:
+
+- `alloy`: Neutral and balanced
+- `ash`: Clear and precise
+- `ballad`: Melodic and smooth
+- `coral`: Warm and friendly
+- `echo`: Resonant and deep
+- `sage`: Calm and thoughtful
+- `shimmer`: Bright and energetic
+- `verse`: Versatile and expressive
+
+## Notes
+
+- API keys can be provided via constructor options or the `OPENAI_API_KEY` environment variable
+- The OpenAI Realtime Voice API uses WebSockets for real-time communication
+- Server-side Voice Activity Detection (VAD) provides better accuracy for speech detection
+- All audio data is processed as Int16Array format
+- The voice instance must be connected with `connect()` before using other methods
+- Always call `close()` when done to properly clean up resources
+- Memory management is handled by OpenAI Realtime API
+
+---
+
+## Reference: voice.addInstructions()
+
+> Documentation for the addInstructions() method available in voice providers, which adds instructions to guide the voice model
+
+The `addInstructions()` method equips a voice provider with instructions that guide the model's behavior during real-time interactions. This is particularly useful for real-time voice providers that maintain context across a conversation.
+
+## Usage Example
+
+```typescript
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+import { Agent } from "@mastra/core/agent";
+
+// Initialize a real-time voice provider
+const voice = new OpenAIRealtimeVoice({
+  realtimeConfig: {
+    model: "gpt-5.1-realtime",
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+});
+
+// Create an agent with the voice provider
+const agent = new Agent({
+  name: "Customer Support Agent",
+  instructions:
+    "You are a helpful customer support agent for a software company.",
+  model: "openai/gpt-5.1",
+  voice,
+});
+
+// Add additional instructions to the voice provider
+voice.addInstructions(`
+  When speaking to customers:
+  - Always introduce yourself as the customer support agent
+  - Speak clearly and concisely
+  - Ask clarifying questions when needed
+  - Summarize the conversation at the end
+`);
+
+// Connect to the real-time service
+await voice.connect();
+```
+
+## Parameters
+
+<br />
+
+## Return Value
+
+This method does not return a value.
+
+## Notes
+
+- Instructions are most effective when they are clear, specific, and relevant to the voice interaction
+- This method is primarily used with real-time voice providers that maintain conversation context
+- If called on a voice provider that doesn't support instructions, it will log a warning and do nothing
+- Instructions added with this method are typically combined with any instructions provided by an associated Agent
+- For best results, add instructions before starting a conversation (before calling `connect()`)
+- Multiple calls to `addInstructions()` may either replace or append to existing instructions, depending on the provider implementation
+
+---
+
+## Reference: voice.addTools()
+
+> Documentation for the addTools() method available in voice providers, which equips voice models with function calling capabilities.
+
+The `addTools()` method equips a voice provider with tools (functions) that can be called by the model during real-time interactions. This enables voice assistants to perform actions like searching for information, making calculations, or interacting with external systems.
+
+## Usage Example
+
+```typescript
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+import { createTool } from "@mastra/core/tools";
+import { z } from "zod";
+
+// Define tools
+const weatherTool = createTool({
+  id: "getWeather",
+  description: "Get the current weather for a location",
+  inputSchema: z.object({
+    location: z.string().describe("The city and state, e.g. San Francisco, CA"),
+  }),
+  outputSchema: z.object({
+    message: z.string(),
+  }),
+  execute: async (inputData) => {
+    // Fetch weather data from an API
+    const response = await fetch(
+      `https://api.weather.com?location=${encodeURIComponent(inputData.location)}`,
+    );
+    const data = await response.json();
+    return {
+      message: `The current temperature in ${inputData.location} is ${data.temperature}°F with ${data.conditions}.`,
+    };
+  },
+});
+
+// Initialize a real-time voice provider
+const voice = new OpenAIRealtimeVoice({
+  realtimeConfig: {
+    model: "gpt-5.1-realtime",
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+});
+
+// Add tools to the voice provider
+voice.addTools({
+  getWeather: weatherTool,
+});
+
+// Connect to the real-time service
+await voice.connect();
+```
+
+## Parameters
+
+<br />
+
+## Return Value
+
+This method does not return a value.
+
+## Notes
+
+- Tools must follow the Mastra tool format with name, description, input schema, and execute function
+- This method is primarily used with real-time voice providers that support function calling
+- If called on a voice provider that doesn't support tools, it will log a warning and do nothing
+- Tools added with this method are typically combined with any tools provided by an associated Agent
+- For best results, add tools before starting a conversation (before calling `connect()`)
+- The voice provider will automatically handle the invocation of tool handlers when the model decides to use them
+- Multiple calls to `addTools()` may either replace or merge with existing tools, depending on the provider implementation
+
+---
+
+## Reference: voice.answer()
+
+> Documentation for the answer() method available in real-time voice providers, which triggers the voice provider to generate a response.
+
+The `answer()` method is used in real-time voice providers to trigger the AI to generate a response. This method is particularly useful in speech-to-speech conversations where you need to explicitly signal the AI to respond after receiving user input.
+
+## Usage Example
+
+```typescript
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+import { getMicrophoneStream } from "@mastra/node-audio";
+import Speaker from "@mastra/node-speaker";
+
+const speaker = new Speaker({
+  sampleRate: 24100, // Audio sample rate in Hz - standard for high-quality audio on MacBook Pro
+  channels: 1, // Mono audio output (as opposed to stereo which would be 2)
+  bitDepth: 16, // Bit depth for audio quality - CD quality standard (16-bit resolution)
+});
+
+// Initialize a real-time voice provider
+const voice = new OpenAIRealtimeVoice({
+  realtimeConfig: {
+    model: "gpt-5.1",
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+  speaker: "alloy", // Default voice
+});
+// Connect to the real-time service
+await voice.connect();
+// Register event listener for responses
+voice.on("speaker", (stream) => {
+  // Handle audio response
+  stream.pipe(speaker);
+});
+// Send user audio input
+const microphoneStream = getMicrophoneStream();
+await voice.send(microphoneStream);
+// Trigger the AI to respond
+await voice.answer();
+```
+
+## Parameters
+
+<br />
+
+## Return Value
+
+Returns a `Promise<void>` that resolves when the response has been triggered.
+
+## Notes
+
+- This method is only implemented by real-time voice providers that support speech-to-speech capabilities
+- If called on a voice provider that doesn't support this functionality, it will log a warning and resolve immediately
+- The response audio will typically be emitted through the 'speaking' event rather than returned directly
+- For providers that support it, you can use this method to send a specific response instead of having the AI generate one
+- This method is commonly used in conjunction with `send()` to create a conversational flow
+
+---
+
+## Reference: voice.close()
+
+> Documentation for the close() method available in voice providers, which disconnects from real-time voice services.
+
+The `close()` method disconnects from a real-time voice service and cleans up resources. This is important for properly ending voice sessions and preventing resource leaks.
+
+## Usage Example
+
+```typescript
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+import { getMicrophoneStream } from "@mastra/node-audio";
+
+// Initialize a real-time voice provider
+const voice = new OpenAIRealtimeVoice({
+  realtimeConfig: {
+    model: "gpt-5.1-realtime",
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+});
+
+// Connect to the real-time service
+await voice.connect();
+
+// Start a conversation
+voice.speak("Hello, I'm your AI assistant!");
+
+// Stream audio from a microphone
+const microphoneStream = getMicrophoneStream();
+voice.send(microphoneStream);
+
+// When the conversation is complete
+setTimeout(() => {
+  // Close the connection and clean up resources
+  voice.close();
+  console.log("Voice session ended");
+}, 60000); // End after 1 minute
+```
+
+## Parameters
+
+This method does not accept any parameters.
+
+## Return Value
+
+This method does not return a value.
+
+## Notes
+
+- Always call `close()` when you're done with a real-time voice session to free up resources
+- After calling `close()`, you'll need to call `connect()` again if you want to start a new session
+- This method is primarily used with real-time voice providers that maintain persistent connections
+- If called on a voice provider that doesn't support real-time connections, it will log a warning and do nothing
+- Failing to close connections can lead to resource leaks and potential billing issues with voice service providers
+
+---
+
+## Reference: voice.connect()
+
+> Documentation for the connect() method available in real-time voice providers, which establishes a connection for speech-to-speech communication.
+
+The `connect()` method establishes a WebSocket or WebRTC connection for real-time speech-to-speech communication. This method must be called before using other real-time features like `send()` or `answer()`.
+
+## Usage Example
+
+```typescript
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+import Speaker from "@mastra/node-speaker";
+
+const speaker = new Speaker({
+  sampleRate: 24100, // Audio sample rate in Hz - standard for high-quality audio on MacBook Pro
+  channels: 1, // Mono audio output (as opposed to stereo which would be 2)
+  bitDepth: 16, // Bit depth for audio quality - CD quality standard (16-bit resolution)
+});
+
+// Initialize a real-time voice provider
+const voice = new OpenAIRealtimeVoice({
+  realtimeConfig: {
+    model: "gpt-5.1-realtime",
+    apiKey: process.env.OPENAI_API_KEY,
+    options: {
+      sessionConfig: {
+        turn_detection: {
+          type: "server_vad",
+          threshold: 0.6,
+          silence_duration_ms: 1200,
+        },
+      },
+    },
+  },
+  speaker: "alloy", // Default voice
+});
+// Connect to the real-time service
+await voice.connect();
+// Now you can use real-time features
+voice.on("speaker", (stream) => {
+  stream.pipe(speaker);
+});
+// With connection options
+await voice.connect({
+  timeout: 10000, // 10 seconds timeout
+  reconnect: true,
+});
+```
+
+## Parameters
+
+## Return Value
+
+Returns a `Promise<void>` that resolves when the connection is successfully established.
+
+## Provider-Specific Options
+
+Each real-time voice provider may support different options for the `connect()` method:
+
+### OpenAI Realtime
+
+## Using with CompositeVoice
+
+When using `CompositeVoice`, the `connect()` method delegates to the configured real-time provider:
+
+```typescript
+import { CompositeVoice } from "@mastra/core/voice";
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+const realtimeVoice = new OpenAIRealtimeVoice();
+const voice = new CompositeVoice({
+  realtimeProvider: realtimeVoice,
+});
+// This will use the OpenAIRealtimeVoice provider
+await voice.connect();
+```
+
+## Notes
+
+- This method is only implemented by real-time voice providers that support speech-to-speech capabilities
+- If called on a voice provider that doesn't support this functionality, it will log a warning and resolve immediately
+- The connection must be established before using other real-time methods like `send()` or `answer()`
+- When you're done with the voice instance, call `close()` to properly clean up resources
+- Some providers may automatically reconnect on connection loss, depending on their implementation
+- Connection errors will typically be thrown as exceptions that should be caught and handled
+
+## Related Methods
+
+- [voice.send()](./voice.send) - Sends audio data to the voice provider
+- [voice.answer()](./voice.answer) - Triggers the voice provider to respond
+- [voice.close()](./voice.close) - Disconnects from the real-time service
+- [voice.on()](./voice.on) - Registers an event listener for voice events
+
+---
+
+## Reference: voice.listen()
+
+> Documentation for the listen() method available in all Mastra voice providers, which converts speech to text.
+
+The `listen()` method is a core function available in all Mastra voice providers that converts speech to text. It takes an audio stream as input and returns the transcribed text.
+
+## Parameters
+
+## Return Value
+
+Returns one of the following:
+
+- `Promise<string>`: A promise that resolves to the transcribed text
+- `Promise<NodeJS.ReadableStream>`: A promise that resolves to a stream of transcribed text (for streaming transcription)
+- `Promise<void>`: For real-time providers that emit 'writing' events instead of returning text directly
+
+## Provider-Specific Options
+
+Each voice provider may support additional options specific to their implementation. Here are some examples:
+
+### OpenAI
+
+### Google
+
+### Deepgram
+
+## Usage Example
+
+```typescript
+import { OpenAIVoice } from "@mastra/voice-openai";
+import { getMicrophoneStream } from "@mastra/node-audio";
+import { createReadStream } from "fs";
+import path from "path";
+
+// Initialize a voice provider
+const voice = new OpenAIVoice({
+  listeningModel: {
+    name: "whisper-1",
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+});
+
+// Basic usage with a file stream
+const audioFilePath = path.join(process.cwd(), "audio.mp3");
+const audioStream = createReadStream(audioFilePath);
+const transcript = await voice.listen(audioStream, {
+  filetype: "mp3",
+});
+console.log("Transcribed text:", transcript);
+
+// Using a microphone stream
+const microphoneStream = getMicrophoneStream(); // Assume this function gets audio input
+const transcription = await voice.listen(microphoneStream);
+
+// With provider-specific options
+const transcriptWithOptions = await voice.listen(audioStream, {
+  language: "en",
+  prompt: "This is a conversation about artificial intelligence.",
+});
+```
+
+## Using with CompositeVoice
+
+When using `CompositeVoice`, the `listen()` method delegates to the configured listening provider:
+
+```typescript
+import { CompositeVoice } from "@mastra/core/voice";
+import { OpenAIVoice } from "@mastra/voice-openai";
+import { PlayAIVoice } from "@mastra/voice-playai";
+
+const voice = new CompositeVoice({
+  input: new OpenAIVoice(),
+  output: new PlayAIVoice(),
+});
+
+// This will use the OpenAIVoice provider
+const transcript = await voice.listen(audioStream);
+```
+
+### Using AI SDK Model Providers
+
+You can also use AI SDK transcription models directly with `CompositeVoice`:
+
+```typescript
+import { CompositeVoice } from "@mastra/core/voice";
+import { openai } from "@ai-sdk/openai";
+import { groq } from "@ai-sdk/groq";
+
+// Use AI SDK transcription models
+const voice = new CompositeVoice({
+  input: openai.transcription('whisper-1'),  // AI SDK model
+  output: new PlayAIVoice(),                 // Mastra provider
+});
+
+// Works the same way
+const transcript = await voice.listen(audioStream);
+
+// Provider-specific options can be passed through
+const transcriptWithOptions = await voice.listen(audioStream, {
+  providerOptions: {
+    openai: {
+      language: 'en',
+      prompt: 'This is about AI',
+    }
+  }
+});
+```
+
+See the [CompositeVoice reference](https://mastra.ai/reference/v1/voice/composite-voice) for more details on AI SDK integration.
+
+## Realtime Voice Providers
+
+When using realtime voice providers like `OpenAIRealtimeVoice`, the `listen()` method behaves differently:
+
+- Instead of returning transcribed text, it emits 'writing' events with the transcribed text
+- You need to register an event listener to receive the transcription
+
+```typescript
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+import { getMicrophoneStream } from "@mastra/node-audio";
+
+const voice = new OpenAIRealtimeVoice();
+await voice.connect();
+
+// Register event listener for transcription
+voice.on("writing", ({ text, role }) => {
+  console.log(`${role}: ${text}`);
+});
+
+// This will emit 'writing' events instead of returning text
+const microphoneStream = getMicrophoneStream();
+await voice.listen(microphoneStream);
+```
+
+## Notes
+
+- Not all voice providers support speech-to-text functionality (e.g., PlayAI, Speechify)
+- The behavior of `listen()` may vary slightly between providers, but all implementations follow the same basic interface
+- When using a realtime voice provider, the method might not return text directly but instead emit a 'writing' event
+- The audio format supported depends on the provider. Common formats include MP3, WAV, and M4A
+- Some providers support streaming transcription, where text is returned as it's transcribed
+- For best performance, consider closing or ending the audio stream when you're done with it
+
+## Related Methods
+
+- [voice.speak()](./voice.speak) - Converts text to speech
+- [voice.send()](./voice.send) - Sends audio data to the voice provider in real-time
+- [voice.on()](./voice.on) - Registers an event listener for voice events
+
+---
+
+## Reference: voice.off()
+
+> Documentation for the off() method available in voice providers, which removes event listeners for voice events.
+
+The `off()` method removes event listeners previously registered with the `on()` method. This is particularly useful for cleaning up resources and preventing memory leaks in long-running applications with real-time voice capabilities.
+
+## Usage Example
+
+```typescript
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+import chalk from "chalk";
+
+// Initialize a real-time voice provider
+const voice = new OpenAIRealtimeVoice({
+  realtimeConfig: {
+    model: "gpt-5.1-realtime",
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+});
+
+// Connect to the real-time service
+await voice.connect();
+
+// Define the callback function
+const writingCallback = ({ text, role }) => {
+  if (role === "user") {
+    process.stdout.write(chalk.green(text));
+  } else {
+    process.stdout.write(chalk.blue(text));
+  }
+};
+
+// Register event listener
+voice.on("writing", writingCallback);
+
+// Later, when you want to remove the listener
+voice.off("writing", writingCallback);
+```
+
+## Parameters
+
+<br />
+
+## Return Value
+
+This method does not return a value.
+
+## Notes
+
+- The callback passed to `off()` must be the same function reference that was passed to `on()`
+- If the callback is not found, the method will have no effect
+- This method is primarily used with real-time voice providers that support event-based communication
+- If called on a voice provider that doesn't support events, it will log a warning and do nothing
+- Removing event listeners is important for preventing memory leaks in long-running applications
+
+---
+
+## Reference: voice.on()
+
+> Documentation for the on() method available in voice providers, which registers event listeners for voice events.
+
+The `on()` method registers event listeners for various voice events. This is particularly important for real-time voice providers, where events are used to communicate transcribed text, audio responses, and other state changes.
+
+## Usage Example
+
+```typescript
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+import Speaker from "@mastra/node-speaker";
+import chalk from "chalk";
+
+// Initialize a real-time voice provider
+const voice = new OpenAIRealtimeVoice({
+  realtimeConfig: {
+    model: "gpt-5.1-realtime",
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+});
+
+// Connect to the real-time service
+await voice.connect();
+
+// Register event listener for transcribed text
+voice.on("writing", (event) => {
+  if (event.role === "user") {
+    process.stdout.write(chalk.green(event.text));
+  } else {
+    process.stdout.write(chalk.blue(event.text));
+  }
+});
+
+// Listen for audio data and play it
+const speaker = new Speaker({
+  sampleRate: 24100,
+  channels: 1,
+  bitDepth: 16,
+});
+
+voice.on("speaker", (stream) => {
+  stream.pipe(speaker);
+});
+
+// Register event listener for errors
+voice.on("error", ({ message, code, details }) => {
+  console.error(`Error ${code}: ${message}`, details);
+});
+```
+
+## Parameters
+
+<br />
+
+## Return Value
+
+This method does not return a value.
+
+## Events
+
+For a comprehensive list of events and their payload structures, see the [Voice Events](./voice.events) documentation.
+
+Common events include:
+
+- `speaking`: Emitted when audio data is available
+- `speaker`: Emitted with a stream that can be piped to audio output
+- `writing`: Emitted when text is transcribed or generated
+- `error`: Emitted when an error occurs
+- `tool-call-start`: Emitted when a tool is about to be executed
+- `tool-call-result`: Emitted when a tool execution is complete
+
+Different voice providers may support different sets of events with varying payload structures.
+
+## Using with CompositeVoice
+
+When using `CompositeVoice`, the `on()` method delegates to the configured real-time provider:
+
+```typescript
+import { CompositeVoice } from "@mastra/core/voice";
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+import Speaker from "@mastra/node-speaker";
+
+const speaker = new Speaker({
+  sampleRate: 24100, // Audio sample rate in Hz - standard for high-quality audio on MacBook Pro
+  channels: 1, // Mono audio output (as opposed to stereo which would be 2)
+  bitDepth: 16, // Bit depth for audio quality - CD quality standard (16-bit resolution)
+});
+
+const realtimeVoice = new OpenAIRealtimeVoice();
+const voice = new CompositeVoice({
+  realtimeProvider: realtimeVoice,
+});
+
+// Connect to the real-time service
+await voice.connect();
+
+// This will register the event listener with the OpenAIRealtimeVoice provider
+voice.on("speaker", (stream) => {
+  stream.pipe(speaker);
+});
+```
+
+## Notes
+
+- This method is primarily used with real-time voice providers that support event-based communication
+- If called on a voice provider that doesn't support events, it will log a warning and do nothing
+- Event listeners should be registered before calling methods that might emit events
+- To remove an event listener, use the [voice.off()](./voice.off) method with the same event name and callback function
+- Multiple listeners can be registered for the same event
+- The callback function will receive different data depending on the event type (see [Voice Events](./voice.events))
+- For best performance, consider removing event listeners when they are no longer needed
+
+---
+
+## Reference: voice.send()
+
+> Documentation for the send() method available in real-time voice providers, which streams audio data for continuous processing.
+
+The `send()` method streams audio data in real-time to voice providers for continuous processing. This method is essential for real-time speech-to-speech conversations, allowing you to send microphone input directly to the AI service.
+
+## Usage Example
+
+```typescript
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+import Speaker from "@mastra/node-speaker";
+import { getMicrophoneStream } from "@mastra/node-audio";
+
+const speaker = new Speaker({
+  sampleRate: 24100, // Audio sample rate in Hz - standard for high-quality audio on MacBook Pro
+  channels: 1, // Mono audio output (as opposed to stereo which would be 2)
+  bitDepth: 16, // Bit depth for audio quality - CD quality standard (16-bit resolution)
+});
+
+// Initialize a real-time voice provider
+const voice = new OpenAIRealtimeVoice({
+  realtimeConfig: {
+    model: "gpt-5.1-realtime",
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+});
+
+// Connect to the real-time service
+await voice.connect();
+
+// Set up event listeners for responses
+voice.on("writing", ({ text, role }) => {
+  console.log(`${role}: ${text}`);
+});
+
+voice.on("speaker", (stream) => {
+  stream.pipe(speaker);
+});
+
+// Get microphone stream (implementation depends on your environment)
+const microphoneStream = getMicrophoneStream();
+
+// Send audio data to the voice provider
+await voice.send(microphoneStream);
+
+// You can also send audio data as Int16Array
+const audioBuffer = getAudioBuffer(); // Assume this returns Int16Array
+await voice.send(audioBuffer);
+```
+
+## Parameters
+
+<br />
+
+## Return Value
+
+Returns a `Promise<void>` that resolves when the audio data has been accepted by the voice provider.
+
+## Notes
+
+- This method is only implemented by real-time voice providers that support speech-to-speech capabilities
+- If called on a voice provider that doesn't support this functionality, it will log a warning and resolve immediately
+- You must call `connect()` before using `send()` to establish the WebSocket connection
+- The audio format requirements depend on the specific voice provider
+- For continuous conversation, you typically call `send()` to transmit user audio, then `answer()` to trigger the AI response
+- The provider will typically emit 'writing' events with transcribed text as it processes the audio
+- When the AI responds, the provider will emit 'speaking' events with the audio response
+
+---
+
+## Reference: voice.speak()
+
+> Documentation for the speak() method available in all Mastra voice providers, which converts text to speech.
+
+The `speak()` method is a core function available in all Mastra voice providers that converts text to speech. It takes text input and returns an audio stream that can be played or saved.
+
+## Parameters
+
+## Return Value
+
+Returns a `Promise<NodeJS.ReadableStream | void>` where:
+
+- `NodeJS.ReadableStream`: A stream of audio data that can be played or saved
+- `void`: When using a realtime voice provider that emits audio through events instead of returning it directly
+
+## Provider-Specific Options
+
+Each voice provider may support additional options specific to their implementation. Here are some examples:
+
+### OpenAI
+
+### ElevenLabs
+
+### Google
+
+### Murf
+
+## Usage Example
+
+```typescript
+import { OpenAIVoice } from "@mastra/voice-openai";
+// Initialize a voice provider
+const voice = new OpenAIVoice({
+  speaker: "alloy", // Default voice
+});
+// Basic usage with default settings
+const audioStream = await voice.speak("Hello, world!");
+// Using a different voice for this specific request
+const audioStreamWithDifferentVoice = await voice.speak("Hello again!", {
+  speaker: "nova",
+});
+// Using provider-specific options
+const audioStreamWithOptions = await voice.speak("Hello with options!", {
+  speaker: "echo",
+  speed: 1.2, // OpenAI-specific option
+});
+// Using a text stream as input
+import { Readable } from "stream";
+const textStream = Readable.from(["Hello", " from", " a", " stream!"]);
+const audioStreamFromTextStream = await voice.speak(textStream);
+```
+
+## Using with CompositeVoice
+
+When using `CompositeVoice`, the `speak()` method delegates to the configured speaking provider:
+
+```typescript
+import { CompositeVoice } from "@mastra/core/voice";
+import { OpenAIVoice } from "@mastra/voice-openai";
+import { PlayAIVoice } from "@mastra/voice-playai";
+
+const voice = new CompositeVoice({
+  output: new PlayAIVoice(),
+  input: new OpenAIVoice(),
+});
+
+// This will use the PlayAIVoice provider
+const audioStream = await voice.speak("Hello, world!");
+```
+
+### Using AI SDK Model Providers
+
+You can also use AI SDK speech models directly with `CompositeVoice`:
+
+```typescript
+import { CompositeVoice } from "@mastra/core/voice";
+import { openai } from "@ai-sdk/openai";
+import { elevenlabs } from "@ai-sdk/elevenlabs";
+
+// Use AI SDK speech models
+const voice = new CompositeVoice({
+  output: elevenlabs.speech('eleven_turbo_v2'),  // AI SDK model
+  input: openai.transcription('whisper-1'),      // AI SDK model
+});
+
+// Works the same way
+const audioStream = await voice.speak("Hello from AI SDK!");
+
+// Provider-specific options can be passed through
+const audioWithOptions = await voice.speak("Hello with options!", {
+  speaker: 'Rachel',  // ElevenLabs voice
+  providerOptions: {
+    elevenlabs: {
+      stability: 0.5,
+      similarity_boost: 0.75,
+    }
+  }
+});
+```
+
+See the [CompositeVoice reference](https://mastra.ai/reference/v1/voice/composite-voice) for more details on AI SDK integration.
+
+## Realtime Voice Providers
+
+When using realtime voice providers like `OpenAIRealtimeVoice`, the `speak()` method behaves differently:
+
+- Instead of returning an audio stream, it emits a 'speaking' event with the audio data
+- You need to register an event listener to receive the audio chunks
+
+```typescript
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+import Speaker from "@mastra/node-speaker";
+
+const speaker = new Speaker({
+  sampleRate: 24100, // Audio sample rate in Hz - standard for high-quality audio on MacBook Pro
+  channels: 1, // Mono audio output (as opposed to stereo which would be 2)
+  bitDepth: 16, // Bit depth for audio quality - CD quality standard (16-bit resolution)
+});
+
+const voice = new OpenAIRealtimeVoice();
+await voice.connect();
+// Register event listener for audio chunks
+voice.on("speaker", (stream) => {
+  // Handle audio chunk (e.g., play it or save it)
+  stream.pipe(speaker);
+});
+// This will emit 'speaking' events instead of returning a stream
+await voice.speak("Hello, this is realtime speech!");
+```
+
+## Notes
+
+- The behavior of `speak()` may vary slightly between providers, but all implementations follow the same basic interface.
+- When using a realtime voice provider, the method might not return an audio stream directly but instead emit a 'speaking' event.
+- If a text stream is provided as input, the provider will typically convert it to a string before processing.
+- The audio format of the returned stream depends on the provider. Common formats include MP3, WAV, and OGG.
+- For best performance, consider closing or ending the audio stream when you're done with it.
+
+---
+
+## Reference: voice.updateConfig()
+
+> Documentation for the updateConfig() method available in voice providers, which updates the configuration of a voice provider at runtime.
+
+The `updateConfig()` method allows you to update the configuration of a voice provider at runtime. This is useful for changing voice settings, API keys, or other provider-specific options without creating a new instance.
+
+## Usage Example
+
+```typescript
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+
+// Initialize a real-time voice provider
+const voice = new OpenAIRealtimeVoice({
+  realtimeConfig: {
+    model: "gpt-5.1-realtime",
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+  speaker: "alloy",
+});
+
+// Connect to the real-time service
+await voice.connect();
+
+// Later, update the configuration
+voice.updateConfig({
+  voice: "nova", // Change the default voice
+  turn_detection: {
+    type: "server_vad",
+    threshold: 0.5,
+    silence_duration_ms: 1000,
+  },
+});
+
+// The next speak() call will use the new configuration
+await voice.speak("Hello with my new voice!");
+```
+
+## Parameters
+
+<br />
+
+## Return Value
+
+This method does not return a value.
+
+## Configuration Options
+
+Different voice providers support different configuration options:
+
+### OpenAI Realtime
+
+<br />
+
+## Notes
+
+- The default implementation logs a warning if the provider doesn't support this method
+- Configuration updates are typically applied to subsequent operations, not ongoing ones
+- Not all properties that can be set in the constructor can be updated at runtime
+- The specific behavior depends on the voice provider implementation
+- For real-time voice providers, some configuration changes may require reconnecting to the service