@mastra/mcp-docs-server 0.0.0-commonjs-20250414101718

package/.docs/raw/reference/voice/voice.connect.mdx

@@ -0,0 +1,124 @@
+---
+title: "Reference: voice.connect() | Voice Providers | Mastra Docs"
+description: "Documentation for the connect() method available in real-time voice providers, which establishes a connection for speech-to-speech communication."
+---
+
+# voice.connect()
+
+The `connect()` method establishes a WebSocket or WebRTC connection for real-time speech-to-speech communication. This method must be called before using other real-time features like `send()` or `answer()`.
+
+## Usage Example
+
+```typescript
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+import Speaker from "@mastra/node-speaker";
+
+const speaker = new Speaker({
+  sampleRate: 24100,  // Audio sample rate in Hz - standard for high-quality audio on MacBook Pro
+  channels: 1,        // Mono audio output (as opposed to stereo which would be 2)
+  bitDepth: 16,       // Bit depth for audio quality - CD quality standard (16-bit resolution)
+});
+
+// Initialize a real-time voice provider
+const voice = new OpenAIRealtimeVoice({
+  realtimeConfig: {
+    model: "gpt-4o-mini-realtime",
+    apiKey: process.env.OPENAI_API_KEY,
+    options: {
+      sessionConfig: {
+        turn_detection: {
+          type: "server_vad",
+          threshold: 0.6,
+          silence_duration_ms: 1200,
+        },
+      },
+    },
+  },
+  speaker: "alloy", // Default voice
+});
+// Connect to the real-time service
+await voice.connect();
+// Now you can use real-time features
+voice.on("speaker", (stream) => {
+  stream.pipe(speaker);
+});
+// With connection options
+await voice.connect({
+  timeout: 10000, // 10 seconds timeout
+  reconnect: true,
+});
+```
+
+
+## Parameters
+
+<PropertiesTable
+  content={[
+    {
+      name: "options",
+      type: "Record<string, unknown>",
+      description: "Provider-specific connection options",
+      isOptional: true,
+    }
+  ]}
+/>
+
+## Return Value
+
+Returns a `Promise<void>` that resolves when the connection is successfully established.
+
+## Provider-Specific Options
+
+Each real-time voice provider may support different options for the `connect()` method:
+
+### OpenAI Realtime
+
+<PropertiesTable
+  content={[
+    {
+      name: "options.timeout",
+      type: "number",
+      description: "Connection timeout in milliseconds",
+      isOptional: true,
+      defaultValue: "30000",
+    },
+    {
+      name: "options.reconnect",
+      type: "boolean",
+      description: "Whether to automatically reconnect on connection loss",
+      isOptional: true,
+      defaultValue: "false",
+    }
+  ]}
+/>
+
+## Using with CompositeVoice
+
+When using `CompositeVoice`, the `connect()` method delegates to the configured real-time provider:
+
+```typescript
+import { CompositeVoice } from "@mastra/core/voice";
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+const realtimeVoice = new OpenAIRealtimeVoice();
+const voice = new CompositeVoice({
+  realtimeProvider: realtimeVoice,
+});
+// This will use the OpenAIRealtimeVoice provider
+await voice.connect();
+```
+
+## Notes
+
+- This method is only implemented by real-time voice providers that support speech-to-speech capabilities
+- If called on a voice provider that doesn't support this functionality, it will log a warning and resolve immediately
+- The connection must be established before using other real-time methods like `send()` or `answer()`
+- When you're done with the voice instance, call `close()` to properly clean up resources
+- Some providers may automatically reconnect on connection loss, depending on their implementation
+- Connection errors will typically be thrown as exceptions that should be caught and handled
+
+## Related Methods
+
+- [voice.send()](./voice.send) - Sends audio data to the voice provider
+- [voice.answer()](./voice.answer) - Triggers the voice provider to respond
+- [voice.close()](./voice.close) - Disconnects from the real-time service
+- [voice.on()](./voice.on) - Registers an event listener for voice events

package/.docs/raw/reference/voice/voice.listen.mdx

@@ -0,0 +1,195 @@
+---
+title: "Reference: voice.listen() | Voice Providers | Mastra Docs"
+description: "Documentation for the listen() method available in all Mastra voice providers, which converts speech to text."
+---
+
+# voice.listen()
+
+The `listen()` method is a core function available in all Mastra voice providers that converts speech to text. It takes an audio stream as input and returns the transcribed text.
+
+## Usage Example
+
+```typescript
+import { OpenAIVoice } from "@mastra/voice-openai";
+import { createReadStream } from "fs";
+import path from "path";
+
+// Initialize a voice provider
+const voice = new OpenAIVoice({
+  listeningModel: {
+    name: "whisper-1",
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+});
+
+// Basic usage with a file stream
+const audioFilePath = path.join(process.cwd(), "audio.mp3");
+const audioStream = createReadStream(audioFilePath);
+const transcript = await voice.listen(audioStream, {
+  filetype: "mp3",
+});
+console.log("Transcribed text:", transcript);
+
+// Using a microphone stream
+const microphoneStream = getMicrophoneStream(); // Assume this function gets audio input
+const transcription = await voice.listen(microphoneStream);
+
+// With provider-specific options
+const transcriptWithOptions = await voice.listen(audioStream, {
+  language: "en",
+  prompt: "This is a conversation about artificial intelligence.",
+});
+```
+
+## Parameters
+
+<PropertiesTable
+  content={[
+    {
+      name: "audioStream",
+      type: "NodeJS.ReadableStream",
+      description: "Audio stream to transcribe. This can be a file stream or a microphone stream.",
+      isOptional: false,
+    },
+    {
+      name: "options",
+      type: "object",
+      description: "Provider-specific options for speech recognition",
+      isOptional: true,
+    }
+  ]}
+/>
+
+## Return Value
+
+Returns one of the following:
+- `Promise<string>`: A promise that resolves to the transcribed text
+- `Promise<NodeJS.ReadableStream>`: A promise that resolves to a stream of transcribed text (for streaming transcription)
+- `Promise<void>`: For real-time providers that emit 'writing' events instead of returning text directly
+
+## Provider-Specific Options
+
+Each voice provider may support additional options specific to their implementation. Here are some examples:
+
+### OpenAI
+
+<PropertiesTable
+  content={[
+    {
+      name: "options.filetype",
+      type: "string",
+      description: "Audio file format (e.g., 'mp3', 'wav', 'm4a')",
+      isOptional: true,
+      defaultValue: "'mp3'",
+    },
+    {
+      name: "options.prompt",
+      type: "string",
+      description: "Text to guide the model's transcription",
+      isOptional: true,
+    },
+    {
+      name: "options.language",
+      type: "string",
+      description: "Language code (e.g., 'en', 'fr', 'de')",
+      isOptional: true,
+    }
+  ]}
+/>
+
+### Google
+
+<PropertiesTable
+  content={[
+    {
+      name: "options.stream",
+      type: "boolean",
+      description: "Whether to use streaming recognition",
+      isOptional: true,
+      defaultValue: "false",
+    },
+    {
+      name: "options.config",
+      type: "object",
+      description: "Recognition configuration from Google Cloud Speech-to-Text API",
+      isOptional: true,
+      defaultValue: "{ encoding: 'LINEAR16', languageCode: 'en-US' }",
+    }
+  ]}
+/>
+
+### Deepgram
+
+<PropertiesTable
+  content={[
+    {
+      name: "options.model",
+      type: "string",
+      description: "Deepgram model to use for transcription",
+      isOptional: true,
+      defaultValue: "'nova-2'",
+    },
+    {
+      name: "options.language",
+      type: "string",
+      description: "Language code for transcription",
+      isOptional: true,
+      defaultValue: "'en'",
+    }
+  ]}
+/>
+
+## Realtime Voice Providers
+
+When using realtime voice providers like `OpenAIRealtimeVoice`, the `listen()` method behaves differently:
+
+- Instead of returning transcribed text, it emits 'writing' events with the transcribed text
+- You need to register an event listener to receive the transcription
+
+```typescript
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+const voice = new OpenAIRealtimeVoice();
+await voice.connect();
+
+// Register event listener for transcription
+voice.on("writing", ({ text, role }) => {
+  console.log(`${role}: ${text}`);
+});
+
+// This will emit 'writing' events instead of returning text
+const microphoneStream = getMicrophoneStream();
+await voice.listen(microphoneStream);
+```
+
+## Using with CompositeVoice
+
+When using `CompositeVoice`, the `listen()` method delegates to the configured listening provider:
+
+```typescript
+import { CompositeVoice } from "@mastra/core/voice";
+import { OpenAIVoice } from "@mastra/voice-openai";
+import { PlayAIVoice } from "@mastra/voice-playai";
+
+const voice = new CompositeVoice({
+  listenProvider: new OpenAIVoice(),
+  speakProvider: new PlayAIVoice(),
+});
+
+// This will use the OpenAIVoice provider
+const transcript = await voice.listen(audioStream);
+```
+
+## Notes
+
+- Not all voice providers support speech-to-text functionality (e.g., PlayAI, Speechify)
+- The behavior of `listen()` may vary slightly between providers, but all implementations follow the same basic interface
+- When using a realtime voice provider, the method might not return text directly but instead emit a 'writing' event
+- The audio format supported depends on the provider. Common formats include MP3, WAV, and M4A
+- Some providers support streaming transcription, where text is returned as it's transcribed
+- For best performance, consider closing or ending the audio stream when you're done with it
+
+## Related Methods
+
+- [voice.speak()](./voice.speak) - Converts text to speech
+- [voice.send()](./voice.send) - Sends audio data to the voice provider in real-time
+- [voice.on()](./voice.on) - Registers an event listener for voice events

package/.docs/raw/reference/voice/voice.on.mdx

@@ -0,0 +1,189 @@
+---
+title: "Reference: voice.on() | Voice Providers | Mastra Docs"
+description: "Documentation for the on() method available in voice providers, which registers event listeners for voice events."
+---
+
+# voice.on()
+
+The `on()` method registers event listeners for various voice events. This is particularly important for real-time voice providers, where events are used to communicate transcribed text, audio responses, and other state changes.
+
+## Usage Example
+
+```typescript
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+import Speaker from "@mastra/node-speaker";
+import chalk from "chalk";
+
+const speaker = new Speaker({
+  sampleRate: 24100,  // Audio sample rate in Hz - standard for high-quality audio on MacBook Pro
+  channels: 1,        // Mono audio output (as opposed to stereo which would be 2)
+  bitDepth: 16,       // Bit depth for audio quality - CD quality standard (16-bit resolution)
+});
+
+// Initialize a real-time voice provider
+const voice = new OpenAIRealtimeVoice({
+  realtimeConfig: {
+    model: "gpt-4o-mini-realtime",
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+});
+
+// Connect to the real-time service
+await voice.connect();
+
+// Register event listener for transcribed text
+voice.on("writing", ({ text, role }) => {
+  if (ev.role === 'user') {
+    process.stdout.write(chalk.green(ev.text));
+  } else {
+    process.stdout.write(chalk.blue(ev.text));
+  }
+});
+
+// Register event listener for speaker responses
+voice.on("speaking", (stream) => {
+  // Stream the audio to node speaker
+  stream.pipe(speaker)
+});
+
+// Register event listener for errors
+voice.on("error", ({ message, code, details }) => {
+  console.error(`Error ${code}: ${message}`, details);
+});
+```
+
+## Parameters
+
+<PropertiesTable
+  content={[
+    {
+      name: "event",
+      type: "string",
+      description: "Name of the event to listen for (e.g., 'speaking', 'writing', 'error')",
+      isOptional: false,
+    },
+    {
+      name: "callback",
+      type: "function",
+      description: "Callback function that will be called when the event occurs",
+      isOptional: false,
+    }
+  ]}
+/>
+
+## Return Value
+
+This method does not return a value.
+
+## Standard Events
+
+All voice providers that implement event handling support these standard events:
+
+<PropertiesTable
+  content={[
+    {
+      name: "speaking",
+      type: "event",
+      description: "Emitted when audio data is available. The callback receives { audio } where audio is typically an Int16Array or Buffer.",
+    },
+    {
+      name: "speaker",
+      type: "event",
+      description: "Emitted when a new audio response is ready to be streamed. The callback receives a buffer that can be piped into node-speaker."
+    },
+    {
+      name: "writing",
+      type: "event",
+      description: "Emitted when text is transcribed or generated. The callback receives { text, role } where role is either 'user' or 'assistant'.",
+    },
+    {
+      name: "error",
+      type: "event",
+      description: "Emitted when an error occurs. The callback receives { message, code, details } with information about the error.",
+    }
+  ]}
+/>
+
+## Provider-Specific Events
+
+Different voice providers may emit additional events specific to their implementation:
+
+### OpenAI Realtime
+
+OpenAI Realtime events are prefixed with `openAIRealtime:` and include:
+
+<PropertiesTable
+  content={[
+    {
+      name: "openAIRealtime:conversation.created",
+      type: "event",
+      description: "Emitted when a new conversation is created.",
+    },
+    {
+      name: "openAIRealtime:conversation.interrupted",
+      type: "event",
+      description: "Emitted when a conversation is interrupted.",
+    },
+    {
+      name: "openAIRealtime:conversation.updated",
+      type: "event",
+      description: "Emitted when a conversation is updated.",
+    },
+    {
+      name: "openAIRealtime:conversation.item.appended",
+      type: "event",
+      description: "Emitted when an item is appended to the conversation.",
+    },
+    {
+      name: "openAIRealtime:conversation.item.completed",
+      type: "event",
+      description: "Emitted when an item in the conversation is completed.",
+    }
+  ]}
+/>
+
+## Using with CompositeVoice
+
+When using `CompositeVoice`, the `on()` method delegates to the configured real-time provider:
+
+```typescript
+import { CompositeVoice } from "@mastra/core/voice";
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+import Speaker from "@mastra/node-speaker";
+
+const speaker = new Speaker({
+  sampleRate: 24100,  // Audio sample rate in Hz - standard for high-quality audio on MacBook Pro
+  channels: 1,        // Mono audio output (as opposed to stereo which would be 2)
+  bitDepth: 16,       // Bit depth for audio quality - CD quality standard (16-bit resolution)
+});
+
+const realtimeVoice = new OpenAIRealtimeVoice();
+const voice = new CompositeVoice({
+  realtimeProvider: realtimeVoice,
+});
+
+// Connect to the real-time service
+await voice.connect();
+
+// This will register the event listener with the OpenAIRealtimeVoice provider
+voice.on("speaker", (stream) => {
+  stream.pipe(speaker)
+});
+```
+
+## Notes
+
+- This method is primarily used with real-time voice providers that support event-based communication
+- If called on a voice provider that doesn't support events, it will log a warning and do nothing
+- Event listeners should be registered before calling methods that might emit events
+- To remove an event listener, use the `off()` method with the same event name and callback function
+- Multiple listeners can be registered for the same event
+- The callback function will receive different data depending on the event type
+- For best performance, consider removing event listeners when they are no longer needed
+
+## Related Methods
+
+- [voice.off()](./voice.off) - Removes an event listener
+- [voice.connect()](./voice.connect) - Establishes a connection to the real-time service
+- [voice.send()](./voice.send) - Sends audio data to the voice provider
+- [voice.answer()](./voice.answer) - Triggers the voice provider to respond

package/.docs/raw/reference/voice/voice.send.mdx

@@ -0,0 +1,118 @@
+---
+title: "Reference: voice.send() | Voice Providers | Mastra Docs"
+description: "Documentation for the send() method available in real-time voice providers, which streams audio data for continuous processing."
+---
+
+# voice.send()
+
+The `send()` method streams audio data in real-time to voice providers for continuous processing. This method is essential for real-time speech-to-speech conversations, allowing you to send microphone input directly to the AI service.
+
+## Usage Example
+
+```typescript
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+import Speaker from "@mastra/node-speaker";
+
+const speaker = new Speaker({
+  sampleRate: 24100,  // Audio sample rate in Hz - standard for high-quality audio on MacBook Pro
+  channels: 1,        // Mono audio output (as opposed to stereo which would be 2)
+  bitDepth: 16,       // Bit depth for audio quality - CD quality standard (16-bit resolution)
+});
+
+
+// Initialize a real-time voice provider
+const voice = new OpenAIRealtimeVoice({
+  realtimeConfig: {
+    model: "gpt-4o-mini-realtime",
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+});
+
+// Connect to the real-time service
+await voice.connect();
+
+// Set up event listeners for responses
+voice.on("writing", ({ text, role }) => {
+  console.log(`${role}: ${text}`);
+});
+
+voice.on("speaker", (stream) => {
+  stream.pipe(speaker)
+});
+
+// Get microphone stream (implementation depends on your environment)
+const microphoneStream = getMicrophoneStream();
+
+// Send audio data to the voice provider
+await voice.send(microphoneStream);
+
+// You can also send audio data as Int16Array
+const audioBuffer = getAudioBuffer(); // Assume this returns Int16Array
+await voice.send(audioBuffer);
+```
+
+## Parameters
+
+<PropertiesTable
+  content={[
+    {
+      name: "audioData",
+      type: "NodeJS.ReadableStream | Int16Array",
+      description: "Audio data to send to the voice provider. Can be a readable stream (like a microphone stream) or an Int16Array of audio samples.",
+      isOptional: false,
+    }
+  ]}
+/>
+
+## Return Value
+
+Returns a `Promise<void>` that resolves when the audio data has been accepted by the voice provider.
+
+## Provider-Specific Behavior
+
+Different real-time voice providers may handle the audio data in different ways:
+
+### OpenAI Realtime
+
+- Accepts both NodeJS.ReadableStream and Int16Array formats
+- Audio should be in PCM format, 16-bit, mono, 16kHz sample rate
+- Automatically handles voice activity detection (VAD) to determine when the user has finished speaking
+- Emits 'writing' events with transcribed text as it processes the audio
+
+## Using with CompositeVoice
+
+When using `CompositeVoice`, the `send()` method delegates to the configured real-time provider:
+
+```typescript
+import { CompositeVoice } from "@mastra/core/voice";
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+
+const realtimeVoice = new OpenAIRealtimeVoice();
+const voice = new CompositeVoice({
+  realtimeProvider: realtimeVoice,
+});
+
+// Connect to the real-time service
+await voice.connect();
+
+// This will use the OpenAIRealtimeVoice provider
+const microphoneStream = getMicrophoneStream();
+await voice.send(microphoneStream);
+```
+
+## Notes
+
+- This method is only implemented by real-time voice providers that support speech-to-speech capabilities
+- If called on a voice provider that doesn't support this functionality, it will log a warning and resolve immediately
+- You must call `connect()` before using `send()` to establish the WebSocket connection
+- The audio format requirements depend on the specific voice provider
+- For continuous conversation, you typically call `send()` to transmit user audio, then `answer()` to trigger the AI response
+- The provider will typically emit 'writing' events with transcribed text as it processes the audio
+- When the AI responds, the provider will emit 'speaking' events with the audio response
+
+## Related Methods
+
+- [voice.connect()](./voice.connect) - Establishes a connection to the real-time service
+- [voice.answer()](./voice.answer) - Triggers the voice provider to respond
+- [voice.listen()](./voice.listen) - Converts speech to text (non-streaming)
+- [voice.on()](./voice.on) - Registers an event listener for voice events