@mastra/mcp-docs-server 1.1.5 → 1.1.6-alpha.0

package/.docs/reference/voice/voice.answer.md

@@ -0,0 +1,54 @@
+# voice.answer()
+
+The `answer()` method is used in real-time voice providers to trigger the AI to generate a response. This method is particularly useful in speech-to-speech conversations where you need to explicitly signal the AI to respond after receiving user input.
+
+## Usage Example
+
+```typescript
+import { OpenAIRealtimeVoice } from '@mastra/voice-openai-realtime'
+import { getMicrophoneStream } from '@mastra/node-audio'
+import Speaker from '@mastra/node-speaker'
+
+const speaker = new Speaker({
+  sampleRate: 24100, // Audio sample rate in Hz - standard for high-quality audio on MacBook Pro
+  channels: 1, // Mono audio output (as opposed to stereo which would be 2)
+  bitDepth: 16, // Bit depth for audio quality - CD quality standard (16-bit resolution)
+})
+
+// Initialize a real-time voice provider
+const voice = new OpenAIRealtimeVoice({
+  realtimeConfig: {
+    model: 'gpt-5.1',
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+  speaker: 'alloy', // Default voice
+})
+// Connect to the real-time service
+await voice.connect()
+// Register event listener for responses
+voice.on('speaker', stream => {
+  // Handle audio response
+  stream.pipe(speaker)
+})
+// Send user audio input
+const microphoneStream = getMicrophoneStream()
+await voice.send(microphoneStream)
+// Trigger the AI to respond
+await voice.answer()
+```
+
+## Parameters
+
+**options?:** (`Record<string, unknown>`): Provider-specific options for the response
+
+## Return Value
+
+Returns a `Promise<void>` that resolves when the response has been triggered.
+
+## Notes
+
+- This method is only implemented by real-time voice providers that support speech-to-speech capabilities
+- If called on a voice provider that doesn't support this functionality, it will log a warning and resolve immediately
+- The response audio will typically be emitted through the 'speaking' event rather than returned directly
+- For providers that support it, you can use this method to send a specific response instead of having the AI generate one
+- This method is commonly used in conjunction with `send()` to create a conversational flow

package/.docs/reference/voice/voice.close.md

@@ -0,0 +1,51 @@
+# voice.close()
+
+The `close()` method disconnects from a real-time voice service and cleans up resources. This is important for properly ending voice sessions and preventing resource leaks.
+
+## Usage Example
+
+```typescript
+import { OpenAIRealtimeVoice } from '@mastra/voice-openai-realtime'
+import { getMicrophoneStream } from '@mastra/node-audio'
+
+// Initialize a real-time voice provider
+const voice = new OpenAIRealtimeVoice({
+  realtimeConfig: {
+    model: 'gpt-5.1-realtime',
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+})
+
+// Connect to the real-time service
+await voice.connect()
+
+// Start a conversation
+voice.speak("Hello, I'm your AI assistant!")
+
+// Stream audio from a microphone
+const microphoneStream = getMicrophoneStream()
+voice.send(microphoneStream)
+
+// When the conversation is complete
+setTimeout(() => {
+  // Close the connection and clean up resources
+  voice.close()
+  console.log('Voice session ended')
+}, 60000) // End after 1 minute
+```
+
+## Parameters
+
+This method does not accept any parameters.
+
+## Return Value
+
+This method does not return a value.
+
+## Notes
+
+- Always call `close()` when you're done with a real-time voice session to free up resources
+- After calling `close()`, you'll need to call `connect()` again if you want to start a new session
+- This method is primarily used with real-time voice providers that maintain persistent connections
+- If called on a voice provider that doesn't support real-time connections, it will log a warning and do nothing
+- Failing to close connections can lead to resource leaks and potential billing issues with voice service providers

package/.docs/reference/voice/voice.connect.md

@@ -0,0 +1,94 @@
+# voice.connect()
+
+The `connect()` method establishes a WebSocket or WebRTC connection for real-time speech-to-speech communication. This method must be called before using other real-time features like `send()` or `answer()`.
+
+## Usage Example
+
+```typescript
+import { OpenAIRealtimeVoice } from '@mastra/voice-openai-realtime'
+import Speaker from '@mastra/node-speaker'
+
+const speaker = new Speaker({
+  sampleRate: 24100, // Audio sample rate in Hz - standard for high-quality audio on MacBook Pro
+  channels: 1, // Mono audio output (as opposed to stereo which would be 2)
+  bitDepth: 16, // Bit depth for audio quality - CD quality standard (16-bit resolution)
+})
+
+// Initialize a real-time voice provider
+const voice = new OpenAIRealtimeVoice({
+  realtimeConfig: {
+    model: 'gpt-5.1-realtime',
+    apiKey: process.env.OPENAI_API_KEY,
+    options: {
+      sessionConfig: {
+        turn_detection: {
+          type: 'server_vad',
+          threshold: 0.6,
+          silence_duration_ms: 1200,
+        },
+      },
+    },
+  },
+  speaker: 'alloy', // Default voice
+})
+// Connect to the real-time service
+await voice.connect()
+// Now you can use real-time features
+voice.on('speaker', stream => {
+  stream.pipe(speaker)
+})
+// With connection options
+await voice.connect({
+  timeout: 10000, // 10 seconds timeout
+  reconnect: true,
+})
+```
+
+## Parameters
+
+**options?:** (`Record<string, unknown>`): Provider-specific connection options
+
+## Return Value
+
+Returns a `Promise<void>` that resolves when the connection is successfully established.
+
+## Provider-Specific Options
+
+Each real-time voice provider may support different options for the `connect()` method:
+
+### OpenAI Realtime
+
+**options.timeout?:** (`number`): Connection timeout in milliseconds (Default: `30000`)
+
+**options.reconnect?:** (`boolean`): Whether to automatically reconnect on connection loss (Default: `false`)
+
+## Using with CompositeVoice
+
+When using `CompositeVoice`, the `connect()` method delegates to the configured real-time provider:
+
+```typescript
+import { CompositeVoice } from '@mastra/core/voice'
+import { OpenAIRealtimeVoice } from '@mastra/voice-openai-realtime'
+const realtimeVoice = new OpenAIRealtimeVoice()
+const voice = new CompositeVoice({
+  realtime: realtimeVoice,
+})
+// This will use the OpenAIRealtimeVoice provider
+await voice.connect()
+```
+
+## Notes
+
+- This method is only implemented by real-time voice providers that support speech-to-speech capabilities
+- If called on a voice provider that doesn't support this functionality, it will log a warning and resolve immediately
+- The connection must be established before using other real-time methods like `send()` or `answer()`
+- When you're done with the voice instance, call `close()` to properly clean up resources
+- Some providers may automatically reconnect on connection loss, depending on their implementation
+- Connection errors will typically be thrown as exceptions that should be caught and handled
+
+## Related Methods
+
+- [voice.send()](https://mastra.ai/reference/voice/voice.send) - Sends audio data to the voice provider
+- [voice.answer()](https://mastra.ai/reference/voice/voice.answer) - Triggers the voice provider to respond
+- [voice.close()](https://mastra.ai/reference/voice/voice.close) - Disconnects from the real-time service
+- [voice.on()](https://mastra.ai/reference/voice/voice.on) - Registers an event listener for voice events

package/.docs/reference/voice/voice.events.md

@@ -0,0 +1,37 @@
+# Voice Events
+
+Voice providers emit various events during real-time voice interactions. These events can be listened to using the [voice.on()](https://mastra.ai/reference/voice/voice.on) method and are particularly important for building interactive voice applications.
+
+## Common Events
+
+These events are commonly implemented across real-time voice providers:
+
+**error:** (`Error`): Emitted when an error occurs during voice processing or when audio data format is unsupported
+
+**session.created:** (`object`): Emitted when a new session is created with the OpenAI service
+
+**session.updated:** (`object`): Emitted when the session configuration is updated
+
+**response.created:** (`object`): Emitted when a new response is created by the AI assistant
+
+**response.done:** (`object`): Emitted when the AI assistant has completed its response
+
+**speaker:** (`StreamWithId`): Emitted with a new audio stream that can be piped to an audio output
+
+**writing:** (`object`): Emitted when text is being transcribed (user) or generated (assistant)
+
+**speaking:** (`object`): Emitted when audio data is available from the voice provider
+
+**speaking.done:** (`object`): Emitted when the voice provider has finished speaking
+
+**tool-call-start:** (`object`): Emitted when the AI assistant starts executing a tool
+
+**tool-call-result:** (`object`): Emitted when a tool execution is complete with its result
+
+## Notes
+
+- Not all events are supported by all voice providers
+- The exact payload structure may vary between providers
+- For non-real-time providers, most of these events will not be emitted
+- Events are useful for building interactive UIs that respond to the conversation state
+- Consider using the [voice.off()](https://mastra.ai/reference/voice/voice.off) method to remove event listeners when they are no longer needed

package/.docs/reference/voice/voice.getSpeakers.md

@@ -0,0 +1,129 @@
+# voice.getSpeakers()
+
+The `getSpeakers()` method retrieves a list of available voice options (speakers) from the voice provider. This allows applications to present users with voice choices or programmatically select the most appropriate voice for different contexts.
+
+## Usage Example
+
+```typescript
+import { OpenAIVoice } from '@mastra/voice-openai'
+import { ElevenLabsVoice } from '@mastra/voice-elevenlabs'
+
+// Initialize voice providers
+const openaiVoice = new OpenAIVoice()
+const elevenLabsVoice = new ElevenLabsVoice({
+  apiKey: process.env.ELEVENLABS_API_KEY,
+})
+
+// Get available speakers from OpenAI
+const openaiSpeakers = await openaiVoice.getSpeakers()
+console.log('OpenAI voices:', openaiSpeakers)
+// Example output: [{ voiceId: "alloy" }, { voiceId: "echo" }, { voiceId: "fable" }, ...]
+
+// Get available speakers from ElevenLabs
+const elevenLabsSpeakers = await elevenLabsVoice.getSpeakers()
+console.log('ElevenLabs voices:', elevenLabsSpeakers)
+// Example output: [{ voiceId: "21m00Tcm4TlvDq8ikWAM", name: "Rachel" }, ...]
+
+// Use a specific voice for speech
+const text = 'Hello, this is a test of different voices.'
+await openaiVoice.speak(text, { speaker: openaiSpeakers[2].voiceId })
+await elevenLabsVoice.speak(text, { speaker: elevenLabsSpeakers[0].voiceId })
+```
+
+## Parameters
+
+This method does not accept any parameters.
+
+## Return Value
+
+**Promise\<Array<{ voiceId: string } & TSpeakerMetadata>>:** (`Promise`): A promise that resolves to an array of voice options, where each option contains at least a voiceId property and may include additional provider-specific metadata.
+
+## Provider-Specific Metadata
+
+Different voice providers return different metadata for their voices:
+
+**OpenAI**:
+
+**voiceId:** (`string`): Unique identifier for the voice (e.g., 'alloy', 'echo', 'fable', 'onyx', 'nova', 'shimmer')
+
+**OpenAI Realtime**:
+
+**voiceId:** (`string`): Unique identifier for the voice (e.g., 'alloy', 'echo', 'fable', 'onyx', 'nova', 'shimmer')
+
+**Deepgram**:
+
+**voiceId:** (`string`): Unique identifier for the voice
+
+**language:** (`string`): Language code embedded in the voice ID (e.g., 'en')
+
+**ElevenLabs**:
+
+**voiceId:** (`string`): Unique identifier for the voice
+
+**name:** (`string`): Human-readable name of the voice
+
+**category:** (`string`): Category of the voice (e.g., 'premade', 'cloned')
+
+**Google**:
+
+**voiceId:** (`string`): Unique identifier for the voice
+
+**languageCodes:** (`string[]`): Array of language codes supported by the voice (e.g., \['en-US'])
+
+**Azure**:
+
+**voiceId:** (`string`): Unique identifier for the voice
+
+**language:** (`string`): Language code extracted from the voice ID (e.g., 'en')
+
+**region:** (`string`): Region code extracted from the voice ID (e.g., 'US')
+
+**Murf**:
+
+**voiceId:** (`string`): Unique identifier for the voice
+
+**name:** (`string`): Name of the voice (same as voiceId)
+
+**language:** (`string`): Language code extracted from the voice ID (e.g., 'en')
+
+**gender:** (`string`): Gender of the voice (always 'neutral' in current implementation)
+
+**PlayAI**:
+
+**voiceId:** (`string`): Unique identifier for the voice (S3 URL to manifest.json)
+
+**name:** (`string`): Human-readable name of the voice (e.g., 'Angelo', 'Arsenio')
+
+**accent:** (`string`): Accent of the voice (e.g., 'US', 'Irish', 'US African American')
+
+**gender:** (`string`): Gender of the voice ('M' or 'F')
+
+**age:** (`string`): Age category of the voice (e.g., 'Young', 'Middle')
+
+**style:** (`string`): Speaking style of the voice (e.g., 'Conversational')
+
+**Speechify**:
+
+**voiceId:** (`string`): Unique identifier for the voice
+
+**name:** (`string`): Human-readable name of the voice
+
+**language:** (`string`): Language code of the voice (e.g., 'en-US')
+
+**Sarvam**:
+
+**voiceId:** (`string`): Unique identifier for the voice
+
+**name:** (`string`): Human-readable name of the voice
+
+**language:** (`string`): Language of the voice (e.g., 'english', 'hindi')
+
+**gender:** (`string`): Gender of the voice ('male' or 'female')
+
+## Notes
+
+- The available voices vary significantly between providers
+- Some providers may require authentication to retrieve the full list of voices
+- The default implementation returns an empty array if the provider doesn't support this method
+- For performance reasons, consider caching the results if you need to display the list frequently
+- The `voiceId` property is guaranteed to be present for all providers, but additional metadata varies

package/.docs/reference/voice/voice.listen.md

@@ -0,0 +1,164 @@
+# voice.listen()
+
+The `listen()` method is a core function available in all Mastra voice providers that converts speech to text. It takes an audio stream as input and returns the transcribed text.
+
+## Parameters
+
+**audioStream:** (`NodeJS.ReadableStream`): Audio stream to transcribe. This can be a file stream or a microphone stream.
+
+**options?:** (`object`): Provider-specific options for speech recognition
+
+## Return Value
+
+Returns one of the following:
+
+- `Promise<string>`: A promise that resolves to the transcribed text
+- `Promise<NodeJS.ReadableStream>`: A promise that resolves to a stream of transcribed text (for streaming transcription)
+- `Promise<void>`: For real-time providers that emit 'writing' events instead of returning text directly
+
+## Provider-Specific Options
+
+Each voice provider may support additional options specific to their implementation. Here are some examples:
+
+### OpenAI
+
+**options.filetype?:** (`string`): Audio file format (e.g., 'mp3', 'wav', 'm4a') (Default: `'mp3'`)
+
+**options.prompt?:** (`string`): Text to guide the model's transcription
+
+**options.language?:** (`string`): Language code (e.g., 'en', 'fr', 'de')
+
+### Google
+
+**options.stream?:** (`boolean`): Whether to use streaming recognition (Default: `false`)
+
+**options.config?:** (`object`): Recognition configuration from Google Cloud Speech-to-Text API (Default: `{ encoding: 'LINEAR16', languageCode: 'en-US' }`)
+
+### Deepgram
+
+**options.model?:** (`string`): Deepgram model to use for transcription (Default: `'nova-2'`)
+
+**options.language?:** (`string`): Language code for transcription (Default: `'en'`)
+
+## Usage Example
+
+```typescript
+import { OpenAIVoice } from '@mastra/voice-openai'
+import { getMicrophoneStream } from '@mastra/node-audio'
+import { createReadStream } from 'fs'
+import path from 'path'
+
+// Initialize a voice provider
+const voice = new OpenAIVoice({
+  listeningModel: {
+    name: 'whisper-1',
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+})
+
+// Basic usage with a file stream
+const audioFilePath = path.join(process.cwd(), 'audio.mp3')
+const audioStream = createReadStream(audioFilePath)
+const transcript = await voice.listen(audioStream, {
+  filetype: 'mp3',
+})
+console.log('Transcribed text:', transcript)
+
+// Using a microphone stream
+const microphoneStream = getMicrophoneStream() // Assume this function gets audio input
+const transcription = await voice.listen(microphoneStream)
+
+// With provider-specific options
+const transcriptWithOptions = await voice.listen(audioStream, {
+  language: 'en',
+  prompt: 'This is a conversation about artificial intelligence.',
+})
+```
+
+## Using with CompositeVoice
+
+When using `CompositeVoice`, the `listen()` method delegates to the configured listening provider:
+
+```typescript
+import { CompositeVoice } from '@mastra/core/voice'
+import { OpenAIVoice } from '@mastra/voice-openai'
+import { PlayAIVoice } from '@mastra/voice-playai'
+
+const voice = new CompositeVoice({
+  input: new OpenAIVoice(),
+  output: new PlayAIVoice(),
+})
+
+// This will use the OpenAIVoice provider
+const transcript = await voice.listen(audioStream)
+```
+
+### Using AI SDK Model Providers
+
+You can also use AI SDK transcription models directly with `CompositeVoice`:
+
+```typescript
+import { CompositeVoice } from '@mastra/core/voice'
+import { openai } from '@ai-sdk/openai'
+import { groq } from '@ai-sdk/groq'
+
+// Use AI SDK transcription models
+const voice = new CompositeVoice({
+  input: openai.transcription('whisper-1'), // AI SDK model
+  output: new PlayAIVoice(), // Mastra provider
+})
+
+// Works the same way
+const transcript = await voice.listen(audioStream)
+
+// Provider-specific options can be passed through
+const transcriptWithOptions = await voice.listen(audioStream, {
+  providerOptions: {
+    openai: {
+      language: 'en',
+      prompt: 'This is about AI',
+    },
+  },
+})
+```
+
+See the [CompositeVoice reference](https://mastra.ai/reference/voice/composite-voice) for more details on AI SDK integration.
+
+## Realtime Voice Providers
+
+When using realtime voice providers like `OpenAIRealtimeVoice`, the `listen()` method behaves differently:
+
+- Instead of returning transcribed text, it emits 'writing' events with the transcribed text
+- You need to register an event listener to receive the transcription
+
+```typescript
+import { OpenAIRealtimeVoice } from '@mastra/voice-openai-realtime'
+import { getMicrophoneStream } from '@mastra/node-audio'
+
+const voice = new OpenAIRealtimeVoice()
+await voice.connect()
+
+// Register event listener for transcription
+voice.on('writing', ({ text, role }) => {
+  console.log(`${role}: ${text}`)
+})
+
+// This will emit 'writing' events instead of returning text
+const microphoneStream = getMicrophoneStream()
+await voice.listen(microphoneStream)
+```
+
+## Notes
+
+- Not all voice providers support speech-to-text functionality (e.g., PlayAI, Speechify)
+- The behavior of `listen()` may vary slightly between providers, but all implementations follow the same basic interface
+- When using a realtime voice provider, the method might not return text directly but instead emit a 'writing' event
+- The audio format supported depends on the provider. Common formats include MP3, WAV, and M4A
+- Some providers support streaming transcription, where text is returned as it's transcribed
+- For best performance, consider closing or ending the audio stream when you're done with it
+
+## Related Methods
+
+- [voice.speak()](https://mastra.ai/reference/voice/voice.speak) - Converts text to speech
+- [voice.send()](https://mastra.ai/reference/voice/voice.send) - Sends audio data to the voice provider in real-time
+- [voice.on()](https://mastra.ai/reference/voice/voice.on) - Registers an event listener for voice events

package/.docs/reference/voice/voice.off.md

@@ -0,0 +1,54 @@
+# voice.off()
+
+The `off()` method removes event listeners previously registered with the `on()` method. This is particularly useful for cleaning up resources and preventing memory leaks in long-running applications with real-time voice capabilities.
+
+## Usage Example
+
+```typescript
+import { OpenAIRealtimeVoice } from '@mastra/voice-openai-realtime'
+import chalk from 'chalk'
+
+// Initialize a real-time voice provider
+const voice = new OpenAIRealtimeVoice({
+  realtimeConfig: {
+    model: 'gpt-5.1-realtime',
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+})
+
+// Connect to the real-time service
+await voice.connect()
+
+// Define the callback function
+const writingCallback = ({ text, role }) => {
+  if (role === 'user') {
+    process.stdout.write(chalk.green(text))
+  } else {
+    process.stdout.write(chalk.blue(text))
+  }
+}
+
+// Register event listener
+voice.on('writing', writingCallback)
+
+// Later, when you want to remove the listener
+voice.off('writing', writingCallback)
+```
+
+## Parameters
+
+**event:** (`string`): Name of the event to stop listening for (e.g., 'speaking', 'writing', 'error')
+
+**callback:** (`function`): The same callback function that was passed to on()
+
+## Return Value
+
+This method does not return a value.
+
+## Notes
+
+- The callback passed to `off()` must be the same function reference that was passed to `on()`
+- If the callback is not found, the method will have no effect
+- This method is primarily used with real-time voice providers that support event-based communication
+- If called on a voice provider that doesn't support events, it will log a warning and do nothing
+- Removing event listeners is important for preventing memory leaks in long-running applications