@mastra/mcp-docs-server 1.1.5 → 1.1.6-alpha.0

package/.docs/reference/voice/composite-voice.md

@@ -0,0 +1,121 @@
+# CompositeVoice
+
+The CompositeVoice class allows you to combine different voice providers for text-to-speech and speech-to-text operations. This is particularly useful when you want to use the best provider for each operation - for example, using OpenAI for speech-to-text and PlayAI for text-to-speech.
+
+CompositeVoice supports both Mastra voice providers and AI SDK model providers
+
+## Constructor Parameters
+
+**config:** (`object`): Configuration object for the composite voice service
+
+**config.input?:** (`MastraVoice | TranscriptionModel`): Voice provider or AI SDK transcription model to use for speech-to-text operations. AI SDK models are automatically wrapped.
+
+**config.output?:** (`MastraVoice | SpeechModel`): Voice provider or AI SDK speech model to use for text-to-speech operations. AI SDK models are automatically wrapped.
+
+**config.realtime?:** (`MastraVoice`): Voice provider to use for real-time speech-to-speech operations
+
+## Methods
+
+### speak()
+
+Converts text to speech using the configured speaking provider.
+
+**input:** (`string | NodeJS.ReadableStream`): Text to convert to speech
+
+**options?:** (`object`): Provider-specific options passed to the speaking provider
+
+Notes:
+
+- If no speaking provider is configured, this method will throw an error
+- Options are passed through to the configured speaking provider
+- Returns a stream of audio data
+
+### listen()
+
+Converts speech to text using the configured listening provider.
+
+**audioStream:** (`NodeJS.ReadableStream`): Audio stream to convert to text
+
+**options?:** (`object`): Provider-specific options passed to the listening provider
+
+Notes:
+
+- If no listening provider is configured, this method will throw an error
+- Options are passed through to the configured listening provider
+- Returns either a string or a stream of transcribed text, depending on the provider
+
+### getSpeakers()
+
+Returns a list of available voices from the speaking provider, where each node contains:
+
+**voiceId:** (`string`): Unique identifier for the voice
+
+**key?:** (`value`): Additional voice properties that vary by provider (e.g., name, language)
+
+Notes:
+
+- Returns voices from the speaking provider only
+- If no speaking provider is configured, returns an empty array
+- Each voice object will have at least a voiceId property
+- Additional voice properties depend on the speaking provider
+
+## Usage Examples
+
+### Using Mastra Voice Providers
+
+```typescript
+import { CompositeVoice } from '@mastra/core/voice'
+import { OpenAIVoice } from '@mastra/voice-openai'
+import { PlayAIVoice } from '@mastra/voice-playai'
+
+// Create voice providers
+const openai = new OpenAIVoice()
+const playai = new PlayAIVoice()
+
+// Use OpenAI for listening (speech-to-text) and PlayAI for speaking (text-to-speech)
+const voice = new CompositeVoice({
+  input: openai,
+  output: playai,
+})
+
+// Convert speech to text using OpenAI
+const text = await voice.listen(audioStream)
+
+// Convert text to speech using PlayAI
+const audio = await voice.speak('Hello, world!')
+```
+
+### Using AI SDK Model Providers
+
+You can pass AI SDK transcription and speech models directly to CompositeVoice:
+
+```typescript
+import { CompositeVoice } from '@mastra/core/voice'
+import { openai } from '@ai-sdk/openai'
+import { elevenlabs } from '@ai-sdk/elevenlabs'
+
+// Use AI SDK models directly - they will be auto-wrapped
+const voice = new CompositeVoice({
+  input: openai.transcription('whisper-1'), // AI SDK transcription
+  output: elevenlabs.speech('eleven_turbo_v2'), // AI SDK speech
+})
+
+// Works the same way as with Mastra providers
+const text = await voice.listen(audioStream)
+const audio = await voice.speak('Hello from AI SDK!')
+```
+
+### Mix and Match
+
+You can combine Mastra providers with AI SDK models:
+
+```typescript
+import { CompositeVoice } from '@mastra/core/voice'
+import { PlayAIVoice } from '@mastra/voice-playai'
+import { groq } from '@ai-sdk/groq'
+
+const voice = new CompositeVoice({
+  input: groq.transcription('whisper-large-v3'), // AI SDK for STT
+  output: new PlayAIVoice(), // Mastra for TTS
+})
+```

package/.docs/reference/voice/deepgram.md

@@ -0,0 +1,79 @@
+# Deepgram
+
+The Deepgram voice implementation in Mastra provides text-to-speech (TTS) and speech-to-text (STT) capabilities using Deepgram's API. It supports multiple voice models and languages, with configurable options for both speech synthesis and transcription.
+
+## Usage Example
+
+```typescript
+import { DeepgramVoice } from '@mastra/voice-deepgram'
+
+// Initialize with default configuration (uses DEEPGRAM_API_KEY environment variable)
+const voice = new DeepgramVoice()
+
+// Initialize with custom configuration
+const voice = new DeepgramVoice({
+  speechModel: {
+    name: 'aura',
+    apiKey: 'your-api-key',
+  },
+  listeningModel: {
+    name: 'nova-2',
+    apiKey: 'your-api-key',
+  },
+  speaker: 'asteria-en',
+})
+
+// Text-to-Speech
+const audioStream = await voice.speak('Hello, world!')
+
+// Speech-to-Text
+const transcript = await voice.listen(audioStream)
+```
+
+## Constructor Parameters
+
+**speechModel?:** (`DeepgramVoiceConfig`): Configuration for text-to-speech functionality. (Default: `{ name: 'aura' }`)
+
+**listeningModel?:** (`DeepgramVoiceConfig`): Configuration for speech-to-text functionality. (Default: `{ name: 'nova' }`)
+
+**speaker?:** (`DeepgramVoiceId`): Default voice to use for text-to-speech (Default: `'asteria-en'`)
+
+### DeepgramVoiceConfig
+
+**name?:** (`DeepgramModel`): The Deepgram model to use
+
+**apiKey?:** (`string`): Deepgram API key. Falls back to DEEPGRAM\_API\_KEY environment variable
+
+**properties?:** (`Record<string, any>`): Additional properties to pass to the Deepgram API
+
+**language?:** (`string`): Language code for the model
+
+## Methods
+
+### speak()
+
+Converts text to speech using the configured speech model and voice.
+
+**input:** (`string | NodeJS.ReadableStream`): Text to convert to speech. If a stream is provided, it will be converted to text first.
+
+**options?:** (`object`): Additional options for speech synthesis
+
+**options.speaker?:** (`string`): Override the default speaker for this request
+
+Returns: `Promise<NodeJS.ReadableStream>`
+
+### listen()
+
+Converts speech to text using the configured listening model.
+
+**audioStream:** (`NodeJS.ReadableStream`): Audio stream to transcribe
+
+**options?:** (`object`): Additional options to pass to the Deepgram API
+
+Returns: `Promise<string>`
+
+### getSpeakers()
+
+Returns a list of available voice options.
+
+**voiceId:** (`string`): Unique identifier for the voice

package/.docs/reference/voice/elevenlabs.md

@@ -0,0 +1,98 @@
+# ElevenLabs
+
+The ElevenLabs voice implementation in Mastra provides high-quality text-to-speech (TTS) and speech-to-text (STT) capabilities using the ElevenLabs API.
+
+## Usage Example
+
+```typescript
+import { ElevenLabsVoice } from '@mastra/voice-elevenlabs'
+
+// Initialize with default configuration (uses ELEVENLABS_API_KEY environment variable)
+const voice = new ElevenLabsVoice()
+
+// Initialize with custom configuration
+const voice = new ElevenLabsVoice({
+  speechModel: {
+    name: 'eleven_multilingual_v2',
+    apiKey: 'your-api-key',
+  },
+  speaker: 'custom-speaker-id',
+})
+
+// Text-to-Speech
+const audioStream = await voice.speak('Hello, world!')
+
+// Get available speakers
+const speakers = await voice.getSpeakers()
+```
+
+## Constructor Parameters
+
+**speechModel?:** (`ElevenLabsVoiceConfig`): Configuration for text-to-speech functionality. (Default: `{ name: 'eleven_multilingual_v2' }`)
+
+**speaker?:** (`string`): ID of the speaker to use for text-to-speech (Default: `'9BWtsMINqrJLrRacOk9x' (Aria voice)`)
+
+### ElevenLabsVoiceConfig
+
+**name?:** (`ElevenLabsModel`): The ElevenLabs model to use (Default: `'eleven_multilingual_v2'`)
+
+**apiKey?:** (`string`): ElevenLabs API key. Falls back to ELEVENLABS\_API\_KEY environment variable
+
+## Methods
+
+### speak()
+
+Converts text to speech using the configured speech model and voice.
+
+**input:** (`string | NodeJS.ReadableStream`): Text to convert to speech. If a stream is provided, it will be converted to text first.
+
+**options?:** (`object`): Additional options for speech synthesis
+
+**options.speaker?:** (`string`): Override the default speaker ID for this request
+
+Returns: `Promise<NodeJS.ReadableStream>`
+
+### getSpeakers()
+
+Returns an array of available voice options, where each node contains:
+
+**voiceId:** (`string`): Unique identifier for the voice
+
+**name:** (`string`): Display name of the voice
+
+**language:** (`string`): Language code for the voice
+
+**gender:** (`string`): Gender of the voice
+
+### listen()
+
+Converts audio input to text using ElevenLabs Speech-to-Text API.
+
+**input:** (`NodeJS.ReadableStream`): A readable stream containing the audio data to transcribe
+
+**options?:** (`object`): Configuration options for the transcription
+
+The options object supports the following properties:
+
+**language\_code?:** (`string`): ISO language code (e.g., 'en', 'fr', 'es')
+
+**tag\_audio\_events?:** (`boolean`): Whether to tag audio events like \[MUSIC], \[LAUGHTER], etc.
+
+**num\_speakers?:** (`number`): Number of speakers to detect in the audio
+
+**filetype?:** (`string`): Audio file format (e.g., 'mp3', 'wav', 'ogg')
+
+**timeoutInSeconds?:** (`number`): Request timeout in seconds
+
+**maxRetries?:** (`number`): Maximum number of retry attempts
+
+**abortSignal?:** (`AbortSignal`): Signal to abort the request
+
+Returns: `Promise<string>` - A Promise that resolves to the transcribed text
+
+## Important Notes
+
+1. An ElevenLabs API key is required. Set it via the `ELEVENLABS_API_KEY` environment variable or pass it in the constructor.
+2. The default speaker is set to Aria (ID: '9BWtsMINqrJLrRacOk9x').
+3. Speech-to-text functionality is not supported by ElevenLabs.
+4. Available speakers can be retrieved using the `getSpeakers()` method, which returns detailed information about each voice including language and gender.

package/.docs/reference/voice/google-gemini-live.md

@@ -0,0 +1,378 @@
+# Google Gemini Live Voice
+
+The GeminiLiveVoice class provides real-time voice interaction capabilities using Google's Gemini Live API. It supports bidirectional audio streaming, tool calling, session management, and both standard Google API and Vertex AI authentication methods.
+
+## Usage Example
+
+```typescript
+import { GeminiLiveVoice } from '@mastra/voice-google-gemini-live'
+import { playAudio, getMicrophoneStream } from '@mastra/node-audio'
+
+// Initialize with Gemini API (using API key)
+const voice = new GeminiLiveVoice({
+  apiKey: process.env.GOOGLE_API_KEY, // Required for Gemini API
+  model: 'gemini-2.0-flash-exp',
+  speaker: 'Puck', // Default voice
+  debug: true,
+})
+
+// Or initialize with Vertex AI (using OAuth)
+const voiceWithVertexAI = new GeminiLiveVoice({
+  vertexAI: true,
+  project: 'your-gcp-project',
+  location: 'us-central1',
+  serviceAccountKeyFile: '/path/to/service-account.json',
+  model: 'gemini-2.0-flash-exp',
+  speaker: 'Puck',
+})
+
+// Or use the VoiceConfig pattern (recommended for consistency with other providers)
+const voiceWithConfig = new GeminiLiveVoice({
+  speechModel: {
+    name: 'gemini-2.0-flash-exp',
+    apiKey: process.env.GOOGLE_API_KEY,
+  },
+  speaker: 'Puck',
+  realtimeConfig: {
+    model: 'gemini-2.0-flash-exp',
+    apiKey: process.env.GOOGLE_API_KEY,
+    options: {
+      debug: true,
+      sessionConfig: {
+        interrupts: { enabled: true },
+      },
+    },
+  },
+})
+
+// Establish connection (required before using other methods)
+await voice.connect()
+
+// Set up event listeners
+voice.on('speaker', audioStream => {
+  // Handle audio stream (NodeJS.ReadableStream)
+  playAudio(audioStream)
+})
+
+voice.on('writing', ({ text, role }) => {
+  // Handle transcribed text
+  console.log(`${role}: ${text}`)
+})
+
+voice.on('turnComplete', ({ timestamp }) => {
+  // Handle turn completion
+  console.log('Turn completed at:', timestamp)
+})
+
+// Convert text to speech
+await voice.speak('Hello, how can I help you today?', {
+  speaker: 'Charon', // Override default voice
+  responseModalities: ['AUDIO', 'TEXT'],
+})
+
+// Process audio input
+const microphoneStream = getMicrophoneStream()
+await voice.send(microphoneStream)
+
+// Update session configuration
+await voice.updateSessionConfig({
+  speaker: 'Kore',
+  instructions: 'Be more concise in your responses',
+})
+
+// When done, disconnect
+await voice.disconnect()
+// Or use the synchronous wrapper
+voice.close()
+```
+
+## Configuration
+
+### Constructor Options
+
+**apiKey?:** (`string`): Google API key for Gemini API authentication. Required unless using Vertex AI.
+
+**model?:** (`GeminiVoiceModel`): The model ID to use for real-time voice interactions. (Default: `'gemini-2.0-flash-exp'`)
+
+**speaker?:** (`GeminiVoiceName`): Default voice ID for speech synthesis. (Default: `'Puck'`)
+
+**vertexAI?:** (`boolean`): Use Vertex AI instead of Gemini API for authentication. (Default: `false`)
+
+**project?:** (`string`): Google Cloud project ID (required for Vertex AI).
+
+**location?:** (`string`): Google Cloud region for Vertex AI. (Default: `'us-central1'`)
+
+**serviceAccountKeyFile?:** (`string`): Path to service account JSON key file for Vertex AI authentication.
+
+**serviceAccountEmail?:** (`string`): Service account email for impersonation (alternative to key file).
+
+**instructions?:** (`string`): System instructions for the model.
+
+**sessionConfig?:** (`GeminiSessionConfig`): Session configuration including interrupt and context settings.
+
+**debug?:** (`boolean`): Enable debug logging for troubleshooting. (Default: `false`)
+
+### Session Configuration
+
+**interrupts?:** (`object`): Interrupt handling configuration.
+
+**interrupts.enabled?:** (`boolean`): Enable interrupt handling. (Default: `true`)
+
+**interrupts.allowUserInterruption?:** (`boolean`): Allow user to interrupt model responses. (Default: `true`)
+
+**contextCompression?:** (`boolean`): Enable automatic context compression. (Default: `false`)
+
+## Methods
+
+### connect()
+
+Establishes a connection to the Gemini Live API. Must be called before using speak, listen, or send methods.
+
+**requestContext?:** (`object`): Optional request context for the connection.
+
+**returns:** (`Promise<void>`): Promise that resolves when the connection is established.
+
+### speak()
+
+Converts text to speech and sends it to the model. Can accept either a string or a readable stream as input.
+
+**input:** (`string | NodeJS.ReadableStream`): Text or text stream to convert to speech.
+
+**options?:** (`GeminiLiveVoiceOptions`): Optional speech configuration.
+
+**options.speaker?:** (`GeminiVoiceName`): Voice ID to use for this specific speech request. (Default: `Constructor's speaker value`)
+
+**options.languageCode?:** (`string`): Language code for the response.
+
+**options.responseModalities?:** (`('AUDIO' | 'TEXT')[]`): Response modalities to receive from the model. (Default: `['AUDIO', 'TEXT']`)
+
+Returns: `Promise<void>` (responses are emitted via `speaker` and `writing` events)
+
+### listen()
+
+Processes audio input for speech recognition. Takes a readable stream of audio data and returns the transcribed text.
+
+**audioStream:** (`NodeJS.ReadableStream`): Audio stream to transcribe.
+
+**options?:** (`GeminiLiveVoiceOptions`): Optional listening configuration.
+
+Returns: `Promise<string>` - The transcribed text
+
+### send()
+
+Streams audio data in real-time to the Gemini service for continuous audio streaming scenarios like live microphone input.
+
+**audioData:** (`NodeJS.ReadableStream | Int16Array`): Audio stream or buffer to send to the service.
+
+Returns: `Promise<void>`
+
+### updateSessionConfig()
+
+Updates the session configuration dynamically. This can be used to modify voice settings, speaker selection, and other runtime configurations.
+
+**config:** (`Partial<GeminiLiveVoiceConfig>`): Configuration updates to apply.
+
+Returns: `Promise<void>`
+
+### addTools()
+
+Adds a set of tools to the voice instance. Tools allow the model to perform additional actions during conversations. When GeminiLiveVoice is added to an Agent, any tools configured for the Agent will automatically be available to the voice interface.
+
+**tools:** (`ToolsInput`): Tools configuration to equip.
+
+Returns: `void`
+
+### addInstructions()
+
+Adds or updates system instructions for the model.
+
+**instructions?:** (`string`): System instructions to set.
+
+Returns: `void`
+
+### answer()
+
+Triggers a response from the model. This method is primarily used internally when integrated with an Agent.
+
+**options?:** (`Record<string, unknown>`): Optional parameters for the answer request.
+
+Returns: `Promise<void>`
+
+### getSpeakers()
+
+Returns a list of available voice speakers for the Gemini Live API.
+
+Returns: `Promise<Array<{ voiceId: string; description?: string }>>`
+
+### disconnect()
+
+Disconnects from the Gemini Live session and cleans up resources. This is the async method that properly handles cleanup.
+
+Returns: `Promise<void>`
+
+### close()
+
+Synchronous wrapper for disconnect(). Calls disconnect() internally without awaiting.
+
+Returns: `void`
+
+### on()
+
+Registers an event listener for voice events.
+
+**event:** (`string`): Name of the event to listen for.
+
+**callback:** (`Function`): Function to call when the event occurs.
+
+Returns: `void`
+
+### off()
+
+Removes a previously registered event listener.
+
+**event:** (`string`): Name of the event to stop listening to.
+
+**callback:** (`Function`): The specific callback function to remove.
+
+Returns: `void`
+
+## Events
+
+The GeminiLiveVoice class emits the following events:
+
+**speaker:** (`event`): Emitted when audio data is received from the model. Callback receives a NodeJS.ReadableStream.
+
+**speaking:** (`event`): Emitted with audio metadata. Callback receives { audioData?: Int16Array, sampleRate?: number }.
+
+**writing:** (`event`): Emitted when transcribed text is available. Callback receives { text: string, role: 'assistant' | 'user' }.
+
+**session:** (`event`): Emitted on session state changes. Callback receives { state: 'connecting' | 'connected' | 'disconnected' | 'disconnecting' | 'updated', config?: object }.
+
+**turnComplete:** (`event`): Emitted when a conversation turn is completed. Callback receives { timestamp: number }.
+
+**toolCall:** (`event`): Emitted when the model requests a tool call. Callback receives { name: string, args: object, id: string }.
+
+**usage:** (`event`): Emitted with token usage information. Callback receives { inputTokens: number, outputTokens: number, totalTokens: number, modality: string }.
+
+**error:** (`event`): Emitted when an error occurs. Callback receives { message: string, code?: string, details?: unknown }.
+
+**interrupt:** (`event`): Interrupt events. Callback receives { type: 'user' | 'model', timestamp: number }.
+
+## Available Models
+
+The following Gemini Live models are available:
+
+- `gemini-2.0-flash-exp` (default)
+- `gemini-2.0-flash-exp-image-generation`
+- `gemini-2.0-flash-live-001`
+- `gemini-live-2.5-flash-preview-native-audio`
+- `gemini-2.5-flash-exp-native-audio-thinking-dialog`
+- `gemini-live-2.5-flash-preview`
+- `gemini-2.6.flash-preview-tts`
+
+## Available Voices
+
+The following voice options are available:
+
+- `Puck` (default): Conversational, friendly
+- `Charon`: Deep, authoritative
+- `Kore`: Neutral, professional
+- `Fenrir`: Warm, approachable
+
+## Authentication Methods
+
+### Gemini API (Development)
+
+The simplest method using an API key from [Google AI Studio](https://makersuite.google.com/app/apikey):
+
+```typescript
+const voice = new GeminiLiveVoice({
+  apiKey: 'your-api-key', // Required for Gemini API
+  model: 'gemini-2.0-flash-exp',
+})
+```
+
+### Vertex AI (Production)
+
+For production use with OAuth authentication and Google Cloud Platform:
+
+```typescript
+// Using service account key file
+const voice = new GeminiLiveVoice({
+  vertexAI: true,
+  project: 'your-gcp-project',
+  location: 'us-central1',
+  serviceAccountKeyFile: '/path/to/service-account.json',
+})
+
+// Using Application Default Credentials
+const voice = new GeminiLiveVoice({
+  vertexAI: true,
+  project: 'your-gcp-project',
+  location: 'us-central1',
+})
+
+// Using service account impersonation
+const voice = new GeminiLiveVoice({
+  vertexAI: true,
+  project: 'your-gcp-project',
+  location: 'us-central1',
+  serviceAccountEmail: 'service-account@project.iam.gserviceaccount.com',
+})
+```
+
+## Advanced Features
+
+### Session Management
+
+The Gemini Live API supports session resumption for handling network interruptions:
+
+```typescript
+voice.on('sessionHandle', ({ handle, expiresAt }) => {
+  // Store session handle for resumption
+  saveSessionHandle(handle, expiresAt)
+})
+
+// Resume a previous session
+const voice = new GeminiLiveVoice({
+  sessionConfig: {
+    enableResumption: true,
+    maxDuration: '2h',
+  },
+})
+```
+
+### Tool Calling
+
+Enable the model to call functions during conversations:
+
+```typescript
+import { z } from 'zod'
+
+voice.addTools({
+  weather: {
+    description: 'Get weather information',
+    parameters: z.object({
+      location: z.string(),
+    }),
+    execute: async ({ location }) => {
+      const weather = await getWeather(location)
+      return weather
+    },
+  },
+})
+
+voice.on('toolCall', ({ name, args, id }) => {
+  console.log(`Tool called: ${name} with args:`, args)
+})
+```
+
+## Notes
+
+- The Gemini Live API uses WebSockets for real-time communication
+- Audio is processed as 16kHz PCM16 for input and 24kHz PCM16 for output
+- The voice instance must be connected with `connect()` before using other methods
+- Always call `close()` when done to properly clean up resources
+- Vertex AI authentication requires appropriate IAM permissions (`aiplatform.user` role)
+- Session resumption allows recovery from network interruptions
+- The API supports real-time interactions with text and audio