@mastra/mcp-docs-server 1.2.15-alpha.19 → 1.2.15-alpha.21

package/.docs/{reference/voice/google-gemini-live.md → integrations/voice/google.md}

@@ -1,10 +1,285 @@
 > Discover all available pages from the documentation index: https://mastra.ai/llms.txt
 
-# Google Gemini Live voice
+# Google
+
+## Speech
+
+The Google Voice implementation in Mastra provides both text-to-speech (TTS) and speech-to-text (STT) capabilities using Google Cloud services. It supports multiple voices, languages, advanced audio configuration options, and both standard API key authentication and Vertex AI mode for enterprise deployments.
+
+### Usage example
+
+```typescript
+import { GoogleVoice } from '@mastra/voice-google'
+
+// Initialize with default configuration (uses GOOGLE_API_KEY environment variable)
+const voice = new GoogleVoice()
+
+// Text-to-Speech (plain text)
+const audioStream = await voice.speak('Hello, world!', {
+  languageCode: 'en-US',
+  audioConfig: {
+    audioEncoding: 'LINEAR16',
+  },
+})
+
+// Text-to-Speech with SSML
+const ssmlStream = await voice.speak('ignored', {
+  input: {
+    ssml: '<speak>Take <say-as interpret-as="unit">5 mg</say-as> daily.</speak>',
+  },
+})
+
+// Text-to-Speech with Gemini-TTS model
+const geminiStream = await voice.speak('Hello from Gemini TTS!', {
+  voice: { name: 'Kore', modelName: 'gemini-2.5-flash-preview-tts' },
+  input: { prompt: 'Warm, calm tone.' },
+})
+
+// Speech-to-Text
+const transcript = await voice.listen(audioStream, {
+  config: {
+    encoding: 'LINEAR16',
+    languageCode: 'en-US',
+  },
+})
+
+// Get available voices for a specific language
+const voices = await voice.getSpeakers({ languageCode: 'en-US' })
+```
+
+### Constructor parameters
+
+**speechModel** (`GoogleModelConfig`): Configuration for text-to-speech functionality (Default: `{ apiKey: process.env.GOOGLE_API_KEY }`)
+
+**speechModel.apiKey** (`string`): Google Cloud API key. Falls back to GOOGLE\_API\_KEY environment variable. Not used when vertexAI is true.
+
+**speechModel.keyFilename** (`string`): Path to service account JSON key file. Falls back to GOOGLE\_APPLICATION\_CREDENTIALS environment variable.
+
+**speechModel.credentials** (`object`): In-memory service account credentials object with client\_email and private\_key properties.
+
+**listeningModel** (`GoogleModelConfig`): Configuration for speech-to-text functionality (Default: `{ apiKey: process.env.GOOGLE_API_KEY }`)
+
+**listeningModel.apiKey** (`string`): Google Cloud API key. Falls back to GOOGLE\_API\_KEY environment variable. Not used when vertexAI is true.
+
+**listeningModel.keyFilename** (`string`): Path to service account JSON key file. Falls back to GOOGLE\_APPLICATION\_CREDENTIALS environment variable.
+
+**listeningModel.credentials** (`object`): In-memory service account credentials object with client\_email and private\_key properties.
+
+**speaker** (`string`): Default voice ID to use for text-to-speech (Default: `'en-US-Casual-K'`)
+
+**vertexAI** (`boolean`): Enable Vertex AI mode for enterprise deployments. Uses project-based authentication instead of API keys. Requires 'project' to be set. (Default: `false`)
+
+**project** (`string`): Google Cloud project ID (required when vertexAI is true). Falls back to GOOGLE\_CLOUD\_PROJECT environment variable.
+
+**location** (`string`): Google Cloud region for Vertex AI. Falls back to GOOGLE\_CLOUD\_LOCATION environment variable. (Default: `'us-central1'`)
+
+### Methods
+
+#### `speak()`
+
+Converts text to speech using Google Cloud Text-to-Speech service.
+
+**input** (`string | NodeJS.ReadableStream`): Text to convert to speech. If a stream is provided, it will be converted to text first.
+
+**options** (`object`): Speech synthesis options
+
+**options.speaker** (`string`): Voice ID to use for this request.
+
+**options.languageCode** (`string`): Language code for the voice (e.g., 'en-US'). Defaults to the language code derived from the speaker ID, or 'en-US'.
+
+**options.input** (`ISynthesizeSpeechRequest['input']`): Rich input object passed through to the Google Cloud TTS API. Supports ssml, markup, prompt (Gemini-TTS style steering), customPronunciations, and multiSpeakerMarkup. When provided without text, ssml, markup, or multiSpeakerMarkup, the positional input argument is used as the text field automatically.
+
+**options.voice** (`ISynthesizeSpeechRequest['voice']`): Voice configuration merged on top of defaults (name and languageCode). Supports modelName (e.g., 'gemini-2.5-flash-preview-tts') and multiSpeakerVoiceConfig.
+
+**options.audioConfig** (`ISynthesizeSpeechRequest['audioConfig']`): Audio configuration options from Google Cloud Text-to-Speech API.
+
+Returns: `Promise<NodeJS.ReadableStream>`
+
+#### `listen()`
+
+Converts speech to text using Google Cloud Speech-to-Text service. Supports both v1 (default) and v2 APIs. The v2 API adds support for AAC-in-MP4 audio (iOS Safari) via auto-decoding.
+
+##### v1 (default)
+
+**audioStream** (`NodeJS.ReadableStream`): Audio stream to transcribe
+
+**options** (`GoogleListenOptionsV1`): v1 recognition options
+
+**options.config** (`IRecognitionConfig`): v1 recognition configuration from Google Cloud Speech-to-Text API
+
+##### v2
+
+Pass `v2: true` to use the Cloud Speech-to-Text v2 API, which supports additional audio formats like AAC-in-MP4 (iOS Safari).
+
+```typescript
+const transcript = await voice.listen(iosSafariAacStream, {
+  v2: true,
+  config: {
+    autoDecodingConfig: {},
+  },
+})
+```
+
+**audioStream** (`NodeJS.ReadableStream`): Audio stream to transcribe
+
+**options** (`GoogleListenOptionsV2`): v2 recognition options
+
+**options.v2** (`true`): Enables the v2 API path
+
+**options.config** (`v2.IRecognitionConfig`): v2 recognition configuration. Defaults to auto-decoding with languageCodes: \['en-US'] and model: 'long'. Set autoDecodingConfig: {} to auto-detect the audio format, or use explicitDecodingConfig to specify an encoding like MP4\_AAC, M4A\_AAC, or MOV\_AAC.
+
+**options.recognizer** (`string`): v2 recognizer resource path. Defaults to projects/{project}/locations/global/recognizers/\_ where {project} is resolved from the constructor project option, GOOGLE\_CLOUD\_PROJECT, or the client's default project.
+
+Returns: `Promise<string>`
+
+#### `getSpeakers()`
+
+Returns an array of available voice options, where each node contains:
+
+**voiceId** (`string`): Unique identifier for the voice
+
+**languageCodes** (`string[]`): List of language codes supported by this voice
+
+#### `isUsingVertexAI()`
+
+Checks if Vertex AI mode is enabled.
+
+Returns: `boolean` - `true` if using Vertex AI, `false` otherwise
+
+#### `getProject()`
+
+Gets the configured Google Cloud project ID.
+
+Returns: `string | undefined` - The project ID or `undefined` if not set
+
+#### `getLocation()`
+
+Gets the configured Google Cloud location/region.
+
+Returns: `string` - The location (default: `'us-central1'`)
+
+### Authentication
+
+The Google Voice provider supports two authentication methods:
+
+#### Standard Mode (API Key)
+
+Uses a Google Cloud API key for authentication. Suitable for development and basic use cases.
+
+```typescript
+// Using environment variable (GOOGLE_API_KEY)
+const voice = new GoogleVoice()
+
+// Using explicit API key
+const voice = new GoogleVoice({
+  speechModel: { apiKey: 'your-api-key' },
+  listeningModel: { apiKey: 'your-api-key' },
+  speaker: 'en-US-Casual-K',
+})
+```
+
+#### Vertex AI Mode (Service Account)
+
+Uses Google Cloud project-based authentication with service accounts. Recommended for production and enterprise deployments.
+
+**Benefits:**
+
+- Better security (no API keys in code)
+- IAM-based access control
+- Project-level billing and quotas
+- Audit logging
+- Enterprise features
+
+**Configuration Options:**
+
+```typescript
+// Using Application Default Credentials (ADC)
+// Set GOOGLE_APPLICATION_CREDENTIALS and GOOGLE_CLOUD_PROJECT env vars
+const voice = new GoogleVoice({
+  vertexAI: true,
+  project: 'your-gcp-project',
+  location: 'us-central1', // Optional, defaults to 'us-central1'
+})
+
+// Using service account key file
+const voice = new GoogleVoice({
+  vertexAI: true,
+  project: 'your-gcp-project',
+  speechModel: {
+    keyFilename: '/path/to/service-account.json',
+  },
+  listeningModel: {
+    keyFilename: '/path/to/service-account.json',
+  },
+})
+
+// Using in-memory credentials
+const voice = new GoogleVoice({
+  vertexAI: true,
+  project: 'your-gcp-project',
+  speechModel: {
+    credentials: {
+      client_email: 'service-account@project.iam.gserviceaccount.com',
+      private_key: '-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----',
+    },
+  },
+})
+```
+
+##### Required Permissions
+
+##### IAM Roles
+
+For Text-to-Speech:
+
+- `roles/texttospeech.admin` - Text-to-Speech Admin (full access)
+- `roles/texttospeech.editor` - Text-to-Speech Editor (create and manage)
+- `roles/texttospeech.viewer` - Text-to-Speech Viewer (read-only)
+
+For Speech-to-Text:
+
+- `roles/speech.client` - Speech-to-Text Client
+
+##### OAuth Scopes
+
+For synchronous Text-to-Speech synthesis:
+
+- `https://www.googleapis.com/auth/cloud-platform` - Full access to Google Cloud Platform services
+
+For long-audio Text-to-Speech operations:
+
+- `locations.longAudioSynthesize` - Create long-audio synthesis operations
+- `operations.get` - Get operation status
+- `operations.list` - List operations
+
+### Important notes
+
+1. **Authentication**: Either a Google Cloud API key (standard mode) or service account credentials (Vertex AI mode) is required.
+
+2. **Environment Variables**:
+
+   - `GOOGLE_API_KEY` - API key for standard mode
+   - `GOOGLE_CLOUD_PROJECT` - Project ID for Vertex AI mode
+   - `GOOGLE_CLOUD_LOCATION` - Location for Vertex AI mode (defaults to 'us-central1')
+   - `GOOGLE_APPLICATION_CREDENTIALS` - Path to service account key file
+
+3. The default voice is set to `'en-US-Casual-K'`.
+
+4. Both text-to-speech and speech-to-text services use LINEAR16 as the default audio encoding.
+
+5. The `speak()` method supports advanced audio configuration through the Google Cloud Text-to-Speech API.
+
+6. The `listen()` method supports various recognition configurations through the Google Cloud Speech-to-Text API.
+
+7. Available voices can be filtered by language code using the `getSpeakers()` method.
+
+8. Vertex AI mode provides enterprise features including IAM control, audit logs, and project-level billing.
+
+## Gemini Live
 
 The GeminiLiveVoice class provides real-time voice interaction capabilities using Google's Gemini Live API. It supports bidirectional audio streaming, tool calling, session management, and both standard Google API and Vertex AI authentication methods.
 
-## Usage example
+### Usage example
 
 ```typescript
 import { GeminiLiveVoice } from '@mastra/voice-google-gemini-live'
@@ -88,9 +363,9 @@ await voice.disconnect()
 voice.close()
 ```
 
-## Configuration
+### Configuration
 
-### Constructor options
+#### Constructor options
 
 **apiKey** (`string`): Google API key for Gemini API authentication. Required unless using Vertex AI.
 
@@ -122,9 +397,9 @@ voice.close()
 
 **debug** (`boolean`): Enable debug logging for troubleshooting. (Default: `false`)
 
-## Methods
+### Methods
 
-### `connect()`
+#### `connect()`
 
 Establishes a connection to the Gemini Live API. Must be called before using speak, listen, or send methods.
 
@@ -132,7 +407,7 @@ Establishes a connection to the Gemini Live API. Must be called before using spe
 
 **returns** (`Promise<void>`): Promise that resolves when the connection is established.
 
-### `speak()`
+#### `speak()`
 
 Converts text to speech and sends it to the model. Can accept either a string or a readable stream as input.
 
@@ -148,7 +423,7 @@ Converts text to speech and sends it to the model. Can accept either a string or
 
 Returns: `Promise<void>` (responses are emitted via `speaker` and `writing` events)
 
-### `sendContext()`
+#### `sendContext()`
 
 Sends conversation history into the live session without triggering a model response. Use this to seed prior turns (e.g. from Mastra Memory) on a cold connect so the model has context before the user speaks.
 
@@ -170,7 +445,7 @@ await voice.send(micStream)
 
 Returns: `Promise<void>`
 
-### `listen()`
+#### `listen()`
 
 Processes audio input for speech recognition. Takes a readable stream of audio data and returns the transcribed text.
 
@@ -180,7 +455,7 @@ Processes audio input for speech recognition. Takes a readable stream of audio d
 
 Returns: `Promise<string>` - The transcribed text
 
-### `send()`
+#### `send()`
 
 Streams audio data in real-time to the Gemini service for continuous audio streaming scenarios like live microphone input.
 
@@ -188,7 +463,7 @@ Streams audio data in real-time to the Gemini service for continuous audio strea
 
 Returns: `Promise<void>`
 
-### `updateSessionConfig()`
+#### `updateSessionConfig()`
 
 Updates the session configuration at runtime. This can modify voice settings and speaker selection. It can also modify other runtime configurations.
 
@@ -196,7 +471,7 @@ Updates the session configuration at runtime. This can modify voice settings and
 
 Returns: `Promise<void>`
 
-### `addTools()`
+#### `addTools()`
 
 Adds a set of tools to the voice instance. Tools allow the model to perform additional actions during conversations. When GeminiLiveVoice is added to an Agent, any tools configured for the Agent will automatically be available to the voice interface.
 
@@ -204,7 +479,7 @@ Adds a set of tools to the voice instance. Tools allow the model to perform addi
 
 Returns: `void`
 
-### `addInstructions()`
+#### `addInstructions()`
 
 Adds or updates system instructions for the model.
 
@@ -212,7 +487,7 @@ Adds or updates system instructions for the model.
 
 Returns: `void`
 
-### `answer()`
+#### `answer()`
 
 Triggers a response from the model. This method is primarily used internally when integrated with an Agent.
 
@@ -220,25 +495,25 @@ Triggers a response from the model. This method is primarily used internally whe
 
 Returns: `Promise<void>`
 
-### `getSpeakers()`
+#### `getSpeakers()`
 
 Returns a list of available voice speakers for the Gemini Live API.
 
 Returns: `Promise<Array<{ voiceId: string; description?: string }>>`
 
-### `disconnect()`
+#### `disconnect()`
 
 Disconnects from the Gemini Live session and cleans up resources. This is the async method that properly handles cleanup.
 
 Returns: `Promise<void>`
 
-### `close()`
+#### `close()`
 
 Synchronous wrapper for disconnect(). Calls disconnect() internally without awaiting.
 
 Returns: `void`
 
-### `on()`
+#### `on()`
 
 Registers an event listener for voice events.
 
@@ -248,7 +523,7 @@ Registers an event listener for voice events.
 
 Returns: `void`
 
-### `off()`
+#### `off()`
 
 Removes a previously registered event listener.
 
@@ -258,7 +533,7 @@ Removes a previously registered event listener.
 
 Returns: `void`
 
-## Events
+### Events
 
 The GeminiLiveVoice class emits the following events:
 
@@ -282,7 +557,7 @@ The GeminiLiveVoice class emits the following events:
 
 **interrupt** (`event`): Emitted on barge-in when the user starts speaking over an in-flight model response. The server cancels any further audio for the current turn. Callback receives { type: 'user', timestamp: number }.
 
-## Native-audio behavior
+### Native-audio behavior
 
 Native-audio Gemini Live models (any model whose ID contains `native-audio`, such as `gemini-2.5-flash-native-audio-preview-12-2025`) split text output across two channels:
 
@@ -293,7 +568,7 @@ On non-native-audio models there is no `output_audio_transcription` channel, so
 
 Input transcription, output transcription, and barge-in detection (`realtime_input_config.activity_handling = 'START_OF_ACTIVITY_INTERRUPTS'`) are enabled automatically in the setup payload. You don't need extra configuration.
 
-## Available models
+### Available models
 
 The following Gemini Live models are available:
 
@@ -305,7 +580,7 @@ The following Gemini Live models are available:
 - `gemini-live-2.5-flash-preview`
 - `gemini-2.6.flash-preview-tts`
 
-## Available voices
+### Available voices
 
 The following voice options are available:
 
@@ -314,9 +589,9 @@ The following voice options are available:
 - `Kore`: Neutral, professional
 - `Fenrir`: Warm, approachable
 
-## Authentication methods
+### Authentication methods
 
-### Gemini API (Development)
+#### Gemini API (Development)
 
 The simplest method using an API key from [Google AI Studio](https://makersuite.google.com/app/apikey):
 
@@ -327,7 +602,7 @@ const voice = new GeminiLiveVoice({
 })
 ```
 
-### Vertex AI (Production)
+#### Vertex AI (Production)
 
 For production use with OAuth authentication and Google Cloud Platform:
 
@@ -356,9 +631,9 @@ const voice = new GeminiLiveVoice({
 })
 ```
 
-## Advanced features
+### Advanced features
 
-### Session Management
+#### Session Management
 
 The Gemini Live API supports session resumption for handling network interruptions:
 
@@ -377,7 +652,7 @@ const voice = new GeminiLiveVoice({
 })
 ```
 
-### Tool Calling
+#### Tool Calling
 
 Enable the model to call functions during conversations:
 
@@ -402,7 +677,7 @@ voice.on('toolCall', ({ name, args, id }) => {
 })
 ```
 
-## Notes
+### Notes
 
 - The Gemini Live API uses WebSockets for real-time communication
 - Audio is processed as 16kHz PCM16 for input and 24kHz PCM16 for output