@mastra/voice-google-gemini-live 0.0.0-add-libsql-changeset-20250910154739

package/CHANGELOG.md

@@ -0,0 +1,159 @@
+# @mastra/voice-google-gemini-live
+
+## 0.0.0-add-libsql-changeset-20250910154739
+
+### Patch Changes
+
+- Updated dependencies [[`b4379f7`](https://github.com/mastra-ai/mastra/commit/b4379f703fd74474f253420e8c3a684f2c4b2f8e), [`d34aaa1`](https://github.com/mastra-ai/mastra/commit/d34aaa1da5d3c5f991740f59e2fe6d28d3e2dd91), [`b2babfa`](https://github.com/mastra-ai/mastra/commit/b2babfa9e75b22f2759179e71d8473f6dc5421ed), [`d8c3ba5`](https://github.com/mastra-ai/mastra/commit/d8c3ba516f4173282d293f7e64769cfc8738d360), [`6424c7e`](https://github.com/mastra-ai/mastra/commit/6424c7ec38b6921d66212431db1e0958f441b2a7), [`db94750`](https://github.com/mastra-ai/mastra/commit/db94750a41fd29b43eb1f7ce8e97ba8b9978c91b), [`a66a371`](https://github.com/mastra-ai/mastra/commit/a66a3716b00553d7f01842be9deb34f720b10fab), [`69fc3cd`](https://github.com/mastra-ai/mastra/commit/69fc3cd0fd814901785bdcf49bf536ab1e7fd975)]:
+  - @mastra/core@0.0.0-add-libsql-changeset-20250910154739
+
+## 0.10.10
+
+### Patch Changes
+
+- 376913a: Update peerdeps
+- Updated dependencies [8fbf79e]
+- Updated dependencies [fd83526]
+- Updated dependencies [d0b90ab]
+- Updated dependencies [6f5eb7a]
+- Updated dependencies [a01cf14]
+- Updated dependencies [a9e50ee]
+- Updated dependencies [5397eb4]
+- Updated dependencies [c9f4e4a]
+- Updated dependencies [0acbc80]
+  - @mastra/core@0.16.0
+
+## 0.10.10-alpha.0
+
+### Patch Changes
+
+- 376913a: Update peerdeps
+- Updated dependencies [8fbf79e]
+  - @mastra/core@0.16.0-alpha.1
+
+## 0.10.9
+
+### Patch Changes
+
+- ab48c97: dependencies updates:
+  - Updated dependency [`zod@^3.25.76` ↗︎](https://www.npmjs.com/package/zod/v/3.25.76) (from `^3.25.67`, in `dependencies`)
+  - Updated dependency [`zod-to-json-schema@^3.24.6` ↗︎](https://www.npmjs.com/package/zod-to-json-schema/v/3.24.6) (from `^3.23.5`, in `dependencies`)
+- de3cbc6: Update the `package.json` file to include additional fields like `repository`, `homepage` or `files`.
+- f0dfcac: updated core peerdep
+- Updated dependencies [ab48c97]
+- Updated dependencies [85ef90b]
+- Updated dependencies [aedbbfa]
+- Updated dependencies [ff89505]
+- Updated dependencies [637f323]
+- Updated dependencies [de3cbc6]
+- Updated dependencies [c19bcf7]
+- Updated dependencies [4474d04]
+- Updated dependencies [183dc95]
+- Updated dependencies [a1111e2]
+- Updated dependencies [b42a961]
+- Updated dependencies [61debef]
+- Updated dependencies [9beaeff]
+- Updated dependencies [29de0e1]
+- Updated dependencies [f643c65]
+- Updated dependencies [00c74e7]
+- Updated dependencies [fef7375]
+- Updated dependencies [e3d8fea]
+- Updated dependencies [45e4d39]
+- Updated dependencies [9eee594]
+- Updated dependencies [7149d8d]
+- Updated dependencies [822c2e8]
+- Updated dependencies [979912c]
+- Updated dependencies [7dcf4c0]
+- Updated dependencies [4106a58]
+- Updated dependencies [ad78bfc]
+- Updated dependencies [0302f50]
+- Updated dependencies [6ac697e]
+- Updated dependencies [74db265]
+- Updated dependencies [0ce418a]
+- Updated dependencies [af90672]
+- Updated dependencies [8387952]
+- Updated dependencies [7f3b8da]
+- Updated dependencies [905352b]
+- Updated dependencies [599d04c]
+- Updated dependencies [56041d0]
+- Updated dependencies [3412597]
+- Updated dependencies [5eca5d2]
+- Updated dependencies [f2cda47]
+- Updated dependencies [5de1555]
+- Updated dependencies [cfd377a]
+- Updated dependencies [1ed5a3e]
+  - @mastra/core@0.15.3
+
+## 0.10.9-alpha.2
+
+### Patch Changes
+
+- [#7394](https://github.com/mastra-ai/mastra/pull/7394) [`f0dfcac`](https://github.com/mastra-ai/mastra/commit/f0dfcac4458bdf789b975e2d63e984f5d1e7c4d3) Thanks [@NikAiyer](https://github.com/NikAiyer)! - updated core peerdep
+
+- Updated dependencies [[`7149d8d`](https://github.com/mastra-ai/mastra/commit/7149d8d4bdc1edf0008e0ca9b7925eb0b8b60dbe)]:
+  - @mastra/core@0.15.3-alpha.7
+
+## 0.10.9-alpha.1
+
+### Patch Changes
+
+- [#7343](https://github.com/mastra-ai/mastra/pull/7343) [`de3cbc6`](https://github.com/mastra-ai/mastra/commit/de3cbc61079211431bd30487982ea3653517278e) Thanks [@LekoArts](https://github.com/LekoArts)! - Update the `package.json` file to include additional fields like `repository`, `homepage` or `files`.
+
+- Updated dependencies [[`85ef90b`](https://github.com/mastra-ai/mastra/commit/85ef90bb2cd4ae4df855c7ac175f7d392c55c1bf), [`de3cbc6`](https://github.com/mastra-ai/mastra/commit/de3cbc61079211431bd30487982ea3653517278e)]:
+  - @mastra/core@0.15.3-alpha.5
+
+## 0.10.9-alpha.0
+
+### Patch Changes
+
+- [#5816](https://github.com/mastra-ai/mastra/pull/5816) [`ab48c97`](https://github.com/mastra-ai/mastra/commit/ab48c979098ea571faf998a55d3a00e7acd7a715) Thanks [@dane-ai-mastra](https://github.com/apps/dane-ai-mastra)! - dependencies updates:
+  - Updated dependency [`zod@^3.25.76` ↗︎](https://www.npmjs.com/package/zod/v/3.25.76) (from `^3.25.67`, in `dependencies`)
+  - Updated dependency [`zod-to-json-schema@^3.24.6` ↗︎](https://www.npmjs.com/package/zod-to-json-schema/v/3.24.6) (from `^3.23.5`, in `dependencies`)
+- Updated dependencies [[`ab48c97`](https://github.com/mastra-ai/mastra/commit/ab48c979098ea571faf998a55d3a00e7acd7a715), [`ff89505`](https://github.com/mastra-ai/mastra/commit/ff895057c8c7e91a5535faef46c5e5391085ddfa), [`183dc95`](https://github.com/mastra-ai/mastra/commit/183dc95596f391b977bd1a2c050b8498dac74891), [`a1111e2`](https://github.com/mastra-ai/mastra/commit/a1111e24e705488adfe5e0a6f20c53bddf26cb22), [`61debef`](https://github.com/mastra-ai/mastra/commit/61debefd80ad3a7ed5737e19df6a23d40091689a), [`9beaeff`](https://github.com/mastra-ai/mastra/commit/9beaeffa4a97b1d5fd01a7f8af8708b16067f67c), [`9eee594`](https://github.com/mastra-ai/mastra/commit/9eee594e35e0ca2a650fcc33fa82009a142b9ed0), [`979912c`](https://github.com/mastra-ai/mastra/commit/979912cfd180aad53287cda08af771df26454e2c), [`7dcf4c0`](https://github.com/mastra-ai/mastra/commit/7dcf4c04f44d9345b1f8bc5d41eae3f11ac61611), [`ad78bfc`](https://github.com/mastra-ai/mastra/commit/ad78bfc4ea6a1fff140432bf4f638e01af7af668), [`0ce418a`](https://github.com/mastra-ai/mastra/commit/0ce418a1ccaa5e125d4483a9651b635046152569), [`8387952`](https://github.com/mastra-ai/mastra/commit/838795227b4edf758c84a2adf6f7fba206c27719), [`5eca5d2`](https://github.com/mastra-ai/mastra/commit/5eca5d2655788863ea0442a46c9ef5d3c6dbe0a8)]:
+  - @mastra/core@0.15.3-alpha.4
+
+## 0.10.8
+
+### Patch Changes
+
+- [`c6113ed`](https://github.com/mastra-ai/mastra/commit/c6113ed7f9df297e130d94436ceee310273d6430) Thanks [@wardpeet](https://github.com/wardpeet)! - Fix peerdpes for @mastra/core
+
+- Updated dependencies []:
+  - @mastra/core@0.15.2
+
+## 0.10.7
+
+### Patch Changes
+
+- dd512a2: Integrates Google Gemini Live API into the Mastra framework
+- Updated dependencies [227c7e6]
+- Updated dependencies [12cae67]
+- Updated dependencies [fd3a3eb]
+- Updated dependencies [6faaee5]
+- Updated dependencies [4232b14]
+- Updated dependencies [a89de7e]
+- Updated dependencies [5a37d0c]
+- Updated dependencies [4bde0cb]
+- Updated dependencies [cf4f357]
+- Updated dependencies [ad888a2]
+- Updated dependencies [481751d]
+- Updated dependencies [2454423]
+- Updated dependencies [194e395]
+- Updated dependencies [a722c0b]
+- Updated dependencies [c30bca8]
+- Updated dependencies [3b5fec7]
+- Updated dependencies [a8f129d]
+  - @mastra/core@0.14.0
+
+## 0.10.7-alpha.0
+
+### Patch Changes
+
+- dd512a2: Integrates Google Gemini Live API into the Mastra framework
+- Updated dependencies [6faaee5]
+- Updated dependencies [4232b14]
+- Updated dependencies [a89de7e]
+- Updated dependencies [cf4f357]
+- Updated dependencies [a722c0b]
+- Updated dependencies [3b5fec7]
+  - @mastra/core@0.14.0-alpha.1

package/LICENSE.md

@@ -0,0 +1,15 @@
+# Apache License 2.0
+
+Copyright (c) 2025 Kepler Software, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

package/README.md

@@ -0,0 +1,459 @@
+# @mastra/voice-google-gemini-live
+
+Google Gemini Live API integration for Mastra, providing real-time multimodal voice interactions with advanced capabilities including video input, tool calling, and session management.
+
+## Installation
+
+```bash
+npm install @mastra/voice-google-gemini-live
+```
+
+## Configuration
+
+The module supports two authentication methods:
+
+### Option 1: Gemini API (Recommended for development)
+
+Use an API key from [Google AI Studio](https://makersuite.google.com/app/apikey):
+
+```bash
+# Set environment variable
+GOOGLE_API_KEY=your_api_key
+```
+
+### Option 2: Vertex AI (Recommended for production)
+
+Use OAuth authentication with Google Cloud Platform. There are multiple ways to authenticate:
+
+#### Application Default Credentials (ADC)
+
+```bash
+# Install gcloud CLI and authenticate
+gcloud auth application-default login
+
+# Set project ID
+GOOGLE_CLOUD_PROJECT=your_project_id
+```
+
+#### Service Account Key File
+
+```bash
+# Set path to service account JSON
+GOOGLE_APPLICATION_CREDENTIALS=path/to/service-account.json
+GOOGLE_CLOUD_PROJECT=your_project_id
+```
+
+#### Service Account in Code
+
+```typescript
+const voice = new GeminiLiveVoice({
+  vertexAI: true,
+  project: 'your-gcp-project',
+  location: 'us-central1',
+  serviceAccountKeyFile: '/path/to/service-account.json',
+  // OR use service account email for impersonation
+  serviceAccountEmail: 'service-account@project.iam.gserviceaccount.com',
+});
+```
+
+### Required Permissions for Vertex AI
+
+When using Vertex AI, ensure your service account or user has these IAM roles:
+
+- `aiplatform.user` or specific permissions:
+  - `aiplatform.endpoints.predict`
+  - `aiplatform.models.predict`
+
+## Usage
+
+```typescript
+import { GeminiLiveVoice } from '@mastra/voice-google-gemini-live';
+
+// Initialize with Gemini API
+const voice = new GeminiLiveVoice({
+  apiKey: 'your-api-key', // Optional, can use GOOGLE_API_KEY env var
+  model: 'gemini-2.0-flash-live-001',
+  speaker: 'Puck', // Default voice
+});
+
+// OR initialize with Vertex AI (recommended for production)
+const voice = new GeminiLiveVoice({
+  vertexAI: true,
+  project: 'your-project-id',
+  model: 'gemini-2.0-flash-live-001',
+  speaker: 'Puck',
+});
+
+// Connect to the Live API
+await voice.connect();
+
+// Listen for responses
+voice.on('speaking', ({ audioData }) => {
+  // Handle audio response as Int16Array
+  playAudio(audioData);
+});
+
+// Or subscribe to a concatenated audio stream per response
+voice.on('speaker', audioStream => {
+  audioStream.pipe(playbackDevice);
+});
+
+voice.on('writing', ({ text, role }) => {
+  // Handle transcribed text
+  console.log(`${role}: ${text}`);
+});
+
+// Send text to speech
+await voice.speak('Hello from Mastra!');
+
+// Send audio stream
+const microphoneStream = getMicrophoneStream();
+await voice.send(microphoneStream);
+
+// When done, disconnect
+voice.disconnect();
+```
+
+## API Reference
+
+### Constructor
+
+**`new GeminiLiveVoice(options?: GeminiLiveVoiceConfig)`**
+
+Creates a new GeminiLiveVoice instance.
+
+**Parameters:**
+
+- `options` (optional): Configuration object
+  - `apiKey?: string` - Google API key (falls back to GOOGLE_API_KEY env var)
+  - `model?: GeminiVoiceModel` - Model to use (default: 'gemini-2.0-flash-exp')
+  - `speaker?: GeminiVoiceName` - Voice to use (default: 'Puck')
+  - `vertexAI?: boolean` - Use Vertex AI instead of Gemini API
+  - `project?: string` - Google Cloud project ID (required for Vertex AI)
+  - `location?: string` - Google Cloud region (default: 'us-central1')
+  - `serviceAccountKeyFile?: string` - Path to service account JSON key file
+  - `serviceAccountEmail?: string` - Service account email for impersonation
+  - `instructions?: string` - System instructions for the model
+  - `tools?: GeminiToolConfig[]` - Tools available to the model
+  - `sessionConfig?: GeminiSessionConfig` - Session configuration
+  - `audioConfig?: Partial<AudioConfig>` - Audio configuration
+  - `debug?: boolean` - Enable debug logging
+
+### Connection Management
+
+**`async connect(): Promise<void>`**
+
+Establishes connection to the Gemini Live API. Must be called before using other methods.
+
+**Returns:** Promise that resolves when connection is established
+
+**Throws:** Error if connection fails or authentication is invalid
+
+---
+
+**`async disconnect(): Promise<void>`**
+
+Disconnects from the Gemini Live API and cleans up resources.
+
+**Returns:** Promise that resolves when disconnection is complete
+
+---
+
+**`getConnectionState(): 'disconnected' | 'connected'`**
+
+Gets the current connection state.
+
+**Returns:** Current connection state
+
+---
+
+**`isConnected(): boolean`**
+
+Checks if currently connected to the API.
+
+**Returns:** true if connected, false otherwise
+
+---
+
+Connection lifecycle transitions such as "connecting", "disconnecting", and "updated" are emitted via the `session` event:
+
+```ts
+voice.on('session', data => {
+  // data.state is one of: 'connecting' | 'connected' | 'disconnected' | 'disconnecting' | 'updated'
+});
+```
+
+### Audio and Speech
+
+**`async speak(input: string | NodeJS.ReadableStream, options?: GeminiLiveVoiceOptions): Promise<void>`**
+
+Converts text to speech and sends it to the model.
+
+**Parameters:**
+
+- `input: string | NodeJS.ReadableStream` - Text to convert to speech
+- `options?: GeminiLiveVoiceOptions` - Optional speech options
+  - `speaker?: GeminiVoiceName` - Override the default speaker
+  - `languageCode?: string` - Language code for the response
+  - `responseModalities?: ('AUDIO' | 'TEXT')[]` - Response modalities
+
+**Returns:** Promise<void> (responses are emitted via `speaker` and `writing` events)
+
+**Throws:** Error if not connected or input is empty
+
+---
+
+**`async send(audioData: NodeJS.ReadableStream | Int16Array): Promise<void>`**
+
+Sends audio data for real-time processing.
+
+**Parameters:**
+
+- `audioData: NodeJS.ReadableStream | Int16Array` - Audio data to send
+
+**Returns:** Promise that resolves when audio is sent
+
+**Throws:** Error if not connected or audio format is invalid
+
+---
+
+**`async listen(audioStream: NodeJS.ReadableStream, options?: GeminiLiveVoiceOptions): Promise<string>`**
+
+Processes audio stream for speech-to-text transcription.
+
+**Parameters:**
+
+- `audioStream: NodeJS.ReadableStream` - Audio stream to transcribe
+- `options?: GeminiLiveVoiceOptions` - Optional transcription options
+
+**Returns:** Promise that resolves to transcribed text
+
+**Throws:** Error if not connected, audio format is invalid, or transcription fails
+
+---
+
+**`getCurrentSpeakerStream(): NodeJS.ReadableStream | null`**
+
+Gets the current concatenated audio stream for the active response.
+
+**Returns:** ReadableStream of concatenated audio chunks, or null if no active stream
+
+### Session Management
+
+**`async updateSessionConfig(config: Partial<GeminiLiveVoiceConfig>): Promise<void>`**
+
+Updates session configuration during an active session.
+
+**Parameters:**
+
+- `config: Partial<GeminiLiveVoiceConfig>` - Configuration to update
+  - `speaker?: GeminiVoiceName` - Change voice/speaker
+  - `instructions?: string` - Update system instructions
+  - `tools?: GeminiToolConfig[]` - Update available tools
+  - `sessionConfig?: GeminiSessionConfig` - Update session settings (e.g. `vad`, `interrupts`, `contextCompression`)
+
+**Returns:** Promise that resolves when configuration is updated
+
+**Throws:** Error if not connected or update fails
+
+---
+
+**`async resumeSession(handle: string): Promise<void>`**
+
+Resumes a previous session using a session handle.
+
+**Parameters:**
+
+- `handle: string` - Session handle from previous session
+
+**Returns:** Promise that resolves when session is resumed
+
+**Note:** Session resumption is not yet fully implemented for Gemini Live API
+
+---
+
+**`getSessionHandle(): string | undefined`**
+
+Gets the current session handle for resumption.
+
+**Returns:** Session handle string, or undefined if not available
+
+**Note:** Session handles are not yet fully supported by Gemini Live API
+
+### Voice and Model Information
+
+**`async getSpeakers(): Promise<Array<{ voiceId: string; description?: string }>>`**
+
+Gets available speakers/voices.
+
+**Returns:** Promise that resolves to array of available voices with descriptions
+
+---
+
+**`async getListener(): Promise<{ enabled: boolean }>`**
+
+Checks if listening capabilities are enabled.
+
+**Returns:** Promise that resolves to listening status
+
+**Note:** Inherits default implementation from MastraVoice base class
+
+### Event Handling
+
+**`on<E extends VoiceEventType>(event: E, callback: (data: E extends keyof GeminiLiveEventMap ? GeminiLiveEventMap[E] : unknown) => void): void`**
+
+Registers an event listener.
+
+**Parameters:**
+
+- `event: E` - Event name to listen for
+- `callback: (data) => void` - Function to call when event occurs
+
+**Available Events:**
+
+- `'speaking'` - Audio response from model
+- `'speaker'` - Readable stream of concatenated audio for the active response
+- `'writing'` - Text response or transcription
+- `'error'` - Error events
+- `'session'` - Session state changes
+- `'toolCall'` - Tool calls from model
+- `'vad'` - Voice activity detection events
+- `'interrupt'` - Interrupt events
+- `'usage'` - Token usage information
+- `'sessionHandle'` - Session resumption handle
+- `'turnComplete'` - Turn completion for the current model response
+
+### Tools
+
+Add tools with `addTools()` using either `@mastra/core/tools` or a plain object matching `ToolsInput`.
+
+Using `createTool`:
+
+```ts
+import { createTool } from '@mastra/core/tools';
+import { z } from 'zod';
+
+const searchTool = createTool({
+  id: 'search',
+  description: 'Search the web',
+  inputSchema: z.object({ query: z.string() }),
+  execute: async ({ context }) => {
+    const { query } = context;
+    // ... perform search
+    return { results: [] };
+  },
+});
+
+voice.addTools({ search: searchTool });
+```
+
+Using a plain object (ensure each tool has an `id`):
+
+```ts
+voice.addTools({
+  search: {
+    id: 'search',
+    description: 'Search the web',
+    inputSchema: { type: 'object', properties: { query: { type: 'string' } } },
+    execute: async ({ context }) => ({ results: [] }),
+  },
+});
+```
+
+Tool call events from the model are emitted as:
+
+```ts
+voice.on('toolCall', ({ name, args, id }) => {
+  // name: string, args: Record<string, any>, id: string
+});
+```
+
+---
+
+**`off<E extends VoiceEventType>(event: E, callback: (data: E extends keyof GeminiLiveEventMap ? GeminiLiveEventMap[E] : unknown) => void): void`**
+
+Removes an event listener.
+
+**Parameters:**
+
+- `event: E` - Event name to stop listening to
+- `callback: (data) => void` - Specific callback function to remove
+
+### Configuration Types
+
+**`GeminiLiveVoiceConfig`**
+
+```typescript
+interface GeminiLiveVoiceConfig {
+  apiKey?: string;
+  model?: GeminiVoiceModel;
+  speaker?: GeminiVoiceName;
+  vertexAI?: boolean;
+  project?: string;
+  location?: string;
+  serviceAccountKeyFile?: string;
+  serviceAccountEmail?: string;
+  instructions?: string;
+  tools?: GeminiToolConfig[];
+  sessionConfig?: GeminiSessionConfig;
+  audioConfig?: Partial<AudioConfig>;
+  debug?: boolean;
+}
+```
+
+**`GeminiLiveVoiceOptions`**
+
+```typescript
+interface GeminiLiveVoiceOptions {
+  speaker?: GeminiVoiceName;
+  languageCode?: string;
+  responseModalities?: ('AUDIO' | 'TEXT')[];
+}
+```
+
+**`GeminiSessionConfig`**
+
+```typescript
+interface GeminiSessionConfig {
+  enableResumption?: boolean;
+  maxDuration?: string;
+  contextCompression?: boolean;
+  vad?: {
+    enabled?: boolean;
+    sensitivity?: number;
+    silenceDurationMs?: number;
+  };
+  interrupts?: {
+    enabled?: boolean;
+    allowUserInterruption?: boolean;
+  };
+}
+```
+
+## Features
+
+- **Real-time bidirectional audio streaming**
+- **Multimodal input support** (audio, video, text)
+- **Built-in Voice Activity Detection (VAD)**
+- **Interrupt handling** - Natural conversation flow
+- **Session management** - Resume conversations after network interruptions
+- **Tool calling support** - Integrate with external APIs and functions
+- **Live transcription** - Real-time speech-to-text
+- **Multiple voice options** - Choose from various voice personalities
+- **Multilingual support** - Support for 30+ languages
+
+## Voice Options
+
+- **Puck** - Conversational, friendly
+- **Charon** - Deep, authoritative
+- **Kore** - Neutral, professional
+- **Fenrir** - Warm, approachable
+
+## Model Options
+
+- `gemini-2.0-flash-exp` - Default model
+- `gemini-2.0-flash-live-001` - Latest production model
+- `gemini-2.5-flash-preview-native-audio-dialog` - Preview with native audio
+- `gemini-live-2.5-flash-preview` - Half-cascade architecture
+
+For detailed API documentation, visit [Google's Gemini Live API docs](https://ai.google.dev/gemini-api/docs/live).