@mastra/core 1.7.0 → 1.8.0

package/dist/docs/references/reference-voice-composite-voice.md

@@ -0,0 +1,121 @@
+# CompositeVoice
+
+The CompositeVoice class allows you to combine different voice providers for text-to-speech and speech-to-text operations. This is particularly useful when you want to use the best provider for each operation - for example, using OpenAI for speech-to-text and PlayAI for text-to-speech.
+
+CompositeVoice supports both Mastra voice providers and AI SDK model providers
+
+## Constructor Parameters
+
+**config:** (`object`): Configuration object for the composite voice service
+
+**config.input?:** (`MastraVoice | TranscriptionModel`): Voice provider or AI SDK transcription model to use for speech-to-text operations. AI SDK models are automatically wrapped.
+
+**config.output?:** (`MastraVoice | SpeechModel`): Voice provider or AI SDK speech model to use for text-to-speech operations. AI SDK models are automatically wrapped.
+
+**config.realtime?:** (`MastraVoice`): Voice provider to use for real-time speech-to-speech operations
+
+## Methods
+
+### speak()
+
+Converts text to speech using the configured speaking provider.
+
+**input:** (`string | NodeJS.ReadableStream`): Text to convert to speech
+
+**options?:** (`object`): Provider-specific options passed to the speaking provider
+
+Notes:
+
+- If no speaking provider is configured, this method will throw an error
+- Options are passed through to the configured speaking provider
+- Returns a stream of audio data
+
+### listen()
+
+Converts speech to text using the configured listening provider.
+
+**audioStream:** (`NodeJS.ReadableStream`): Audio stream to convert to text
+
+**options?:** (`object`): Provider-specific options passed to the listening provider
+
+Notes:
+
+- If no listening provider is configured, this method will throw an error
+- Options are passed through to the configured listening provider
+- Returns either a string or a stream of transcribed text, depending on the provider
+
+### getSpeakers()
+
+Returns a list of available voices from the speaking provider, where each node contains:
+
+**voiceId:** (`string`): Unique identifier for the voice
+
+**key?:** (`value`): Additional voice properties that vary by provider (e.g., name, language)
+
+Notes:
+
+- Returns voices from the speaking provider only
+- If no speaking provider is configured, returns an empty array
+- Each voice object will have at least a voiceId property
+- Additional voice properties depend on the speaking provider
+
+## Usage Examples
+
+### Using Mastra Voice Providers
+
+```typescript
+import { CompositeVoice } from '@mastra/core/voice'
+import { OpenAIVoice } from '@mastra/voice-openai'
+import { PlayAIVoice } from '@mastra/voice-playai'
+
+// Create voice providers
+const openai = new OpenAIVoice()
+const playai = new PlayAIVoice()
+
+// Use OpenAI for listening (speech-to-text) and PlayAI for speaking (text-to-speech)
+const voice = new CompositeVoice({
+  input: openai,
+  output: playai,
+})
+
+// Convert speech to text using OpenAI
+const text = await voice.listen(audioStream)
+
+// Convert text to speech using PlayAI
+const audio = await voice.speak('Hello, world!')
+```
+
+### Using AI SDK Model Providers
+
+You can pass AI SDK transcription and speech models directly to CompositeVoice:
+
+```typescript
+import { CompositeVoice } from '@mastra/core/voice'
+import { openai } from '@ai-sdk/openai'
+import { elevenlabs } from '@ai-sdk/elevenlabs'
+
+// Use AI SDK models directly - they will be auto-wrapped
+const voice = new CompositeVoice({
+  input: openai.transcription('whisper-1'), // AI SDK transcription
+  output: elevenlabs.speech('eleven_turbo_v2'), // AI SDK speech
+})
+
+// Works the same way as with Mastra providers
+const text = await voice.listen(audioStream)
+const audio = await voice.speak('Hello from AI SDK!')
+```
+
+### Mix and Match
+
+You can combine Mastra providers with AI SDK models:
+
+```typescript
+import { CompositeVoice } from '@mastra/core/voice'
+import { PlayAIVoice } from '@mastra/voice-playai'
+import { groq } from '@ai-sdk/groq'
+
+const voice = new CompositeVoice({
+  input: groq.transcription('whisper-large-v3'), // AI SDK for STT
+  output: new PlayAIVoice(), // Mastra for TTS
+})
+```

package/dist/docs/references/reference-voice-mastra-voice.md

@@ -0,0 +1,311 @@
+# MastraVoice
+
+The MastraVoice class is an abstract base class that defines the core interface for voice services in Mastra. All voice provider implementations (like OpenAI, Deepgram, PlayAI, Speechify) extend this class to provide their specific functionality. The class now includes support for real-time speech-to-speech capabilities through WebSocket connections.
+
+## Usage Example
+
+```typescript
+import { MastraVoice } from '@mastra/core/voice'
+
+// Create a voice provider implementation
+class MyVoiceProvider extends MastraVoice {
+  constructor(config: {
+    speechModel?: BuiltInModelConfig
+    listeningModel?: BuiltInModelConfig
+    speaker?: string
+    realtimeConfig?: {
+      model?: string
+      apiKey?: string
+      options?: unknown
+    }
+  }) {
+    super({
+      speechModel: config.speechModel,
+      listeningModel: config.listeningModel,
+      speaker: config.speaker,
+      realtimeConfig: config.realtimeConfig,
+    })
+  }
+
+  // Implement required abstract methods
+  async speak(
+    input: string | NodeJS.ReadableStream,
+    options?: { speaker?: string },
+  ): Promise<NodeJS.ReadableStream | void> {
+    // Implement text-to-speech conversion
+  }
+
+  async listen(
+    audioStream: NodeJS.ReadableStream,
+    options?: unknown,
+  ): Promise<string | NodeJS.ReadableStream | void> {
+    // Implement speech-to-text conversion
+  }
+
+  async getSpeakers(): Promise<Array<{ voiceId: string; [key: string]: unknown }>> {
+    // Return list of available voices
+  }
+
+  // Optional speech-to-speech methods
+  async connect(): Promise<void> {
+    // Establish WebSocket connection for speech-to-speech communication
+  }
+
+  async send(audioData: NodeJS.ReadableStream | Int16Array): Promise<void> {
+    // Stream audio data in speech-to-speech
+  }
+
+  async answer(): Promise<void> {
+    // Trigger voice provider to respond
+  }
+
+  addTools(tools: Array<unknown>): void {
+    // Add tools for the voice provider to use
+  }
+
+  close(): void {
+    // Close WebSocket connection
+  }
+
+  on(event: string, callback: (data: unknown) => void): void {
+    // Register event listener
+  }
+
+  off(event: string, callback: (data: unknown) => void): void {
+    // Remove event listener
+  }
+}
+```
+
+## Constructor Parameters
+
+**config?:** (`VoiceConfig`): Configuration object for the voice service
+
+**config.speechModel?:** (`BuiltInModelConfig`): Configuration for the text-to-speech model
+
+**config.listeningModel?:** (`BuiltInModelConfig`): Configuration for the speech-to-text model
+
+**config.speaker?:** (`string`): Default speaker/voice ID to use
+
+**config.name?:** (`string`): Name for the voice provider instance
+
+**config.realtimeConfig?:** (`object`): Configuration for real-time speech-to-speech capabilities
+
+### BuiltInModelConfig
+
+**name:** (`string`): Name of the model to use
+
+**apiKey?:** (`string`): API key for the model service
+
+### RealtimeConfig
+
+**model?:** (`string`): Model to use for real-time speech-to-speech capabilities
+
+**apiKey?:** (`string`): API key for the real-time service
+
+**options?:** (`unknown`): Provider-specific options for real-time capabilities
+
+## Abstract Methods
+
+These methods must be implemented by unknown class extending MastraVoice.
+
+### speak()
+
+Converts text to speech using the configured speech model.
+
+```typescript
+abstract speak(
+  input: string | NodeJS.ReadableStream,
+  options?: {
+    speaker?: string;
+    [key: string]: unknown;
+  }
+): Promise<NodeJS.ReadableStream | void>
+```
+
+Purpose:
+
+- Takes text input and converts it to speech using the provider's text-to-speech service
+- Supports both string and stream input for flexibility
+- Allows overriding the default speaker/voice through options
+- Returns a stream of audio data that can be played or saved
+- May return void if the audio is handled by emitting 'speaking' event
+
+### listen()
+
+Converts speech to text using the configured listening model.
+
+```typescript
+abstract listen(
+  audioStream: NodeJS.ReadableStream,
+  options?: {
+    [key: string]: unknown;
+  }
+): Promise<string | NodeJS.ReadableStream | void>
+```
+
+Purpose:
+
+- Takes an audio stream and converts it to text using the provider's speech-to-text service
+- Supports provider-specific options for transcription configuration
+- Can return either a complete text transcription or a stream of transcribed text
+- Not all providers support this functionality (e.g., PlayAI, Speechify)
+- May return void if the transcription is handled by emitting 'writing' event
+
+### getSpeakers()
+
+Returns a list of available voices supported by the provider.
+
+```typescript
+abstract getSpeakers(): Promise<Array<{ voiceId: string; [key: string]: unknown }>>
+```
+
+Purpose:
+
+- Retrieves the list of available voices/speakers from the provider
+- Each voice must have at least a voiceId property
+- Providers can include additional metadata about each voice
+- Used to discover available voices for text-to-speech conversion
+
+## Optional Methods
+
+These methods have default implementations but can be overridden by voice providers that support speech-to-speech capabilities.
+
+### connect()
+
+Establishes a WebSocket or WebRTC connection for communication.
+
+```typescript
+connect(config?: unknown): Promise<void>
+```
+
+Purpose:
+
+- Initializes a connection to the voice service for communication
+- Must be called before using features like send() or answer()
+- Returns a Promise that resolves when the connection is established
+- Configuration is provider-specific
+
+### send()
+
+Streams audio data in real-time to the voice provider.
+
+```typescript
+send(audioData: NodeJS.ReadableStream | Int16Array): Promise<void>
+```
+
+Purpose:
+
+- Sends audio data to the voice provider for real-time processing
+- Useful for continuous audio streaming scenarios like live microphone input
+- Supports both ReadableStream and Int16Array audio formats
+- Must be in connected state before calling this method
+
+### answer()
+
+Triggers the voice provider to generate a response.
+
+```typescript
+answer(): Promise<void>
+```
+
+Purpose:
+
+- Sends a signal to the voice provider to generate a response
+- Used in real-time conversations to prompt the AI to respond
+- Response will be emitted through the event system (e.g., 'speaking' event)
+
+### addTools()
+
+Equips the voice provider with tools that can be used during conversations.
+
+```typescript
+addTools(tools: Array<Tool>): void
+```
+
+Purpose:
+
+- Adds tools that the voice provider can use during conversations
+- Tools can extend the capabilities of the voice provider
+- Implementation is provider-specific
+
+### close()
+
+Disconnects from the WebSocket or WebRTC connection.
+
+```typescript
+close(): void
+```
+
+Purpose:
+
+- Closes the connection to the voice service
+- Cleans up resources and stops any ongoing real-time processing
+- Should be called when you're done with the voice instance
+
+### on()
+
+Registers an event listener for voice events.
+
+```typescript
+on<E extends VoiceEventType>(
+  event: E,
+  callback: (data: E extends keyof VoiceEventMap ? VoiceEventMap[E] : unknown) => void,
+): void
+```
+
+Purpose:
+
+- Registers a callback function to be called when the specified event occurs
+- Standard events include 'speaking', 'writing', and 'error'
+- Providers can emit custom events as well
+- Event data structure depends on the event type
+
+### off()
+
+Removes an event listener.
+
+```typescript
+off<E extends VoiceEventType>(
+  event: E,
+  callback: (data: E extends keyof VoiceEventMap ? VoiceEventMap[E] : unknown) => void,
+): void
+```
+
+Purpose:
+
+- Removes a previously registered event listener
+- Used to clean up event handlers when they're no longer needed
+
+## Event System
+
+The MastraVoice class includes an event system for real-time communication. Standard event types include:
+
+**speaking:** (`{ text: string; audioStream?: NodeJS.ReadableStream; audio?: Int16Array }`): Emitted when the voice provider is speaking, contains audio data
+
+**writing:** (`{ text: string, role: string }`): Emitted when text is transcribed from speech
+
+**error:** (`{ message: string; code?: string; details?: unknown }`): Emitted when an error occurs
+
+## Protected Properties
+
+**listeningModel?:** (`BuiltInModelConfig | undefined`): Configuration for the speech-to-text model
+
+**speechModel?:** (`BuiltInModelConfig | undefined`): Configuration for the text-to-speech model
+
+**speaker?:** (`string | undefined`): Default speaker/voice ID
+
+**realtimeConfig?:** (`{ model?: string; apiKey?: string; options?: unknown } | undefined`): Configuration for real-time speech-to-speech capabilities
+
+## Telemetry Support
+
+MastraVoice includes built-in telemetry support through the `traced` method, which wraps method calls with performance tracking and error monitoring.
+
+## Notes
+
+- MastraVoice is an abstract class and cannot be instantiated directly
+- Implementations must provide concrete implementations for all abstract methods
+- The class provides a consistent interface across different voice service providers
+- Speech-to-speech capabilities are optional and provider-specific
+- The event system enables asynchronous communication for real-time interactions
+- Telemetry is automatically handled for all method calls

package/dist/docs/references/reference-voice-voice.addInstructions.md

@@ -0,0 +1,55 @@
+# voice.addInstructions()
+
+The `addInstructions()` method equips a voice provider with instructions that guide the model's behavior during real-time interactions. This is particularly useful for real-time voice providers that maintain context across a conversation.
+
+## Usage Example
+
+```typescript
+import { OpenAIRealtimeVoice } from '@mastra/voice-openai-realtime'
+import { Agent } from '@mastra/core/agent'
+
+// Initialize a real-time voice provider
+const voice = new OpenAIRealtimeVoice({
+  realtimeConfig: {
+    model: 'gpt-5.1-realtime',
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+})
+
+// Create an agent with the voice provider
+const agent = new Agent({
+  name: 'Customer Support Agent',
+  instructions: 'You are a helpful customer support agent for a software company.',
+  model: 'openai/gpt-5.1',
+  voice,
+})
+
+// Add additional instructions to the voice provider
+voice.addInstructions(`
+  When speaking to customers:
+  - Always introduce yourself as the customer support agent
+  - Speak clearly and concisely
+  - Ask clarifying questions when needed
+  - Summarize the conversation at the end
+`)
+
+// Connect to the real-time service
+await voice.connect()
+```
+
+## Parameters
+
+**instructions:** (`string`): Instructions to guide the voice model's behavior
+
+## Return Value
+
+This method does not return a value.
+
+## Notes
+
+- Instructions are most effective when they are clear, specific, and relevant to the voice interaction
+- This method is primarily used with real-time voice providers that maintain conversation context
+- If called on a voice provider that doesn't support instructions, it will log a warning and do nothing
+- Instructions added with this method are typically combined with any instructions provided by an associated Agent
+- For best results, add instructions before starting a conversation (before calling `connect()`)
+- Multiple calls to `addInstructions()` may either replace or append to existing instructions, depending on the provider implementation

package/dist/docs/references/reference-voice-voice.addTools.md

@@ -0,0 +1,67 @@
+# voice.addTools()
+
+The `addTools()` method equips a voice provider with tools (functions) that can be called by the model during real-time interactions. This enables voice assistants to perform actions like searching for information, making calculations, or interacting with external systems.
+
+## Usage Example
+
+```typescript
+import { OpenAIRealtimeVoice } from '@mastra/voice-openai-realtime'
+import { createTool } from '@mastra/core/tools'
+import { z } from 'zod'
+
+// Define tools
+const weatherTool = createTool({
+  id: 'getWeather',
+  description: 'Get the current weather for a location',
+  inputSchema: z.object({
+    location: z.string().describe('The city and state, e.g. San Francisco, CA'),
+  }),
+  outputSchema: z.object({
+    message: z.string(),
+  }),
+  execute: async inputData => {
+    // Fetch weather data from an API
+    const response = await fetch(
+      `https://api.weather.com?location=${encodeURIComponent(inputData.location)}`,
+    )
+    const data = await response.json()
+    return {
+      message: `The current temperature in ${inputData.location} is ${data.temperature}°F with ${data.conditions}.`,
+    }
+  },
+})
+
+// Initialize a real-time voice provider
+const voice = new OpenAIRealtimeVoice({
+  realtimeConfig: {
+    model: 'gpt-5.1-realtime',
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+})
+
+// Add tools to the voice provider
+voice.addTools({
+  getWeather: weatherTool,
+})
+
+// Connect to the real-time service
+await voice.connect()
+```
+
+## Parameters
+
+**tools:** (`ToolsInput`): Object containing tool definitions that can be called by the voice model
+
+## Return Value
+
+This method does not return a value.
+
+## Notes
+
+- Tools must follow the Mastra tool format with name, description, input schema, and execute function
+- This method is primarily used with real-time voice providers that support function calling
+- If called on a voice provider that doesn't support tools, it will log a warning and do nothing
+- Tools added with this method are typically combined with any tools provided by an associated Agent
+- For best results, add tools before starting a conversation (before calling `connect()`)
+- The voice provider will automatically handle the invocation of tool handlers when the model decides to use them
+- Multiple calls to `addTools()` may either replace or merge with existing tools, depending on the provider implementation

package/dist/docs/references/reference-voice-voice.connect.md

@@ -0,0 +1,94 @@
+# voice.connect()
+
+The `connect()` method establishes a WebSocket or WebRTC connection for real-time speech-to-speech communication. This method must be called before using other real-time features like `send()` or `answer()`.
+
+## Usage Example
+
+```typescript
+import { OpenAIRealtimeVoice } from '@mastra/voice-openai-realtime'
+import Speaker from '@mastra/node-speaker'
+
+const speaker = new Speaker({
+  sampleRate: 24100, // Audio sample rate in Hz - standard for high-quality audio on MacBook Pro
+  channels: 1, // Mono audio output (as opposed to stereo which would be 2)
+  bitDepth: 16, // Bit depth for audio quality - CD quality standard (16-bit resolution)
+})
+
+// Initialize a real-time voice provider
+const voice = new OpenAIRealtimeVoice({
+  realtimeConfig: {
+    model: 'gpt-5.1-realtime',
+    apiKey: process.env.OPENAI_API_KEY,
+    options: {
+      sessionConfig: {
+        turn_detection: {
+          type: 'server_vad',
+          threshold: 0.6,
+          silence_duration_ms: 1200,
+        },
+      },
+    },
+  },
+  speaker: 'alloy', // Default voice
+})
+// Connect to the real-time service
+await voice.connect()
+// Now you can use real-time features
+voice.on('speaker', stream => {
+  stream.pipe(speaker)
+})
+// With connection options
+await voice.connect({
+  timeout: 10000, // 10 seconds timeout
+  reconnect: true,
+})
+```
+
+## Parameters
+
+**options?:** (`Record<string, unknown>`): Provider-specific connection options
+
+## Return Value
+
+Returns a `Promise<void>` that resolves when the connection is successfully established.
+
+## Provider-Specific Options
+
+Each real-time voice provider may support different options for the `connect()` method:
+
+### OpenAI Realtime
+
+**options.timeout?:** (`number`): Connection timeout in milliseconds (Default: `30000`)
+
+**options.reconnect?:** (`boolean`): Whether to automatically reconnect on connection loss (Default: `false`)
+
+## Using with CompositeVoice
+
+When using `CompositeVoice`, the `connect()` method delegates to the configured real-time provider:
+
+```typescript
+import { CompositeVoice } from '@mastra/core/voice'
+import { OpenAIRealtimeVoice } from '@mastra/voice-openai-realtime'
+const realtimeVoice = new OpenAIRealtimeVoice()
+const voice = new CompositeVoice({
+  realtime: realtimeVoice,
+})
+// This will use the OpenAIRealtimeVoice provider
+await voice.connect()
+```
+
+## Notes
+
+- This method is only implemented by real-time voice providers that support speech-to-speech capabilities
+- If called on a voice provider that doesn't support this functionality, it will log a warning and resolve immediately
+- The connection must be established before using other real-time methods like `send()` or `answer()`
+- When you're done with the voice instance, call `close()` to properly clean up resources
+- Some providers may automatically reconnect on connection loss, depending on their implementation
+- Connection errors will typically be thrown as exceptions that should be caught and handled
+
+## Related Methods
+
+- [voice.send()](https://mastra.ai/reference/voice/voice.send) - Sends audio data to the voice provider
+- [voice.answer()](https://mastra.ai/reference/voice/voice.answer) - Triggers the voice provider to respond
+- [voice.close()](https://mastra.ai/reference/voice/voice.close) - Disconnects from the real-time service
+- [voice.on()](https://mastra.ai/reference/voice/voice.on) - Registers an event listener for voice events

package/dist/docs/references/reference-voice-voice.events.md

@@ -0,0 +1,37 @@
+# Voice Events
+
+Voice providers emit various events during real-time voice interactions. These events can be listened to using the [voice.on()](https://mastra.ai/reference/voice/voice.on) method and are particularly important for building interactive voice applications.
+
+## Common Events
+
+These events are commonly implemented across real-time voice providers:
+
+**error:** (`Error`): Emitted when an error occurs during voice processing or when audio data format is unsupported
+
+**session.created:** (`object`): Emitted when a new session is created with the OpenAI service
+
+**session.updated:** (`object`): Emitted when the session configuration is updated
+
+**response.created:** (`object`): Emitted when a new response is created by the AI assistant
+
+**response.done:** (`object`): Emitted when the AI assistant has completed its response
+
+**speaker:** (`StreamWithId`): Emitted with a new audio stream that can be piped to an audio output
+
+**writing:** (`object`): Emitted when text is being transcribed (user) or generated (assistant)
+
+**speaking:** (`object`): Emitted when audio data is available from the voice provider
+
+**speaking.done:** (`object`): Emitted when the voice provider has finished speaking
+
+**tool-call-start:** (`object`): Emitted when the AI assistant starts executing a tool
+
+**tool-call-result:** (`object`): Emitted when a tool execution is complete with its result
+
+## Notes
+
+- Not all events are supported by all voice providers
+- The exact payload structure may vary between providers
+- For non-real-time providers, most of these events will not be emitted
+- Events are useful for building interactive UIs that respond to the conversation state
+- Consider using the [voice.off()](https://mastra.ai/reference/voice/voice.off) method to remove event listeners when they are no longer needed