@mastra/core 1.7.0 → 1.8.0

package/dist/docs/references/docs-voice-speech-to-speech.md

@@ -0,0 +1,102 @@
+# Speech-to-Speech Capabilities in Mastra
+
+## Introduction
+
+Speech-to-Speech (STS) in Mastra provides a standardized interface for real-time interactions across multiple providers. STS enables continuous bidirectional audio communication through listening to events from Realtime models. Unlike separate TTS and STT operations, STS maintains an open connection that processes speech continuously in both directions.
+
+## Configuration
+
+- **`apiKey`**: Your OpenAI API key. Falls back to the `OPENAI_API_KEY` environment variable.
+- **`model`**: The model ID to use for real-time voice interactions (e.g., `gpt-5.1-realtime`).
+- **`speaker`**: The default voice ID for speech synthesis. This allows you to specify which voice to use for the speech output.
+
+```typescript
+const voice = new OpenAIRealtimeVoice({
+  apiKey: 'your-openai-api-key',
+  model: 'gpt-5.1-realtime',
+  speaker: 'alloy', // Default voice
+})
+
+// If using default settings the configuration can be simplified to:
+const voice = new OpenAIRealtimeVoice()
+```
+
+## Using STS
+
+```typescript
+import { Agent } from '@mastra/core/agent'
+import { OpenAIRealtimeVoice } from '@mastra/voice-openai-realtime'
+import { playAudio, getMicrophoneStream } from '@mastra/node-audio'
+
+const agent = new Agent({
+  id: 'agent',
+  name: 'OpenAI Realtime Agent',
+  instructions: `You are a helpful assistant with real-time voice capabilities.`,
+  model: 'openai/gpt-5.1',
+  voice: new OpenAIRealtimeVoice(),
+})
+
+// Connect to the voice service
+await agent.voice.connect()
+
+// Listen for agent audio responses
+agent.voice.on('speaker', ({ audio }) => {
+  playAudio(audio)
+})
+
+// Initiate the conversation
+await agent.voice.speak('How can I help you today?')
+
+// Send continuous audio from the microphone
+const micStream = getMicrophoneStream()
+await agent.voice.send(micStream)
+```
+
+For integrating Speech-to-Speech capabilities with agents, refer to the [Adding Voice to Agents](https://mastra.ai/docs/agents/adding-voice) documentation.
+
+## Google Gemini Live (Realtime)
+
+```typescript
+import { Agent } from '@mastra/core/agent'
+import { GeminiLiveVoice } from '@mastra/voice-google-gemini-live'
+import { playAudio, getMicrophoneStream } from '@mastra/node-audio'
+
+const agent = new Agent({
+  id: 'agent',
+  name: 'Gemini Live Agent',
+  instructions: 'You are a helpful assistant with real-time voice capabilities.',
+  // Model used for text generation; voice provider handles realtime audio
+  model: 'openai/gpt-5.1',
+  voice: new GeminiLiveVoice({
+    apiKey: process.env.GOOGLE_API_KEY,
+    model: 'gemini-2.0-flash-exp',
+    speaker: 'Puck',
+    debug: true,
+    // Vertex AI option:
+    // vertexAI: true,
+    // project: 'your-gcp-project',
+    // location: 'us-central1',
+    // serviceAccountKeyFile: '/path/to/service-account.json',
+  }),
+})
+
+await agent.voice.connect()
+
+agent.voice.on('speaker', ({ audio }) => {
+  playAudio(audio)
+})
+
+agent.voice.on('writing', ({ role, text }) => {
+  console.log(`${role}: ${text}`)
+})
+
+await agent.voice.speak('How can I help you today?')
+
+const micStream = getMicrophoneStream()
+await agent.voice.send(micStream)
+```
+
+Note:
+
+- Live API requires `GOOGLE_API_KEY`. Vertex AI requires project/location and service account credentials.
+- Events: `speaker` (audio stream), `writing` (text), `turnComplete`, `usage`, and `error`.

package/dist/docs/references/docs-voice-speech-to-text.md

@@ -0,0 +1,79 @@
+# Speech-to-Text (STT)
+
+Speech-to-Text (STT) in Mastra provides a standardized interface for converting audio input into text across multiple service providers. STT helps create voice-enabled applications that can respond to human speech, enabling hands-free interaction, accessibility for users with disabilities, and more natural human-computer interfaces.
+
+## Configuration
+
+To use STT in Mastra, you need to provide a `listeningModel` when initializing the voice provider. This includes parameters such as:
+
+- **`name`**: The specific STT model to use.
+- **`apiKey`**: Your API key for authentication.
+- **Provider-specific options**: Additional options that may be required or supported by the specific voice provider.
+
+**Note**: All of these parameters are optional. You can use the default settings provided by the voice provider, which will depend on the specific provider you are using.
+
+```typescript
+const voice = new OpenAIVoice({
+  listeningModel: {
+    name: 'whisper-1',
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+})
+
+// If using default settings the configuration can be simplified to:
+const voice = new OpenAIVoice()
+```
+
+## Available Providers
+
+Mastra supports several Speech-to-Text providers, each with their own capabilities and strengths:
+
+- [**OpenAI**](https://mastra.ai/reference/voice/openai) - High-accuracy transcription with Whisper models
+- [**Azure**](https://mastra.ai/reference/voice/azure) - Microsoft's speech recognition with enterprise-grade reliability
+- [**ElevenLabs**](https://mastra.ai/reference/voice/elevenlabs) - Advanced speech recognition with support for multiple languages
+- [**Google**](https://mastra.ai/reference/voice/google) - Google's speech recognition with extensive language support
+- [**Cloudflare**](https://mastra.ai/reference/voice/cloudflare) - Edge-optimized speech recognition for low-latency applications
+- [**Deepgram**](https://mastra.ai/reference/voice/deepgram) - AI-powered speech recognition with high accuracy for various accents
+- [**Sarvam**](https://mastra.ai/reference/voice/sarvam) - Specialized in Indic languages and accents
+
+Each provider is implemented as a separate package that you can install as needed:
+
+```bash
+pnpm add @mastra/voice-openai@latest  # Example for OpenAI
+```
+
+## Using the Listen Method
+
+The primary method for STT is the `listen()` method, which converts spoken audio into text. Here's how to use it:
+
+```typescript
+import { Agent } from '@mastra/core/agent'
+import { OpenAIVoice } from '@mastra/voice-openai'
+import { getMicrophoneStream } from '@mastra/node-audio'
+
+const voice = new OpenAIVoice()
+
+const agent = new Agent({
+  id: 'voice-agent',
+  name: 'Voice Agent',
+  instructions: 'You are a voice assistant that provides recommendations based on user input.',
+  model: 'openai/gpt-5.1',
+  voice,
+})
+
+const audioStream = getMicrophoneStream() // Assume this function gets audio input
+
+const transcript = await agent.voice.listen(audioStream, {
+  filetype: 'm4a', // Optional: specify the audio file type
+})
+
+console.log(`User said: ${transcript}`)
+
+const { text } = await agent.generate(
+  `Based on what the user said, provide them a recommendation: ${transcript}`,
+)
+
+console.log(`Recommendation: ${text}`)
+```
+
+Check out the [Adding Voice to Agents](https://mastra.ai/docs/agents/adding-voice) documentation to learn how to use STT in an agent.

package/dist/docs/references/docs-voice-text-to-speech.md

@@ -0,0 +1,83 @@
+# Text-to-Speech (TTS)
+
+Text-to-Speech (TTS) in Mastra offers a unified API for synthesizing spoken audio from text using various providers. By incorporating TTS into your applications, you can enhance user experience with natural voice interactions, improve accessibility for users with visual impairments, and create more engaging multimodal interfaces.
+
+TTS is a core component of any voice application. Combined with STT (Speech-to-Text), it forms the foundation of voice interaction systems. Newer models support STS ([Speech-to-Speech](https://mastra.ai/docs/voice/speech-to-speech)) which can be used for real-time interactions but come at high cost ($).
+
+## Configuration
+
+To use TTS in Mastra, you need to provide a `speechModel` when initializing the voice provider. This includes parameters such as:
+
+- **`name`**: The specific TTS model to use.
+- **`apiKey`**: Your API key for authentication.
+- **Provider-specific options**: Additional options that may be required or supported by the specific voice provider.
+
+The **`speaker`** option allows you to select different voices for speech synthesis. Each provider offers a variety of voice options with distinct characteristics for **Voice diversity**, **Quality**, **Voice personality**, and **Multilingual support**
+
+**Note**: All of these parameters are optional. You can use the default settings provided by the voice provider, which will depend on the specific provider you are using.
+
+```typescript
+const voice = new OpenAIVoice({
+  speechModel: {
+    name: 'tts-1-hd',
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+  speaker: 'alloy',
+})
+
+// If using default settings the configuration can be simplified to:
+const voice = new OpenAIVoice()
+```
+
+## Available Providers
+
+Mastra supports a wide range of Text-to-Speech providers, each with their own unique capabilities and voice options. You can choose the provider that best suits your application's needs:
+
+- [**OpenAI**](https://mastra.ai/reference/voice/openai) - High-quality voices with natural intonation and expression
+- [**Azure**](https://mastra.ai/reference/voice/azure) - Microsoft's speech service with a wide range of voices and languages
+- [**ElevenLabs**](https://mastra.ai/reference/voice/elevenlabs) - Ultra-realistic voices with emotion and fine-grained control
+- [**PlayAI**](https://mastra.ai/reference/voice/playai) - Specialized in natural-sounding voices with various styles
+- [**Google**](https://mastra.ai/reference/voice/google) - Google's speech synthesis with multilingual support
+- [**Cloudflare**](https://mastra.ai/reference/voice/cloudflare) - Edge-optimized speech synthesis for low-latency applications
+- [**Deepgram**](https://mastra.ai/reference/voice/deepgram) - AI-powered speech technology with high accuracy
+- [**Speechify**](https://mastra.ai/reference/voice/speechify) - Text-to-speech optimized for readability and accessibility
+- [**Sarvam**](https://mastra.ai/reference/voice/sarvam) - Specialized in Indic languages and accents
+- [**Murf**](https://mastra.ai/reference/voice/murf) - Studio-quality voice overs with customizable parameters
+
+Each provider is implemented as a separate package that you can install as needed:
+
+```bash
+pnpm add @mastra/voice-openai@latest  # Example for OpenAI
+```
+
+## Using the Speak Method
+
+The primary method for TTS is the `speak()` method, which converts text to speech. This method can accept options that allows you to specify the speaker and other provider-specific options. Here's how to use it:
+
+```typescript
+import { Agent } from '@mastra/core/agent'
+import { OpenAIVoice } from '@mastra/voice-openai'
+
+const voice = new OpenAIVoice()
+
+const agent = new Agent({
+  id: 'voice-agent',
+  name: 'Voice Agent',
+  instructions: 'You are a voice assistant that can help users with their tasks.',
+  model: 'openai/gpt-5.1',
+  voice,
+})
+
+const { text } = await agent.generate('What color is the sky?')
+
+// Convert text to speech to an Audio Stream
+const readableStream = await voice.speak(text, {
+  speaker: 'default', // Optional: specify a speaker
+  properties: {
+    speed: 1.0, // Optional: adjust speech speed
+    pitch: 'default', // Optional: specify pitch if supported
+  },
+})
+```
+
+Check out the [Adding Voice to Agents](https://mastra.ai/docs/agents/adding-voice) documentation to learn how to use TTS in an agent.

package/dist/docs/references/docs-workflows-agents-and-tools.md

@@ -0,0 +1,166 @@
+# Agents and Tools
+
+Workflow steps can call agents to leverage LLM reasoning or call tools for type-safe logic. You can either invoke them from within a step's `execute` function or compose them directly as steps using `createStep()`.
+
+## Using agents in workflows
+
+Use agents in workflow steps when you need reasoning, language generation, or other LLM-based tasks. Call from a step's `execute` function for more control over the agent call (e.g., track message history or return structured output). Compose agents as steps when you don't need to modify how the agent is invoked.
+
+### Calling agents
+
+Call agents inside a step's `execute` function using `.generate()` or `.stream()`. This lets you modify the agent call and handle the response before passing it to the next step.
+
+```typescript
+const step1 = createStep({
+  execute: async ({ inputData, mastra }) => {
+    const { message } = inputData
+
+    const testAgent = mastra.getAgent('testAgent')
+    const response = await testAgent.generate(
+      `Convert this message into bullet points: ${message}`,
+      {
+        memory: {
+          thread: 'user-123',
+          resource: 'test-123',
+        },
+      },
+    )
+
+    return {
+      list: response.text,
+    }
+  },
+})
+```
+
+### Agents as steps
+
+Compose an agent as a step using `createStep()` when you don't need to modify the agent call. Use `.map()` to transform the previous step's output into a `prompt` the agent can use.
+
+![Agent as step](/assets/images/workflows-agent-tools-agent-step-b2f5be22552ce514f7f8cd785ffc5604.jpg)
+
+```typescript
+import { testAgent } from '../agents/test-agent'
+
+const step1 = createStep(testAgent)
+
+export const testWorkflow = createWorkflow({})
+  .map(async ({ inputData }) => {
+    const { message } = inputData
+    return {
+      prompt: `Convert this message into bullet points: ${message}`,
+    }
+  })
+  .then(step1)
+  .then(step2)
+  .commit()
+```
+
+> **Info:** Visit [Input Data Mapping](https://mastra.ai/docs/workflows/control-flow) for more information.
+
+When no `structuredOutput` option is provided, Mastra agents use a default schema that expects a `prompt` string as input and returns a `text` string as output:
+
+```typescript
+{
+  inputSchema: {
+    prompt: string
+  },
+  outputSchema: {
+    text: string
+  }
+}
+```
+
+### Agents with structured output
+
+When you need the agent to return structured data instead of plain text, pass the `structuredOutput` option to `createStep()`. The step's output schema will match your provided schema, enabling type-safe chaining to subsequent steps.
+
+```typescript
+const articleSchema = z.object({
+  title: z.string(),
+  summary: z.string(),
+  tags: z.array(z.string()),
+})
+
+const agentStep = createStep(testAgent, {
+  structuredOutput: { schema: articleSchema },
+})
+
+// Next step receives typed structured data
+const processStep = createStep({
+  id: 'process',
+  inputSchema: articleSchema, // Matches agent's outputSchema
+  outputSchema: z.object({ tagCount: z.number() }),
+  execute: async ({ inputData }) => ({
+    tagCount: inputData.tags.length, // Fully typed
+  }),
+})
+
+export const testWorkflow = createWorkflow({})
+  .map(async ({ inputData }) => ({
+    prompt: `Generate an article about: ${inputData.topic}`,
+  }))
+  .then(agentStep)
+  .then(processStep)
+  .commit()
+```
+
+The `structuredOutput.schema` option accepts any Zod schema. The agent will generate output conforming to this schema, and the step's `outputSchema` will be automatically set to match.
+
+> **Info:** Visit [Structured Output](https://mastra.ai/docs/agents/structured-output) for more options like error handling strategies and streaming with structured output.
+
+## Using tools in workflows
+
+Use tools in workflow steps to leverage existing tool logic. Call from a step's `execute` function when you need to prepare context or process responses. Compose tools as steps when you don't need to modify how the tool is used.
+
+### Calling tools
+
+Call tools inside a step's `execute` function using `.execute()`. This gives you more control over the tool's input context, or process its response before passing it to the next step.
+
+```typescript
+import { testTool } from '../tools/test-tool'
+
+const step2 = createStep({
+  execute: async ({ inputData, requestContext }) => {
+    const { formatted } = inputData
+
+    const response = await testTool.execute({ text: formatted }, { requestContext })
+
+    return {
+      emphasized: response.emphasized,
+    }
+  },
+})
+```
+
+> **Info:** Visit [Calling Tools](https://mastra.ai/docs/workflows/agents-and-tools) for more examples.
+
+### Tools as steps
+
+Compose a tool as a step using `createStep()` when the previous step's output matches the tool's input context. You can use `.map()` to transform the previous step's output if they don't.
+
+![Tool as step](/assets/images/workflows-agent-tools-tool-step-cfd56227ce83c2d03a8c8d0496faeeef.jpg)
+
+```typescript
+import { testTool } from '../tools/test-tool'
+
+const step2 = createStep(testTool)
+
+export const testWorkflow = createWorkflow({})
+  .then(step1)
+  .map(async ({ inputData }) => {
+    const { formatted } = inputData
+    return {
+      text: formatted,
+    }
+  })
+  .then(step2)
+  .commit()
+```
+
+> **Info:** Visit [Input Data Mapping](https://mastra.ai/docs/workflows/control-flow) for more information.
+
+## Related
+
+- [Using Agents](https://mastra.ai/docs/agents/overview)
+- [MCP Overview](https://mastra.ai/docs/mcp/overview)