@mastra/voice-openai 0.12.0-beta.1 → 0.12.0

package/dist/docs/voice/02-text-to-speech.md

@@ -0,0 +1,87 @@
+> Overview of Text-to-Speech capabilities in Mastra, including configuration, usage, and integration with voice providers.
+
+# Text-to-Speech (TTS)
+
+Text-to-Speech (TTS) in Mastra offers a unified API for synthesizing spoken audio from text using various providers.
+By incorporating TTS into your applications, you can enhance user experience with natural voice interactions, improve accessibility for users with visual impairments, and create more engaging multimodal interfaces.
+
+TTS is a core component of any voice application. Combined with STT (Speech-to-Text), it forms the foundation of voice interaction systems. Newer models support STS ([Speech-to-Speech](./speech-to-speech)) which can be used for real-time interactions but come at high cost ($).
+
+## Configuration
+
+To use TTS in Mastra, you need to provide a `speechModel` when initializing the voice provider. This includes parameters such as:
+
+- **`name`**: The specific TTS model to use.
+- **`apiKey`**: Your API key for authentication.
+- **Provider-specific options**: Additional options that may be required or supported by the specific voice provider.
+
+The **`speaker`** option allows you to select different voices for speech synthesis. Each provider offers a variety of voice options with distinct characteristics for **Voice diversity**, **Quality**, **Voice personality**, and **Multilingual support**
+
+**Note**: All of these parameters are optional. You can use the default settings provided by the voice provider, which will depend on the specific provider you are using.
+
+```typescript
+const voice = new OpenAIVoice({
+  speechModel: {
+    name: "tts-1-hd",
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+  speaker: "alloy",
+});
+
+// If using default settings the configuration can be simplified to:
+const voice = new OpenAIVoice();
+```
+
+## Available Providers
+
+Mastra supports a wide range of Text-to-Speech providers, each with their own unique capabilities and voice options. You can choose the provider that best suits your application's needs:
+
+- [**OpenAI**](https://mastra.ai/reference/v1/voice/openai/) - High-quality voices with natural intonation and expression
+- [**Azure**](https://mastra.ai/reference/v1/voice/azure/) - Microsoft's speech service with a wide range of voices and languages
+- [**ElevenLabs**](https://mastra.ai/reference/v1/voice/elevenlabs/) - Ultra-realistic voices with emotion and fine-grained control
+- [**PlayAI**](https://mastra.ai/reference/v1/voice/playai/) - Specialized in natural-sounding voices with various styles
+- [**Google**](https://mastra.ai/reference/v1/voice/google/) - Google's speech synthesis with multilingual support
+- [**Cloudflare**](https://mastra.ai/reference/v1/voice/cloudflare/) - Edge-optimized speech synthesis for low-latency applications
+- [**Deepgram**](https://mastra.ai/reference/v1/voice/deepgram/) - AI-powered speech technology with high accuracy
+- [**Speechify**](https://mastra.ai/reference/v1/voice/speechify/) - Text-to-speech optimized for readability and accessibility
+- [**Sarvam**](https://mastra.ai/reference/v1/voice/sarvam/) - Specialized in Indic languages and accents
+- [**Murf**](https://mastra.ai/reference/v1/voice/murf/) - Studio-quality voice overs with customizable parameters
+
+Each provider is implemented as a separate package that you can install as needed:
+
+```bash
+pnpm add @mastra/voice-openai@beta  # Example for OpenAI
+```
+
+## Using the Speak Method
+
+The primary method for TTS is the `speak()` method, which converts text to speech. This method can accept options that allows you to specify the speaker and other provider-specific options. Here's how to use it:
+
+```typescript
+import { Agent } from "@mastra/core/agent";
+import { OpenAIVoice } from "@mastra/voice-openai";
+
+const voice = new OpenAIVoice();
+
+const agent = new Agent({
+  id: "voice-agent",
+  name: "Voice Agent",
+  instructions:
+    "You are a voice assistant that can help users with their tasks.",
+  model: "openai/gpt-5.1",
+  voice,
+});
+
+const { text } = await agent.generate("What color is the sky?");
+
+// Convert text to speech to an Audio Stream
+const readableStream = await voice.speak(text, {
+  speaker: "default", // Optional: specify a speaker
+  properties: {
+    speed: 1.0, // Optional: adjust speech speed
+    pitch: "default", // Optional: specify pitch if supported
+  },
+});
+```
+
+Check out the [Adding Voice to Agents](https://mastra.ai/docs/v1/agents/adding-voice) documentation to learn how to use TTS in an agent.

package/dist/docs/voice/03-speech-to-text.md

@@ -0,0 +1,83 @@
+> Overview of Speech-to-Text capabilities in Mastra, including configuration, usage, and integration with voice providers.
+
+# Speech-to-Text (STT)
+
+Speech-to-Text (STT) in Mastra provides a standardized interface for converting audio input into text across multiple service providers.
+STT helps create voice-enabled applications that can respond to human speech, enabling hands-free interaction, accessibility for users with disabilities, and more natural human-computer interfaces.
+
+## Configuration
+
+To use STT in Mastra, you need to provide a `listeningModel` when initializing the voice provider. This includes parameters such as:
+
+- **`name`**: The specific STT model to use.
+- **`apiKey`**: Your API key for authentication.
+- **Provider-specific options**: Additional options that may be required or supported by the specific voice provider.
+
+**Note**: All of these parameters are optional. You can use the default settings provided by the voice provider, which will depend on the specific provider you are using.
+
+```typescript
+const voice = new OpenAIVoice({
+  listeningModel: {
+    name: "whisper-1",
+    apiKey: process.env.OPENAI_API_KEY,
+  },
+});
+
+// If using default settings the configuration can be simplified to:
+const voice = new OpenAIVoice();
+```
+
+## Available Providers
+
+Mastra supports several Speech-to-Text providers, each with their own capabilities and strengths:
+
+- [**OpenAI**](https://mastra.ai/reference/v1/voice/openai/) - High-accuracy transcription with Whisper models
+- [**Azure**](https://mastra.ai/reference/v1/voice/azure/) - Microsoft's speech recognition with enterprise-grade reliability
+- [**ElevenLabs**](https://mastra.ai/reference/v1/voice/elevenlabs/) - Advanced speech recognition with support for multiple languages
+- [**Google**](https://mastra.ai/reference/v1/voice/google/) - Google's speech recognition with extensive language support
+- [**Cloudflare**](https://mastra.ai/reference/v1/voice/cloudflare/) - Edge-optimized speech recognition for low-latency applications
+- [**Deepgram**](https://mastra.ai/reference/v1/voice/deepgram/) - AI-powered speech recognition with high accuracy for various accents
+- [**Sarvam**](https://mastra.ai/reference/v1/voice/sarvam/) - Specialized in Indic languages and accents
+
+Each provider is implemented as a separate package that you can install as needed:
+
+```bash
+pnpm add @mastra/voice-openai@beta  # Example for OpenAI
+```
+
+## Using the Listen Method
+
+The primary method for STT is the `listen()` method, which converts spoken audio into text. Here's how to use it:
+
+```typescript
+import { Agent } from "@mastra/core/agent";
+import { OpenAIVoice } from "@mastra/voice-openai";
+import { getMicrophoneStream } from "@mastra/node-audio";
+
+const voice = new OpenAIVoice();
+
+const agent = new Agent({
+  id: "voice-agent",
+  name: "Voice Agent",
+  instructions:
+    "You are a voice assistant that provides recommendations based on user input.",
+  model: "openai/gpt-5.1",
+  voice,
+});
+
+const audioStream = getMicrophoneStream(); // Assume this function gets audio input
+
+const transcript = await agent.voice.listen(audioStream, {
+  filetype: "m4a", // Optional: specify the audio file type
+});
+
+console.log(`User said: ${transcript}`);
+
+const { text } = await agent.generate(
+  `Based on what the user said, provide them a recommendation: ${transcript}`,
+);
+
+console.log(`Recommendation: ${text}`);
+```
+
+Check out the [Adding Voice to Agents](https://mastra.ai/docs/v1/agents/adding-voice) documentation to learn how to use STT in an agent.