@mastra/voice-openai-realtime 0.11.12 → 0.12.0-beta.1

package/dist/docs/voice/02-speech-to-speech.md

@@ -0,0 +1,106 @@
+> Overview of speech-to-speech capabilities in Mastra, including real-time interactions and event-driven architecture.
+
+# Speech-to-Speech Capabilities in Mastra
+
+## Introduction
+
+Speech-to-Speech (STS) in Mastra provides a standardized interface for real-time interactions across multiple providers.
+STS enables continuous bidirectional audio communication through listening to events from Realtime models. Unlike separate TTS and STT operations, STS maintains an open connection that processes speech continuously in both directions.
+
+## Configuration
+
+- **`apiKey`**: Your OpenAI API key. Falls back to the `OPENAI_API_KEY` environment variable.
+- **`model`**: The model ID to use for real-time voice interactions (e.g., `gpt-5.1-realtime`).
+- **`speaker`**: The default voice ID for speech synthesis. This allows you to specify which voice to use for the speech output.
+
+```typescript
+const voice = new OpenAIRealtimeVoice({
+  apiKey: "your-openai-api-key",
+  model: "gpt-5.1-realtime",
+  speaker: "alloy", // Default voice
+});
+
+// If using default settings the configuration can be simplified to:
+const voice = new OpenAIRealtimeVoice();
+```
+
+## Using STS
+
+```typescript
+import { Agent } from "@mastra/core/agent";
+import { OpenAIRealtimeVoice } from "@mastra/voice-openai-realtime";
+import { playAudio, getMicrophoneStream } from "@mastra/node-audio";
+
+const agent = new Agent({
+  id: "agent",
+  name: "OpenAI Realtime Agent",
+  instructions: `You are a helpful assistant with real-time voice capabilities.`,
+  model: "openai/gpt-5.1",
+  voice: new OpenAIRealtimeVoice(),
+});
+
+// Connect to the voice service
+await agent.voice.connect();
+
+// Listen for agent audio responses
+agent.voice.on("speaker", ({ audio }) => {
+  playAudio(audio);
+});
+
+// Initiate the conversation
+await agent.voice.speak("How can I help you today?");
+
+// Send continuous audio from the microphone
+const micStream = getMicrophoneStream();
+await agent.voice.send(micStream);
+```
+
+For integrating Speech-to-Speech capabilities with agents, refer to the [Adding Voice to Agents](https://mastra.ai/docs/v1/agents/adding-voice) documentation.
+
+## Google Gemini Live (Realtime)
+
+```typescript
+import { Agent } from "@mastra/core/agent";
+import { GeminiLiveVoice } from "@mastra/voice-google-gemini-live";
+import { playAudio, getMicrophoneStream } from "@mastra/node-audio";
+
+const agent = new Agent({
+  id: "agent",
+  name: "Gemini Live Agent",
+  instructions:
+    "You are a helpful assistant with real-time voice capabilities.",
+  // Model used for text generation; voice provider handles realtime audio
+  model: "openai/gpt-5.1",
+  voice: new GeminiLiveVoice({
+    apiKey: process.env.GOOGLE_API_KEY,
+    model: "gemini-2.0-flash-exp",
+    speaker: "Puck",
+    debug: true,
+    // Vertex AI option:
+    // vertexAI: true,
+    // project: 'your-gcp-project',
+    // location: 'us-central1',
+    // serviceAccountKeyFile: '/path/to/service-account.json',
+  }),
+});
+
+await agent.voice.connect();
+
+agent.voice.on("speaker", ({ audio }) => {
+  playAudio(audio);
+});
+
+agent.voice.on("writing", ({ role, text }) => {
+  console.log(`${role}: ${text}`);
+});
+
+await agent.voice.speak("How can I help you today?");
+
+const micStream = getMicrophoneStream();
+await agent.voice.send(micStream);
+```
+
+Note:
+
+- Live API requires `GOOGLE_API_KEY`. Vertex AI requires project/location and service account credentials.
+- Events: `speaker` (audio stream), `writing` (text), `turnComplete`, `usage`, and `error`.