opencode-skills-collection 1.0.186 → 1.0.187

package/bundled-skills/voice-ai-development/SKILL.md

@@ -1,13 +1,21 @@
 ---
 name: voice-ai-development
-description: "You are an expert in building real-time voice applications. You think in terms of latency budgets, audio quality, and user experience. You know that voice apps feel magical when fast and broken when slow."
+description: Expert in building voice AI applications - from real-time voice
+  agents to voice-enabled apps. Covers OpenAI Realtime API, Vapi for voice
+  agents, Deepgram for transcription, ElevenLabs for synthesis, LiveKit for
+  real-time infrastructure, and WebRTC fundamentals.
 risk: unknown
-source: "vibeship-spawner-skills (Apache 2.0)"
-date_added: "2026-02-27"
+source: vibeship-spawner-skills (Apache 2.0)
+date_added: 2026-02-27
 ---
 
 # Voice AI Development
 
+Expert in building voice AI applications - from real-time voice agents to voice-enabled apps.
+Covers OpenAI Realtime API, Vapi for voice agents, Deepgram for transcription, ElevenLabs
+for synthesis, LiveKit for real-time infrastructure, and WebRTC fundamentals. Knows how to
+build low-latency, production-ready voice experiences.
+
 **Role**: Voice AI Architect
 
 You are an expert in building real-time voice applications. You think in terms of
@@ -15,6 +23,14 @@ latency budgets, audio quality, and user experience. You know that voice apps fe
 magical when fast and broken when slow. You choose the right combination of providers
 for each use case and optimize relentlessly for perceived responsiveness.
 
+### Expertise
+
+- Real-time audio streaming
+- Voice agent architecture
+- Provider selection
+- Latency optimization
+- Audio quality tuning
+
 ## Capabilities
 
 - OpenAI Realtime API
@@ -26,11 +42,47 @@ for each use case and optimize relentlessly for perceived responsiveness.
 - Voice agent design
 - Latency optimization
 
-## Requirements
+## Prerequisites
+
+- 0: Async programming
+- 1: WebSocket basics
+- 2: Audio concepts (sample rate, codec)
+- Required skills: Python or Node.js, API keys for providers, Audio handling knowledge
+
+## Scope
+
+- 0: Latency varies by provider
+- 1: Cost per minute adds up
+- 2: Quality depends on network
+- 3: Complex debugging
+
+## Ecosystem
+
+### Primary
+
+- OpenAI Realtime API
+- Vapi
+- Deepgram
+- ElevenLabs
+
+### Infrastructure
 
-- Python or Node.js
-- API keys for providers
-- Audio handling knowledge
+- LiveKit
+- Daily.co
+- Twilio
+
+### Common_integrations
+
+- WebRTC
+- WebSockets
+- Telephony (SIP/PSTN)
+
+### Platforms
+
+- Web applications
+- Mobile apps
+- Call centers
+- Voice assistants
 
 ## Patterns
 
@@ -40,7 +92,6 @@ Native voice-to-voice with GPT-4o
 
 **When to use**: When you want integrated voice AI without separate STT/TTS
 
-```python
 import asyncio
 import websockets
 import json
@@ -100,8 +151,30 @@ async def voice_session():
         async for message in ws:
             event = json.loads(message)
 
-            if event["type"] == "resp
-```
+            if event["type"] == "response.audio.delta":
+                # Play audio chunk
+                audio = base64.b64decode(event["delta"])
+                play_audio(audio)
+
+            elif event["type"] == "response.audio_transcript.done":
+                print(f"Assistant said: {event['transcript']}")
+
+            elif event["type"] == "input_audio_buffer.speech_started":
+                print("User started speaking")
+
+            elif event["type"] == "response.function_call_arguments.done":
+                # Handle tool call
+                name = event["name"]
+                args = json.loads(event["arguments"])
+                result = call_function(name, args)
+                await ws.send(json.dumps({
+                    "type": "conversation.item.create",
+                    "item": {
+                        "type": "function_call_output",
+                        "call_id": event["call_id"],
+                        "output": json.dumps(result)
+                    }
+                }))
 
 ### Vapi Voice Agent
 
@@ -109,7 +182,6 @@ Build voice agents with Vapi platform
 
 **When to use**: Phone-based agents, quick deployment
 
-```python
 # Vapi provides hosted voice agents with webhooks
 
 from flask import Flask, request, jsonify
@@ -180,7 +252,6 @@ web_call = client.calls.create(
     type="web"
 )
 # Returns URL for WebRTC connection
-```
 
 ### Deepgram STT + ElevenLabs TTS
 
@@ -188,7 +259,6 @@ Best-in-class transcription and synthesis
 
 **When to use**: High quality voice, custom pipeline
 
-```python
 import asyncio
 from deepgram import DeepgramClient, LiveTranscriptionEvents
 from elevenlabs import ElevenLabs
@@ -254,54 +324,313 @@ async def tts_websocket(text_stream):
         # Flush remaining audio
         final_audio = await tts.flush()
         yield final_audio
-```
 
-## Anti-Patterns
+### LiveKit Real-time Infrastructure
+
+WebRTC infrastructure for voice apps
+
+**When to use**: Building custom real-time voice apps
+
+from livekit import api, rtc
+import asyncio
+
+# Server-side: Create room and tokens
+lk_api = api.LiveKitAPI(
+    url="wss://your-livekit.livekit.cloud",
+    api_key="...",
+    api_secret="..."
+)
+
+async def create_room(room_name: str):
+    room = await lk_api.room.create_room(
+        api.CreateRoomRequest(name=room_name)
+    )
+    return room
+
+def create_token(room_name: str, participant_name: str):
+    token = api.AccessToken(
+        api_key="...",
+        api_secret="..."
+    )
+    token.with_identity(participant_name)
+    token.with_grants(api.VideoGrants(
+        room_join=True,
+        room=room_name
+    ))
+    return token.to_jwt()
+
+# Agent-side: Connect and process audio
+async def voice_agent(room_name: str):
+    room = rtc.Room()
+
+    @room.on("track_subscribed")
+    def on_track(track, publication, participant):
+        if track.kind == rtc.TrackKind.KIND_AUDIO:
+            # Process incoming audio
+            audio_stream = rtc.AudioStream(track)
+            asyncio.create_task(process_audio(audio_stream))
+
+    token = create_token(room_name, "agent")
+    await room.connect("wss://your-livekit.livekit.cloud", token)
+
+    # Publish agent's audio
+    source = rtc.AudioSource(sample_rate=24000, num_channels=1)
+    track = rtc.LocalAudioTrack.create_audio_track("agent-voice", source)
+    await room.local_participant.publish_track(track)
+
+    # Send audio from TTS
+    async def speak(text: str):
+        for audio_chunk in text_to_speech(text):
+            await source.capture_frame(rtc.AudioFrame(
+                data=audio_chunk,
+                sample_rate=24000,
+                num_channels=1,
+                samples_per_channel=len(audio_chunk) // 2
+            ))
+
+    return room, speak
+
+# Process audio with STT
+async def process_audio(audio_stream):
+    async for frame in audio_stream:
+        # Send to Deepgram or other STT
+        await transcriber.send(frame.data)
+
+### Full Voice Agent Pipeline
+
+Complete voice agent with all components
+
+**When to use**: Custom production voice agent
+
+import asyncio
+from dataclasses import dataclass
+from typing import AsyncIterator
+
+@dataclass
+class VoiceAgentConfig:
+    stt_provider: str = "deepgram"
+    tts_provider: str = "elevenlabs"
+    llm_provider: str = "openai"
+    vad_enabled: bool = True
+    interrupt_enabled: bool = True
+
+class VoiceAgent:
+    def __init__(self, config: VoiceAgentConfig):
+        self.config = config
+        self.is_speaking = False
+        self.conversation_history = []
+
+    async def process_audio_stream(
+        self,
+        audio_in: AsyncIterator[bytes],
+        audio_out: asyncio.Queue
+    ):
+        """Main audio processing loop."""
+
+        # STT streaming
+        async def transcribe():
+            transcript_buffer = ""
+            async for audio_chunk in audio_in:
+                # Check for interruption
+                if self.is_speaking and self.config.interrupt_enabled:
+                    if await self.detect_speech(audio_chunk):
+                        await self.stop_speaking()
+
+                result = await self.stt.transcribe(audio_chunk)
+                if result.is_final:
+                    yield result.transcript
+
+        # Process transcripts
+        async for user_text in transcribe():
+            if not user_text.strip():
+                continue
+
+            self.conversation_history.append({
+                "role": "user",
+                "content": user_text
+            })
+
+            # Generate response with streaming
+            self.is_speaking = True
+            async for audio_chunk in self.generate_response(user_text):
+                await audio_out.put(audio_chunk)
+            self.is_speaking = False
+
+    async def generate_response(self, text: str) -> AsyncIterator[bytes]:
+        """Stream LLM response through TTS."""
+
+        # Stream LLM tokens
+        llm_stream = self.llm.stream_chat(self.conversation_history)
+
+        # Buffer for TTS (need ~50 chars for good prosody)
+        text_buffer = ""
+        full_response = ""
+
+        async for token in llm_stream:
+            text_buffer += token
+            full_response += token
+
+            # Send to TTS when we have enough text
+            if len(text_buffer) > 50 or token in ".!?":
+                async for audio in self.tts.synthesize_stream(text_buffer):
+                    yield audio
+                text_buffer = ""
+
+        # Flush remaining
+        if text_buffer:
+            async for audio in self.tts.synthesize_stream(text_buffer):
+                yield audio
+
+        self.conversation_history.append({
+            "role": "assistant",
+            "content": full_response
+        })
+
+    async def detect_speech(self, audio: bytes) -> bool:
+        """Voice activity detection."""
+        # Use WebRTC VAD or Silero VAD
+        return self.vad.is_speech(audio)
+
+    async def stop_speaking(self):
+        """Handle interruption."""
+        self.is_speaking = False
+        # Clear audio queue
+        # Stop TTS generation
 
-### ❌ Non-streaming Pipeline
+# Latency optimization tips:
+# 1. Use streaming everywhere (STT, LLM, TTS)
+# 2. Start TTS before LLM finishes (~50 char buffer)
+# 3. Use PCM audio format (no encoding overhead)
+# 4. Keep WebSocket connections alive
+# 5. Use regional endpoints close to users
 
-**Why bad**: Adds seconds of latency.
-User perceives as slow.
-Loses conversation flow.
+## Validation Checks
 
-**Instead**: Stream everything:
-- STT: interim results
-- LLM: token streaming
-- TTS: chunk streaming
-Start TTS before LLM finishes.
+### Non-Streaming TTS
 
-### ❌ Ignoring Interruptions
+Severity: HIGH
 
-**Why bad**: Frustrating user experience.
-Feels like talking to a machine.
-Wastes time.
+Message: Non-streaming TTS adds significant latency.
 
-**Instead**: Implement barge-in detection.
-Use VAD to detect user speech.
-Stop TTS immediately.
-Clear audio queue.
+Fix action: Use tts.synthesize_stream() or tts.convert_as_stream()
 
-### ❌ Single Provider Lock-in
+### Hardcoded Sample Rate
 
-**Why bad**: May not be best quality.
-Single point of failure.
-Harder to optimize.
+Severity: MEDIUM
 
-**Instead**: Mix best providers:
-- Deepgram for STT (speed + accuracy)
-- ElevenLabs for TTS (voice quality)
-- OpenAI/Anthropic for LLM
+Message: Hardcoded sample rate may cause format mismatches.
 
-## Limitations
+Fix action: Define sample rates as constants, document expected formats
 
-- Latency varies by provider
-- Cost per minute adds up
-- Quality depends on network
-- Complex debugging
+### WebSocket Without Reconnection
+
+Severity: HIGH
+
+Message: WebSocket connections need reconnection logic.
+
+Fix action: Add retry loop with exponential backoff
+
+### Missing VAD Configuration
+
+Severity: MEDIUM
+
+Message: VAD needs tuning for good user experience.
+
+Fix action: Configure threshold and silence_duration_ms
+
+### Blocking Audio Processing
+
+Severity: HIGH
+
+Message: Audio processing should be async to avoid blocking.
+
+Fix action: Use async def and await for audio operations
+
+### Missing Interruption Handling
+
+Severity: MEDIUM
+
+Message: Voice agents should handle user interruptions.
+
+Fix action: Add barge-in detection and cancel current response
+
+### Audio Queue Without Clear
+
+Severity: LOW
+
+Message: Audio queues should be clearable for interruptions.
+
+Fix action: Add method to clear queue on interruption
+
+### WebSocket Without Error Handling
+
+Severity: HIGH
+
+Message: WebSocket operations need error handling.
+
+Fix action: Wrap in try/except for ConnectionClosed
+
+## Collaboration
+
+### Delegation Triggers
+
+- agent graph|workflow|state -> langgraph (Need complex agent logic behind voice)
+- extract|structured|json -> structured-output (Need to extract structured data from voice)
+- observability|tracing|monitoring -> langfuse (Need to monitor voice agent quality)
+- frontend|web|react -> nextjs-app-router (Need web interface for voice agent)
+
+### Intelligent Voice Agent
+
+Skills: voice-ai-development, langgraph, structured-output
+
+Workflow:
+
+```
+1. Design agent graph with tools
+2. Add voice interface layer
+3. Use structured output for tool responses
+4. Optimize for voice latency
+```
+
+### Monitored Voice Agent
+
+Skills: voice-ai-development, langfuse
+
+Workflow:
+
+```
+1. Build voice agent with provider of choice
+2. Add Langfuse callbacks
+3. Track latency, quality, conversation flow
+4. Iterate based on metrics
+```
+
+### Phone-based Agent
+
+Skills: voice-ai-development, twilio
+
+Workflow:
+
+```
+1. Set up Vapi or custom agent
+2. Connect to Twilio for PSTN
+3. Handle inbound/outbound calls
+4. Implement call routing logic
+```
 
 ## Related Skills
 
 Works well with: `langgraph`, `structured-output`, `langfuse`
 
 ## When to Use
-This skill is applicable to execute the workflow or actions described in the overview.
+
+- User mentions or implies: voice ai
+- User mentions or implies: voice agent
+- User mentions or implies: speech to text
+- User mentions or implies: text to speech
+- User mentions or implies: realtime voice
+- User mentions or implies: vapi
+- User mentions or implies: deepgram
+- User mentions or implies: elevenlabs
+- User mentions or implies: livekit
+- User mentions or implies: openai realtime