opencode-skills-collection 1.0.185 → 1.0.187

package/bundled-skills/voice-agents/SKILL.md

@@ -1,22 +1,36 @@
 ---
 name: voice-agents
-description: "You are a voice AI architect who has shipped production voice agents handling millions of calls. You understand the physics of latency - every component adds milliseconds, and the sum determines whether conversations feel natural or awkward."
+description: Voice agents represent the frontier of AI interaction - humans
+  speaking naturally with AI systems.
 risk: safe
-source: "vibeship-spawner-skills (Apache 2.0)"
-date_added: "2026-02-27"
+source: vibeship-spawner-skills (Apache 2.0)
+date_added: 2026-02-27
 ---
 
 # Voice Agents
 
-You are a voice AI architect who has shipped production voice agents handling
-millions of calls. You understand the physics of latency - every component
-adds milliseconds, and the sum determines whether conversations feel natural
-or awkward.
+Voice agents represent the frontier of AI interaction - humans speaking
+naturally with AI systems. The challenge isn't just speech recognition
+and synthesis, it's achieving natural conversation flow with sub-800ms
+latency while handling interruptions, background noise, and emotional
+nuance.
 
-Your core insight: Two architectures exist. Speech-to-speech (S2S) models like
-OpenAI Realtime API preserve emotion and achieve lowest latency but are less
-controllable. Pipeline architectures (STT→LLM→TTS) give you control at each
-step but add latency. Mos
+This skill covers two architectures: speech-to-speech (OpenAI Realtime API,
+lowest latency, most natural) and pipeline (STT→LLM→TTS, more control,
+easier to debug). Key insight: latency is the constraint. Humans expect
+responses in 500ms. Every millisecond matters.
+
+84% of organizations are increasing voice AI budgets in 2025. This is the
+year voice agents go mainstream.
+
+## Principles
+
+- Latency is the constraint - target <800ms end-to-end
+- Jitter (variance) matters as much as absolute latency
+- VAD quality determines conversation flow
+- Interruption handling makes or breaks the experience
+- Start with focused MVP, iterate based on real conversations
+- Combine best-in-class components (Deepgram STT + ElevenLabs TTS)
 
 ## Capabilities
 
@@ -30,44 +44,940 @@ step but add latency. Mos
 - barge-in-detection
 - voice-interfaces
 
+## Scope
+
+- phone-system-integration → backend
+- audio-processing-dsp → audio-specialist
+- music-generation → audio-specialist
+- accessibility-compliance → accessibility-specialist
+
+## Tooling
+
+### Speech_to_speech
+
+- OpenAI Realtime API - When: Lowest latency, most natural conversation Note: gpt-4o-realtime-preview, native voice, sub-500ms
+- Pipecat - When: Open-source voice orchestration Note: Daily-backed, enterprise-grade, modular
+
+### Speech_to_text
+
+- OpenAI Whisper - When: Highest accuracy, multilingual Note: gpt-4o-transcribe for best results
+- Deepgram Nova-3 - When: Production workloads, 54% lower WER Note: 150-184ms TTFT, 90%+ accuracy on noisy audio
+- AssemblyAI - When: Real-time streaming, speaker diarization Note: Good accuracy-latency balance
+
+### Text_to_speech
+
+- ElevenLabs - When: Most natural voice, emotional control Note: Flash model 75ms latency, V3 for expression
+- OpenAI TTS - When: Integrated with OpenAI stack Note: gpt-4o-mini-tts, 13 voices, streaming
+- Deepgram Aura-2 - When: Cost-effective production TTS Note: 40% cheaper than ElevenLabs, 184ms TTFB
+
+### Frameworks
+
+- Pipecat - When: Open-source voice agent orchestration Note: Silero VAD, SmartTurn, interruption handling
+- Vapi - When: Managed voice agent platform Note: No infrastructure management
+- Retell AI - When: Low-latency voice agents Note: Best context preservation on interruption
+
 ## Patterns
 
 ### Speech-to-Speech Architecture
 
 Direct audio-to-audio processing for lowest latency
 
+**When to use**: Maximum naturalness, emotional preservation, real-time conversation
+
+# SPEECH-TO-SPEECH ARCHITECTURE:
+
+"""
+[User Audio] → [S2S Model] → [Agent Audio]
+
+Advantages:
+- Lowest latency (sub-500ms)
+- Preserves emotion, emphasis, accents
+- Most natural conversation flow
+
+Disadvantages:
+- Less control over responses
+- Harder to debug/audit
+- Can't easily modify what's said
+"""
+
+## OpenAI Realtime API
+"""
+import { RealtimeClient } from '@openai/realtime-api-beta';
+
+const client = new RealtimeClient({
+  apiKey: process.env.OPENAI_API_KEY,
+});
+
+// Configure for voice conversation
+client.updateSession({
+  modalities: ['text', 'audio'],
+  voice: 'alloy',
+  input_audio_format: 'pcm16',
+  output_audio_format: 'pcm16',
+  instructions: `You are a helpful customer service agent.
+    Be concise and friendly. If you don't know something,
+    say so rather than making things up.`,
+  turn_detection: {
+    type: 'server_vad',  // or 'semantic_vad'
+    threshold: 0.5,
+    prefix_padding_ms: 300,
+    silence_duration_ms: 500,
+  },
+});
+
+// Handle audio streams
+client.on('conversation.item.input_audio_transcription', (event) => {
+  console.log('User said:', event.transcript);
+});
+
+client.on('response.audio.delta', (event) => {
+  // Stream audio to speaker
+  audioPlayer.write(Buffer.from(event.delta, 'base64'));
+});
+
+// Send user audio
+client.appendInputAudio(audioBuffer);
+"""
+
+## Use Cases:
+- Real-time customer support
+- Voice assistants
+- Interactive voice response (IVR)
+- Live language translation
+
 ### Pipeline Architecture
 
 Separate STT → LLM → TTS for maximum control
 
+**When to use**: Need to know/control exactly what's said, debugging, compliance
+
+# PIPELINE ARCHITECTURE:
+
+"""
+[Audio] → [STT] → [Text] → [LLM] → [Text] → [TTS] → [Audio]
+
+Advantages:
+- Full control at each step
+- Can log/audit all text
+- Easier to debug
+- Mix best-in-class components
+
+Disadvantages:
+- Higher latency (700-1200ms typical)
+- Loses some emotion/nuance
+- More components to manage
+"""
+
+## Production Pipeline Example
+"""
+import { Deepgram } from '@deepgram/sdk';
+import { ElevenLabsClient } from 'elevenlabs';
+import OpenAI from 'openai';
+
+// Initialize clients
+const deepgram = new Deepgram(process.env.DEEPGRAM_API_KEY);
+const elevenlabs = new ElevenLabsClient();
+const openai = new OpenAI();
+
+async function processVoiceInput(audioStream) {
+  // 1. Speech-to-Text (Deepgram Nova-3)
+  const transcription = await deepgram.transcription.live({
+    model: 'nova-3',
+    punctuate: true,
+    endpointing: 300,  // ms of silence before end
+  });
+
+  transcription.on('transcript', async (data) => {
+    if (data.is_final && data.speech_final) {
+      const userText = data.channel.alternatives[0].transcript;
+      console.log('User:', userText);
+
+      // 2. LLM Processing
+      const completion = await openai.chat.completions.create({
+        model: 'gpt-4o-mini',
+        messages: [
+          { role: 'system', content: 'You are a concise voice assistant.' },
+          { role: 'user', content: userText }
+        ],
+        max_tokens: 150,  // Keep responses short for voice
+      });
+
+      const agentText = completion.choices[0].message.content;
+      console.log('Agent:', agentText);
+
+      // 3. Text-to-Speech (ElevenLabs)
+      const audioStream = await elevenlabs.textToSpeech.stream({
+        voice_id: 'voice_id_here',
+        text: agentText,
+        model_id: 'eleven_flash_v2_5',  // Lowest latency
+      });
+
+      // Stream to user
+      playAudioStream(audioStream);
+    }
+  });
+
+  // Pipe audio to transcription
+  audioStream.pipe(transcription);
+}
+"""
+
+## Optimization Tips:
+- Start TTS while LLM still generating (streaming)
+- Pre-compute first response segment during user speech
+- Use Flash/turbo models for latency
+
 ### Voice Activity Detection Pattern
 
 Detect when user starts/stops speaking
 
-## Anti-Patterns
+**When to use**: All voice agents need VAD for turn-taking
+
+# VOICE ACTIVITY DETECTION (VAD):
+
+"""
+VAD Types:
+1. Energy-based: Simple, fast, noise-sensitive
+2. Model-based: Silero VAD, more accurate
+3. Semantic VAD: Understands meaning, best for conversation
+"""
+
+## Silero VAD (Popular Open Source)
+"""
+import { SileroVAD } from '@pipecat-ai/silero-vad';
+
+const vad = new SileroVAD({
+  threshold: 0.5,           // Speech probability threshold
+  min_speech_duration: 250, // ms before speech confirmed
+  min_silence_duration: 500, // ms of silence = end of turn
+});
+
+vad.on('speech_start', () => {
+  console.log('User started speaking');
+  // Stop any playing TTS (barge-in)
+  audioPlayer.stop();
+});
+
+vad.on('speech_end', () => {
+  console.log('User finished speaking');
+  // Trigger response generation
+  processTranscript();
+});
+
+// Feed audio to VAD
+audioStream.on('data', (chunk) => {
+  vad.process(chunk);
+});
+"""
+
+## OpenAI Semantic VAD
+"""
+// In Realtime API session config
+client.updateSession({
+  turn_detection: {
+    type: 'semantic_vad',  // Uses meaning, not just silence
+    // Model waits longer after "ummm..."
+    // Responds faster after "Yes, that's correct."
+  },
+});
+"""
+
+## Barge-In Handling
+"""
+// When user interrupts:
+function handleBargeIn() {
+  // 1. Stop TTS immediately
+  audioPlayer.stop();
+
+  // 2. Cancel pending LLM generation
+  llmController.abort();
+
+  // 3. Reset state
+  conversationState.checkpoint();
+
+  // 4. Listen to new input
+  startListening();
+}
+
+// VAD triggers barge-in
+vad.on('speech_start', () => {
+  if (audioPlayer.isPlaying) {
+    handleBargeIn();
+  }
+});
+"""
+
+### Latency Optimization Pattern
+
+Achieving <800ms end-to-end response time
+
+**When to use**: Production voice agents
+
+# LATENCY OPTIMIZATION:
+
+"""
+Target Metrics:
+- End-to-end: <800ms (ideal: <500ms)
+- Time-to-First-Token (TTFT): <300ms
+- Barge-in response: <200ms
+- Jitter variance: <100ms std dev
+"""
+
+## Pipeline Latency Breakdown
+"""
+Typical breakdown:
+- VAD processing: 50-100ms
+- STT first result: 150-200ms
+- LLM TTFT: 100-300ms
+- TTS TTFA: 75-200ms
+- Audio buffering: 50-100ms
+
+Total: 425-900ms
+"""
+
+## Optimization Strategies
+
+### 1. Streaming Everything
+"""
+// Stream STT results as they come
+stt.on('partial_transcript', (text) => {
+  // Start processing before final transcript
+  llmPreprocessor.prepare(text);
+});
+
+// Stream LLM output to TTS
+const llmStream = await openai.chat.completions.create({
+  stream: true,
+  // ...
+});
+
+for await (const chunk of llmStream) {
+  tts.appendText(chunk.choices[0].delta.content);
+}
+"""
+
+### 2. Pre-computation
+"""
+// While user is speaking, predict and prepare
+stt.on('partial_transcript', async (text) => {
+  // Pre-fetch relevant context
+  const context = await retrieveContext(text);
+
+  // Pre-compute likely first sentence
+  const firstSentence = await generateOpener(context);
+});
+"""
+
+### 3. Use Low-Latency Models
+"""
+// STT: Deepgram Nova-3 (150ms TTFT)
+// LLM: gpt-4o-mini (fastest GPT-4 class)
+// TTS: ElevenLabs Flash (75ms) or Deepgram Aura-2 (184ms)
+"""
+
+### 4. Edge Deployment
+"""
+// Run inference closer to user
+// - Cloud regions near user
+// - Edge computing for VAD/STT
+// - WebSocket over HTTP for lower overhead
+"""
+
+### Conversation Design Pattern
+
+Designing natural voice conversations
+
+**When to use**: Building voice UX
+
+# CONVERSATION DESIGN:
+
+## Voice-First Principles
+"""
+Voice is different from text:
+- No undo button - say it right the first time
+- Linear - user can't scroll back
+- Ephemeral - easy to miss information
+- Emotional - tone matters as much as words
+"""
+
+## Response Design
+"""
+# Keep responses short (10-20 seconds max)
+# Front-load the answer
+# Use signposting for lists
+
+Bad: "I found several options. The first is... second is..."
+Good: "I found 3 options. Want me to go through them?"
+
+# Confirm understanding
+Bad: "I'll transfer $500 to John."
+Good: "So that's $500 to John Smith. Should I proceed?"
+"""
+
+## Prompting for Voice
+"""
+system_prompt = '''
+You are a voice assistant. Follow these rules:
+
+1. Be concise - keep responses under 30 words
+2. Use natural speech - contractions, casual language
+3. Never use formatting (bullets, numbers in lists)
+4. Spell out numbers and abbreviations
+5. End with a question to keep conversation flowing
+6. If unclear, ask for clarification
+7. Never say "I'm an AI" unless asked
+
+Good: "Got it. I'll set that reminder for three pm. Anything else?"
+Bad: "I have set a reminder for 3:00 PM. Is there anything else I can assist you with today?"
+'''
+"""
+
+## Error Recovery
+"""
+// Handle recognition errors gracefully
+const errorResponses = {
+  no_speech: "I didn't catch that. Could you say it again?",
+  unclear: "Sorry, I'm not sure I understood. You said [repeat]. Is that right?",
+  timeout: "Still there? I'm here when you're ready.",
+};
+
+// Always offer human fallback for complex issues
+if (confidenceScore < 0.6) {
+  response = "I want to make sure I get this right. Would you like to speak with a human agent?";
+}
+"""
+
+## Sharp Edges
+
+### Response Latency Exceeds 800ms
+
+Severity: CRITICAL
+
+Situation: Building a voice agent pipeline
+
+Symptoms:
+Conversations feel awkward. Users repeat themselves. "Are you
+there?" questions. Users hang up or give up. Low satisfaction
+scores despite correct answers.
+
+Why this breaks:
+In human conversation, responses typically arrive within 500ms.
+Anything over 800ms feels like the agent is slow or confused.
+Users lose confidence and patience. Every component adds latency:
+VAD (100ms) + STT (200ms) + LLM (300ms) + TTS (200ms) = 800ms.
+
+Recommended fix:
+
+# Measure and budget latency for each component:
+
+## Target latencies:
+- VAD processing: <100ms
+- STT time-to-first-token: <200ms
+- LLM time-to-first-token: <300ms
+- TTS time-to-first-audio: <150ms
+- Total end-to-end: <800ms
+
+## Optimization strategies:
+
+1. Use low-latency models:
+   - STT: Deepgram Nova-3 (150ms) vs Whisper (500ms+)
+   - TTS: ElevenLabs Flash (75ms) vs standard (200ms+)
+   - LLM: gpt-4o-mini streaming
+
+2. Stream everything:
+   - Don't wait for full STT transcript
+   - Stream LLM output to TTS
+   - Start audio playback before TTS finishes
+
+3. Pre-compute:
+   - While user speaks, prepare context
+   - Generate opening phrase in parallel
+
+4. Edge deployment:
+   - Run VAD/STT at edge
+   - Use nearest cloud region
+
+## Measure continuously:
+Log timestamps at each stage, track P50/P95 latency
+
+### Response Time Variance Disrupts Rhythm
+
+Severity: HIGH
+
+Situation: Voice agent with inconsistent response times
+
+Symptoms:
+Conversations feel unpredictable. User doesn't know when to speak.
+Sometimes agent responds immediately, sometimes after long pause.
+Users talk over agent. Agent talks over users.
+
+Why this breaks:
+Jitter (variance in response time) disrupts conversational rhythm
+more than absolute latency. Consistent 800ms feels better than
+alternating 400ms and 1200ms. Users can't adapt to unpredictable
+timing.
+
+Recommended fix:
+
+# Target jitter metrics:
+- Standard deviation: <100ms
+- P95-P50 gap: <200ms
+
+## Reduce jitter sources:
+
+1. Consistent model loading:
+   - Keep models warm
+   - Pre-load on connection start
+
+2. Buffer audio output:
+   - Small buffer (50-100ms) smooths playback
+   - Don't start playing until buffer filled
+
+3. Handle LLM variance:
+   - gpt-4o-mini more consistent than larger models
+   - Set max_tokens to limit long responses
+
+4. Monitor and alert:
+   - Track response time distribution
+   - Alert on jitter spikes
+
+## Implementation:
+const MIN_RESPONSE_TIME = 400;  // ms
+
+async function respondWithConsistentTiming(text) {
+  const startTime = Date.now();
+  const audio = await generateSpeech(text);
+
+  const elapsed = Date.now() - startTime;
+  if (elapsed < MIN_RESPONSE_TIME) {
+    await delay(MIN_RESPONSE_TIME - elapsed);
+  }
+
+  playAudio(audio);
+}
+
+### Using Silence Duration for Turn Detection
+
+Severity: HIGH
 
-### ❌ Ignoring Latency Budget
+Situation: Detecting when user finishes speaking
 
-### ❌ Silence-Only Turn Detection
+Symptoms:
+Agent interrupts user mid-thought. Or waits too long after user
+finishes. "Let me think..." triggers premature response. Short
+answers have awkward pause before response.
 
-### ❌ Long Responses
+Why this breaks:
+Simple silence detection (e.g., "end turn after 500ms silence")
+doesn't understand conversation. Humans pause mid-sentence.
+"Yes." needs fast response, "Well, let me think about that..."
+needs patience. Fixed timeout fits neither.
 
-## ⚠️ Sharp Edges
+Recommended fix:
 
-| Issue | Severity | Solution |
-|-------|----------|----------|
-| Issue | critical | # Measure and budget latency for each component: |
-| Issue | high | # Target jitter metrics: |
-| Issue | high | # Use semantic VAD: |
-| Issue | high | # Implement barge-in detection: |
-| Issue | medium | # Constrain response length in prompts: |
-| Issue | medium | # Prompt for spoken format: |
-| Issue | medium | # Implement noise handling: |
-| Issue | medium | # Mitigate STT errors: |
+# Use semantic VAD:
+
+## OpenAI Semantic VAD:
+client.updateSession({
+  turn_detection: {
+    type: 'semantic_vad',
+    // Waits longer after "umm..."
+    // Responds faster after "Yes, that's correct."
+  },
+});
+
+## Pipecat SmartTurn:
+const pipeline = new Pipeline({
+  vad: new SileroVAD(),
+  turnDetection: new SmartTurn(),
+});
+
+// SmartTurn considers:
+// - Speech content (complete sentence?)
+// - Prosody (falling intonation?)
+// - Context (question asked?)
+
+## Fallback: Adaptive silence threshold:
+function calculateSilenceThreshold(transcript) {
+  const endsWithComplete = transcript.match(/[.!?]$/);
+  const hasFillers = transcript.match(/um|uh|like|well/i);
+
+  if (endsWithComplete && !hasFillers) {
+    return 300;  // Fast response
+  } else if (hasFillers) {
+    return 1500;  // Wait for continuation
+  }
+  return 700;  // Default
+}
+
+### Agent Doesn't Stop When User Interrupts
+
+Severity: HIGH
+
+Situation: User tries to interrupt agent mid-sentence
+
+Symptoms:
+Agent talks over user. User has to wait for agent to finish.
+Frustrating experience. Users give up and abandon call.
+"STOP! STOP!" doesn't work.
+
+Why this breaks:
+Without barge-in handling, the TTS plays to completion regardless
+of user input. This violates basic conversational norms - in human
+conversation, we stop when interrupted.
+
+Recommended fix:
+
+# Implement barge-in detection:
+
+## Basic barge-in:
+vad.on('speech_start', () => {
+  if (ttsPlayer.isPlaying) {
+    // 1. Stop audio immediately
+    ttsPlayer.stop();
+
+    // 2. Cancel pending TTS generation
+    ttsController.abort();
+
+    // 3. Checkpoint conversation state
+    conversationState.save();
+
+    // 4. Listen to new input
+    startTranscription();
+  }
+});
+
+## Advanced: Distinguish interruption types:
+vad.on('speech_start', async () => {
+  if (!ttsPlayer.isPlaying) return;
+
+  // Wait 200ms to get first words
+  await delay(200);
+  const firstWords = getTranscriptSoFar();
+
+  if (isBackchannel(firstWords)) {
+    // "uh-huh", "yeah" - don't interrupt
+    return;
+  }
+
+  if (isClarification(firstWords)) {
+    // "What?", "Sorry?" - repeat last sentence
+    repeatLastSentence();
+  } else {
+    // Real interruption - stop and listen
+    handleFullInterruption();
+  }
+});
+
+## Response time target:
+- Barge-in response: <200ms
+- User should feel heard immediately
+
+### Generating Text-Length Responses for Voice
+
+Severity: MEDIUM
+
+Situation: Prompting LLM for voice agent responses
+
+Symptoms:
+Agent rambles. Users lose track of information. "Can you repeat
+that?" requests. Users interrupt to ask for shorter version.
+Low comprehension of conveyed information.
+
+Why this breaks:
+Text can be scanned and re-read. Voice is linear and ephemeral.
+A 3-paragraph response that works in chat is overwhelming in voice.
+Users can only hold ~7 items in working memory.
+
+Recommended fix:
+
+# Constrain response length in prompts:
+
+system_prompt = '''
+You are a voice assistant. Keep responses UNDER 30 WORDS.
+For complex information, break into chunks and confirm
+understanding between each.
+
+Instead of: "Here are the three options. First, you could...
+Second... Third..."
+
+Say: "I found 3 options. Want me to go through them?"
+
+Never list more than 3 items without pausing for confirmation.
+'''
+
+## Enforce at generation:
+const response = await openai.chat.completions.create({
+  max_tokens: 100,  // Hard limit
+  // ...
+});
+
+## Chunking pattern:
+if (information.length > 3) {
+  response = `I have ${information.length} items. Let's go through them one at a time. First: ${information[0]}. Ready for the next?`;
+}
+
+## Progressive disclosure:
+"I found your account. Want the balance, recent transactions, or something else?"
+// Don't dump all info at once
+
+### Using Bullets/Numbers/Markdown in Voice
+
+Severity: MEDIUM
+
+Situation: Formatting LLM output for voice
+
+Symptoms:
+"First bullet point: item one" read aloud. Numbers read as "one
+two three" instead of "one, two, three." Markdown artifacts in
+speech. Robotic, unnatural delivery.
+
+Why this breaks:
+TTS models read what they're given. Text formatting intended for
+visual display sounds robotic when read aloud. Users can't "see"
+structure in audio.
+
+Recommended fix:
+
+# Prompt for spoken format:
+
+system_prompt = '''
+Format responses for SPOKEN delivery:
+- No bullet points, numbered lists, or markdown
+- Spell out numbers: "twenty-three" not "23"
+- Spell out abbreviations: "United States" not "US"
+- Use verbal signposting: "There are three things. First..."
+- Never use asterisks, dashes, or special characters
+'''
+
+## Post-processing:
+function prepareForSpeech(text) {
+  return text
+    // Remove markdown
+    .replace(/[*_#`]/g, '')
+    // Convert numbers
+    .replace(/\d+/g, numToWords)
+    // Expand abbreviations
+    .replace(/\betc\b/gi, 'et cetera')
+    .replace(/\be\.g\./gi, 'for example')
+    // Add pauses
+    .replace(/\. /g, '... ')
+    .replace(/, /g, '... ');
+}
+
+## SSML for precise control:
+<speak>
+  The total is <say-as interpret-as="currency">$49.99</say-as>.
+  <break time="500ms"/>
+  Want to proceed?
+</speak>
+
+### VAD/STT Fails in Noisy Environments
+
+Severity: MEDIUM
+
+Situation: Users in cars, cafes, outdoors
+
+Symptoms:
+"I didn't catch that" frequently. Background noise triggers
+false starts. Fan/AC causes continuous listening. Car engine
+noise confuses STT.
+
+Why this breaks:
+Default VAD thresholds work for quiet environments. Real-world
+usage includes background noise that triggers false positives
+or masks speech, causing false negatives.
+
+Recommended fix:
+
+# Implement noise handling:
+
+## 1. Noise reduction in STT:
+const transcription = await deepgram.transcription.live({
+  model: 'nova-3',
+  noise_reduction: true,
+  // or
+  smart_format: true,
+});
+
+## 2. Adaptive VAD threshold:
+// Measure ambient noise level
+const ambientLevel = measureAmbientNoise(5000);  // 5 sec sample
+
+vad.setThreshold(ambientLevel * 1.5);  // Above ambient
+
+## 3. Confidence filtering:
+stt.on('transcript', (data) => {
+  if (data.confidence < 0.7) {
+    // Low confidence - probably noise
+    askForRepeat();
+    return;
+  }
+  processTranscript(data.transcript);
+});
+
+## 4. Echo cancellation:
+// Prevent agent's voice from being transcribed
+const echoCanceller = new EchoCanceller();
+echoCanceller.reference(ttsOutput);
+const cleanedAudio = echoCanceller.process(userAudio);
+
+### STT Produces Incorrect or Hallucinated Text
+
+Severity: MEDIUM
+
+Situation: Processing unclear or accented speech
+
+Symptoms:
+Agent responds to something user didn't say. Names consistently
+wrong. Technical terms misheard. "I said X, not Y" frustration.
+
+Why this breaks:
+STT models can hallucinate, especially on proper nouns, technical
+terms, or accented speech. These errors propagate through the
+pipeline and produce nonsensical responses.
+
+Recommended fix:
+
+# Mitigate STT errors:
+
+## 1. Use keywords/biasing:
+const transcription = await deepgram.transcription.live({
+  keywords: ['Acme Corp', 'ProductName', 'John Smith'],
+  keyword_boost: 'high',
+});
+
+## 2. Confirmation for critical info:
+if (containsNameOrNumber(transcript)) {
+  response = `I heard "${name}". Is that correct?`;
+}
+
+## 3. Confidence-based fallback:
+if (confidence < 0.8) {
+  response = `I think you said "${transcript}". Did I get that right?`;
+}
+
+## 4. Multiple hypothesis handling:
+// Some STT APIs return n-best list
+const alternatives = transcription.alternatives;
+if (alternatives[0].confidence - alternatives[1].confidence < 0.1) {
+  // Ambiguous - ask for clarification
+}
+
+## 5. Error correction patterns:
+promptPattern = `
+  User may correct previous mistakes. If they say "no, I said X"
+  or "not Y, Z", update your understanding accordingly.
+`;
+
+## Validation Checks
+
+### Missing Latency Measurement
+
+Severity: ERROR
+
+Voice agents must track latency at each stage
+
+Message: Voice pipeline without latency tracking. Add timestamps at each stage to measure performance.
+
+### Using Batch STT Instead of Streaming
+
+Severity: WARNING
+
+Streaming STT reduces latency significantly
+
+Message: Using batch transcription. Consider streaming for lower latency in voice agents.
+
+### TTS Without Streaming Output
+
+Severity: WARNING
+
+Streaming TTS reduces time to first audio
+
+Message: TTS without streaming. Stream audio to reduce time to first audio.
+
+### Hardcoded VAD Silence Threshold
+
+Severity: WARNING
+
+Fixed silence thresholds don't adapt to conversation
+
+Message: Fixed silence threshold. Consider semantic VAD or adaptive thresholds for better turn-taking.
+
+### Missing Barge-In Handling
+
+Severity: WARNING
+
+Voice agents should stop when user interrupts
+
+Message: VAD without barge-in handling. Stop TTS when user starts speaking.
+
+### Voice Prompt Without Length Constraints
+
+Severity: WARNING
+
+Voice prompts should constrain response length
+
+Message: Voice prompt without length constraints. Add 'Keep responses under 30 words' to system prompt.
+
+### Markdown Formatting Sent to TTS
+
+Severity: WARNING
+
+Markdown will be read literally by TTS
+
+Message: Check for markdown in TTS input. Strip formatting before sending to TTS.
+
+### STT Without Error Handling
+
+Severity: WARNING
+
+STT can fail or return low confidence
+
+Message: STT without error handling. Check confidence scores and handle failures.
+
+### WebSocket Without Reconnection
+
+Severity: WARNING
+
+Realtime APIs need reconnection handling
+
+Message: Realtime connection without reconnection logic. Handle disconnects gracefully.
+
+### Missing Noise Handling
+
+Severity: INFO
+
+Real-world audio includes background noise
+
+Message: Consider adding noise handling for real-world audio quality.
+
+## Collaboration
+
+### Delegation Triggers
+
+- user needs phone/telephony integration -> backend (Twilio, Vonage, SIP integration)
+- user needs LLM optimization -> llm-architect (Model selection, prompting, fine-tuning)
+- user needs tools for voice agent -> agent-tool-builder (Tool design for voice context)
+- user needs multi-agent voice system -> multi-agent-orchestration (Voice agents working together)
+- user needs accessibility compliance -> accessibility-specialist (Voice interface accessibility)
 
 ## Related Skills
 
 Works well with: `agent-tool-builder`, `multi-agent-orchestration`, `llm-architect`, `backend`
 
 ## When to Use
-This skill is applicable to execute the workflow or actions described in the overview.
+
+- User mentions or implies: voice agent
+- User mentions or implies: speech to text
+- User mentions or implies: text to speech
+- User mentions or implies: whisper
+- User mentions or implies: elevenlabs
+- User mentions or implies: deepgram
+- User mentions or implies: realtime api
+- User mentions or implies: voice assistant
+- User mentions or implies: voice ai
+- User mentions or implies: conversational ai
+- User mentions or implies: tts
+- User mentions or implies: stt
+- User mentions or implies: asr