aillom-vox-client 1.0.0

package/docs/providers/AILLOMVOX.md

@@ -0,0 +1,185 @@
+# AillomVox Provider
+
+The default, high-performance provider. Uses our proprietary **Hybrid Engine** (Groq LLM + Inworld TTS) to deliver the fastest response times and highest stability at the lowest cost.
+
+## Models
+
+| Component | Model |
+| :--- | :--- |
+| **STT** | `stt-rt-v4` (Soniox) |
+| **LLM** | `openai/gpt-oss-120b` (via Groq) |
+| **TTS** | `inworld-tts-1.5-mini` (Inworld) |
+
+## Configuration
+
+```json
+{
+  "provider": "aillomvox",
+  "voice": "Edward",
+  "system_prompt": "You are a helpful assistant.",
+  "language": "en-US",
+  "sample_rate": 16000
+}
+```
+
+## Available Voices
+
+AillomVox uses **Inworld TTS 1.5** with **65 voices** across 15 languages. All voices support multilingual synthesis.
+
+### English (25 voices)
+
+#### Male
+| Voice | Style |
+| :--- | :--- |
+| **Edward** | Fast-talking, emphatic (default EN) |
+| **Dennis** | Smooth, calm, friendly |
+| **Alex** | Energetic, expressive |
+| **Craig** | Older British, refined, articulate |
+| **Mark** | Energetic, rapid delivery |
+| **Ronald** | Confident British, deep, gravelly |
+| **Shaun** | Friendly, dynamic |
+| **Theodore** | Gravelly, time-worn |
+| **Timothy** | Lively, upbeat American |
+| **Carter** | Mature radio announcer |
+| **Blake** | Rich, intimate |
+| **Clive** | British, calm, cordial |
+| **Dominus** | Robotic, deep, menacing |
+| **Hades** | Commanding, gruff narrator |
+
+#### Female
+| Voice | Style |
+| :--- | :--- |
+| **Ashley** | Warm, natural |
+| **Deborah** | Gentle, elegant |
+| **Elizabeth** | Professional, perfect for narrations |
+| **Julia** | Quirky, high-pitched, playful |
+| **Olivia** | Young British, upbeat, friendly |
+| **Priya** | Even-toned, Indian accent |
+| **Sarah** | Fast-talking, curious |
+| **Wendy** | Posh British |
+| **Luna** | Calm, relaxing, mindfulness |
+| **Hana** | Bright, expressive, young |
+| **Pixie** | High-pitched, childlike |
+
+### Portuguese (2 voices)
+| Voice | Gender | Style |
+| :--- | :--- | :--- |
+| **Heitor** | Male | Composed, neutral (default PT) |
+| **Maitê** | Female | Middle-aged, professional |
+
+### Spanish (4 voices)
+| Voice | Gender | Style |
+| :--- | :--- | :--- |
+| **Diego** | Male | Soothing, gentle (default ES) |
+| **Miguel** | Male | Calm, storytelling |
+| **Rafael** | Male | Deep, composed, narrations |
+| **Lupita** | Female | Vibrant, energetic |
+
+### French (4 voices)
+| Voice | Gender | Style |
+| :--- | :--- | :--- |
+| **Alain** | Male | Deep, smooth, composed |
+| **Mathieu** | Male | Nasal quality |
+| **Étienne** | Male | Calm, young adult |
+| **Hélène** | Female | Smooth, musical, graceful |
+
+### German (2 voices)
+| Voice | Gender | Style |
+| :--- | :--- | :--- |
+| **Josef** | Male | Articulate, announcer-like |
+| **Johanna** | Female | Calm, low, smoky |
+
+### Italian (2 voices)
+| Voice | Gender | Style |
+| :--- | :--- | :--- |
+| **Gianni** | Male | Deep, smooth, rapid |
+| **Orietta** | Female | Calm, soothing cadence |
+
+### Chinese (4 voices)
+| Voice | Gender | Style |
+| :--- | :--- | :--- |
+| **Yichen** | Male | Calm, flat, young adult |
+| **Xiaoyin** | Female | Youthful, gentle, sweet |
+| **Xinyi** | Female | Neutral, narrations |
+| **Jing** | Female | Energetic, fast-paced |
+
+### Dutch (4 voices)
+| Voice | Gender | Style |
+| :--- | :--- | :--- |
+| **Erik** | Male | Older, weathered edge |
+| **Lennart** | Male | Confident, calm, relaxed |
+| **Katrien** | Female | Expressive |
+| **Lore** | Female | Clear, calm, professional |
+
+### Japanese (2 voices)
+| Voice | Gender | Style |
+| :--- | :--- | :--- |
+| **Satoshi** | Male | Dramatic, expressive |
+| **Asuka** | Female | Friendly, young adult |
+
+### Korean (4 voices)
+| Voice | Gender | Style |
+| :--- | :--- | :--- |
+| **Hyunwoo** | Male | Young adult |
+| **Seojun** | Male | Clear, deep, mature |
+| **Minji** | Female | Energetic, friendly |
+| **Yoona** | Female | Gentle, soothing |
+
+### Polish (2 voices)
+| Voice | Gender | Style |
+| :--- | :--- | :--- |
+| **Szymon** | Male | Warm, friendly |
+| **Wojciech** | Male | Middle-aged |
+
+### Russian (4 voices)
+| Voice | Gender | Style |
+| :--- | :--- | :--- |
+| **Dmitry** | Male | Deep, commanding |
+| **Nikolai** | Male | Deep, theatrical |
+| **Svetlana** | Female | Soft, high-pitched |
+| **Elena** | Female | Clear, mid-range, smooth |
+
+### Hindi (2 voices)
+| Voice | Gender | Style |
+| :--- | :--- | :--- |
+| **Manoj** | Male | Clear, professional |
+| **Riya** | Female | Professional, polished |
+
+### Hebrew (2 voices)
+| Voice | Gender | Style |
+| :--- | :--- | :--- |
+| **Oren** | Male | Steady, podcasts |
+| **Yael** | Female | Mid-range, narrations |
+
+### Arabic (2 voices)
+| Voice | Gender | Style |
+| :--- | :--- | :--- |
+| **Omar** | Male | Bright, confident |
+| **Nour** | Female | Polished, friendly |
+
+## Default Voice by Language
+| Language | Default Voice |
+| :--- | :--- |
+| English (`en`) | Edward |
+| Portuguese (`pt`) | Heitor |
+| Spanish (`es`) | Diego |
+| All others | Edward |
+
+## Features
+
+- **Smart Fillers**: Automatically plays filler phrases ("Just a moment...", "Let me check...") during LLM processing.
+- **Dynamic Voice Switching**: Change voice mid-conversation with the `update_voice` tool.
+- **Silence Breakers**: Re-engages the user automatically if they go silent.
+- **Jitter Buffer**: Native handling of network instability.
+- **Native 8kHz**: Perfect for telephony (SIP/Asterisk) with zero resampling overhead.
+- **Adaptive Response Profiles**: Automatically adjusts buffer timing based on response length.
+- **Speed Control**: Server-side speed adjustment (0.5x–1.5x, default 1.2x).
+
+## Languages
+
+Supports 15 languages: `en`, `pt`, `es`, `fr`, `de`, `it`, `ja`, `zh`, `ko`, `hi`, `ar`, `ru`, `pl`, `nl`, `he`
+
+## Best For
+- **General Purpose**: Customer support, sales, virtual assistants
+- **Telephony**: Extremely robust 8kHz support for SIP/Asterisk
+- **High Volume**: Lowest cost per minute ($0.03/min)

package/docs/providers/AWS.md

@@ -0,0 +1,32 @@
+# AWS Bedrock (Nova Sonic)
+
+Enterprise-grade Speech-to-Speech using Amazon's latest **Nova Sonic** model (`amazon.nova-2-sonic-v1:0`).
+
+## Configuration
+
+```json
+{
+  "provider": "aws",
+  "voice": "matthew",
+  "system_prompt": "You are a helpful assistant.",
+  "sample_rate": 16000
+}
+```
+
+## Voices
+
+| Voice | Gender | Style |
+| :--- | :--- | :--- |
+| **matthew** | Male | Neutral, professional |
+| **ruth** | Female | Professional, clear |
+| **tiffany** | Female | Warm, friendly |
+
+## Features
+- **Low Latency**: Faster than previous Polly+Bedrock chains.
+- **Reliability**: Highest uptime guarantee.
+- **Security**: Data privacy compliance (HIPAA, GDPR options available via AWS config).
+- **Tool Use**: Full support for function calling.
+
+## Best For
+- **Enterprise**: Banking, healthcare, corporate environments.
+- **Stability**: When 99.99% uptime is required.

package/docs/providers/GEMINI.md

@@ -0,0 +1,33 @@
+# Google Gemini (Multimodal)
+
+Leverages `gemini-2.5-flash-native-audio-preview-12-2025` for massive context and multimodal capabilities.
+
+## Configuration
+
+```json
+{
+  "provider": "gemini",
+  "voice": "Puck",
+  "system_prompt": "You are a helpful assistant.",
+  "sample_rate": 24000
+}
+```
+
+## Voices
+
+| Voice | Style |
+| :--- | :--- |
+| **Puck** | Soft, higher pitch |
+| **Kore** | Soft, higher pitch |
+| **Charon** | Deep, confident |
+| **Fenrir** | Deep, confident |
+| **Aoede** | Confident, higher pitch |
+
+## Features
+- **Large Context**: Can process huge system prompts or conversation history.
+- **Multimodal**: Can technically process images if sent (though SDK focuses on Audio).
+- **Tool Use**: Robust function calling.
+
+## Best For
+- **Long Context**: Analyzing documents or long previous conversations.
+- **Complex Instructions**: Following very detailed, multi-step system prompts.

package/docs/providers/GROK.md

@@ -0,0 +1,25 @@
+# xAI Grok
+
+A witty, casual, and capable voice model from xAI (`grok-beta`).
+
+## Configuration
+
+```json
+{
+  "provider": "grok",
+  "system_prompt": "You are a helpful assistant.",
+  "sample_rate": 16000
+}
+```
+
+## Voices
+
+Model-dependent. Voice selection depends on the underlying model version.
+
+## Features
+- **Casual Tone**: Designed to be less robotic and more conversational ("witty").
+- **Native 16kHz**: Matches standard VoIP/WebRTC wideband perfectly.
+
+## Best For
+- **Entertainment/Casual**: Chatbots, companions.
+- **News/Twitter**: Real-time information (via RAG/WebSearch tools).

package/docs/providers/OPENAI.md

@@ -0,0 +1,39 @@
+# OpenAI Realtime Provider
+
+Direct integration with the `gpt-realtime-mini` model via WebSocket.
+
+## Configuration
+
+```json
+{
+  "provider": "openai",
+  "voice": "alloy",
+  "system_prompt": "You are a helpful assistant.",
+  "sample_rate": 24000,
+  "max_duration": 300
+}
+```
+
+## Voices
+
+| Voice | Style |
+| :--- | :--- |
+| **alloy** | Neutral, balanced |
+| **ash** | Warm, conversational |
+| **coral** | Clear, professional |
+| **echo** | Smooth, calm |
+| **sage** | Wise, measured |
+| **shimmer** | Bright, energetic |
+
+## Features
+- **Function Calling**: Full support for tool calling.
+- **Native VAD**: Uses OpenAI's server-side voice activity detection.
+- **24kHz High Fidelity**: Best used with `sample_rate: 24000`.
+
+## Audio Notes
+- Native rate is **24kHz**. 
+- If you request 8kHz (telephony), the SDK automatically resamples it, but **24kHz** gives the best results for web calls.
+
+## Best For
+- **Complex Reasoning**: Logic-heavy tasks, math, coding assistance.
+- **English/Multilingual**: Excellent accent capability.

package/docs/providers/QWEN.md

@@ -0,0 +1,27 @@
+# Qwen (Alibaba Cloud)
+
+Open-source based, highly efficient model (`qwen3-omni-flash-realtime`).
+
+## Configuration
+
+```json
+{
+  "provider": "qwen",
+  "system_prompt": "You are a helpful assistant.",
+  "language": "en-US",
+  "sample_rate": 16000
+}
+```
+
+## Voices
+
+Model-dependent. Voice selection depends on the underlying model version.
+
+## Features
+- **Cost Effective**: Generally lower cost than OpenAI.
+- **Fast**: "Flash" model is optimized for speed.
+- **No Tool Support**: Function calling / Client Tools are **not supported** in WebSocket Realtime mode. Use AWS, OpenAI, or Gemini for scenarios requiring tools.
+
+## Best For
+- **Cost-Sensitive**: High volume conversational AI.
+- **Asian Markets**: Excellent support for Mandarin/English/Asian languages.

package/docs/providers/ULTRAVOX.md

@@ -0,0 +1,29 @@
+# UltraVox
+
+Specialized Speech-to-Speech model (`ultravox-v0.7`).
+
+## Configuration
+
+```json
+{
+  "provider": "ultravox",
+  "voice": "Mark",
+  "system_prompt": "You are a helpful assistant.",
+  "language": "pt-BR",
+  "sample_rate": 16000
+}
+```
+
+## Voices
+
+| Voice | Style |
+| :--- | :--- |
+| **Mark** | Male |
+| **Jessica** | Female |
+
+## Features
+- **Nuance**: High capability in understanding tone and emotion.
+- **Dynamic Voices**: Voice list is fetched dynamically from their API.
+
+## Best For
+- **Emotional Intelligence**: Empathetic support or counseling bots.

package/examples/01-basic/app.js

@@ -0,0 +1,196 @@
+// Basic AillomVox Client
+const connectBtn = document.getElementById('connectBtn');
+const disconnectBtn = document.getElementById('disconnectBtn');
+const statusDiv = document.getElementById('status');
+const apiKeyInput = document.getElementById('apiKey');
+
+let socket;
+let audioContext;
+let processor;
+let mediaStream;
+
+// 🎯 ULTRAVOX PATTERN: Track scheduled audio sources for instant barge-in clearing
+let scheduledSources = [];
+let nextPlayTime = 0;
+
+connectBtn.onclick = async () => {
+    const apiKey = apiKeyInput.value.trim();
+    if (!apiKey) return alert('Please enter an API Key');
+
+    // 1. Initialize Audio Context (Must be user-initiated)
+    audioContext = new (window.AudioContext || window.webkitAudioContext)({ sampleRate: 16000 });
+
+    // 2. Connect to WebSocket
+    // Note: Replace 'your-server-url' with actual server if hosted elsewhere
+    // For local dev with aillom-vox, use localhost:8080
+    // For production, use wss://vox.aillom.com/ws
+    const wsUrl = window.location.hostname === 'localhost'
+        ? 'ws://localhost:8080/ws'
+        : 'wss://vox.aillom.com/ws';
+
+    socket = new WebSocket(wsUrl);
+    socket.binaryType = 'arraybuffer';
+
+    socket.onopen = async () => {
+        statusDiv.textContent = 'Connected. Handshaking...';
+
+        // 3. Send Configuration Handshake
+        const handshake = {
+            type: 'config',
+            apikey: apiKey,
+            provider: 'aillomvox',
+            voice: 'Edward',
+            language: 'en-US',
+            sample_rate: 16000,
+            system_prompt: 'You are a helpful assistant. Be concise and friendly.',
+            tools: []
+        };
+        socket.send(JSON.stringify(handshake));
+
+        // 4. Start Microphone and Audio Processing
+        await startMicrophone();
+
+        statusDiv.textContent = '🟢 Online - Speak now!';
+        toggleButtons(true);
+    };
+
+    socket.onmessage = (event) => {
+        if (typeof event.data === 'string') {
+            const msg = JSON.parse(event.data);
+            console.log('Server Message:', msg);
+
+            switch (msg.type) {
+                case 'hangup':
+                    disconnect();
+                    break;
+
+                case 'playback_clear_buffer':
+                    // 🎯 ULTRAVOX PATTERN: Instant barge-in — clear all buffered audio
+                    clearPlaybackBuffer();
+                    break;
+
+                case 'transcript':
+                    if (msg.final) {
+                        console.log(`[${msg.role}] ${msg.text}`);
+                    }
+                    break;
+
+                case 'error':
+                    console.error('Server error:', msg.message);
+                    break;
+
+                case 'state':
+                    // 🎯 ULTRAVOX P1: Conversation state machine
+                    statusDiv.textContent = msg.state === 'listening' ? '🟢 Listening...'
+                        : msg.state === 'thinking' ? '🟡 Thinking...'
+                            : msg.state === 'speaking' ? '🟠 Speaking...'
+                                : `🟢 ${msg.state}`;
+                    break;
+            }
+        } else {
+            // Audio Data (PCM 16-bit) received from server -> Play it
+            playAudioChunk(event.data);
+        }
+    };
+
+    socket.onclose = () => {
+        statusDiv.textContent = '🔴 Disconnected';
+        disconnect();
+    };
+};
+
+disconnectBtn.onclick = disconnect;
+
+function disconnect() {
+    clearPlaybackBuffer();
+    if (socket) socket.close();
+    if (audioContext) audioContext.close();
+    if (mediaStream) mediaStream.getTracks().forEach(t => t.stop());
+    toggleButtons(false);
+}
+
+function toggleButtons(connected) {
+    connectBtn.disabled = connected;
+    disconnectBtn.disabled = !connected;
+    apiKeyInput.disabled = connected;
+}
+
+async function startMicrophone() {
+    mediaStream = await navigator.mediaDevices.getUserMedia({ audio: true });
+    const source = audioContext.createMediaStreamSource(mediaStream);
+
+    // Simple Processor (Buffer Size 4096)
+    processor = audioContext.createScriptProcessor(4096, 1, 1);
+
+    processor.onaudioprocess = (e) => {
+        if (socket.readyState !== WebSocket.OPEN) return;
+
+        const inputData = e.inputBuffer.getChannelData(0);
+        // Convert Float32 to Int16 for Server
+        const pcmData = floatTo16BitPCM(inputData);
+        socket.send(pcmData);
+    };
+
+    source.connect(processor);
+    processor.connect(audioContext.destination);
+}
+
+/**
+ * 🎯 ULTRAVOX PATTERN: Clear all buffered/scheduled audio instantly
+ * Called when server detects barge-in (user speaking while AI is talking)
+ * Stops all AudioBufferSourceNodes that haven't finished playing yet
+ */
+function clearPlaybackBuffer() {
+    for (const source of scheduledSources) {
+        try { source.stop(); } catch (e) { /* already stopped */ }
+    }
+    scheduledSources = [];
+    nextPlayTime = 0;
+    console.log('[AillomVox] 🔇 Playback buffer cleared (barge-in)');
+}
+
+/**
+ * 🎯 ULTRAVOX PATTERN: Sequential audio scheduling
+ * Instead of calling source.start() immediately (which causes overlap),
+ * schedule each chunk to play after the previous one finishes.
+ * This allows proper cancellation via clearPlaybackBuffer().
+ */
+function playAudioChunk(arrayBuffer) {
+    if (!audioContext || audioContext.state === 'closed') return;
+
+    const float32Data = new Float32Array(arrayBuffer.byteLength / 2);
+    const dataView = new DataView(arrayBuffer);
+
+    for (let i = 0; i < float32Data.length; i++) {
+        const int16 = dataView.getInt16(i * 2, true); // Little Endian
+        float32Data[i] = int16 < 0 ? int16 / 0x8000 : int16 / 0x7FFF;
+    }
+
+    const buffer = audioContext.createBuffer(1, float32Data.length, 16000);
+    buffer.getChannelData(0).set(float32Data);
+
+    const source = audioContext.createBufferSource();
+    source.buffer = buffer;
+    source.connect(audioContext.destination);
+
+    // Schedule sequentially: each chunk plays after the previous one ends
+    const now = audioContext.currentTime;
+    const startTime = Math.max(now, nextPlayTime);
+    source.start(startTime);
+    nextPlayTime = startTime + buffer.duration;
+
+    // Track for cancellation on barge-in
+    scheduledSources.push(source);
+    source.onended = () => {
+        scheduledSources = scheduledSources.filter(s => s !== source);
+    };
+}
+
+function floatTo16BitPCM(input) {
+    const output = new Int16Array(input.length);
+    for (let i = 0; i < input.length; i++) {
+        const s = Math.max(-1, Math.min(1, input[i]));
+        output[i] = s < 0 ? s * 0x8000 : s * 0x7FFF;
+    }
+    return output.buffer;
+}

package/examples/01-basic/index.html

@@ -0,0 +1,27 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>AillomVox - Basic Client</title>
+    <style>
+        body { font-family: sans-serif; display: flex; flex-direction: column; align-items: center; justify-content: center; height: 100vh; background: #f0f0f0; }
+        .container { background: white; padding: 2rem; border-radius: 8px; box-shadow: 0 4px 6px rgba(0,0,0,0.1); text-align: center; }
+        input { padding: 8px; margin: 10px 0; width: 100%; box-sizing: border-box; }
+        button { padding: 10px 20px; cursor: pointer; background: #007bff; color: white; border: none; border-radius: 4px; font-size: 16px; }
+        button:disabled { background: #ccc; }
+        #status { margin-top: 15px; font-weight: bold; color: #666; }
+    </style>
+</head>
+<body>
+    <div class="container">
+        <h1>🎙️ AillomVox Basic</h1>
+        <input type="password" id="apiKey" placeholder="Enter API Key">
+        <button id="connectBtn">Connect</button>
+        <button id="disconnectBtn" disabled>Disconnect</button>
+        <div id="status">Disconnected</div>
+    </div>
+
+    <script src="app.js"></script>
+</body>
+</html>