use-voice-control 0.1.0 → 0.1.2

package/speech/docs/ARCHITECTURE.md

@@ -0,0 +1,349 @@
+# Speech Library Architecture
+
+## System Overview
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                         Client Layer                         │
+├─────────────────────────────────────────────────────────────┤
+│  • React Components (client-example.tsx)                     │
+│  • Browser Audio APIs                                        │
+│  • Fetch API                                                 │
+└──────────────────────┬──────────────────────────────────────┘
+                       │ HTTP POST
+                       │ /api/agent/voice
+                       ▼
+┌─────────────────────────────────────────────────────────────┐
+│                      API Route Layer                         │
+├─────────────────────────────────────────────────────────────┤
+│  app/api/agent/voice/route.ts                               │
+│  • Request validation                                        │
+│  • Rate limiting (10/day guests)                            │
+│  • Error handling                                            │
+└──────────────────────┬──────────────────────────────────────┘
+                       │
+                       │ generateSpeech()
+                       ▼
+┌─────────────────────────────────────────────────────────────┐
+│                   Speech Library Core                        │
+├─────────────────────────────────────────────────────────────┤
+│  lib/speech/index.ts                                        │
+│  • Provider routing                                          │
+│  • Input normalization                                       │
+│  • Unified interface                                         │
+└──────────────┬──────────────────────┬───────────────────────┘
+               │                      │
+               │                      │
+       ┌───────▼────────┐    ┌───────▼────────┐
+       │                 │    │                 │
+       │  Kokoro Provider│    │Deepgram Provider│
+       │                 │    │                 │
+       │  kokoro.ts      │    │  deepgram.ts   │
+       │                 │    │                 │
+       └────────┬────────┘    └────────┬────────┘
+                │                      │
+                │                      │
+        ┌───────▼────────┐    ┌────────▼────────┐
+        │                 │    │                  │
+        │   kokoro-js     │    │ Cloudflare AI    │
+        │   (Node CPU)    │    │   Workers AI     │
+        │                 │    │                  │
+        │  • ONNX Runtime │    │  • Deepgram Aura │
+        │  • 82M params   │    │  • MP3 encoder   │
+        │  • WAV output   │    │                  │
+        └─────────────────┘    └──────────────────┘
+```
+
+## Data Flow
+
+### Request Flow
+```
+1. Client sends POST /api/agent/voice
+   {
+     "text": "Hello world",
+     "provider": "kokoro",
+     "voice": "af_heart"
+   }
+
+2. API Route validates & rate-limits
+   ↓
+3. Calls generateSpeech()
+   ↓
+4. Routes to kokoro.ts or deepgram.ts
+   ↓
+5. Provider generates audio
+   ↓
+6. Returns { audio: ArrayBuffer, contentType: string }
+   ↓
+7. API Route streams audio to client
+```
+
+### Response Flow
+```
+HTTP 200 OK
+Content-Type: audio/wav (Kokoro) or audio/mpeg (Deepgram)
+Cache-Control: public, max-age=86400
+Content-Disposition: inline; filename="speech.wav"
+
+[Binary audio data]
+```
+
+## Module Dependencies
+
+```
+route.ts
+├── @/lib/speech (unified interface)
+│   ├── types.ts (shared types)
+│   ├── kokoro.ts
+│   │   └── kokoro-js (npm package)
+│   │       └── onnxruntime-web
+│   └── deepgram.ts
+│       └── @/lib/cloudflare-context
+├── @/lib/auth/session (getUserId)
+└── @/lib/rate-limit/guestRateLimiter
+```
+
+## Type System
+
+```ts
+// Core types
+type TTSProvider = "kokoro" | "deepgram";
+
+interface TTSOptions {
+  text: string;
+  provider?: TTSProvider;
+  voice?: string;
+}
+
+interface TTSResult {
+  audio: ArrayBuffer;
+  contentType: string;
+}
+
+// Voice types
+type KokoroVoice = "af_heart" | "af_alloy" | ... (16 total)
+type DeepgramSpeaker = "angus" | "asteria" | ... (12 total)
+```
+
+## State Management
+
+### Kokoro Model Loading
+```ts
+let ttsInstance: KokoroTTS | null = null;
+let modelLoading: Promise<KokoroTTS> | null = null;
+
+// Lazy singleton pattern
+async function getKokoroTTS() {
+  if (ttsInstance) return ttsInstance;
+  if (modelLoading) await modelLoading;
+  
+  modelLoading = KokoroTTS.from_pretrained(...);
+  ttsInstance = await modelLoading;
+  modelLoading = null;
+  
+  return ttsInstance;
+}
+```
+
+**Benefits:**
+- Model loads once per server instance
+- Concurrent requests wait for single download
+- No memory leaks or duplicate loads
+
+### Rate Limiting State
+```ts
+// In-memory counter per IP/userId
+const rateLimitMap = new Map<string, {
+  count: number;
+  resetAt: number;
+}>();
+```
+
+## Error Handling
+
+```
+┌─────────────────────────────────────┐
+│         Error Categories             │
+├─────────────────────────────────────┤
+│                                      │
+│  1. Client Errors (400)              │
+│     • Missing text                   │
+│     • Invalid JSON                   │
+│                                      │
+│  2. Rate Limiting (429)              │
+│     • Daily limit exceeded           │
+│                                      │
+│  3. Provider Errors (503)            │
+│     • Model loading failed (Kokoro)  │
+│     • CF binding missing (Deepgram)  │
+│                                      │
+│  4. Runtime Errors (500)             │
+│     • Generation failed              │
+│     • Unknown errors                 │
+│                                      │
+└─────────────────────────────────────┘
+```
+
+## Performance Characteristics
+
+### Kokoro
+```
+First Request:
+  Model Download: ~2-3s (82MB, one-time)
+  Generation: ~100-300ms
+  Total: ~2-3s
+
+Subsequent Requests:
+  Generation: ~100-300ms
+  (No network calls)
+```
+
+### Deepgram
+```
+All Requests:
+  Network RTT: ~50-200ms
+  Generation: ~200-400ms
+  Total: ~300-500ms
+  (Depends on edge location)
+```
+
+## Deployment Considerations
+
+### Development
+```yaml
+Environment: Node.js
+Model Storage: ~/.cache/huggingface/
+Network: Required for first Kokoro download
+Cloudflare: Optional (Deepgram won't work)
+```
+
+### Production
+```yaml
+Build:
+  - No prebuild needed
+  - Model downloads on first request
+  
+Runtime:
+  - Node.js (not Edge)
+  - CPU: 2+ cores recommended
+  - RAM: 512MB+ for Kokoro model
+  - Disk: 100MB for cached model
+  
+Cloudflare Workers:
+  - Deepgram requires AI binding
+  - Kokoro requires Node runtime
+```
+
+## Security
+
+### Rate Limiting
+```ts
+// IP-based for guests
+const rateLimitKey = 
+  userId ??
+  req.headers.get("x-forwarded-for")?.split(",")[0] ??
+  req.headers.get("x-real-ip") ??
+  "unknown";
+```
+
+### Input Sanitization
+```ts
+// Max length enforcement
+text: text.slice(0, 5000)
+
+// Type validation
+if (!text || typeof text !== "string" || text.trim().length === 0) {
+  throw new Error("Text is required");
+}
+
+// Voice validation
+const voice = VOICES.includes(requested) ? requested : DEFAULT;
+```
+
+### Headers
+```ts
+// Cache control
+"Cache-Control": "public, max-age=86400"
+
+// Content type enforcement
+"Content-Type": result.contentType  // Never user-controlled
+
+// Safe filename
+"Content-Disposition": `inline; filename="speech.${ext}"`
+```
+
+## Testing Strategy
+
+### Unit Tests
+```ts
+// test.ts - Basic functionality
+- Model loading
+- Voice generation
+- File output
+```
+
+### Integration Tests
+```bash
+# API endpoint
+curl -X POST /api/agent/voice -d '{"text":"test"}'
+
+# Expected: 200 OK, audio/wav
+```
+
+### Load Tests
+```bash
+# Concurrent requests (model singleton)
+ab -n 100 -c 10 http://localhost:3000/api/agent/voice
+
+# Expected: All requests succeed, first is slower
+```
+
+## Future Enhancements
+
+### Potential Improvements
+1. **Streaming Audio**: Chunked transfer encoding
+2. **Voice Cloning**: Custom voice training
+3. **SSML Support**: Prosody control
+4. **Multi-language**: International voices
+5. **Caching**: Redis for repeated phrases
+6. **Metrics**: Prometheus exports
+7. **A/B Testing**: Quality comparisons
+
+### Provider Additions
+- **ElevenLabs**: Ultra-realistic voices
+- **Azure TTS**: Enterprise features
+- **Google Cloud TTS**: WaveNet quality
+- **AWS Polly**: Neural voices
+
+## Monitoring
+
+### Key Metrics
+```ts
+// Track these in production
+- Request rate (req/s)
+- Error rate (%)
+- P50/P95/P99 latency (ms)
+- Provider distribution (kokoro vs deepgram)
+- Cache hit rate (if implemented)
+- Model load time (first request)
+- Rate limit hits (per user)
+```
+
+### Logging
+```ts
+console.log("[TTS]", {
+  provider,
+  voice,
+  textLength: text.length,
+  userId,
+  latency: Date.now() - start,
+  error: error?.message,
+});
+```
+
+## References
+
+- [Kokoro Model](https://huggingface.co/onnx-community/Kokoro-82M-v1.0-ONNX)
+- [kokoro-js](https://www.npmjs.com/package/kokoro-js)
+- [Deepgram Aura](https://developers.cloudflare.com/workers-ai/models/deepgram-aura/)
+- [ONNX Runtime](https://onnxruntime.ai/)

package/speech/docs/HUGGINGFACE_MIGRATION.md

@@ -0,0 +1,172 @@
+# Kokoro TTS Migration: kokoro-js → Hugging Face Transformers
+
+## Overview
+
+Kokoro TTS has been migrated from the `kokoro-js` package to use Hugging Face transformers library loaded from CDN, providing better flexibility and potentially improved performance.
+
+## What Changed
+
+### Browser Implementation (Client-Side)
+- **Before**: Used `kokoro-js` npm package
+- **After**: Uses Hugging Face transformers loaded from CDN (`https://cdn.jsdelivr.net/npm/@huggingface/transformers@3.5.1/dist/transformers.min.js`)
+
+### New Files Created
+
+1. **KokoroTTS.js** - Main TTS class implementing audio generation
+   - Loads model and tokenizer from Hugging Face Hub
+   - Generates phonemes from text
+   - Applies voice styling
+   - Produces audio output
+
+2. **main.js** - Main entry point for browser TTS
+   - Manages Web Worker for model loading
+   - Handles text-to-speech requests
+   - Provides `textToSpeech()` function
+   - Cleans markdown formatting from input text
+
+3. **worker.js** - Web Worker for model management
+   - Loads TTS model in background thread
+   - Handles audio generation without blocking UI
+   - Communicates via message passing
+
+4. **AudioPlayer.js** - Audio playback system
+   - Web Audio API integration
+   - Audio queuing and playback
+   - WAV format conversion
+   - RawAudio class for handling audio buffers
+
+5. **phonemize.js** - Text-to-phoneme conversion
+   - Converts input text to phonetic representation
+   - Language-specific processing (English/other)
+   - Can be extended with g2p-en library for better phonemization
+
+6. **voices.js** - Voice definitions and data management
+   - Voice catalog with metadata
+   - Voice data caching
+   - Fetches voice embeddings from Hugging Face
+
+### Updated Files
+
+1. **kokoro.ts** - Server-side TTS (apps/qwksearch-web)
+   - Updated to use Hugging Face transformers
+   - Currently shows placeholder (server-side needs additional setup)
+
+2. **kokoro.ts** - React client integration (packages/research-agent-ui)
+   - Updated imports to use Hugging Face transformers directly
+   - Simplified backend selection logic
+   - Removed kokoro-js specific handling
+
+## Model Details
+
+- **Model**: `hexgrad/Kokoro-82M` (on Hugging Face Hub)
+- **Format**: Transformers compatible
+- **Backend**: WASM by default, WebGPU if available
+- **Sample Rate**: 24000 Hz
+- **Voice Style Dimension**: 256
+
+## Usage
+
+### Browser Usage
+
+```javascript
+import { textToSpeech, ttsModelReadyPromise } from './main.js';
+
+// Wait for model to load
+await ttsModelReadyPromise;
+
+// Generate and play speech
+textToSpeech("Hello, world!", "af_heart");
+```
+
+### React Component
+
+```typescript
+import { preloadKokoro, getKokoro } from '@lib/kokoro';
+
+// Preload model
+await preloadKokoro();
+
+// Generate speech
+const tts = await getKokoro();
+const audio = await tts.generate("Hello, world!", { voice: "af_heart" });
+```
+
+## Available Voices
+
+### Female Voices (English)
+- `af`, `af_heart`, `af_alloy`, `af_aoede`, `af_bella`
+- `af_jessica`, `af_nicole`, `af_river`, `af_sarah`, `af_sky`
+
+### Male Voices (English)
+- `am`, `am_adam`, `am_echo`, `am_fable`, `am_fenrir`
+- `am_liam`, `am_michael`, `am_onyx`
+
+### Other Languages
+- `bf`, `bm` (language variant 'b')
+
+## Backend Performance
+
+### WASM (Default)
+- More compatible
+- Lower memory usage
+- Slightly slower
+- Quantized to q8 for smaller model size
+
+### WebGPU (Fallback to WASM if unavailable)
+- Faster on supported hardware
+- Requires modern GPU support
+- Full precision (fp32)
+
+## Dependencies
+
+### Browser
+- Hugging Face transformers library (CDN)
+- Web Audio API support
+
+### Node.js (Future)
+- `@huggingface/transformers` npm package
+- Additional setup required for server-side TTS
+
+## Migration Checklist
+
+- [x] Create KokoroTTS class using HF transformers
+- [x] Implement browser-based audio generation
+- [x] Create Web Worker for background loading
+- [x] Implement audio playback
+- [x] Create supporting modules (phonemize, voices, AudioPlayer)
+- [x] Update React client integration
+- [x] Update Node.js server-side file
+
+## Next Steps
+
+1. **Testing**: Test audio generation with various voices and texts
+2. **Phonemization**: Consider using `g2p-en` library for better phoneme accuracy
+3. **Voice Data**: Verify voice embedding data loading from Hugging Face
+4. **Server-Side**: Implement server-side TTS if needed
+5. **Performance**: Monitor and optimize model loading time
+6. **Fallback**: Ensure Deepgram fallback still works if Kokoro fails
+
+## Known Limitations
+
+- Voice data fetching requires internet connectivity
+- Large model size (may cause initial load delay)
+- Phonemization is simplified (could be improved with g2p-en)
+- Server-side implementation placeholder (requires more work)
+
+## Troubleshooting
+
+### Model fails to load
+- Check browser GPU/WASM support
+- Verify Hugging Face Hub connectivity
+- Try refreshing the page
+- Check browser console for detailed error messages
+
+### Audio playback issues
+- Ensure Web Audio API is available
+- Check audio context state
+- Verify speaker/audio output devices
+
+### Voice not found errors
+- Use valid voice keys from the VOICES object
+- Check for typos in voice name
+- See Available Voices section above