@tryhamster/gerbil 1.0.0-rc.0 → 1.0.0-rc.1

package/docs/tts.md

@@ -0,0 +1,569 @@
+# Text-to-Speech
+
+Gerbil provides on-device text-to-speech using the [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) model. Generate natural-sounding speech locally without API keys or internet connection.
+
+## Quick Start
+
+### Node.js / CLI
+
+```typescript
+import { Gerbil } from "@tryhamster/gerbil";
+
+const g = new Gerbil();
+
+// Generate speech
+const result = await g.speak("Hello, I'm Gerbil!", { 
+  voice: "af_heart",  // American female (highest quality)
+  speed: 1.0 
+});
+
+// result.audio = Float32Array (PCM samples)
+// result.sampleRate = 24000
+// result.duration = seconds
+```
+
+### React (Browser)
+
+```tsx
+import { useSpeech } from "@tryhamster/gerbil/browser";
+
+function SpeechDemo() {
+  // Default: Kokoro TTS (24kHz, 28 voices)
+  const { speak, stop, isSpeaking, isLoading, listVoices, setVoice } = useSpeech();
+
+  // Or use Supertonic (44.1kHz, 4 voices, faster)
+  // const { speak, listVoices } = useSpeech({ model: "supertonic-66m" });
+
+  if (isLoading) return <div>Loading TTS model...</div>;
+
+  return (
+    <div>
+      <select onChange={e => setVoice(e.target.value)}>
+        {listVoices().map(v => (
+          <option key={v.id} value={v.id}>{v.name} ({v.language})</option>
+        ))}
+      </select>
+      
+      <button onClick={() => speak("Hello world!")}>
+        {isSpeaking ? "Speaking..." : "Speak"}
+      </button>
+      
+      {isSpeaking && <button onClick={stop}>Stop</button>}
+    </div>
+  );
+}
+```
+
+### AI SDK
+
+```typescript
+import { experimental_generateSpeech as generateSpeech } from "ai";
+import { gerbil } from "@tryhamster/gerbil/ai";
+
+const audio = await generateSpeech({
+  model: gerbil.speech(),
+  text: "Hello from Gerbil!",
+  voice: "bf_emma",  // British female
+});
+
+// audio.audioData = Uint8Array (WAV format)
+```
+
+### CLI
+
+```bash
+# Speak text with default voice
+gerbil speak "Hello world"
+
+# Specify voice and speed
+gerbil speak --voice bf_emma --speed 1.2 "Cheerio!"
+```
+
+### Skills
+
+```typescript
+import { speak, announce, readAloud } from "@tryhamster/gerbil/skills";
+
+// Simple speech
+await speak({ text: "Hello world", voice: "af_heart" });
+
+// AI-crafted announcement
+await announce({ 
+  message: "Build completed successfully",
+  style: "excited",  // casual, formal, excited, calm, urgent
+  voice: "af_bella"
+});
+
+// Read file aloud
+await readAloud({ 
+  content: "./README.md",
+  voice: "bf_emma",
+  summarizeIfLong: true  // Summarize if > 5000 chars
+});
+```
+
+## Available TTS Models
+
+Gerbil supports multiple TTS backends:
+
+| Model | Size | Sample Rate | Voices | Speed | Notes |
+|-------|------|-------------|--------|-------|-------|
+| `kokoro-82m` | ~330MB | 24000 Hz | 28 | ~100x RT | Default, highest quality |
+| `supertonic-66m` | ~250MB | 44100 Hz | 4 | ~167x RT | Faster, HiFi output |
+
+### Selecting a Model
+
+```typescript
+// Use default (Kokoro)
+await g.loadTTS();
+
+// Use Supertonic (faster, higher sample rate)
+await g.loadTTS({ model: "supertonic-66m" });
+
+// List available models
+const models = await g.listTTSModels();
+// [{ id: "kokoro-82m", sampleRate: 24000, voiceCount: 28 }, ...]
+```
+
+## Available Voices
+
+### Kokoro Voices
+
+Kokoro provides 28 voices across American and British English:
+
+### American English - Female (Recommended)
+
+| Voice ID | Name | Quality | Description |
+|----------|------|---------|-------------|
+| `af_heart` | Heart | A | Highest quality, warm tone |
+| `af_bella` | Bella | A- | Warm and friendly |
+| `af_nicole` | Nicole | B- | Soft and gentle |
+| `af_sarah` | Sarah | C+ | Clear and professional |
+
+### American English - Male
+
+| Voice ID | Name | Quality | Description |
+|----------|------|---------|-------------|
+| `am_fenrir` | Fenrir | C+ | Best male quality |
+| `am_michael` | Michael | C+ | Warm and friendly |
+| `am_puck` | Puck | C+ | Neutral tone |
+
+### British English - Female
+
+| Voice ID | Name | Quality | Description |
+|----------|------|---------|-------------|
+| `bf_emma` | Emma | B- | Elegant and clear |
+| `bf_isabella` | Isabella | C | Sophisticated |
+
+### British English - Male
+
+| Voice ID | Name | Quality | Description |
+|----------|------|---------|-------------|
+| `bm_george` | George | C | Distinguished |
+| `bm_fable` | Fable | C | Storyteller tone |
+
+> **Tip:** `af_heart` is the highest quality voice. Start there and experiment with others.
+
+### Supertonic Voices
+
+Supertonic provides 4 high-quality voices:
+
+| Voice | Gender | Description |
+|-------|--------|-------------|
+| `F1` | Female | Clear and natural |
+| `F2` | Female | Warm and expressive |
+| `M1` | Male | Deep and confident |
+| `M2` | Male | Friendly and casual |
+
+## API Reference
+
+### Gerbil Class Methods
+
+```typescript
+class Gerbil {
+  // Load TTS model (auto-called by speak if needed)
+  async loadTTS(options?: LoadTTSOptions): Promise<void>;
+  
+  // Check if TTS is loaded
+  isTTSLoaded(): boolean;
+  
+  // Generate speech
+  async speak(text: string, options?: SpeakOptions): Promise<SpeakResult>;
+  
+  // Stream speech (yields chunks as generated)
+  async *speakStream(text: string, options?: SpeakOptions): AsyncGenerator<AudioChunk>;
+  
+  // Get available voices
+  listVoices(): VoiceInfo[];
+  getVoice(voiceId: string): VoiceInfo | null;
+}
+```
+
+### SpeakOptions
+
+```typescript
+interface SpeakOptions {
+  /** Voice ID (default: "af_heart") */
+  voice?: string;
+  
+  /** Speed multiplier 0.5-2.0 (default: 1.0) */
+  speed?: number;
+  
+  /** Progress callback during loading */
+  onProgress?: (info: ProgressInfo) => void;
+  
+  /** Callback for each audio chunk (streaming) */
+  onAudioChunk?: (chunk: AudioChunk) => void;
+}
+```
+
+### SpeakResult
+
+```typescript
+interface SpeakResult {
+  /** PCM audio samples (mono, float32, -1 to 1) */
+  audio: Float32Array;
+  
+  /** Sample rate (always 24000 for Kokoro) */
+  sampleRate: number;
+  
+  /** Duration in seconds */
+  duration: number;
+  
+  /** Voice ID used */
+  voice: string;
+  
+  /** Generation time in milliseconds */
+  totalTime: number;
+}
+```
+
+## Playing Audio
+
+### Browser (Web Audio API)
+
+```typescript
+import { playAudio } from "@tryhamster/gerbil/browser";
+
+const result = await gerbil.speak("Hello!");
+
+// One-liner playback
+const controller = await playAudio(result.audio, result.sampleRate);
+
+// Stop early
+controller.stop();
+
+// Wait for completion
+await controller.onEnded;
+```
+
+### Node.js (Save to File)
+
+```typescript
+import { writeFileSync } from "fs";
+import { execSync } from "child_process";
+
+const result = await gerbil.speak("Hello!");
+
+// Convert to WAV
+function saveWav(filename: string, audio: Float32Array, sampleRate: number) {
+  const buffer = Buffer.alloc(44 + audio.length * 2);
+  
+  // WAV header
+  buffer.write("RIFF", 0);
+  buffer.writeUInt32LE(36 + audio.length * 2, 4);
+  buffer.write("WAVE", 8);
+  buffer.write("fmt ", 12);
+  buffer.writeUInt32LE(16, 16);
+  buffer.writeUInt16LE(1, 20);  // PCM
+  buffer.writeUInt16LE(1, 22);  // Mono
+  buffer.writeUInt32LE(sampleRate, 24);
+  buffer.writeUInt32LE(sampleRate * 2, 28);
+  buffer.writeUInt16LE(2, 32);
+  buffer.writeUInt16LE(16, 34);
+  buffer.write("data", 36);
+  buffer.writeUInt32LE(audio.length * 2, 40);
+  
+  // Audio data
+  for (let i = 0; i < audio.length; i++) {
+    const s = Math.max(-1, Math.min(1, audio[i]));
+    buffer.writeInt16LE(Math.round(s * 32767), 44 + i * 2);
+  }
+  
+  writeFileSync(filename, buffer);
+}
+
+saveWav("output.wav", result.audio, result.sampleRate);
+
+// Play on macOS
+execSync("afplay output.wav");
+
+// Play on Linux
+// execSync("aplay output.wav");
+```
+
+## Streaming Audio
+
+For long text, stream audio chunks as they're generated:
+
+```typescript
+// Node.js streaming
+for await (const chunk of gerbil.speakStream("Long paragraph of text...")) {
+  console.log(`Chunk ${chunk.index}: ${chunk.samples.length} samples`);
+  // Process/play chunk.samples
+  
+  if (chunk.isFinal) {
+    console.log("Done!");
+  }
+}
+
+// Browser streaming with seamless playback
+import { createAudioPlayer } from "@tryhamster/gerbil/browser";
+
+const player = createAudioPlayer(24000);
+
+for await (const chunk of gerbil.speakStream("Long text...")) {
+  player.queue(chunk.samples);
+}
+```
+
+## useSpeech Hook Reference
+
+```typescript
+const {
+  // Actions
+  speak,          // (text: string, opts?) => Promise<void>
+  stop,           // () => void
+  load,           // () => void (manual load trigger)
+  
+  // State
+  isLoading,      // boolean - model loading
+  isSpeaking,     // boolean - currently speaking
+  isReady,        // boolean - model ready
+  error,          // string | null
+  
+  // Voice control
+  listVoices,     // () => VoiceInfo[] - voices for current model
+  currentVoice,   // string
+  setVoice,       // (id: string) => void
+  currentSpeed,   // number
+  setSpeed,       // (speed: number) => void
+  
+  // Model info
+  currentModel,   // TTSModelId - "kokoro-82m" | "supertonic-66m"
+  sampleRate,     // number - 24000 (Kokoro) or 44100 (Supertonic)
+  
+  // Loading progress
+  loadingProgress, // { status, file?, progress? }
+} = useSpeech({
+  model: "kokoro-82m", // TTS model: "kokoro-82m" (default) or "supertonic-66m"
+  voice: "af_heart",   // default voice (model-specific)
+  speed: 1.0,          // default speed
+  autoLoad: false,     // load on first speak() call
+  onReady: () => {},
+  onError: (err) => {},
+  onStart: () => {},
+  onEnd: () => {},
+});
+```
+
+### Model-Specific Voices
+
+Each model has its own set of voices:
+
+```typescript
+// Kokoro (default) - 28 voices
+const { listVoices } = useSpeech(); // or model: "kokoro-82m"
+// → af_heart, af_bella, am_fenrir, bf_emma, bm_george, ...
+
+// Supertonic - 4 voices  
+const { listVoices } = useSpeech({ model: "supertonic-66m" });
+// → F1, F2, M1, M2
+```
+
+**Note:** Voice IDs are model-specific. Using a Kokoro voice with Supertonic (or vice versa) will throw an error.
+
+## Performance
+
+| Metric | Value |
+|--------|-------|
+| Model size | ~330MB |
+| First load | 3-10s (downloads model) |
+| Cached load | <1s |
+| Generation speed | ~8x realtime on M1 Mac |
+| Sample rate | 24kHz |
+| Audio format | Mono Float32 PCM |
+
+## Troubleshooting
+
+### Audio sounds like gibberish
+
+This happens when using raw transformers.js without proper phoneme conversion. Gerbil uses `kokoro-js` which handles grapheme-to-phoneme (G2P) conversion automatically.
+
+### Audio is too quiet
+
+Kokoro output can be quiet. The audio is normalized by default, but if you're processing raw audio, ensure proper normalization:
+
+```typescript
+function normalizeAudio(audio: Float32Array, targetRms = 0.15): Float32Array {
+  const currentRms = Math.sqrt(audio.reduce((a, b) => a + b * b, 0) / audio.length);
+  const gain = targetRms / currentRms;
+  return audio.map(s => Math.max(-1, Math.min(1, s * gain)));
+}
+```
+
+### Browser autoplay blocked
+
+Browsers require user interaction before playing audio. Trigger speech from a click handler:
+
+```tsx
+<button onClick={() => speak("Hello!")}>Speak</button>
+```
+
+### Voice not found
+
+Use `listVoices()` to see available voice IDs. Voice IDs follow the pattern `{language}{gender}_{name}`:
+- `af_` = American female
+- `am_` = American male
+- `bf_` = British female
+- `bm_` = British male
+
+## Model Info
+
+### Kokoro-82M
+
+Uses [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) via the [ONNX export](https://huggingface.co/onnx-community/Kokoro-82M-v1.0-ONNX) and [kokoro-js](https://www.npmjs.com/package/kokoro-js) wrapper.
+
+- **Architecture:** StyleTTS2-based
+- **Parameters:** 82M
+- **Languages:** English (US + UK)
+- **License:** Apache 2.0
+
+### Supertonic-66M
+
+Uses [Supertonic-TTS](https://huggingface.co/onnx-community/Supertonic-TTS-ONNX) via transformers.js pipeline.
+
+- **Parameters:** 66M
+- **Languages:** English
+- **Sample Rate:** 44.1kHz (HiFi)
+- **Speed:** ~167x realtime
+- **License:** Apache 2.0
+
+---
+
+## Adding New TTS Models (Developer Guide)
+
+To add a new TTS model to Gerbil, implement it in these locations:
+
+### 1. Core TTS Class (`src/core/tts.ts`)
+
+Add your model to the TTS registry and create a class that implements the TTS interface:
+
+```typescript
+// 1. Add voice definitions
+export const MY_MODEL_VOICES: VoiceInfo[] = [
+  { id: "voice1", name: "Voice 1", gender: "female", language: "en", description: "..." },
+  // ...
+];
+
+// 2. Add to TTS_MODELS registry
+export const TTS_MODELS: Record<string, TTSModelConfig> = {
+  // ...existing models...
+  "my-model-50m": {
+    id: "my-model-50m",
+    repo: "hf-org/my-model-onnx",
+    description: "My Model 50M - Description",
+    size: "~150MB",
+    sampleRate: 22050,
+    voices: MY_MODEL_VOICES,
+    defaultVoice: "voice1",
+    languages: ["en"],
+  },
+};
+
+// 3. Create TTS class (see KokoroTTS or SupertonicTTS as examples)
+export class MyModelTTS {
+  async load(options: LoadTTSOptions): Promise<void> { /* ... */ }
+  async speak(text: string, options: SpeakOptions): Promise<SpeakResult> { /* ... */ }
+  listVoices(): VoiceInfo[] { /* ... */ }
+  // ...
+}
+
+// 4. Update createTTS factory
+export function createTTS(modelId: string = "kokoro-82m"): TTSBackend {
+  if (modelId.startsWith("my-model")) {
+    return new MyModelTTS(modelId);
+  }
+  // ...existing logic...
+}
+```
+
+### 2. Browser Hook (`src/browser/index.ts`)
+
+Add browser support in the useSpeech hook:
+
+```typescript
+// 1. Add voice definitions for browser
+const MY_MODEL_BROWSER_VOICES: BrowserVoiceInfo[] = [
+  { id: "voice1", name: "Voice 1", gender: "female", language: "en", description: "..." },
+];
+
+// 2. Update TTS_MODELS in browser
+const TTS_MODELS: Record<TTSModelId, {...}> = {
+  // ...existing models...
+  "my-model-50m": {
+    repo: "hf-org/my-model-onnx",
+    defaultVoice: "voice1",
+    sampleRate: 22050,
+    voices: MY_MODEL_BROWSER_VOICES,
+  },
+};
+
+// 3. Update TTSModelId type
+export type TTSModelId = "kokoro-82m" | "supertonic-66m" | "my-model-50m";
+
+// 4. Add loading logic in useSpeech useEffect
+if (modelId === "my-model-50m") {
+  // Load using appropriate method (pipeline, custom loader, etc.)
+}
+```
+
+### 3. Gerbil Class (`src/core/gerbil.ts`)
+
+Update the main Gerbil class to support the new model:
+
+```typescript
+// Add type import
+type MyModelTTSType = import("./tts.js").MyModelTTS;
+
+// Update TTSBackendType union
+type TTSBackendType = KokoroTTSType | SupertonicTTSType | MyModelTTSType;
+```
+
+### 4. Integration Points
+
+Update these integration files if applicable:
+
+- **AI SDK** (`src/integrations/ai-sdk.ts`): Add speech model support
+- **Skills** (`src/skills/builtin/speak.ts`, etc.): Update voice enums
+- **CLI** (`src/cli/index.ts`): Add model to CLI options
+
+### 5. Documentation
+
+Update documentation:
+
+- `docs/tts.md`: Add model to tables and examples
+- `.cursor/rules/gerbil/tts.mdc`: Update implementation patterns
+
+### File Summary
+
+| File | Purpose |
+|------|---------|
+| `src/core/tts.ts` | Core TTS classes, voice registry, model config |
+| `src/browser/index.ts` | React hooks (useSpeech, useVoiceChat) |
+| `src/core/gerbil.ts` | Main Gerbil class integration |
+| `src/integrations/ai-sdk.ts` | AI SDK v5 speech model |
+| `src/skills/builtin/speak.ts` | Speak skill voice options |
+| `docs/tts.md` | User documentation |
+| `.cursor/rules/gerbil/tts.mdc` | Developer patterns |
+