@tryhamster/gerbil 1.0.0-rc.8 → 1.0.0

package/docs/tts.md

@@ -1,569 +1,147 @@
 # Text-to-Speech
 
-Gerbil provides on-device text-to-speech using the [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) model. Generate natural-sounding speech locally without API keys or internet connection.
+Gerbil's native WebGPU engine synthesizes speech with **Kani-TTS-2** — an LFM2-350M
+codec-LM backbone driving NVIDIA's NeMo **NanoCodec** decoder (FSQ + causal HiFi-GAN).
+`engine.speak()` returns **22.05 kHz mono PCM**, fully on-device, no ONNX.
 
-## Quick Start
-
-### Node.js / CLI
-
-```typescript
-import { Gerbil } from "@tryhamster/gerbil";
-
-const g = new Gerbil();
-
-// Generate speech
-const result = await g.speak("Hello, I'm Gerbil!", { 
-  voice: "af_heart",  // American female (highest quality)
-  speed: 1.0 
-});
+> **Pre-1.0.** Kani-TTS-2 is the only TTS path. The old Kokoro / Supertonic ONNX /
+> transformers.js lane has been removed. The `Gerbil`-class `speak()` method still works for
+> backward compatibility but is now a thin wrapper over the native engine (see
+> [below](#gerbil-class-speak-native-wrapper)).
 
-// result.audio = Float32Array (PCM samples)
-// result.sampleRate = 24000
-// result.duration = seconds
-```
-
-### React (Browser)
-
-```tsx
-import { useSpeech } from "@tryhamster/gerbil/browser";
-
-function SpeechDemo() {
-  // Default: Kokoro TTS (24kHz, 28 voices)
-  const { speak, stop, isSpeaking, isLoading, listVoices, setVoice } = useSpeech();
-
-  // Or use Supertonic (44.1kHz, 4 voices, faster)
-  // const { speak, listVoices } = useSpeech({ model: "supertonic-66m" });
-
-  if (isLoading) return <div>Loading TTS model...</div>;
-
-  return (
-    <div>
-      <select onChange={e => setVoice(e.target.value)}>
-        {listVoices().map(v => (
-          <option key={v.id} value={v.id}>{v.name} ({v.language})</option>
-        ))}
-      </select>
-      
-      <button onClick={() => speak("Hello world!")}>
-        {isSpeaking ? "Speaking..." : "Speak"}
-      </button>
-      
-      {isSpeaking && <button onClick={stop}>Stop</button>}
-    </div>
-  );
-}
-```
+## Quick Start
 
-### AI SDK
+### Node
 
 ```typescript
-import { experimental_generateSpeech as generateSpeech } from "ai";
-import { gerbil } from "@tryhamster/gerbil/ai";
-
-const audio = await generateSpeech({
-  model: gerbil.speech(),
-  text: "Hello from Gerbil!",
-  voice: "bf_emma",  // British female
-});
-
-// audio.audioData = Uint8Array (WAV format)
-```
+import { WebGPUEngine } from "@tryhamster/gerbil/gpu";
 
-### CLI
+const engine = await WebGPUEngine.create({ repo: "nineninesix/kani-tts-2-en" });
 
-```bash
-# Speak text with default voice
-gerbil speak "Hello world"
+const { pcm, sampleRate, audioSeconds } = await engine.speak("Hello, I'm Gerbil!");
+// pcm: Float32Array in [-1, 1], sampleRate === 22050
+console.log(`${audioSeconds.toFixed(2)}s of audio`);
 
-# Specify voice and speed
-gerbil speak --voice bf_emma --speed 1.2 "Cheerio!"
+engine.destroy();
 ```
 
-### Skills
+### Options
 
 ```typescript
-import { speak, announce, readAloud } from "@tryhamster/gerbil/skills";
-
-// Simple speech
-await speak({ text: "Hello world", voice: "af_heart" });
-
-// AI-crafted announcement
-await announce({ 
-  message: "Build completed successfully",
-  style: "excited",  // casual, formal, excited, calm, urgent
-  voice: "af_bella"
-});
-
-// Read file aloud
-await readAloud({ 
-  content: "./README.md",
-  voice: "bf_emma",
-  summarizeIfLong: true  // Summarize if > 5000 chars
+const result = await engine.speak("Speak with feeling.", {
+  languageTag: "en_us",     // prepended as "{tag}: {text}" (default "en_us")
+  temperature: 1.0,         // sampling temperature (default 1.0)
+  topP: 0.95,               // nucleus threshold (default 0.95)
+  repetitionPenalty: 1.1,   // (default 1.1)
+  maxFrames: 2000,          // cap audio length (default: unbounded up to maxSeqLen)
 });
 ```
 
-## Available TTS Models
-
-Gerbil supports multiple TTS backends:
-
-| Model | Size | Sample Rate | Voices | Speed | Notes |
-|-------|------|-------------|--------|-------|-------|
-| `kokoro-82m` | ~330MB | 24000 Hz | 28 | ~100x RT | Default, highest quality |
-| `supertonic-66m` | ~250MB | 44100 Hz | 4 | ~167x RT | Faster, HiFi output |
-
-### Selecting a Model
-
-```typescript
-// Use default (Kokoro)
-await g.loadTTS();
-
-// Use Supertonic (faster, higher sample rate)
-await g.loadTTS({ model: "supertonic-66m" });
-
-// List available models
-const models = await g.listTTSModels();
-// [{ id: "kokoro-82m", sampleRate: 24000, voiceCount: 28 }, ...]
-```
-
-## Available Voices
-
-### Kokoro Voices
-
-Kokoro provides 28 voices across American and British English:
-
-### American English - Female (Recommended)
+`speak()` requires a Kani-TTS-2 checkpoint (architecture `KaniTTS2ForCausalLM`). The
+NanoCodec decoder checkpoint is downloaded lazily on first `speak()` call.
 
-| Voice ID | Name | Quality | Description |
-|----------|------|---------|-------------|
-| `af_heart` | Heart | A | Highest quality, warm tone |
-| `af_bella` | Bella | A- | Warm and friendly |
-| `af_nicole` | Nicole | B- | Soft and gentle |
-| `af_sarah` | Sarah | C+ | Clear and professional |
-
-### American English - Male
-
-| Voice ID | Name | Quality | Description |
-|----------|------|---------|-------------|
-| `am_fenrir` | Fenrir | C+ | Best male quality |
-| `am_michael` | Michael | C+ | Warm and friendly |
-| `am_puck` | Puck | C+ | Neutral tone |
-
-### British English - Female
-
-| Voice ID | Name | Quality | Description |
-|----------|------|---------|-------------|
-| `bf_emma` | Emma | B- | Elegant and clear |
-| `bf_isabella` | Isabella | C | Sophisticated |
-
-### British English - Male
-
-| Voice ID | Name | Quality | Description |
-|----------|------|---------|-------------|
-| `bm_george` | George | C | Distinguished |
-| `bm_fable` | Fable | C | Storyteller tone |
-
-> **Tip:** `af_heart` is the highest quality voice. Start there and experiment with others.
-
-### Supertonic Voices
-
-Supertonic provides 4 high-quality voices:
-
-| Voice | Gender | Description |
-|-------|--------|-------------|
-| `F1` | Female | Clear and natural |
-| `F2` | Female | Warm and expressive |
-| `M1` | Male | Deep and confident |
-| `M2` | Male | Friendly and casual |
-
-## API Reference
-
-### Gerbil Class Methods
+## Result
 
 ```typescript
-class Gerbil {
-  // Load TTS model (auto-called by speak if needed)
-  async loadTTS(options?: LoadTTSOptions): Promise<void>;
-  
-  // Check if TTS is loaded
-  isTTSLoaded(): boolean;
-  
-  // Generate speech
-  async speak(text: string, options?: SpeakOptions): Promise<SpeakResult>;
-  
-  // Stream speech (yields chunks as generated)
-  async *speakStream(text: string, options?: SpeakOptions): AsyncGenerator<AudioChunk>;
-  
-  // Get available voices
-  listVoices(): VoiceInfo[];
-  getVoice(voiceId: string): VoiceInfo | null;
+interface SpeakResult {
+  /** Mono PCM in [-1, 1]. */
+  pcm: Float32Array;
+  /** Sample rate — always 22050. */
+  sampleRate: number;
+  /** Number of audio frames decoded. */
+  frames: number;
+  /** Audio duration in seconds. */
+  audioSeconds: number;
 }
 ```
 
-### SpeakOptions
+## Standalone KaniTTS
+
+`engine.speak()` lazily builds a `KaniTTS` engine under the hood. You can also use it
+directly (e.g. without a separate text model):
 
 ```typescript
-interface SpeakOptions {
-  /** Voice ID (default: "af_heart") */
-  voice?: string;
-  
-  /** Speed multiplier 0.5-2.0 (default: 1.0) */
-  speed?: number;
-  
-  /** Progress callback during loading */
-  onProgress?: (info: ProgressInfo) => void;
-  
-  /** Callback for each audio chunk (streaming) */
-  onAudioChunk?: (chunk: AudioChunk) => void;
-}
-```
+import { KaniTTS } from "@tryhamster/gerbil/gpu";
 
-### SpeakResult
+const tts = await KaniTTS.create({
+  repo: "nineninesix/kani-tts-2-en",
+  // codecRepo defaults to the NeMo 22 kHz NanoCodec MLX checkpoint
+});
 
-```typescript
-interface SpeakResult {
-  /** PCM audio samples (mono, float32, -1 to 1) */
-  audio: Float32Array;
-  
-  /** Sample rate (always 24000 for Kokoro) */
-  sampleRate: number;
-  
-  /** Duration in seconds */
-  duration: number;
-  
-  /** Voice ID used */
-  voice: string;
-  
-  /** Generation time in milliseconds */
-  totalTime: number;
-}
+const { pcm, sampleRate } = await tts.speak("Hello world!");
+tts.destroy();
 ```
 
-## Playing Audio
+## Playing / Saving Audio
 
 ### Browser (Web Audio API)
 
 ```typescript
-import { playAudio } from "@tryhamster/gerbil/browser";
-
-const result = await gerbil.speak("Hello!");
-
-// One-liner playback
-const controller = await playAudio(result.audio, result.sampleRate);
-
-// Stop early
-controller.stop();
-
-// Wait for completion
-await controller.onEnded;
+const ctx = new AudioContext({ sampleRate });
+const buffer = ctx.createBuffer(1, pcm.length, sampleRate);
+buffer.copyToChannel(pcm, 0);
+const source = ctx.createBufferSource();
+source.buffer = buffer;
+source.connect(ctx.destination);
+source.start();
 ```
 
-### Node.js (Save to File)
+### Node (save to WAV)
 
 ```typescript
-import { writeFileSync } from "fs";
-import { execSync } from "child_process";
+import { writeFileSync } from "node:fs";
 
-const result = await gerbil.speak("Hello!");
-
-// Convert to WAV
 function saveWav(filename: string, audio: Float32Array, sampleRate: number) {
   const buffer = Buffer.alloc(44 + audio.length * 2);
-  
-  // WAV header
   buffer.write("RIFF", 0);
   buffer.writeUInt32LE(36 + audio.length * 2, 4);
   buffer.write("WAVE", 8);
   buffer.write("fmt ", 12);
   buffer.writeUInt32LE(16, 16);
-  buffer.writeUInt16LE(1, 20);  // PCM
-  buffer.writeUInt16LE(1, 22);  // Mono
+  buffer.writeUInt16LE(1, 20); // PCM
+  buffer.writeUInt16LE(1, 22); // mono
   buffer.writeUInt32LE(sampleRate, 24);
   buffer.writeUInt32LE(sampleRate * 2, 28);
   buffer.writeUInt16LE(2, 32);
   buffer.writeUInt16LE(16, 34);
   buffer.write("data", 36);
   buffer.writeUInt32LE(audio.length * 2, 40);
-  
-  // Audio data
   for (let i = 0; i < audio.length; i++) {
     const s = Math.max(-1, Math.min(1, audio[i]));
     buffer.writeInt16LE(Math.round(s * 32767), 44 + i * 2);
   }
-  
   writeFileSync(filename, buffer);
 }
 
-saveWav("output.wav", result.audio, result.sampleRate);
-
-// Play on macOS
-execSync("afplay output.wav");
-
-// Play on Linux
-// execSync("aplay output.wav");
+const { pcm, sampleRate } = await engine.speak("Hello!");
+saveWav("output.wav", pcm, sampleRate); // 22.05 kHz mono
+// macOS: execSync("afplay output.wav");
 ```
 
-## Streaming Audio
-
-For long text, stream audio chunks as they're generated:
-
-```typescript
-// Node.js streaming
-for await (const chunk of gerbil.speakStream("Long paragraph of text...")) {
-  console.log(`Chunk ${chunk.index}: ${chunk.samples.length} samples`);
-  // Process/play chunk.samples
-  
-  if (chunk.isFinal) {
-    console.log("Done!");
-  }
-}
+## Model
 
-// Browser streaming with seamless playback
-import { createAudioPlayer } from "@tryhamster/gerbil/browser";
+### Kani-TTS-2
 
-const player = createAudioPlayer(24000);
-
-for await (const chunk of gerbil.speakStream("Long text...")) {
-  player.queue(chunk.samples);
-}
-```
-
-## useSpeech Hook Reference
-
-```typescript
-const {
-  // Actions
-  speak,          // (text: string, opts?) => Promise<void>
-  stop,           // () => void
-  load,           // () => void (manual load trigger)
-  
-  // State
-  isLoading,      // boolean - model loading
-  isSpeaking,     // boolean - currently speaking
-  isReady,        // boolean - model ready
-  error,          // string | null
-  
-  // Voice control
-  listVoices,     // () => VoiceInfo[] - voices for current model
-  currentVoice,   // string
-  setVoice,       // (id: string) => void
-  currentSpeed,   // number
-  setSpeed,       // (speed: number) => void
-  
-  // Model info
-  currentModel,   // TTSModelId - "kokoro-82m" | "supertonic-66m"
-  sampleRate,     // number - 24000 (Kokoro) or 44100 (Supertonic)
-  
-  // Loading progress
-  loadingProgress, // { status, file?, progress? }
-} = useSpeech({
-  model: "kokoro-82m", // TTS model: "kokoro-82m" (default) or "supertonic-66m"
-  voice: "af_heart",   // default voice (model-specific)
-  speed: 1.0,          // default speed
-  autoLoad: false,     // load on first speak() call
-  onReady: () => {},
-  onError: (err) => {},
-  onStart: () => {},
-  onEnd: () => {},
-});
-```
-
-### Model-Specific Voices
-
-Each model has its own set of voices:
-
-```typescript
-// Kokoro (default) - 28 voices
-const { listVoices } = useSpeech(); // or model: "kokoro-82m"
-// → af_heart, af_bella, am_fenrir, bf_emma, bm_george, ...
-
-// Supertonic - 4 voices  
-const { listVoices } = useSpeech({ model: "supertonic-66m" });
-// → F1, F2, M1, M2
-```
-
-**Note:** Voice IDs are model-specific. Using a Kokoro voice with Supertonic (or vice versa) will throw an error.
-
-## Performance
-
-| Metric | Value |
-|--------|-------|
-| Model size | ~330MB |
-| First load | 3-10s (downloads model) |
-| Cached load | <1s |
-| Generation speed | ~8x realtime on M1 Mac |
-| Sample rate | 24kHz |
-| Audio format | Mono Float32 PCM |
-
-## Troubleshooting
-
-### Audio sounds like gibberish
-
-This happens when using raw transformers.js without proper phoneme conversion. Gerbil uses `kokoro-js` which handles grapheme-to-phoneme (G2P) conversion automatically.
-
-### Audio is too quiet
-
-Kokoro output can be quiet. The audio is normalized by default, but if you're processing raw audio, ensure proper normalization:
-
-```typescript
-function normalizeAudio(audio: Float32Array, targetRms = 0.15): Float32Array {
-  const currentRms = Math.sqrt(audio.reduce((a, b) => a + b * b, 0) / audio.length);
-  const gain = targetRms / currentRms;
-  return audio.map(s => Math.max(-1, Math.min(1, s * gain)));
-}
-```
-
-### Browser autoplay blocked
-
-Browsers require user interaction before playing audio. Trigger speech from a click handler:
-
-```tsx
-<button onClick={() => speak("Hello!")}>Speak</button>
-```
-
-### Voice not found
-
-Use `listVoices()` to see available voice IDs. Voice IDs follow the pattern `{language}{gender}_{name}`:
-- `af_` = American female
-- `am_` = American male
-- `bf_` = British female
-- `bm_` = British male
-
-## Model Info
-
-### Kokoro-82M
-
-Uses [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) via the [ONNX export](https://huggingface.co/onnx-community/Kokoro-82M-v1.0-ONNX) and [kokoro-js](https://www.npmjs.com/package/kokoro-js) wrapper.
-
-- **Architecture:** StyleTTS2-based
-- **Parameters:** 82M
-- **Languages:** English (US + UK)
-- **License:** Apache 2.0
-
-### Supertonic-66M
-
-Uses [Supertonic-TTS](https://huggingface.co/onnx-community/Supertonic-TTS-ONNX) via transformers.js pipeline.
-
-- **Parameters:** 66M
-- **Languages:** English
-- **Sample Rate:** 44.1kHz (HiFi)
-- **Speed:** ~167x realtime
-- **License:** Apache 2.0
+- **Backbone:** LFM2-350M codec-LM (architecture `KaniTTS2ForCausalLM`)
+- **Codec:** NVIDIA NeMo NanoCodec (FSQ + causal HiFi-GAN), validated bit-exact
+- **Sample rate:** 22.05 kHz
+- **Repos:** `nineninesix/kani-tts-2-en`. License varies by variant (the `kani-tts-2-en`
+  checkpoint is LFM1.0/other; a 450M variant is Apache 2.0).
 
 ---
 
-## Adding New TTS Models (Developer Guide)
+## `Gerbil`-class `speak()` (native wrapper)
 
-To add a new TTS model to Gerbil, implement it in these locations:
-
-### 1. Core TTS Class (`src/core/tts.ts`)
-
-Add your model to the TTS registry and create a class that implements the TTS interface:
+> The Kokoro-82M / Supertonic-66M ONNX/transformers.js TTS lane has been removed. The
+> `Gerbil`-class `speak()` method below now runs the native Kani-TTS-2 engine under the hood
+> (it requires WebGPU and returns 22.05 kHz PCM). The browser `useSpeech` hook is gone — use
+> `useTTS` from `@tryhamster/gerbil/gpu/hooks`. The AI SDK `gerbil.speech()` provider also
+> routes through this native path.
 
 ```typescript
-// 1. Add voice definitions
-export const MY_MODEL_VOICES: VoiceInfo[] = [
-  { id: "voice1", name: "Voice 1", gender: "female", language: "en", description: "..." },
-  // ...
-];
-
-// 2. Add to TTS_MODELS registry
-export const TTS_MODELS: Record<string, TTSModelConfig> = {
-  // ...existing models...
-  "my-model-50m": {
-    id: "my-model-50m",
-    repo: "hf-org/my-model-onnx",
-    description: "My Model 50M - Description",
-    size: "~150MB",
-    sampleRate: 22050,
-    voices: MY_MODEL_VOICES,
-    defaultVoice: "voice1",
-    languages: ["en"],
-  },
-};
-
-// 3. Create TTS class (see KokoroTTS or SupertonicTTS as examples)
-export class MyModelTTS {
-  async load(options: LoadTTSOptions): Promise<void> { /* ... */ }
-  async speak(text: string, options: SpeakOptions): Promise<SpeakResult> { /* ... */ }
-  listVoices(): VoiceInfo[] { /* ... */ }
-  // ...
-}
-
-// 4. Update createTTS factory
-export function createTTS(modelId: string = "kokoro-82m"): TTSBackend {
-  if (modelId.startsWith("my-model")) {
-    return new MyModelTTS(modelId);
-  }
-  // ...existing logic...
-}
-```
-
-### 2. Browser Hook (`src/browser/index.ts`)
-
-Add browser support in the useSpeech hook:
-
-```typescript
-// 1. Add voice definitions for browser
-const MY_MODEL_BROWSER_VOICES: BrowserVoiceInfo[] = [
-  { id: "voice1", name: "Voice 1", gender: "female", language: "en", description: "..." },
-];
-
-// 2. Update TTS_MODELS in browser
-const TTS_MODELS: Record<TTSModelId, {...}> = {
-  // ...existing models...
-  "my-model-50m": {
-    repo: "hf-org/my-model-onnx",
-    defaultVoice: "voice1",
-    sampleRate: 22050,
-    voices: MY_MODEL_BROWSER_VOICES,
-  },
-};
-
-// 3. Update TTSModelId type
-export type TTSModelId = "kokoro-82m" | "supertonic-66m" | "my-model-50m";
-
-// 4. Add loading logic in useSpeech useEffect
-if (modelId === "my-model-50m") {
-  // Load using appropriate method (pipeline, custom loader, etc.)
-}
-```
-
-### 3. Gerbil Class (`src/core/gerbil.ts`)
-
-Update the main Gerbil class to support the new model:
-
-```typescript
-// Add type import
-type MyModelTTSType = import("./tts.js").MyModelTTS;
+import { Gerbil } from "@tryhamster/gerbil";
 
-// Update TTSBackendType union
-type TTSBackendType = KokoroTTSType | SupertonicTTSType | MyModelTTSType;
+const g = new Gerbil();
+const result = await g.speak("Hello!");
+// result.audio = Float32Array, result.sampleRate = 22050 (native Kani-TTS-2)
 ```
-
-### 4. Integration Points
-
-Update these integration files if applicable:
-
-- **AI SDK** (`src/integrations/ai-sdk.ts`): Add speech model support
-- **Skills** (`src/skills/builtin/speak.ts`, etc.): Update voice enums
-- **CLI** (`src/cli/index.ts`): Add model to CLI options
-
-### 5. Documentation
-
-Update documentation:
-
-- `docs/tts.md`: Add model to tables and examples
-- `.cursor/rules/gerbil/tts.mdc`: Update implementation patterns
-
-### File Summary
-
-| File | Purpose |
-|------|---------|
-| `src/core/tts.ts` | Core TTS classes, voice registry, model config |
-| `src/browser/index.ts` | React hooks (useSpeech, useVoiceChat) |
-| `src/core/gerbil.ts` | Main Gerbil class integration |
-| `src/integrations/ai-sdk.ts` | AI SDK v5 speech model |
-| `src/skills/builtin/speak.ts` | Speak skill voice options |
-| `docs/tts.md` | User documentation |
-| `.cursor/rules/gerbil/tts.mdc` | Developer patterns |
-