@tryhamster/gerbil 1.0.0-rc.0 → 1.0.0-rc.2

package/docs/stt.md

@@ -0,0 +1,494 @@
+# Speech-to-Text (STT)
+
+Gerbil provides local speech-to-text transcription using Whisper ONNX models via transformers.js. No API keys required - everything runs on-device.
+
+## Quick Start
+
+### Node.js
+
+```typescript
+import { Gerbil } from "@tryhamster/gerbil";
+import { readFileSync } from "fs";
+
+const g = new Gerbil();
+
+// Transcribe a WAV file
+const audioData = new Uint8Array(readFileSync("audio.wav"));
+const result = await g.transcribe(audioData);
+console.log(result.text);
+```
+
+### CLI
+
+```bash
+# Transcribe audio file
+gerbil transcribe audio.wav
+
+# With timestamps
+gerbil transcribe audio.wav --timestamps
+
+# List available models
+gerbil transcribe --list-models
+
+# Use a different model
+gerbil transcribe audio.wav --model whisper-base.en
+```
+
+## Available Models
+
+| Model | Size | Languages | Description |
+|-------|------|-----------|-------------|
+| `whisper-tiny.en` | 39M | English | Fastest, good for simple audio |
+| `whisper-tiny` | 39M | Multi | Multilingual tiny model |
+| `whisper-base.en` | 74M | English | Good balance of speed/accuracy |
+| `whisper-base` | 74M | Multi | Multilingual base model |
+| `whisper-small.en` | 244M | English | High quality transcription |
+| `whisper-small` | 244M | Multi | High quality multilingual |
+| `whisper-large-v3-turbo` | 809M | 80+ langs | Best quality, 5.4x faster than v3 |
+
+## API Reference
+
+### Gerbil Class Methods
+
+```typescript
+// Load STT model explicitly (optional - auto-loads on first transcribe)
+await g.loadSTT("whisper-tiny.en", {
+  onProgress: (p) => console.log(p.status),
+});
+
+// Transcribe audio
+const result = await g.transcribe(audioData, {
+  language: "en",      // Language hint (multilingual models only)
+  timestamps: true,    // Return segment timestamps
+  onProgress: (p) => console.log(p.status),
+});
+
+// Result structure
+console.log(result.text);       // Full transcription
+console.log(result.duration);   // Audio duration in seconds
+console.log(result.totalTime);  // Processing time in ms
+console.log(result.segments);   // Timestamped segments (if requested)
+
+// Check if loaded
+console.log(g.isSTTLoaded());
+
+// List available models
+const models = await g.listSTTModels();
+```
+
+### Audio Input Formats
+
+```typescript
+// WAV file (Uint8Array) - automatically decoded
+const wavData = new Uint8Array(fs.readFileSync("audio.wav"));
+await g.transcribe(wavData);
+
+// Raw audio (Float32Array at 16kHz mono)
+const raw16k = new Float32Array([...]); // 16kHz mono samples
+await g.transcribe(raw16k);
+
+// WAV at any sample rate - automatically resampled to 16kHz
+const wav44k = new Uint8Array(fs.readFileSync("audio-44khz.wav"));
+await g.transcribe(wav44k); // Resampled internally
+```
+
+### Transcription with Timestamps
+
+```typescript
+const result = await g.transcribe(audioData, { timestamps: true });
+
+for (const segment of result.segments || []) {
+  console.log(`[${segment.start.toFixed(1)}s - ${segment.end.toFixed(1)}s] ${segment.text}`);
+}
+// [0.0s - 5.2s] Hello world, this is a test
+// [5.2s - 8.0s] of the transcription system
+```
+
+### Language Detection (Multilingual Models)
+
+```typescript
+// Use multilingual model
+await g.loadSTT("whisper-base");
+
+// Let Whisper detect language
+const result = await g.transcribe(audioData);
+console.log(result.language); // "en", "es", "fr", etc.
+
+// Or provide a language hint
+const spanishResult = await g.transcribe(audioData, { language: "es" });
+```
+
+## Direct WhisperSTT Class
+
+For lower-level control:
+
+```typescript
+import { WhisperSTT, decodeWav, resampleAudio } from "@tryhamster/gerbil/core/stt";
+
+const stt = new WhisperSTT("whisper-tiny.en");
+
+await stt.load({
+  onProgress: (p) => console.log(p.status),
+});
+
+const result = await stt.transcribe(audioData, {
+  timestamps: true,
+});
+
+console.log(result.text);
+
+stt.dispose();
+```
+
+### Audio Utilities
+
+```typescript
+import { decodeWav, resampleAudio } from "@tryhamster/gerbil/core/stt";
+
+// Decode WAV to Float32Array
+const { audio, sampleRate } = decodeWav(wavUint8Array);
+
+// Resample to 16kHz (Whisper requirement)
+const audio16k = resampleAudio(audio, sampleRate, 16000);
+```
+
+## Streaming Transcription (Real-time)
+
+Transcribe audio in chunks as it comes in - perfect for live captioning, call transcription, or real-time subtitles:
+
+```typescript
+import { Gerbil } from "@tryhamster/gerbil";
+
+const g = new Gerbil();
+await g.loadSTT("whisper-tiny.en");
+
+// Create streaming session
+const session = await g.createStreamingTranscription({
+  chunkDuration: 1500,  // Transcribe every 1.5 seconds (default)
+  onChunk: (text, chunkIndex) => {
+    console.log(`Chunk ${chunkIndex}: ${text}`);
+  },
+  onTranscript: (fullText) => {
+    console.log("Full transcript:", fullText);
+  },
+  onError: (err) => console.error(err),
+});
+
+// Start the session (begins automatic interval-based transcription)
+session.start();
+
+// Feed audio data as it comes in (Float32Array at 16kHz)
+// This could come from a microphone, WebRTC stream, etc.
+session.feedAudio(audioChunk);
+session.feedAudio(moreAudioChunk);
+
+// Or manually trigger transcription of buffered audio
+await session.flush();
+
+// Get current state
+console.log(session.getTranscript());   // Full accumulated text
+console.log(session.getChunkCount());   // Number of chunks processed
+console.log(session.isRunning());       // Whether session is active
+
+// Stop and get final transcript
+const finalText = await session.stop();
+console.log("Final:", finalText);
+
+// Reset for reuse
+session.reset();
+```
+
+### Session API
+
+```typescript
+interface StreamingTranscriptionSession {
+  feedAudio(audio: Float32Array): void;  // Add audio to buffer
+  flush(): Promise<string>;              // Transcribe buffer now
+  start(): void;                         // Start auto-transcription
+  stop(): Promise<string>;               // Stop and get final text
+  isRunning(): boolean;                  // Check if running
+  getTranscript(): string;               // Get current full text
+  getChunkCount(): number;               // Number of chunks processed
+  reset(): void;                         // Clear buffer and transcript
+}
+
+interface StreamingTranscriptionOptions {
+  chunkDuration?: number;    // Interval in ms (default: 1500)
+  minChunkSize?: number;     // Min samples before transcribing (default: 8000)
+  onChunk?: (text: string, chunkIndex: number) => void;
+  onTranscript?: (fullText: string) => void;
+  onError?: (error: string) => void;
+  language?: string;         // Language hint for multilingual models
+}
+```
+
+### Streaming with Microphone (Node.js)
+
+```typescript
+import { Gerbil, StreamingMicrophone } from "@tryhamster/gerbil";
+
+const g = new Gerbil();
+await g.loadSTT("whisper-tiny.en");
+
+// Create streaming transcription session
+const session = await g.createStreamingTranscription({
+  chunkDuration: 2500,
+  onChunk: (text, idx) => console.log(`[${idx}] ${text}`),
+  onTranscript: (full) => console.log("Transcript:", full),
+});
+
+session.start();
+
+// Create streaming microphone that feeds to session
+const mic = new StreamingMicrophone({
+  onAudioChunk: (audio) => session.feedAudio(audio),
+  chunkDuration: 500,  // 0.5s audio chunks
+});
+
+await mic.start();
+console.log("Recording... Press Ctrl+C to stop");
+
+// Handle graceful shutdown
+process.on("SIGINT", async () => {
+  await mic.stop();
+  const transcript = await session.stop();
+  console.log("\nFinal:", transcript);
+  process.exit(0);
+});
+```
+
+## React Hook (Browser)
+
+Use `useVoiceInput` for browser-based voice recording and transcription:
+
+```tsx
+import { useVoiceInput } from "@tryhamster/gerbil/browser";
+
+function VoiceInput() {
+  const {
+    startRecording,
+    stopRecording,
+    isRecording,
+    isTranscribing,
+    transcript,
+    error,
+  } = useVoiceInput({
+    model: "whisper-tiny.en",
+    onTranscript: (text) => console.log("User said:", text),
+  });
+
+  return (
+    <div>
+      <button onClick={isRecording ? stopRecording : startRecording}>
+        {isRecording ? "🔴 Stop" : "🎤 Record"}
+      </button>
+      {isTranscribing && <span>Transcribing...</span>}
+      {transcript && <p>{transcript}</p>}
+    </div>
+  );
+}
+```
+
+### Hook Options
+
+```typescript
+const {
+  startRecording,   // () => Promise<void> - start recording
+  stopRecording,    // () => Promise<string> - stop and transcribe
+  cancelRecording,  // () => void - stop without transcribing
+  transcribe,       // (audio: Float32Array) => Promise<string>
+  isRecording,      // boolean - currently recording
+  isTranscribing,   // boolean - transcribing audio
+  isLoading,        // boolean - model loading
+  isReady,          // boolean - model ready
+  transcript,       // string - latest result
+  loadingProgress,  // STTProgress - loading status
+  error,            // string | null
+  load,             // () => void - manually load model
+} = useVoiceInput({
+  model: "whisper-tiny.en",  // STT model ID
+  autoLoad: false,           // loads on first record
+  onReady: () => {},         // called when model ready
+  onTranscript: (text) => {},// called on each transcription
+  onError: (error) => {},    // called on errors
+  onProgress: (p) => {},     // loading progress
+});
+```
+
+## CLI Commands
+
+### Transcribe
+
+```bash
+# Basic transcription
+gerbil transcribe recording.wav
+
+# With timestamps
+gerbil transcribe recording.wav --timestamps
+
+# Save to file
+gerbil transcribe recording.wav --output transcript.txt
+
+# Use specific model
+gerbil transcribe recording.wav --model whisper-base.en
+
+# Multilingual with language hint
+gerbil transcribe recording.wav --model whisper-base --language es
+
+# List models
+gerbil transcribe --list-models
+```
+
+### Voice Chat (STT → LLM → TTS)
+
+Complete voice conversation loop from audio file:
+
+```bash
+# Voice chat with audio file
+gerbil voice question.wav
+
+# Customize models and voice
+gerbil voice question.wav --model qwen3-1.7b --voice bf_emma
+
+# With custom system prompt
+gerbil voice question.wav --system "You are a pirate. Speak like one!"
+
+# Enable thinking mode
+gerbil voice question.wav --thinking
+```
+
+This command:
+1. Transcribes your audio (Whisper)
+2. Generates a response (LLM)
+3. Speaks the response (Kokoro TTS)
+
+### Microphone Recording
+
+Record directly from your microphone:
+
+```typescript
+import { Gerbil } from "@tryhamster/gerbil";
+
+const g = new Gerbil();
+
+// Record for 5 seconds and transcribe
+const result = await g.listen(5000);
+console.log(result.text);
+
+// Check if microphone is available
+const available = await g.isMicrophoneAvailable();
+```
+
+**Requirements:** SoX must be installed for microphone recording:
+- macOS: `brew install sox`
+- Ubuntu: `sudo apt install sox`
+- Windows: Download from https://sox.sourceforge.net/
+
+### REPL Voice Input
+
+In the Gerbil REPL chat:
+- Press `Ctrl+R` to record from microphone (5 seconds)
+- The transcribed text appears in the input field
+- Combined with voice mode (`Ctrl+V`), enables full voice conversations
+
+```bash
+gerbil chat
+# In chat: Press ^R to record, ^V to enable voice responses
+```
+
+## Performance
+
+Transcription speed on Apple M1:
+
+| Model | Audio Length | Time | Speed |
+|-------|--------------|------|-------|
+| whisper-tiny.en | 11s | ~250ms | 44x realtime |
+| whisper-base.en | 11s | ~500ms | 22x realtime |
+| whisper-small.en | 11s | ~1.5s | 7x realtime |
+
+## Troubleshooting
+
+### "Unsupported bit depth"
+Only 16-bit WAV files are supported. Convert with:
+```bash
+ffmpeg -i audio.mp3 -ar 16000 -ac 1 -sample_fmt s16 output.wav
+```
+
+### Slow first transcription
+The first call downloads the model (~40-250MB). Subsequent calls use the cached model.
+
+### Memory usage
+Whisper models require:
+- tiny: ~150MB RAM
+- base: ~300MB RAM
+- small: ~600MB RAM
+- large-v3-turbo: ~1.5GB RAM
+
+## useVoiceChat Hook (Full Voice Conversation)
+
+For complete voice-to-voice conversations (STT → LLM → TTS):
+
+```tsx
+import { useVoiceChat } from "@tryhamster/gerbil/browser";
+
+function VoiceAssistant() {
+  const {
+    messages,
+    startListening,
+    stopListening,
+    isListening,
+    isSpeaking,
+    stage,  // "idle" | "listening" | "transcribing" | "thinking" | "speaking"
+    isReady,
+  } = useVoiceChat({
+    llmModel: "qwen3-0.6b",
+    sttModel: "whisper-tiny.en",
+    voice: "af_bella",
+    system: "You are a helpful voice assistant.",
+  });
+
+  return (
+    <div>
+      {messages.map(m => <p key={m.id}>{m.role}: {m.content}</p>)}
+      <button
+        onMouseDown={startListening}
+        onMouseUp={stopListening}
+      >
+        {stage === "idle" ? "🎤 Hold to Speak" : stage}
+      </button>
+    </div>
+  );
+}
+```
+
+## Adding Custom Models
+
+Gerbil supports any ONNX model with the `transformers.js` tag on HuggingFace:
+
+```typescript
+// Use any onnx-community model
+await g.loadSTT("whisper-large-v3-turbo");
+
+// Browse compatible models at:
+// https://huggingface.co/models?library=transformers.js&pipeline_tag=automatic-speech-recognition
+```
+
+To add a new model to the registry, edit `src/core/stt.ts`:
+
+```typescript
+{
+  id: "my-custom-model",
+  repo: "onnx-community/my-model",
+  description: "My custom model",
+  size: "100M",
+  multilingual: true,
+  languages: ["en", "es"],
+  sampleRate: 16000,
+}
+```
+
+## See Also
+
+- [Text-to-Speech (TTS)](./tts.md) - Speech synthesis
+- [Browser Hooks](./browser.md) - useChat, useVoiceInput, useVoiceChat
+