@tryhamster/gerbil 1.1.2 → 1.1.3

package/docs/architecture/overview.md

@@ -1,189 +0,0 @@
-# Architecture Overview
-
-Gerbil is a local LLM inference library that runs entirely on-device, with no API calls or cloud dependencies.
-
-## Core Components
-
-### 1. Gerbil Class (`src/core/gerbil.ts`)
-
-The main entry point for Node.js applications:
-
-```typescript
-const g = new Gerbil();
-await g.loadModel("qwen3-0.6b");
-const result = await g.generate("Hello");
-```
-
-Responsibilities:
-- Model loading and lifecycle management
-- Device selection (WebGPU vs CPU)
-- Generation orchestration
-- Streaming coordination
-- Session statistics
-- Request queue (concurrency control)
-- Telemetry hooks for observability
-
-### 2. Model Registry (`src/core/models.ts`)
-
-Maps friendly model IDs to Hugging Face paths:
-
-```typescript
-const BUILTIN_MODELS = {
-  "qwen3-0.6b": {
-    id: "qwen3-0.6b",
-    path: "onnx-community/Qwen3-0.6B-ONNX",
-    family: "qwen",
-    size: "0.6B",
-    contextLength: 32768,
-    supportsThinking: true,
-  },
-  // ...
-};
-```
-
-### 3. Chrome GPU Backend (`src/core/chrome-backend.ts`)
-
-Enables WebGPU in Node.js by using headless Chrome as a GPU accelerator:
-
-```
-┌─────────────┐    HTTP     ┌──────────────────┐
-│   Node.js   │◄──────────►│  Headless Chrome  │
-│  (Gerbil)   │   :43724   │  (WebGPU worker)  │
-└─────────────┘            └──────────────────┘
-       │                          │
-       │ CDP (DevTools)           │ WebGPU
-       └──────────────────────────┘
-                                  │
-                            ┌─────▼─────┐
-                            │    GPU    │
-                            └───────────┘
-```
-
-### 4. Browser Module (`src/browser/index.ts`)
-
-Provides React hooks and workers for browser applications:
-
-```typescript
-// LLM worker
-const gerbil = await createGerbilWorker({
-  modelId: "qwen3-0.6b",
-  onToken: (token) => console.log(token.text),
-});
-
-// React hooks
-const { messages, handleSubmit } = useChat();
-const { speak, isSpeaking } = useSpeech();
-const { startRecording, transcript } = useVoiceInput();
-```
-
-Uses inline Web Workers for:
-- **LLM**: Model loading, token streaming, GPU memory management
-- **TTS**: Kokoro/Supertonic speech synthesis
-- **STT**: Whisper transcription
-
-All workers load dependencies from CDN to avoid bundler issues with onnxruntime-web.
-
-## Execution Paths
-
-### Browser Path
-
-```
-User Code
-    │
-    ▼
-createGerbilWorker()
-    │
-    ▼
-Web Worker (inline blob)
-    │
-    ▼
-transformers.js
-    │
-    ▼
-ONNX Runtime (WebGPU)
-    │
-    ▼
-GPU
-```
-
-### Node.js CPU Path
-
-```
-User Code
-    │
-    ▼
-Gerbil.generate()
-    │
-    ▼
-transformers.js pipeline
-    │
-    ▼
-ONNX Runtime (CPU)
-    │
-    ▼
-CPU (with SIMD)
-```
-
-### Node.js WebGPU Path
-
-```
-User Code
-    │
-    ▼
-Gerbil.generate()
-    │
-    ▼
-ChromeGPUBackend
-    │
-    ▼
-Headless Chrome (via CDP)
-    │
-    ▼
-transformers.js (in Chrome)
-    │
-    ▼
-ONNX Runtime (WebGPU)
-    │
-    ▼
-GPU
-```
-
-## Device Selection
-
-Gerbil automatically selects the best available backend:
-
-```typescript
-// Explicit selection
-await g.loadModel("qwen3-0.6b", { device: "webgpu" }); // or "cpu"
-
-// Check current device
-g.getDeviceMode(); // "webgpu" | "cpu" | "wasm"
-```
-
-Priority:
-1. **Browser**: WebGPU → WASM fallback
-2. **Node.js with --gpu**: ChromeGPUBackend (headless Chrome)
-3. **Node.js default**: CPU via ONNX Runtime
-
-## File Structure
-
-```
-src/
-├── core/
-│   ├── gerbil.ts         # Main Gerbil class
-│   ├── models.ts         # Model registry
-│   ├── types.ts          # TypeScript types
-│   ├── tools.ts          # Tool calling system
-│   └── chrome-backend.ts # Node.js WebGPU via Chrome
-├── browser/
-│   └── index.ts          # React hooks + LLM/TTS/STT workers
-├── skills/
-│   └── ...               # Built-in skills (commit, summarize, etc.)
-├── integrations/
-│   └── ...               # AI SDK, LangChain, MCP
-├── frameworks/
-│   └── ...               # Next.js, Express, React, etc.
-└── cli/
-    └── repl/             # Interactive terminal UI
-```
-

package/docs/architecture/streaming.md

@@ -1,261 +0,0 @@
-# Streaming Architecture
-
-How Gerbil streams tokens in real-time without blocking the UI.
-
-## The Problem
-
-LLM inference is computationally intensive:
-- Model loading: 1-30 seconds
-- Token generation: 10-100ms per token
-- Total generation: 1-30 seconds
-
-Running this on the main thread would freeze the UI completely.
-
-## Solution: Web Workers
-
-Gerbil runs inference in a Web Worker, keeping the main thread free for UI updates.
-
-```
-┌─────────────────────────────────────────────────────────────┐
-│                        Main Thread                           │
-│                                                               │
-│  ┌─────────────┐         ┌─────────────────────────────────┐ │
-│  │   Your UI   │◄───────►│      GerbilWorker API           │ │
-│  │  (React,    │ tokens  │  - generate()                   │ │
-│  │   Vue, etc) │         │  - interrupt()                  │ │
-│  └─────────────┘         │  - reset()                      │ │
-│                          └──────────────┬──────────────────┘ │
-│                                         │                     │
-│                                         │ postMessage         │
-│                                         ▼                     │
-├─────────────────────────────────────────────────────────────┤
-│                        Web Worker                            │
-│                                                               │
-│  ┌─────────────────────────────────────────────────────────┐ │
-│  │                   ModelPipeline                          │ │
-│  │                                                          │ │
-│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────────┐  │ │
-│  │  │  Tokenizer  │  │    Model    │  │  TextStreamer   │  │ │
-│  │  └─────────────┘  └─────────────┘  └────────┬────────┘  │ │
-│  │                                              │           │ │
-│  │                                    tokens    │           │ │
-│  │                                              ▼           │ │
-│  │                                    postMessage(token)    │ │
-│  └─────────────────────────────────────────────────────────┘ │
-│                                                               │
-└─────────────────────────────────────────────────────────────┘
-```
-
-## Browser Implementation
-
-### Inline Worker
-
-Gerbil creates an inline Web Worker from a blob URL:
-
-```typescript
-const workerCode = `
-  import { AutoModelForCausalLM, TextStreamer } from "@huggingface/transformers";
-  
-  // ... worker code ...
-  
-  self.addEventListener("message", async (e) => {
-    const { type, ...data } = e.data;
-    switch (type) {
-      case "load": await load(data); break;
-      case "generate": await generate(data); break;
-      case "interrupt": stoppingCriteria.interrupt(); break;
-    }
-  });
-`;
-
-const blob = new Blob([workerCode], { type: "application/javascript" });
-const workerUrl = URL.createObjectURL(blob);
-const worker = new Worker(workerUrl, { type: "module" });
-```
-
-### Message Protocol
-
-```typescript
-// Main → Worker
-{ type: "load", modelId: "qwen3-0.6b" }
-{ type: "generate", messages: [...], options: {...} }
-{ type: "interrupt" }
-{ type: "reset" }
-
-// Worker → Main
-{ status: "loading", message: "Loading model..." }
-{ status: "progress", file: "model.onnx", progress: 50 }
-{ status: "ready" }
-{ status: "token", text: "Hello", state: "answering", tps: 75 }
-{ status: "complete", text: "Hello world!", numTokens: 3, tps: 75 }
-{ status: "error", error: "Out of memory" }
-```
-
-### Token Streaming
-
-The `TextStreamer` class from transformers.js calls back for each token:
-
-```typescript
-const streamer = new TextStreamer(tokenizer, {
-  skip_prompt: true,
-  skip_special_tokens: true,
-  callback_function: (text) => {
-    self.postMessage({ status: "token", text, state, tps });
-  },
-  token_callback_function: (tokens) => {
-    // Track thinking state from special tokens
-    const tokenId = Number(tokens[0]);
-    if (tokenId === START_THINKING_TOKEN_ID) state = "thinking";
-    if (tokenId === END_THINKING_TOKEN_ID) state = "answering";
-  },
-});
-```
-
-## Node.js Implementation
-
-### CPU Mode
-
-For CPU inference, Gerbil uses "fake streaming" - generate all tokens first, then yield:
-
-```typescript
-async *stream(prompt, options) {
-  const result = await this.generateRaw(prompt, options);
-  
-  // Yield word by word for streaming effect
-  const words = result.rawText.split(/(\s+)/);
-  for (const word of words) {
-    yield word;
-    options.onToken?.(word);
-  }
-  
-  return result.result;
-}
-```
-
-### WebGPU Mode (ChromeGPUBackend)
-
-For WebGPU, real streaming via Chrome DevTools Protocol:
-
-```typescript
-async *stream(prompt, options) {
-  const tokenQueue: string[] = [];
-  let resolveNext: ((value: string | null) => void) | null = null;
-  let done = false;
-  
-  // Start generation with streaming callback
-  const generatePromise = this.chromeBackend.generate(prompt, {
-    onToken: (token) => {
-      if (resolveNext) {
-        resolveNext(token.text);
-        resolveNext = null;
-      } else {
-        tokenQueue.push(token.text);
-      }
-    },
-  });
-  
-  // Yield tokens as they arrive
-  while (!done || tokenQueue.length > 0) {
-    if (tokenQueue.length > 0) {
-      yield tokenQueue.shift()!;
-    } else if (!done) {
-      const token = await new Promise<string | null>((resolve) => {
-        resolveNext = resolve;
-      });
-      if (token) yield token;
-    }
-  }
-  
-  await generatePromise;
-}
-```
-
-## Interruption
-
-Users can interrupt generation mid-stream:
-
-```typescript
-// Browser
-gerbil.interrupt();
-
-// Internally uses InterruptableStoppingCriteria
-const stoppingCriteria = new InterruptableStoppingCriteria();
-
-// On interrupt message:
-stoppingCriteria.interrupt();
-```
-
-The model stops generating after the current token.
-
-## Thinking State Tracking
-
-For Qwen3 thinking mode, Gerbil tracks whether the model is "thinking" or "answering":
-
-```typescript
-const [START_THINKING_TOKEN_ID, END_THINKING_TOKEN_ID] = 
-  tokenizer.encode("<think></think>", { add_special_tokens: false });
-
-let state = "answering";
-
-const tokenCallback = (tokens) => {
-  const tokenId = Number(tokens[0]);
-  if (tokenId === START_THINKING_TOKEN_ID) state = "thinking";
-  if (tokenId === END_THINKING_TOKEN_ID) state = "answering";
-};
-```
-
-This allows the UI to display thinking content differently:
-
-```typescript
-onToken: (token) => {
-  if (token.state === "thinking") {
-    setThinking(t => t + token.text);
-  } else {
-    setResponse(r => r + token.text);
-  }
-}
-```
-
-## Performance Considerations
-
-### Batching
-
-Tokens arrive very fast (~100/sec). Consider batching UI updates:
-
-```typescript
-let buffer = "";
-let updateScheduled = false;
-
-onToken: (token) => {
-  buffer += token.text;
-  if (!updateScheduled) {
-    updateScheduled = true;
-    requestAnimationFrame(() => {
-      setResponse(r => r + buffer);
-      buffer = "";
-      updateScheduled = false;
-    });
-  }
-}
-```
-
-### Memory
-
-The Web Worker has its own memory space. Large models may need:
-- Closing other tabs
-- Using smaller models
-- Falling back to server-side inference
-
-### Cleanup
-
-Always terminate workers when done:
-
-```typescript
-useEffect(() => {
-  const worker = await createGerbilWorker({ ... });
-  workerRef.current = worker;
-  
-  return () => worker.terminate();
-}, []);
-```
-

package/docs/architecture/webgpu.md

@@ -1,213 +0,0 @@
-# WebGPU Acceleration
-
-How Gerbil uses WebGPU for GPU-accelerated inference.
-
-## What is WebGPU?
-
-WebGPU is a modern graphics and compute API that provides access to GPU hardware from web browsers and (experimentally) Node.js.
-
-Key features:
-- **Cross-platform**: Works on Windows, macOS, Linux, ChromeOS
-- **Modern**: Designed for ML workloads, not just graphics
-- **Standardized**: W3C specification, implemented by all major browsers
-
-## Browser WebGPU
-
-In browsers, WebGPU is available via `navigator.gpu`:
-
-```typescript
-// Check support
-if (!navigator.gpu) {
-  console.log("WebGPU not supported");
-  return;
-}
-
-// Get adapter
-const adapter = await navigator.gpu.requestAdapter();
-const device = await adapter.requestDevice();
-```
-
-### Browser Support
-
-| Browser | Version | Status |
-|---------|---------|--------|
-| Chrome | 113+ | ✅ Full support |
-| Edge | 113+ | ✅ Full support |
-| Safari | 18+ | ⚠️ Partial (some bugs) |
-| Firefox | Nightly | 🔧 Behind flag |
-
-### Gerbil Browser API
-
-```typescript
-import { createGerbilWorker, isWebGPUSupported } from "@tryhamster/gerbil/browser";
-
-if (!isWebGPUSupported()) {
-  // Fall back to server-side inference
-  return;
-}
-
-const gerbil = await createGerbilWorker({
-  modelId: "qwen3-0.6b",
-  onToken: (token) => console.log(token.text),
-});
-```
-
-## Node.js WebGPU (ChromeGPUBackend)
-
-Node.js doesn't have native WebGPU support, so Gerbil uses a creative solution: **headless Chrome as a WebGPU accelerator**.
-
-### Architecture
-
-```
-┌─────────────────────────────────────────────────────────────┐
-│                        Node.js Process                       │
-│                                                               │
-│  ┌─────────────┐         ┌─────────────────────────────────┐ │
-│  │   Gerbil    │◄───────►│         HTTP Server             │ │
-│  │   (your     │         │       (port 43724)              │ │
-│  │    code)    │         │                                 │ │
-│  └──────┬──────┘         └─────────────────────────────────┘ │
-│         │                              ▲                      │
-│         │ CDP                          │ Serves worker.html   │
-│         ▼                              │                      │
-│  ┌─────────────────────────────────────┴───────────────────┐ │
-│  │                   Headless Chrome                        │ │
-│  │                                                          │ │
-│  │  ┌────────────────────────────────────────────────────┐ │ │
-│  │  │                  Worker Page                        │ │ │
-│  │  │                                                     │ │ │
-│  │  │  ┌─────────────┐  ┌─────────────┐  ┌────────────┐ │ │ │
-│  │  │  │transformers │  │ ONNX Runtime│  │   WebGPU   │ │ │ │
-│  │  │  │     .js     │  │   (WASM)    │  │  Shaders   │ │ │ │
-│  │  │  └─────────────┘  └─────────────┘  └────────────┘ │ │ │
-│  │  └────────────────────────────────────────────────────┘ │ │
-│  │                           │                              │ │
-│  │                           ▼                              │ │
-│  │                    ┌─────────────┐                       │ │
-│  │                    │     GPU     │                       │ │
-│  │                    └─────────────┘                       │ │
-│  └──────────────────────────────────────────────────────────┘ │
-└─────────────────────────────────────────────────────────────┘
-```
-
-### How It Works
-
-1. **HTTP Server**: Gerbil starts a local HTTP server on port 43724
-2. **Chrome Launch**: Puppeteer launches headless Chrome with WebGPU flags
-3. **Page Load**: Chrome navigates to the HTTP server, loading the worker page
-4. **Model Load**: transformers.js loads the model with WebGPU backend
-5. **CDP Communication**: Gerbil sends prompts via Chrome DevTools Protocol
-6. **Streaming**: Tokens stream back via console.log → CDP events
-
-### Why Port 43724?
-
-The port is fixed (43724 = "GERBI" on a phone keypad) for a critical reason: **IndexedDB cache consistency**.
-
-IndexedDB caches are origin-specific. A fixed port means:
-- Same origin every time → same cache
-- Model downloads are cached and reused
-- ~1.5s startup when cached vs ~20s first run
-
-### Chrome Launch Flags
-
-```typescript
-const args = [
-  "--enable-unsafe-webgpu",      // Enable WebGPU
-  "--enable-gpu",                 // Enable GPU acceleration
-  "--no-sandbox",                 // Required for some environments
-  "--disable-setuid-sandbox",
-  "--headless=new",               // New headless mode
-  // Platform-specific...
-];
-
-// Linux: Use Vulkan backend
-if (process.platform === "linux") {
-  args.push("--enable-features=Vulkan");
-  args.push("--use-vulkan");
-}
-```
-
-### Singleton Pattern
-
-The Chrome browser and HTTP server are singletons:
-
-```typescript
-let globalBrowser: Browser | null = null;
-let globalServer: Server | null = null;
-
-// Multiple Gerbil instances share the same browser
-const backend1 = await ChromeGPUBackend.create({ modelId: "qwen3-0.6b" });
-const backend2 = await ChromeGPUBackend.create({ modelId: "smollm2-360m" });
-// Both use the same Chrome process
-```
-
-This avoids:
-- Multiple Chrome processes consuming GPU memory
-- Port conflicts from multiple HTTP servers
-- Redundant model downloads
-
-## Performance Comparison
-
-Tested on Apple M4 Max with Qwen3-0.6B:
-
-| Backend | Tokens/sec | First Token | Notes |
-|---------|------------|-------------|-------|
-| WebGPU (Browser) | ~80 tok/s | ~50ms | Native WebGPU |
-| WebGPU (Node.js) | ~75 tok/s | ~60ms | Via Chrome |
-| CPU (Node.js) | ~100+ tok/s | ~500ms | Apple Neural Engine |
-| WASM (Browser) | ~10 tok/s | ~200ms | Fallback only |
-
-Note: On Apple Silicon, CPU can be faster than WebGPU for small models due to the unified memory architecture and optimized ONNX Runtime.
-
-## Shader Compilation
-
-WebGPU compiles shaders (GPU programs) at runtime. This causes a one-time delay:
-
-```
-First run:
-  Model download:     ~15-30s (cached after)
-  Shader compilation: ~5-10s  (cached in browser)
-  First token:        ~50ms
-
-Subsequent runs:
-  Model load:         ~1-2s   (from IndexedDB)
-  Shader load:        ~0.5s   (from GPU cache)
-  First token:        ~50ms
-```
-
-### Warm-up
-
-Gerbil performs a warm-up generation after model load:
-
-```typescript
-// Compile shaders with a minimal generation
-const warmupInputs = tokenizer("a");
-await model.generate({ ...warmupInputs, max_new_tokens: 1 });
-```
-
-This ensures shaders are compiled before the user's first prompt.
-
-## Troubleshooting
-
-### "WebGPU not supported"
-
-**Browser:**
-- Update to Chrome/Edge 113+
-- Check `chrome://gpu` for WebGPU status
-- Try enabling `chrome://flags/#enable-unsafe-webgpu`
-
-**Node.js:**
-- Ensure Chrome is installed
-- Set `CHROME_PATH` environment variable if Chrome is in a non-standard location
-- Check GPU drivers are up to date
-
-### Slow shader compilation
-
-This is normal on first run. The shaders are cached after compilation.
-
-### Out of GPU memory
-
-- Close other GPU-intensive applications
-- Use a smaller model (`smollm2-135m`)
-- Fall back to CPU mode
-