@tryhamster/gerbil 1.0.0-rc.0

package/docs/architecture/streaming.md

@@ -0,0 +1,261 @@
+# Streaming Architecture
+
+How Gerbil streams tokens in real-time without blocking the UI.
+
+## The Problem
+
+LLM inference is computationally intensive:
+- Model loading: 1-30 seconds
+- Token generation: 10-100ms per token
+- Total generation: 1-30 seconds
+
+Running this on the main thread would freeze the UI completely.
+
+## Solution: Web Workers
+
+Gerbil runs inference in a Web Worker, keeping the main thread free for UI updates.
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                        Main Thread                           │
+│                                                               │
+│  ┌─────────────┐         ┌─────────────────────────────────┐ │
+│  │   Your UI   │◄───────►│      GerbilWorker API           │ │
+│  │  (React,    │ tokens  │  - generate()                   │ │
+│  │   Vue, etc) │         │  - interrupt()                  │ │
+│  └─────────────┘         │  - reset()                      │ │
+│                          └──────────────┬──────────────────┘ │
+│                                         │                     │
+│                                         │ postMessage         │
+│                                         ▼                     │
+├─────────────────────────────────────────────────────────────┤
+│                        Web Worker                            │
+│                                                               │
+│  ┌─────────────────────────────────────────────────────────┐ │
+│  │                   ModelPipeline                          │ │
+│  │                                                          │ │
+│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────────┐  │ │
+│  │  │  Tokenizer  │  │    Model    │  │  TextStreamer   │  │ │
+│  │  └─────────────┘  └─────────────┘  └────────┬────────┘  │ │
+│  │                                              │           │ │
+│  │                                    tokens    │           │ │
+│  │                                              ▼           │ │
+│  │                                    postMessage(token)    │ │
+│  └─────────────────────────────────────────────────────────┘ │
+│                                                               │
+└─────────────────────────────────────────────────────────────┘
+```
+
+## Browser Implementation
+
+### Inline Worker
+
+Gerbil creates an inline Web Worker from a blob URL:
+
+```typescript
+const workerCode = `
+  import { AutoModelForCausalLM, TextStreamer } from "@huggingface/transformers";
+  
+  // ... worker code ...
+  
+  self.addEventListener("message", async (e) => {
+    const { type, ...data } = e.data;
+    switch (type) {
+      case "load": await load(data); break;
+      case "generate": await generate(data); break;
+      case "interrupt": stoppingCriteria.interrupt(); break;
+    }
+  });
+`;
+
+const blob = new Blob([workerCode], { type: "application/javascript" });
+const workerUrl = URL.createObjectURL(blob);
+const worker = new Worker(workerUrl, { type: "module" });
+```
+
+### Message Protocol
+
+```typescript
+// Main → Worker
+{ type: "load", modelId: "qwen3-0.6b" }
+{ type: "generate", messages: [...], options: {...} }
+{ type: "interrupt" }
+{ type: "reset" }
+
+// Worker → Main
+{ status: "loading", message: "Loading model..." }
+{ status: "progress", file: "model.onnx", progress: 50 }
+{ status: "ready" }
+{ status: "token", text: "Hello", state: "answering", tps: 75 }
+{ status: "complete", text: "Hello world!", numTokens: 3, tps: 75 }
+{ status: "error", error: "Out of memory" }
+```
+
+### Token Streaming
+
+The `TextStreamer` class from transformers.js calls back for each token:
+
+```typescript
+const streamer = new TextStreamer(tokenizer, {
+  skip_prompt: true,
+  skip_special_tokens: true,
+  callback_function: (text) => {
+    self.postMessage({ status: "token", text, state, tps });
+  },
+  token_callback_function: (tokens) => {
+    // Track thinking state from special tokens
+    const tokenId = Number(tokens[0]);
+    if (tokenId === START_THINKING_TOKEN_ID) state = "thinking";
+    if (tokenId === END_THINKING_TOKEN_ID) state = "answering";
+  },
+});
+```
+
+## Node.js Implementation
+
+### CPU Mode
+
+For CPU inference, Gerbil uses "fake streaming" - generate all tokens first, then yield:
+
+```typescript
+async *stream(prompt, options) {
+  const result = await this.generateRaw(prompt, options);
+  
+  // Yield word by word for streaming effect
+  const words = result.rawText.split(/(\s+)/);
+  for (const word of words) {
+    yield word;
+    options.onToken?.(word);
+  }
+  
+  return result.result;
+}
+```
+
+### WebGPU Mode (ChromeGPUBackend)
+
+For WebGPU, real streaming via Chrome DevTools Protocol:
+
+```typescript
+async *stream(prompt, options) {
+  const tokenQueue: string[] = [];
+  let resolveNext: ((value: string | null) => void) | null = null;
+  let done = false;
+  
+  // Start generation with streaming callback
+  const generatePromise = this.chromeBackend.generate(prompt, {
+    onToken: (token) => {
+      if (resolveNext) {
+        resolveNext(token.text);
+        resolveNext = null;
+      } else {
+        tokenQueue.push(token.text);
+      }
+    },
+  });
+  
+  // Yield tokens as they arrive
+  while (!done || tokenQueue.length > 0) {
+    if (tokenQueue.length > 0) {
+      yield tokenQueue.shift()!;
+    } else if (!done) {
+      const token = await new Promise<string | null>((resolve) => {
+        resolveNext = resolve;
+      });
+      if (token) yield token;
+    }
+  }
+  
+  await generatePromise;
+}
+```
+
+## Interruption
+
+Users can interrupt generation mid-stream:
+
+```typescript
+// Browser
+gerbil.interrupt();
+
+// Internally uses InterruptableStoppingCriteria
+const stoppingCriteria = new InterruptableStoppingCriteria();
+
+// On interrupt message:
+stoppingCriteria.interrupt();
+```
+
+The model stops generating after the current token.
+
+## Thinking State Tracking
+
+For Qwen3 thinking mode, Gerbil tracks whether the model is "thinking" or "answering":
+
+```typescript
+const [START_THINKING_TOKEN_ID, END_THINKING_TOKEN_ID] = 
+  tokenizer.encode("<think></think>", { add_special_tokens: false });
+
+let state = "answering";
+
+const tokenCallback = (tokens) => {
+  const tokenId = Number(tokens[0]);
+  if (tokenId === START_THINKING_TOKEN_ID) state = "thinking";
+  if (tokenId === END_THINKING_TOKEN_ID) state = "answering";
+};
+```
+
+This allows the UI to display thinking content differently:
+
+```typescript
+onToken: (token) => {
+  if (token.state === "thinking") {
+    setThinking(t => t + token.text);
+  } else {
+    setResponse(r => r + token.text);
+  }
+}
+```
+
+## Performance Considerations
+
+### Batching
+
+Tokens arrive very fast (~100/sec). Consider batching UI updates:
+
+```typescript
+let buffer = "";
+let updateScheduled = false;
+
+onToken: (token) => {
+  buffer += token.text;
+  if (!updateScheduled) {
+    updateScheduled = true;
+    requestAnimationFrame(() => {
+      setResponse(r => r + buffer);
+      buffer = "";
+      updateScheduled = false;
+    });
+  }
+}
+```
+
+### Memory
+
+The Web Worker has its own memory space. Large models may need:
+- Closing other tabs
+- Using smaller models
+- Falling back to server-side inference
+
+### Cleanup
+
+Always terminate workers when done:
+
+```typescript
+useEffect(() => {
+  const worker = await createGerbilWorker({ ... });
+  workerRef.current = worker;
+  
+  return () => worker.terminate();
+}, []);
+```
+

package/docs/architecture/webgpu.md

@@ -0,0 +1,213 @@
+# WebGPU Acceleration
+
+How Gerbil uses WebGPU for GPU-accelerated inference.
+
+## What is WebGPU?
+
+WebGPU is a modern graphics and compute API that provides access to GPU hardware from web browsers and (experimentally) Node.js.
+
+Key features:
+- **Cross-platform**: Works on Windows, macOS, Linux, ChromeOS
+- **Modern**: Designed for ML workloads, not just graphics
+- **Standardized**: W3C specification, implemented by all major browsers
+
+## Browser WebGPU
+
+In browsers, WebGPU is available via `navigator.gpu`:
+
+```typescript
+// Check support
+if (!navigator.gpu) {
+  console.log("WebGPU not supported");
+  return;
+}
+
+// Get adapter
+const adapter = await navigator.gpu.requestAdapter();
+const device = await adapter.requestDevice();
+```
+
+### Browser Support
+
+| Browser | Version | Status |
+|---------|---------|--------|
+| Chrome | 113+ | ✅ Full support |
+| Edge | 113+ | ✅ Full support |
+| Safari | 18+ | ⚠️ Partial (some bugs) |
+| Firefox | Nightly | 🔧 Behind flag |
+
+### Gerbil Browser API
+
+```typescript
+import { createGerbilWorker, isWebGPUSupported } from "@tryhamster/gerbil/browser";
+
+if (!isWebGPUSupported()) {
+  // Fall back to server-side inference
+  return;
+}
+
+const gerbil = await createGerbilWorker({
+  modelId: "qwen3-0.6b",
+  onToken: (token) => console.log(token.text),
+});
+```
+
+## Node.js WebGPU (ChromeGPUBackend)
+
+Node.js doesn't have native WebGPU support, so Gerbil uses a creative solution: **headless Chrome as a WebGPU accelerator**.
+
+### Architecture
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                        Node.js Process                       │
+│                                                               │
+│  ┌─────────────┐         ┌─────────────────────────────────┐ │
+│  │   Gerbil    │◄───────►│         HTTP Server             │ │
+│  │   (your     │         │       (port 43724)              │ │
+│  │    code)    │         │                                 │ │
+│  └──────┬──────┘         └─────────────────────────────────┘ │
+│         │                              ▲                      │
+│         │ CDP                          │ Serves worker.html   │
+│         ▼                              │                      │
+│  ┌─────────────────────────────────────┴───────────────────┐ │
+│  │                   Headless Chrome                        │ │
+│  │                                                          │ │
+│  │  ┌────────────────────────────────────────────────────┐ │ │
+│  │  │                  Worker Page                        │ │ │
+│  │  │                                                     │ │ │
+│  │  │  ┌─────────────┐  ┌─────────────┐  ┌────────────┐ │ │ │
+│  │  │  │transformers │  │ ONNX Runtime│  │   WebGPU   │ │ │ │
+│  │  │  │     .js     │  │   (WASM)    │  │  Shaders   │ │ │ │
+│  │  │  └─────────────┘  └─────────────┘  └────────────┘ │ │ │
+│  │  └────────────────────────────────────────────────────┘ │ │
+│  │                           │                              │ │
+│  │                           ▼                              │ │
+│  │                    ┌─────────────┐                       │ │
+│  │                    │     GPU     │                       │ │
+│  │                    └─────────────┘                       │ │
+│  └──────────────────────────────────────────────────────────┘ │
+└─────────────────────────────────────────────────────────────┘
+```
+
+### How It Works
+
+1. **HTTP Server**: Gerbil starts a local HTTP server on port 43724
+2. **Chrome Launch**: Puppeteer launches headless Chrome with WebGPU flags
+3. **Page Load**: Chrome navigates to the HTTP server, loading the worker page
+4. **Model Load**: transformers.js loads the model with WebGPU backend
+5. **CDP Communication**: Gerbil sends prompts via Chrome DevTools Protocol
+6. **Streaming**: Tokens stream back via console.log → CDP events
+
+### Why Port 43724?
+
+The port is fixed (43724 = "GERBI" on a phone keypad) for a critical reason: **IndexedDB cache consistency**.
+
+IndexedDB caches are origin-specific. A fixed port means:
+- Same origin every time → same cache
+- Model downloads are cached and reused
+- ~1.5s startup when cached vs ~20s first run
+
+### Chrome Launch Flags
+
+```typescript
+const args = [
+  "--enable-unsafe-webgpu",      // Enable WebGPU
+  "--enable-gpu",                 // Enable GPU acceleration
+  "--no-sandbox",                 // Required for some environments
+  "--disable-setuid-sandbox",
+  "--headless=new",               // New headless mode
+  // Platform-specific...
+];
+
+// Linux: Use Vulkan backend
+if (process.platform === "linux") {
+  args.push("--enable-features=Vulkan");
+  args.push("--use-vulkan");
+}
+```
+
+### Singleton Pattern
+
+The Chrome browser and HTTP server are singletons:
+
+```typescript
+let globalBrowser: Browser | null = null;
+let globalServer: Server | null = null;
+
+// Multiple Gerbil instances share the same browser
+const backend1 = await ChromeGPUBackend.create({ modelId: "qwen3-0.6b" });
+const backend2 = await ChromeGPUBackend.create({ modelId: "smollm2-360m" });
+// Both use the same Chrome process
+```
+
+This avoids:
+- Multiple Chrome processes consuming GPU memory
+- Port conflicts from multiple HTTP servers
+- Redundant model downloads
+
+## Performance Comparison
+
+Tested on Apple M4 Max with Qwen3-0.6B:
+
+| Backend | Tokens/sec | First Token | Notes |
+|---------|------------|-------------|-------|
+| WebGPU (Browser) | ~80 tok/s | ~50ms | Native WebGPU |
+| WebGPU (Node.js) | ~75 tok/s | ~60ms | Via Chrome |
+| CPU (Node.js) | ~100+ tok/s | ~500ms | Apple Neural Engine |
+| WASM (Browser) | ~10 tok/s | ~200ms | Fallback only |
+
+Note: On Apple Silicon, CPU can be faster than WebGPU for small models due to the unified memory architecture and optimized ONNX Runtime.
+
+## Shader Compilation
+
+WebGPU compiles shaders (GPU programs) at runtime. This causes a one-time delay:
+
+```
+First run:
+  Model download:     ~15-30s (cached after)
+  Shader compilation: ~5-10s  (cached in browser)
+  First token:        ~50ms
+
+Subsequent runs:
+  Model load:         ~1-2s   (from IndexedDB)
+  Shader load:        ~0.5s   (from GPU cache)
+  First token:        ~50ms
+```
+
+### Warm-up
+
+Gerbil performs a warm-up generation after model load:
+
+```typescript
+// Compile shaders with a minimal generation
+const warmupInputs = tokenizer("a");
+await model.generate({ ...warmupInputs, max_new_tokens: 1 });
+```
+
+This ensures shaders are compiled before the user's first prompt.
+
+## Troubleshooting
+
+### "WebGPU not supported"
+
+**Browser:**
+- Update to Chrome/Edge 113+
+- Check `chrome://gpu` for WebGPU status
+- Try enabling `chrome://flags/#enable-unsafe-webgpu`
+
+**Node.js:**
+- Ensure Chrome is installed
+- Set `CHROME_PATH` environment variable if Chrome is in a non-standard location
+- Check GPU drivers are up to date
+
+### Slow shader compilation
+
+This is normal on first run. The shaders are cached after compilation.
+
+### Out of GPU memory
+
+- Close other GPU-intensive applications
+- Use a smaller model (`smollm2-135m`)
+- Fall back to CPU mode
+