@tryhamster/gerbil 1.1.2 → 1.2.0

package/docs/architecture/caching.md

@@ -1,227 +0,0 @@
-# Model Caching
-
-How Gerbil caches models for fast subsequent loads.
-
-## Overview
-
-Model files are large (100MB - 500MB). Gerbil caches them locally to avoid re-downloading:
-
-| Environment | Cache Location | Mechanism |
-|-------------|----------------|-----------|
-| Browser | IndexedDB | transformers.js built-in |
-| Node.js (CPU) | `~/.cache/huggingface/hub` | transformers.js built-in |
-| Node.js (WebGPU) | Chrome's IndexedDB | Via ChromeGPUBackend |
-
-## Browser Caching
-
-### IndexedDB
-
-transformers.js automatically caches model files in IndexedDB:
-
-```
-IndexedDB
-└── transformers-cache
-    └── onnx-community/Qwen3-0.6B-ONNX
-        ├── tokenizer.json
-        ├── config.json
-        ├── model_q4f16.onnx
-        └── ...
-```
-
-### Cache Behavior
-
-1. **First load**: Downloads from Hugging Face Hub (~15-30s)
-2. **Subsequent loads**: Reads from IndexedDB (~1-2s)
-
-### Checking Cache
-
-```typescript
-import { getWebGPUInfo } from "@tryhamster/gerbil/browser";
-
-// Models are cached per-origin
-// Same origin = same cache
-```
-
-### Clearing Cache
-
-```javascript
-// In browser DevTools:
-indexedDB.deleteDatabase("transformers-cache");
-```
-
-## Node.js CPU Caching
-
-### Hugging Face Hub Cache
-
-transformers.js uses the standard HF cache directory:
-
-```
-~/.cache/huggingface/hub/
-└── models--onnx-community--Qwen3-0.6B-ONNX/
-    ├── blobs/
-    │   └── [sha256 hashes]
-    ├── refs/
-    │   └── main
-    └── snapshots/
-        └── [commit hash]/
-            ├── tokenizer.json
-            ├── config.json
-            └── model_q4.onnx
-```
-
-### Environment Variables
-
-```bash
-# Custom cache directory
-export HF_HOME=/path/to/cache
-export TRANSFORMERS_CACHE=/path/to/cache
-
-# Offline mode (use cache only)
-export TRANSFORMERS_OFFLINE=1
-```
-
-### CLI Cache Management
-
-```bash
-# View cache
-npx @tryhamster/gerbil cache
-
-# Clear cache
-npx @tryhamster/gerbil cache --clean
-
-# Clear old models
-npx @tryhamster/gerbil cache --older-than 30
-```
-
-## Node.js WebGPU Caching
-
-### ChromeGPUBackend Cache
-
-When using WebGPU in Node.js, models are cached in Chrome's IndexedDB:
-
-```
-~/.gerbil/chrome-cache/
-└── Default/
-    └── IndexedDB/
-        └── http_127.0.0.1_43724.indexeddb.leveldb/
-            └── [model cache]
-```
-
-### Why a Fixed Port?
-
-The ChromeGPUBackend uses port 43724 ("GERBI") for a critical reason:
-
-IndexedDB caches are **origin-specific**. The origin includes:
-- Protocol: `http://`
-- Host: `127.0.0.1`
-- Port: `43724`
-
-A fixed port ensures the same origin every time → same cache.
-
-```typescript
-const GERBIL_LOCAL_PORT = 43724; // "GERBI" on phone keypad
-
-// Always same origin:
-// http://127.0.0.1:43724
-```
-
-### Cache Persistence
-
-The Chrome user data directory persists between runs:
-
-```typescript
-this.userDataDir = join(homedir(), ".gerbil", "chrome-cache");
-```
-
-This means:
-- Model downloads are cached
-- Shader compilations are cached
-- ~1.5s startup when cached vs ~20s first run
-
-## Cache Sizes
-
-| Model | Download Size | Cache Size |
-|-------|--------------|------------|
-| qwen3-0.6b | ~400MB | ~400MB |
-| smollm2-360m | ~250MB | ~250MB |
-| smollm2-135m | ~100MB | ~100MB |
-
-## Preloading Models
-
-### Browser
-
-```typescript
-// Preload during idle time
-const gerbil = await createGerbilWorker({
-  modelId: "qwen3-0.6b",
-  onProgress: (p) => {
-    if (p.status === "ready") {
-      console.log("Model cached and ready");
-    }
-  },
-});
-
-// Model is now in IndexedDB for instant loads
-```
-
-### Node.js
-
-```typescript
-// Preload in background
-const g = new Gerbil();
-await g.loadModel("qwen3-0.6b");
-// Model is now in HF cache
-```
-
-### CLI
-
-```bash
-# Download without running
-npx @tryhamster/gerbil info -m qwen3-0.6b
-# Model is now cached
-```
-
-## Offline Usage
-
-Once cached, models work offline:
-
-```typescript
-// Browser: Works if model is in IndexedDB
-const gerbil = await createGerbilWorker({ modelId: "qwen3-0.6b" });
-
-// Node.js: Set offline mode
-process.env.TRANSFORMERS_OFFLINE = "1";
-const g = new Gerbil();
-await g.loadModel("qwen3-0.6b"); // Uses cache only
-```
-
-## Troubleshooting
-
-### "Model not found" after cache clear
-
-Re-download by loading the model:
-
-```typescript
-await g.loadModel("qwen3-0.6b"); // Will re-download
-```
-
-### Cache taking too much space
-
-```bash
-# View cache size
-npx @tryhamster/gerbil cache
-
-# Clear old models
-npx @tryhamster/gerbil cache --older-than 7
-
-# Clear everything
-npx @tryhamster/gerbil cache --clean
-```
-
-### Browser cache not persisting
-
-Check browser settings:
-- Cookies/site data must be allowed
-- IndexedDB must not be blocked
-- Storage quota must not be exceeded
-

package/docs/architecture/inference.md

@@ -1,176 +0,0 @@
-# Inference Pipeline
-
-How Gerbil runs LLM inference using ONNX Runtime and transformers.js.
-
-## The Stack
-
-```
-┌─────────────────────────────────────┐
-│           transformers.js           │  ← Tokenization, model loading, generation
-├─────────────────────────────────────┤
-│            ONNX Runtime             │  ← Neural network execution
-├─────────────────────────────────────┤
-│     WebGPU  │   CPU   │   WASM      │  ← Execution backends
-└─────────────────────────────────────┘
-```
-
-## transformers.js
-
-[transformers.js](https://huggingface.co/docs/transformers.js) is a JavaScript port of Hugging Face Transformers that runs ONNX models in the browser and Node.js.
-
-### Model Loading
-
-```typescript
-import { AutoModelForCausalLM, AutoTokenizer } from "@huggingface/transformers";
-
-const tokenizer = await AutoTokenizer.from_pretrained(modelId);
-const model = await AutoModelForCausalLM.from_pretrained(modelId, {
-  dtype: "q4f16",   // Quantization
-  device: "webgpu", // Backend
-});
-```
-
-### Generation
-
-```typescript
-const inputs = tokenizer.apply_chat_template(messages, {
-  add_generation_prompt: true,
-  return_dict: true,
-});
-
-const output = await model.generate({
-  ...inputs,
-  max_new_tokens: 256,
-  temperature: 0.7,
-  do_sample: true,
-});
-
-const text = tokenizer.decode(output[0], { skip_special_tokens: true });
-```
-
-### Streaming with TextStreamer
-
-```typescript
-const streamer = new TextStreamer(tokenizer, {
-  skip_prompt: true,
-  skip_special_tokens: true,
-  callback_function: (text) => {
-    console.log(text); // Called for each token
-  },
-});
-
-await model.generate({ ...inputs, streamer });
-```
-
-## ONNX Runtime
-
-ONNX Runtime is the execution engine that runs the neural network operations.
-
-### Backends
-
-| Backend | Environment | Speed | Notes |
-|---------|-------------|-------|-------|
-| **WebGPU** | Browser, Chrome | ~70-100 tok/s | Fastest, requires GPU |
-| **CPU** | Node.js | ~10-30 tok/s | Uses SIMD, good on Apple Silicon |
-| **WASM** | Browser fallback | ~5-10 tok/s | Works everywhere |
-
-### Execution Providers
-
-ONNX Runtime selects execution providers based on availability:
-
-```
-WebGPU EP → WASM EP → CPU EP
-```
-
-Gerbil explicitly requests the desired backend:
-
-```typescript
-// For WebGPU
-await AutoModelForCausalLM.from_pretrained(modelId, { device: "webgpu" });
-
-// For CPU
-await pipeline("text-generation", modelId, { device: "cpu" });
-```
-
-## Quantization
-
-Quantization reduces model size and improves inference speed by using lower-precision numbers.
-
-### Quantization Types
-
-| Type | Weights | Compute | Size Reduction | Use Case |
-|------|---------|---------|----------------|----------|
-| **fp32** | 32-bit float | 32-bit | 1x (baseline) | Training |
-| **fp16** | 16-bit float | 16-bit | 2x | GPU inference |
-| **q4f16** | 4-bit int | 16-bit | ~4x | WebGPU inference |
-| **q4** | 4-bit int | 32-bit | ~4x | CPU inference |
-
-### Why q4f16 for WebGPU?
-
-WebGPU shaders work best with fp16 compute. The `q4f16` format:
-- Stores weights as 4-bit integers (small download)
-- Dequantizes to fp16 during inference (fast on GPU)
-- Maintains good quality for small models
-
-### Model Sizes
-
-| Model | Original | q4f16 | Download |
-|-------|----------|-------|----------|
-| Qwen3-0.6B | ~2.4GB | ~400MB | ~400MB |
-| SmolLM2-360M | ~1.4GB | ~250MB | ~250MB |
-| SmolLM2-135M | ~540MB | ~100MB | ~100MB |
-
-## Tokenization
-
-### Chat Templates
-
-Gerbil uses model-specific chat templates to format conversations:
-
-```typescript
-const messages = [
-  { role: "system", content: "You are helpful." },
-  { role: "user", content: "Hello!" },
-];
-
-const inputs = tokenizer.apply_chat_template(messages, {
-  add_generation_prompt: true,  // Add assistant turn start
-  return_dict: true,            // Return input_ids + attention_mask
-  enable_thinking: true,        // Qwen3 thinking mode
-});
-```
-
-### Thinking Mode (Qwen3)
-
-Qwen3 models support a "thinking" mode where the model shows reasoning:
-
-```
-<think>
-Let me work through this step by step...
-127 × 43 = 127 × 40 + 127 × 3 = 5080 + 381 = 5461
-</think>
-The answer is 5461.
-```
-
-Enabled via `enable_thinking: true` in the chat template.
-
-## KV Cache
-
-The Key-Value cache stores intermediate attention states to speed up autoregressive generation:
-
-```typescript
-const { past_key_values, sequences } = await model.generate({
-  ...inputs,
-  past_key_values: previousCache,  // Reuse from last turn
-  return_dict_in_generate: true,
-});
-
-// Save for next turn
-cache = past_key_values;
-```
-
-Benefits:
-- Faster multi-turn conversations
-- Reduced compute for long contexts
-
-Gerbil manages the KV cache automatically for multi-turn chat.
-

package/docs/architecture/overview.md

@@ -1,189 +0,0 @@
-# Architecture Overview
-
-Gerbil is a local LLM inference library that runs entirely on-device, with no API calls or cloud dependencies.
-
-## Core Components
-
-### 1. Gerbil Class (`src/core/gerbil.ts`)
-
-The main entry point for Node.js applications:
-
-```typescript
-const g = new Gerbil();
-await g.loadModel("qwen3-0.6b");
-const result = await g.generate("Hello");
-```
-
-Responsibilities:
-- Model loading and lifecycle management
-- Device selection (WebGPU vs CPU)
-- Generation orchestration
-- Streaming coordination
-- Session statistics
-- Request queue (concurrency control)
-- Telemetry hooks for observability
-
-### 2. Model Registry (`src/core/models.ts`)
-
-Maps friendly model IDs to Hugging Face paths:
-
-```typescript
-const BUILTIN_MODELS = {
-  "qwen3-0.6b": {
-    id: "qwen3-0.6b",
-    path: "onnx-community/Qwen3-0.6B-ONNX",
-    family: "qwen",
-    size: "0.6B",
-    contextLength: 32768,
-    supportsThinking: true,
-  },
-  // ...
-};
-```
-
-### 3. Chrome GPU Backend (`src/core/chrome-backend.ts`)
-
-Enables WebGPU in Node.js by using headless Chrome as a GPU accelerator:
-
-```
-┌─────────────┐    HTTP     ┌──────────────────┐
-│   Node.js   │◄──────────►│  Headless Chrome  │
-│  (Gerbil)   │   :43724   │  (WebGPU worker)  │
-└─────────────┘            └──────────────────┘
-       │                          │
-       │ CDP (DevTools)           │ WebGPU
-       └──────────────────────────┘
-                                  │
-                            ┌─────▼─────┐
-                            │    GPU    │
-                            └───────────┘
-```
-
-### 4. Browser Module (`src/browser/index.ts`)
-
-Provides React hooks and workers for browser applications:
-
-```typescript
-// LLM worker
-const gerbil = await createGerbilWorker({
-  modelId: "qwen3-0.6b",
-  onToken: (token) => console.log(token.text),
-});
-
-// React hooks
-const { messages, handleSubmit } = useChat();
-const { speak, isSpeaking } = useSpeech();
-const { startRecording, transcript } = useVoiceInput();
-```
-
-Uses inline Web Workers for:
-- **LLM**: Model loading, token streaming, GPU memory management
-- **TTS**: Kokoro/Supertonic speech synthesis
-- **STT**: Whisper transcription
-
-All workers load dependencies from CDN to avoid bundler issues with onnxruntime-web.
-
-## Execution Paths
-
-### Browser Path
-
-```
-User Code
-    │
-    ▼
-createGerbilWorker()
-    │
-    ▼
-Web Worker (inline blob)
-    │
-    ▼
-transformers.js
-    │
-    ▼
-ONNX Runtime (WebGPU)
-    │
-    ▼
-GPU
-```
-
-### Node.js CPU Path
-
-```
-User Code
-    │
-    ▼
-Gerbil.generate()
-    │
-    ▼
-transformers.js pipeline
-    │
-    ▼
-ONNX Runtime (CPU)
-    │
-    ▼
-CPU (with SIMD)
-```
-
-### Node.js WebGPU Path
-
-```
-User Code
-    │
-    ▼
-Gerbil.generate()
-    │
-    ▼
-ChromeGPUBackend
-    │
-    ▼
-Headless Chrome (via CDP)
-    │
-    ▼
-transformers.js (in Chrome)
-    │
-    ▼
-ONNX Runtime (WebGPU)
-    │
-    ▼
-GPU
-```
-
-## Device Selection
-
-Gerbil automatically selects the best available backend:
-
-```typescript
-// Explicit selection
-await g.loadModel("qwen3-0.6b", { device: "webgpu" }); // or "cpu"
-
-// Check current device
-g.getDeviceMode(); // "webgpu" | "cpu" | "wasm"
-```
-
-Priority:
-1. **Browser**: WebGPU → WASM fallback
-2. **Node.js with --gpu**: ChromeGPUBackend (headless Chrome)
-3. **Node.js default**: CPU via ONNX Runtime
-
-## File Structure
-
-```
-src/
-├── core/
-│   ├── gerbil.ts         # Main Gerbil class
-│   ├── models.ts         # Model registry
-│   ├── types.ts          # TypeScript types
-│   ├── tools.ts          # Tool calling system
-│   └── chrome-backend.ts # Node.js WebGPU via Chrome
-├── browser/
-│   └── index.ts          # React hooks + LLM/TTS/STT workers
-├── skills/
-│   └── ...               # Built-in skills (commit, summarize, etc.)
-├── integrations/
-│   └── ...               # AI SDK, LangChain, MCP
-├── frameworks/
-│   └── ...               # Next.js, Express, React, etc.
-└── cli/
-    └── repl/             # Interactive terminal UI
-```
-