@tryhamster/gerbil 1.0.0-rc.0

package/docs/architecture/caching.md

@@ -0,0 +1,227 @@
+# Model Caching
+
+How Gerbil caches models for fast subsequent loads.
+
+## Overview
+
+Model files are large (100MB - 500MB). Gerbil caches them locally to avoid re-downloading:
+
+| Environment | Cache Location | Mechanism |
+|-------------|----------------|-----------|
+| Browser | IndexedDB | transformers.js built-in |
+| Node.js (CPU) | `~/.cache/huggingface/hub` | transformers.js built-in |
+| Node.js (WebGPU) | Chrome's IndexedDB | Via ChromeGPUBackend |
+
+## Browser Caching
+
+### IndexedDB
+
+transformers.js automatically caches model files in IndexedDB:
+
+```
+IndexedDB
+└── transformers-cache
+    └── onnx-community/Qwen3-0.6B-ONNX
+        ├── tokenizer.json
+        ├── config.json
+        ├── model_q4f16.onnx
+        └── ...
+```
+
+### Cache Behavior
+
+1. **First load**: Downloads from Hugging Face Hub (~15-30s)
+2. **Subsequent loads**: Reads from IndexedDB (~1-2s)
+
+### Checking Cache
+
+```typescript
+import { getWebGPUInfo } from "@tryhamster/gerbil/browser";
+
+// Models are cached per-origin
+// Same origin = same cache
+```
+
+### Clearing Cache
+
+```javascript
+// In browser DevTools:
+indexedDB.deleteDatabase("transformers-cache");
+```
+
+## Node.js CPU Caching
+
+### Hugging Face Hub Cache
+
+transformers.js uses the standard HF cache directory:
+
+```
+~/.cache/huggingface/hub/
+└── models--onnx-community--Qwen3-0.6B-ONNX/
+    ├── blobs/
+    │   └── [sha256 hashes]
+    ├── refs/
+    │   └── main
+    └── snapshots/
+        └── [commit hash]/
+            ├── tokenizer.json
+            ├── config.json
+            └── model_q4.onnx
+```
+
+### Environment Variables
+
+```bash
+# Custom cache directory
+export HF_HOME=/path/to/cache
+export TRANSFORMERS_CACHE=/path/to/cache
+
+# Offline mode (use cache only)
+export TRANSFORMERS_OFFLINE=1
+```
+
+### CLI Cache Management
+
+```bash
+# View cache
+npx @tryhamster/gerbil cache
+
+# Clear cache
+npx @tryhamster/gerbil cache --clean
+
+# Clear old models
+npx @tryhamster/gerbil cache --older-than 30
+```
+
+## Node.js WebGPU Caching
+
+### ChromeGPUBackend Cache
+
+When using WebGPU in Node.js, models are cached in Chrome's IndexedDB:
+
+```
+~/.gerbil/chrome-cache/
+└── Default/
+    └── IndexedDB/
+        └── http_127.0.0.1_43724.indexeddb.leveldb/
+            └── [model cache]
+```
+
+### Why a Fixed Port?
+
+The ChromeGPUBackend uses port 43724 ("GERBI") for a critical reason:
+
+IndexedDB caches are **origin-specific**. The origin includes:
+- Protocol: `http://`
+- Host: `127.0.0.1`
+- Port: `43724`
+
+A fixed port ensures the same origin every time → same cache.
+
+```typescript
+const GERBIL_LOCAL_PORT = 43724; // "GERBI" on phone keypad
+
+// Always same origin:
+// http://127.0.0.1:43724
+```
+
+### Cache Persistence
+
+The Chrome user data directory persists between runs:
+
+```typescript
+this.userDataDir = join(homedir(), ".gerbil", "chrome-cache");
+```
+
+This means:
+- Model downloads are cached
+- Shader compilations are cached
+- ~1.5s startup when cached vs ~20s first run
+
+## Cache Sizes
+
+| Model | Download Size | Cache Size |
+|-------|--------------|------------|
+| qwen3-0.6b | ~400MB | ~400MB |
+| smollm2-360m | ~250MB | ~250MB |
+| smollm2-135m | ~100MB | ~100MB |
+
+## Preloading Models
+
+### Browser
+
+```typescript
+// Preload during idle time
+const gerbil = await createGerbilWorker({
+  modelId: "qwen3-0.6b",
+  onProgress: (p) => {
+    if (p.status === "ready") {
+      console.log("Model cached and ready");
+    }
+  },
+});
+
+// Model is now in IndexedDB for instant loads
+```
+
+### Node.js
+
+```typescript
+// Preload in background
+const g = new Gerbil();
+await g.loadModel("qwen3-0.6b");
+// Model is now in HF cache
+```
+
+### CLI
+
+```bash
+# Download without running
+npx @tryhamster/gerbil info -m qwen3-0.6b
+# Model is now cached
+```
+
+## Offline Usage
+
+Once cached, models work offline:
+
+```typescript
+// Browser: Works if model is in IndexedDB
+const gerbil = await createGerbilWorker({ modelId: "qwen3-0.6b" });
+
+// Node.js: Set offline mode
+process.env.TRANSFORMERS_OFFLINE = "1";
+const g = new Gerbil();
+await g.loadModel("qwen3-0.6b"); // Uses cache only
+```
+
+## Troubleshooting
+
+### "Model not found" after cache clear
+
+Re-download by loading the model:
+
+```typescript
+await g.loadModel("qwen3-0.6b"); // Will re-download
+```
+
+### Cache taking too much space
+
+```bash
+# View cache size
+npx @tryhamster/gerbil cache
+
+# Clear old models
+npx @tryhamster/gerbil cache --older-than 7
+
+# Clear everything
+npx @tryhamster/gerbil cache --clean
+```
+
+### Browser cache not persisting
+
+Check browser settings:
+- Cookies/site data must be allowed
+- IndexedDB must not be blocked
+- Storage quota must not be exceeded
+

package/docs/architecture/inference.md

@@ -0,0 +1,176 @@
+# Inference Pipeline
+
+How Gerbil runs LLM inference using ONNX Runtime and transformers.js.
+
+## The Stack
+
+```
+┌─────────────────────────────────────┐
+│           transformers.js           │  ← Tokenization, model loading, generation
+├─────────────────────────────────────┤
+│            ONNX Runtime             │  ← Neural network execution
+├─────────────────────────────────────┤
+│     WebGPU  │   CPU   │   WASM      │  ← Execution backends
+└─────────────────────────────────────┘
+```
+
+## transformers.js
+
+[transformers.js](https://huggingface.co/docs/transformers.js) is a JavaScript port of Hugging Face Transformers that runs ONNX models in the browser and Node.js.
+
+### Model Loading
+
+```typescript
+import { AutoModelForCausalLM, AutoTokenizer } from "@huggingface/transformers";
+
+const tokenizer = await AutoTokenizer.from_pretrained(modelId);
+const model = await AutoModelForCausalLM.from_pretrained(modelId, {
+  dtype: "q4f16",   // Quantization
+  device: "webgpu", // Backend
+});
+```
+
+### Generation
+
+```typescript
+const inputs = tokenizer.apply_chat_template(messages, {
+  add_generation_prompt: true,
+  return_dict: true,
+});
+
+const output = await model.generate({
+  ...inputs,
+  max_new_tokens: 256,
+  temperature: 0.7,
+  do_sample: true,
+});
+
+const text = tokenizer.decode(output[0], { skip_special_tokens: true });
+```
+
+### Streaming with TextStreamer
+
+```typescript
+const streamer = new TextStreamer(tokenizer, {
+  skip_prompt: true,
+  skip_special_tokens: true,
+  callback_function: (text) => {
+    console.log(text); // Called for each token
+  },
+});
+
+await model.generate({ ...inputs, streamer });
+```
+
+## ONNX Runtime
+
+ONNX Runtime is the execution engine that runs the neural network operations.
+
+### Backends
+
+| Backend | Environment | Speed | Notes |
+|---------|-------------|-------|-------|
+| **WebGPU** | Browser, Chrome | ~70-100 tok/s | Fastest, requires GPU |
+| **CPU** | Node.js | ~10-30 tok/s | Uses SIMD, good on Apple Silicon |
+| **WASM** | Browser fallback | ~5-10 tok/s | Works everywhere |
+
+### Execution Providers
+
+ONNX Runtime selects execution providers based on availability:
+
+```
+WebGPU EP → WASM EP → CPU EP
+```
+
+Gerbil explicitly requests the desired backend:
+
+```typescript
+// For WebGPU
+await AutoModelForCausalLM.from_pretrained(modelId, { device: "webgpu" });
+
+// For CPU
+await pipeline("text-generation", modelId, { device: "cpu" });
+```
+
+## Quantization
+
+Quantization reduces model size and improves inference speed by using lower-precision numbers.
+
+### Quantization Types
+
+| Type | Weights | Compute | Size Reduction | Use Case |
+|------|---------|---------|----------------|----------|
+| **fp32** | 32-bit float | 32-bit | 1x (baseline) | Training |
+| **fp16** | 16-bit float | 16-bit | 2x | GPU inference |
+| **q4f16** | 4-bit int | 16-bit | ~4x | WebGPU inference |
+| **q4** | 4-bit int | 32-bit | ~4x | CPU inference |
+
+### Why q4f16 for WebGPU?
+
+WebGPU shaders work best with fp16 compute. The `q4f16` format:
+- Stores weights as 4-bit integers (small download)
+- Dequantizes to fp16 during inference (fast on GPU)
+- Maintains good quality for small models
+
+### Model Sizes
+
+| Model | Original | q4f16 | Download |
+|-------|----------|-------|----------|
+| Qwen3-0.6B | ~2.4GB | ~400MB | ~400MB |
+| SmolLM2-360M | ~1.4GB | ~250MB | ~250MB |
+| SmolLM2-135M | ~540MB | ~100MB | ~100MB |
+
+## Tokenization
+
+### Chat Templates
+
+Gerbil uses model-specific chat templates to format conversations:
+
+```typescript
+const messages = [
+  { role: "system", content: "You are helpful." },
+  { role: "user", content: "Hello!" },
+];
+
+const inputs = tokenizer.apply_chat_template(messages, {
+  add_generation_prompt: true,  // Add assistant turn start
+  return_dict: true,            // Return input_ids + attention_mask
+  enable_thinking: true,        // Qwen3 thinking mode
+});
+```
+
+### Thinking Mode (Qwen3)
+
+Qwen3 models support a "thinking" mode where the model shows reasoning:
+
+```
+<think>
+Let me work through this step by step...
+127 × 43 = 127 × 40 + 127 × 3 = 5080 + 381 = 5461
+</think>
+The answer is 5461.
+```
+
+Enabled via `enable_thinking: true` in the chat template.
+
+## KV Cache
+
+The Key-Value cache stores intermediate attention states to speed up autoregressive generation:
+
+```typescript
+const { past_key_values, sequences } = await model.generate({
+  ...inputs,
+  past_key_values: previousCache,  // Reuse from last turn
+  return_dict_in_generate: true,
+});
+
+// Save for next turn
+cache = past_key_values;
+```
+
+Benefits:
+- Faster multi-turn conversations
+- Reduced compute for long contexts
+
+Gerbil manages the KV cache automatically for multi-turn chat.
+

package/docs/architecture/overview.md

@@ -0,0 +1,179 @@
+# Architecture Overview
+
+Gerbil is a local LLM inference library that runs entirely on-device, with no API calls or cloud dependencies.
+
+## Core Components
+
+### 1. Gerbil Class (`src/core/gerbil.ts`)
+
+The main entry point for Node.js applications:
+
+```typescript
+const g = new Gerbil();
+await g.loadModel("qwen3-0.6b");
+const result = await g.generate("Hello");
+```
+
+Responsibilities:
+- Model loading and lifecycle management
+- Device selection (WebGPU vs CPU)
+- Generation orchestration
+- Streaming coordination
+- Session statistics
+
+### 2. Model Registry (`src/core/models.ts`)
+
+Maps friendly model IDs to Hugging Face paths:
+
+```typescript
+const BUILTIN_MODELS = {
+  "qwen3-0.6b": {
+    id: "qwen3-0.6b",
+    path: "onnx-community/Qwen3-0.6B-ONNX",
+    family: "qwen",
+    size: "0.6B",
+    contextLength: 32768,
+    supportsThinking: true,
+  },
+  // ...
+};
+```
+
+### 3. Chrome GPU Backend (`src/core/chrome-backend.ts`)
+
+Enables WebGPU in Node.js by using headless Chrome as a GPU accelerator:
+
+```
+┌─────────────┐    HTTP     ┌──────────────────┐
+│   Node.js   │◄──────────►│  Headless Chrome  │
+│  (Gerbil)   │   :43724   │  (WebGPU worker)  │
+└─────────────┘            └──────────────────┘
+       │                          │
+       │ CDP (DevTools)           │ WebGPU
+       └──────────────────────────┘
+                                  │
+                            ┌─────▼─────┐
+                            │    GPU    │
+                            └───────────┘
+```
+
+### 4. Browser Worker (`src/browser/index.ts`)
+
+Provides `createGerbilWorker()` for browser applications:
+
+```typescript
+const gerbil = await createGerbilWorker({
+  modelId: "qwen3-0.6b",
+  onToken: (token) => console.log(token.text),
+});
+```
+
+Uses an inline Web Worker to:
+- Load models without blocking the UI
+- Stream tokens in real-time
+- Manage GPU memory separately from main thread
+
+## Execution Paths
+
+### Browser Path
+
+```
+User Code
+    │
+    ▼
+createGerbilWorker()
+    │
+    ▼
+Web Worker (inline blob)
+    │
+    ▼
+transformers.js
+    │
+    ▼
+ONNX Runtime (WebGPU)
+    │
+    ▼
+GPU
+```
+
+### Node.js CPU Path
+
+```
+User Code
+    │
+    ▼
+Gerbil.generate()
+    │
+    ▼
+transformers.js pipeline
+    │
+    ▼
+ONNX Runtime (CPU)
+    │
+    ▼
+CPU (with SIMD)
+```
+
+### Node.js WebGPU Path
+
+```
+User Code
+    │
+    ▼
+Gerbil.generate()
+    │
+    ▼
+ChromeGPUBackend
+    │
+    ▼
+Headless Chrome (via CDP)
+    │
+    ▼
+transformers.js (in Chrome)
+    │
+    ▼
+ONNX Runtime (WebGPU)
+    │
+    ▼
+GPU
+```
+
+## Device Selection
+
+Gerbil automatically selects the best available backend:
+
+```typescript
+// Explicit selection
+await g.loadModel("qwen3-0.6b", { device: "webgpu" }); // or "cpu"
+
+// Check current device
+g.getDeviceMode(); // "webgpu" | "cpu" | "wasm"
+```
+
+Priority:
+1. **Browser**: WebGPU → WASM fallback
+2. **Node.js with --gpu**: ChromeGPUBackend (headless Chrome)
+3. **Node.js default**: CPU via ONNX Runtime
+
+## File Structure
+
+```
+src/
+├── core/
+│   ├── gerbil.ts         # Main Gerbil class
+│   ├── models.ts         # Model registry
+│   ├── types.ts          # TypeScript types
+│   ├── tools.ts          # Tool calling system
+│   └── chrome-backend.ts # Node.js WebGPU via Chrome
+├── browser/
+│   └── index.ts          # createGerbilWorker + utilities
+├── skills/
+│   └── ...               # Built-in skills (commit, summarize, etc.)
+├── integrations/
+│   └── ...               # AI SDK, LangChain, MCP
+├── frameworks/
+│   └── ...               # Next.js, Express, React, etc.
+└── cli/
+    └── repl/             # Interactive terminal UI
+```
+