@tryhamster/gerbil 1.0.0-rc.9 → 1.0.0

package/docs/research/native-vs-chromium-decision.md

@@ -0,0 +1,152 @@
+# Native WebGPU Engine vs. the Chromium Approach — Decision Analysis
+
+**Date:** 2026-06-12
+**Status:** Recommendation — deprecate `chrome-backend.ts` now; keep the transformers.js worker path indefinitely (it is not "the chromium approach")
+**Context:** The native engine (`src/gpu/`) now runs on desktop Node via node-dawn (145 tok/s INT4) and on mobile WebKit (iPad, 31–36 tok/s, byte-correct after the Metal coherence fixes in `38bc674`, `2f0cabc`, `d574cdb`).
+
+---
+
+## 1. What "the chromium approach" actually is in this codebase
+
+There are **two separate things** that get lumped together, and they have completely different removal calculus. Only one of them is actually Chromium.
+
+### 1a. `src/core/chrome-backend.ts` (`ChromeGPUBackend`) — the real chromium approach
+
+1,836 lines. **Node.js only.** What it does, concretely:
+
+- Finds an installed Chrome/Edge/Brave/Chromium binary on the host (`findChrome()`, `CHROME_PATHS`), or fails with "Chrome not found."
+- Launches it headless via **puppeteer-core** (a hard runtime dependency in `package.json`), with a persistent profile at `~/.gerbil/chrome-cache` and a saved WebSocket endpoint file for cross-process reconnection.
+- Starts a localhost HTTP server on fixed port **43724** that serves a generated HTML page (`getWorkerPageHTML()`). The page imports **transformers.js 4.0.0-next.7 from the jsdelivr CDN**, loads an ONNX model with `dtype: "q4f16", device: "webgpu"`, and exposes `window.gerbilGenerate` / `gerbilGenerateVision` / `gerbilInterrupt` / `gerbilReset`.
+- Streams tokens back to Node by **JSON-encoding them through `console.log` and parsing CDP `Runtime.consoleAPICalled` events**.
+- Carries a large operational surface to keep this stable: orphan-page cleanup, `SingletonLock` removal, WS-endpoint staleness handling, `MAX_CONCURRENT_PAGES` limits, JS-heap polling via CDP `Performance.getMetrics`, `checkMemoryAndCleanup`, `killAllBackends`, SIGINT/SIGTERM/exit kill handlers, and the `gerbil cleanup` CLI command (`src/cli/index.ts:1041–1130`) whose entire job is killing zombie Chrome processes with `pkill -f "gerbil/chrome-cache"`.
+- Side-jobs unrelated to inference: it owns `~/.gerbil/cached-models.json` bookkeeping (`getChromeCachedModels` / `trackCachedModel` / `refreshCachedModelSizes`), which `gerbil.ts:786` and `src/cli/repl/utils.ts` use to answer "is this model cached?" in the REPL/CLI.
+
+**When it is actually used today** (`src/core/gerbil.ts:382–505`, `592–629`):
+
+| Path | Condition | Backend chosen |
+|---|---|---|
+| Node text gen, `device: auto/webgpu` | default | **native engine** (`effectiveBackend = "webgpu-native"`) |
+| Node text gen, explicit `backend: "onnx"` + WebGPU | opt-in | **ChromeGPUBackend** |
+| Node **vision model** + WebGPU (e.g. `ministral-3b`) | automatic | **ChromeGPUBackend** (`loadVisionModel`, line 610) |
+| Native engine load fails (unsupported arch, GPU error) | automatic | falls back to **CPU transformers.js pipeline** — *not* Chrome |
+| Browser | any | never (puppeteer is dynamically imported, Node-only) |
+
+Key fact: **the chrome backend is already off the default text-generation path.** `device: "auto"` in Node routes to the native engine. The catch-block fallback for unsupported architectures goes to CPU, not Chrome. Chrome is only reachable via (a) explicit `backend: "onnx"` with GPU, or (b) vision models in Node.
+
+### 1b. The transformers.js worker path (`src/browser/worker.ts`, `worker-entry.ts`) — NOT chromium
+
+This runs in **the end user's actual browser**, whatever it is. `worker-entry.ts` is bundled into a self-contained classic-worker IIFE (`worker-code.generated.ts`, built by `scripts/build-worker.mjs`) so it works in iOS WKWebView where module workers from blob URLs fail. It uses transformers.js + onnxruntime-web with three execution tiers chosen by `backend-selector.ts` from the device profile:
+
+- **WebGPU + q4f16** (≥2 GB GPU est.), **WebGPU + q4** (1–2 GB),
+- **WASM + q4** (≥1.5 GB RAM, "5–10× slower" warning),
+- **WASM/CPU + q4i** (last resort, 256-token context).
+
+Plus an iOS-specific **main-thread** variant (`createIOSMainThreadWorker`) with crash breadcrumbs in localStorage, single-threaded WASM, 128-token caps, and a device-memory-tiered model **fallback chain** (qwen3-1.7b → … → smollm2-135m).
+
+Calling this "the chromium approach" is a category error — it's the *any-browser, any-hardware* compatibility layer. It happens to share transformers.js with the Chrome backend, which is the only family resemblance.
+
+### 1c. Where each feature actually lives
+
+- `useChat` / `useCompletion` (`use-chat.ts`) → `createGerbilWorker` → **transformers.js worker** (1b). Hard-gated on `isWebGPUSupported()` at the hook level (arguably a bug — the worker itself supports WASM, see §2 notes).
+- `useEmbedding` (`use-embedding.ts`) → its own CDN-loaded transformers.js worker, `feature-extraction` pipeline, WebGPU-if-available else WASM. **No native path exists.**
+- `useSpeech` (`use-speech.ts`) → **kokoro-js** (Kokoro 82M) or transformers.js `text-to-speech` pipeline (Supertonic, WebGPU). **No native path.**
+- `useVoiceInput` / `useVoiceChat` (`use-voice-input.ts`) → `WhisperSTT` (`src/core/stt.ts`), transformers.js `automatic-speech-recognition`, CPU/WASM only. **No native path.**
+- `useNativeEngine` (`use-native-engine.ts`) → **native WebGPUEngine** directly on the main thread; deliberately *not* exported from `browser/index.ts` to keep GPU code out of the main bundle (import from `/gpu`).
+- Node `g.embed()` → transformers.js `feature-extraction` pipeline (CPU). Node STT/TTS → same ONNX story.
+- Tool calling / JSON mode → prompt-level (`core/tools.ts`), works on **any** text backend including native.
+
+---
+
+## 2. Capability matrix
+
+Backends: **N** = native engine (`src/gpu`), **T** = transformers.js (browser worker / Node pipeline), **C** = ChromeGPUBackend. ✅ works today, ⚠️ works with caveats, ❌ unavailable.
+
+| Environment × Feature | Text gen | Embeddings | STT (Whisper) | TTS | Vision (VLM) | Tool calling/JSON |
+|---|---|---|---|---|---|---|
+| **Node + GPU (desktop)** | **N** ✅ 145 tok/s (Qwen only) · C ⚠️ legacy opt-in (~100 tok/s, any ONNX) | T ✅ (CPU) | T ✅ (CPU) | T/kokoro ✅ (CPU) | **C ✅ (only GPU path)** · T ⚠️ CPU fallback | N/T/C ✅ (prompt-level) |
+| **Node, no GPU / headless CI** | T ✅ (CPU pipeline) | T ✅ | T ✅ | T ✅ | T ⚠️ slow | ✅ |
+| **Browser: Chrome/Edge (WebGPU)** | T ✅ q4f16 default · N ✅ opt-in via `useNativeEngine` | T ✅ (WebGPU) | T ✅ (WASM) | T/kokoro ✅ | T ✅ | ✅ |
+| **Browser: Safari/WebKit desktop** | **N ✅ (packed-f16, post-Metal-fixes)** · T ⚠️ ORT-WebGPU flaky on WebKit → selector mostly lands on WASM | T ✅ (WASM) | T ✅ (WASM) | T ⚠️ | T ⚠️ WASM, slow | ✅ |
+| **iPhone/iPad (WKWebView)** | **N ✅ 31–36 tok/s** · T ⚠️ main-thread WASM, 128-token cap, crash breadcrumbs | T ⚠️ WASM | T ⚠️ | T ⚠️ | ❌ practically (memory) | ✅ |
+| **Browser, no WebGPU at all** (old Android, Firefox ESR, locked-down enterprise) | T ✅ WASM q4/q4i **only option** · N ❌ (hard-requires WebGPU) | T ✅ | T ✅ | T ⚠️ | T ⚠️ | ✅ |
+
+### What breaks if each backend is removed
+
+**Remove ChromeGPUBackend (C):**
+- Node GPU **vision** inference is the only real loss — `ministral-3b` etc. drop to CPU transformers.js.
+- `backend: "onnx"` + GPU in Node stops working (callers get native-or-CPU).
+- Node GPU text gen for **non-Qwen ONNX models** loses its (opt-in) GPU path — but note `device:"auto"` already sends those to CPU today, because the native-engine failure falls back to CPU, not Chrome.
+- `gerbil cleanup` CLI command and `~/.gerbil/cached-models.json` tracking need extraction/replacement (small, self-contained).
+- **Gains:** delete 1,836 lines + zombie-process management; drop `puppeteer-core` from hard `dependencies`; remove the "you must have Chrome installed" runtime requirement; remove the CDN dependency at inference time; remove a whole bug class (the file is ~40% lifecycle defensive code).
+
+**Remove transformers.js worker path (T):**
+- Browser loses: vision, embeddings, STT, TTS, the entire ONNX model zoo, the WASM fallback for no-WebGPU/low-memory devices, and the device-tiered model fallback chain. `useChat`/`useCompletion`/`useEmbedding`/`useSpeech`/`useVoiceInput` all die.
+- Node loses: CPU inference entirely (the only no-GPU path), embeddings, STT, TTS.
+- **This is not removable.** It is the project's breadth layer, and it is not chromium.
+
+**Remove native engine (N):** Node GPU text gen reverts to headless Chrome; Safari/iPad on-device inference reverts to "barely works on WASM." Obviously not on the table.
+
+---
+
+## 3. Model coverage gap: native vs. the ONNX zoo
+
+`src/gpu/architectures/index.ts` registers exactly three architecture strings:
+
+```
+Qwen2ForCausalLM   → generateQwen2Graph
+Qwen3ForCausalLM   → generateQwen2Graph
+Qwen3_5ForConditionalGeneration → generateQwen3_5Graph
+```
+
+Quant/weight formats (`model-loader.ts`): f32 safetensors, on-the-fly INT4 (`dtype: "q4"`), **pre-quantized GPTQ** (auto-detected, repacked), **MLX 4-bit affine** (auto-detected, group_size-aware repack). Capabilities struct is `{ text: true, vision: false, moe: false }` — text only. `maxSeqLen` capped at 4096 (512 default on WebKit) vs. the 32k–262k contexts the ONNX path advertises.
+
+Against the built-in registry (`src/core/models.ts`):
+
+| Built-in | Architecture | Native? |
+|---|---|---|
+| qwen3-0.6b / 1.7b | Qwen3ForCausalLM | ✅ |
+| qwen3.5-0.8b / 2b | Qwen3_5ForConditionalGeneration | ✅ |
+| qwen2.5-0.5b / coder | Qwen2ForCausalLM | ✅ |
+| smollm2-135m / 360m | LlamaForCausalLM | ❌ (TODO stub in registry) |
+| phi-3-mini | Phi3ForCausalLM | ❌ |
+| lfm2.5-1.2b-thinking | LFM2 hybrid | ❌ |
+| ministral-3b (vision) | Mistral3/vision | ❌ |
+
+So **6 of 11 built-ins run native; 5 don't** — and the 5 include the smallest models (SmolLM2, the low-memory fallback chain backbone) and the only vision model. Beyond built-ins, users lose the `hf:onnx-community/*` long tail (anything transformers.js can load), Whisper, Kokoro/Supertonic, and every embedding model (MiniLM etc.). Notably, the SmolLM2 gap matters more for the *browser fallback chain* than for Node: `getModelFallbackChain` bottoms out at smollm2-135m on <3 GB devices, and those devices are exactly the ones using the WASM path anyway — so the practical native gap is: **Llama-family graph, Phi graph, vision, embeddings, audio.**
+
+The Llama graph is the highest-leverage addition: one graph generator (`llama.ts`, already stubbed in the registry comments) unlocks SmolLM2, Llama 3.x, Mistral 7B-class, and most of the open-weights world, because the Qwen2 generator is already ~90% of a Llama generator (RoPE + GQA + SwiGLU + RMSNorm).
+
+---
+
+## 4. Recommendation
+
+**Short answer: yes for text, no overall — but the thing you can kill is `chrome-backend.ts`, and you should kill it now.** The transformers.js worker path is not "the chromium approach" and must stay; it's what makes the "anywhere JavaScript runs" promise true on the 40% of surfaces the native engine can't touch (no-WebGPU browsers, embeddings, audio, vision, non-Qwen models).
+
+The chrome backend, by contrast, is now a redundant middleman for its main job: it was built to give Node a WebGPU path before node-dawn worked, and the native engine is now *faster* (145 vs ~100 tok/s), dependency-free at runtime (no Chrome install, no CDN fetch, no puppeteer), and already the default. What remains load-bearing is narrow: Node+GPU **vision**, and opt-in Node+GPU **ONNX** for non-Qwen models. Neither justifies 1,836 lines, a hard puppeteer-core dependency, a localhost server, and a zombie-process janitor in every install.
+
+### Phased plan
+
+**Phase 1 — now (next rc):** Deprecate, demote, and stop auto-routing.
+1. Mark `ChromeGPUBackend` and `backend: "onnx"` + GPU-in-Node as deprecated (console.warn + README). Keep them working.
+2. Change `loadVisionModel` in Node to **CPU transformers.js by default**, with Chrome behind explicit `backend: "chrome"` opt-in. Vision-on-Node-GPU users are rare; the ones that exist can opt in for two more releases.
+3. Move `puppeteer-core` from `dependencies` to `optionalDependencies`/peer with a clear error message ("install puppeteer-core and Chrome to use backend:'chrome'"). This alone removes the heaviest install cost for everyone else.
+4. Extract `cached-models.json` bookkeeping (`getChromeCachedModels`, `trackCachedModel`, `refreshCachedModelSizes`) out of `chrome-backend.ts` into `src/core/model-cache.ts` — it's used by the REPL/CLI and `gerbil.ts:786` and has nothing to do with Chrome. Also fix the bookkeeping to cover native-engine downloads (it currently only tracks Chrome-IndexedDB models).
+5. While in there: declare `@huggingface/transformers` properly. It's statically imported by `src/core/gerbil.ts` but listed only in `devDependencies` and externalized by tsdown for Node builds — that's a latent packaging bug independent of this decision.
+
+**Phase 2 — gate for full removal (target: 2 releases):** the native engine must gain, in priority order:
+1. **Llama-family graph** (`LlamaForCausalLM`, `MistralForCausalLM`, `SmolLM`) — unlocks SmolLM2 built-ins + most of HF. This is the cheapest, highest-value gap closure.
+2. **Phi graph** (phi-3-mini built-in) or drop phi-3-mini from built-ins (it's the weakest entry in the registry anyway — 4k context).
+3. A decision on vision: either (a) accept CPU vision on Node and document it, or (b) implement the vision encoder natively (large effort — not worth blocking removal on; choose (a)).
+
+**Phase 3 — delete:** remove `chrome-backend.ts`, the `gerbil cleanup` Chrome plumbing, `puppeteer-core`, and the `backend: "chrome"` option. Net: −1,836 lines core, −CLI cruft, one fewer process manager to debug, no host-Chrome requirement, no inference-time CDN dependency.
+
+**Explicit non-goals:** do not deprecate the browser worker path, the WASM tiers, the iOS main-thread path, or the embedding/STT/TTS workers. If anything, invest the freed maintenance budget there: route `useChat` to the native engine *when the device profile says WebKit-with-WebGPU* (where ORT-WebGPU is flaky but native now shines), keep transformers.js as the chooser's other arm, and relax the `isWebGPUSupported()` hard-gate in `use-chat.ts` (the worker has a perfectly good WASM tier the hook currently refuses to use).
+
+### Decision summary
+
+| Component | Verdict |
+|---|---|
+| `chrome-backend.ts` (headless Chrome via puppeteer) | **Deprecate now, delete in 2 releases.** Gate: Llama graph in native engine; vision goes to CPU on Node. |
+| transformers.js browser worker (+ WASM tiers, iOS path) | **Keep indefinitely.** It's the breadth layer, not the chromium approach. |
+| transformers.js Node CPU pipeline | **Keep.** Only no-GPU Node path; also embeddings/STT/TTS. |
+| Native engine | **Default everywhere it can run; grow Llama graph next.** |