@tryhamster/gerbil 1.1.2 → 1.1.3

package/docs/PROJECT-STATE.md

@@ -1,321 +0,0 @@
-# Gerbil — Project State & Architecture Decision
-
-**As of 2026-06-14.** The single authoritative snapshot. When something here conflicts
-with an older doc, this wins. Supersedes scattered findings in `docs/research/*` and
-the paper's roadmap. **Newest material is §12** (native audio begins — Moonshine STT
-+ Kani-TTS-2 NanoCodec decoder; Gemma 4 E2B text decode; on-device memory/RAG; the
-text+ViT autoresearch campaign; gerbil-site live on the native engine). §11 captured
-EmbeddingGemma on-device, LFM2.5, the SPM tokenizer fix, MLX/DWQ loader, progress fix.
-
----
-
-## 1. What Gerbil is
-
-A local LLM inference library that runs models in the browser and Node on a
-**single native WebGPU engine** behind one task API (no fallback lane — §2). The
-headline achievements this cycle: the from-scratch native engine **works on mobile**
-(iPad/iOS Safari 26.5+, WebKit) — previously it crashed — a desktop optimization
-pass roughly doubled its throughput, and the engine went **multimodal natively**:
-text embeddings ship (Qwen3-Embedding-0.6B) and the Qwen3.5 vision encoder is built
-**bit-exact vs HuggingFace** (vision LM-integration is phase 2).
-
-## 2. The decided architecture — NATIVE-ONLY (owner decision, overrides the panel)
-
-**Decision (owner, 2026-06-13):** Gerbil is a single native WebGPU engine. **No
-tfjs fallback lane** — a permanent fallback "assumes defeat to begin with." The
-panel proposed keeping tfjs as a breadth lane; the owner rejected that. Instead:
-
-- **Launch set = text + vision + embeddings, ALL native.** One model per modality
-  by default, expandable to other families via the add-model-family process.
-- **Audio (TTS/STT) is deferred, not delegated to tfjs** — it ships later as
-  small *native* models (candidates under eval: VibeVoice-1.5B, OmniVoice,
-  dots.tts-soar for TTS — IF they publish safetensors; Moonshine for STT). "What's
-  the big deal" — launching without audio is fine; a permanent second engine is not.
-- **tfjs is at most temporary dev scaffolding, not a destination.** It may stay
-  briefly to keep desktop demos working during the native build, but it is being
-  removed, not kept as a lane. `chrome-backend.ts` is deleted outright.
-- **Vision uses Qwen3.5's OWN built-in ViT** (we currently skip its 192MB tower) —
-  one multimodal model, not a separate vision model. Stop dropping it "like idiots."
-- **A thin onnxruntime-web bridge is a break-glass option only** — used solely if
-  a needed model has no extractable weights AND no native alternative. Not a lane.
-
-**Status update (2026-06-13):** both next modalities have landed natively.
-**Embeddings ship** (Qwen3-Embedding-0.6B, validated). The **vision encoder is
-DONE** — the Qwen3.5 ViT runs natively and is **bit-exact vs HF transformers 5.12**
-(per-token cosine 1.000000). The earlier "parallel attention kernel first" gate was
-based on a misread of dead code (§5): the attention kernel was already parallel, and
-non-causal was a one-line `is_causal` flag, not a rewrite. The remaining vision work
-is **LM-side integration** (M-RoPE, token splice, image preprocessing) — phase 2,
-plumbing over a verified core. Audio follows once a native small model is validated.
-
-## 3. Capability matrix (what's true today)
-
-Native-only. The "tfjs lane" is gone — tfjs is temporary dev scaffolding being
-removed (§2), not a capability path.
-
-| Modality | Native status |
-|---|---|
-| Text | ✅ ~51 tok/s mobile (sustained 200-tok), ~207 desktop, bit-correct |
-| Text (alt family) | ✅ **LFM2.5-350M LANDED** (`Lfm2ForCausalLM`, hybrid conv/attn): ~600 tok/s desktop (2.8× Qwen), ~46 tok/s mobile, ~199MB q4, no new kernels |
-| Embeddings | ✅ **DONE + ON iPad** — **EmbeddingGemma-300M** (bidirectional Gemma3 encoder, 173MB MLX-4bit) runs on iPad Safari; cos=1.00000 vs NumPy ref, Mars/bread margin >0.1, dim 768. First non-Qwen embedder. (Qwen3-Embedding-0.6B still works on desktop but OOMs iPad at 1.2GB BF16.) |
-| Vision (image) | ✅ **END-TO-END DONE** — Qwen3.5 ViT, bit-exact vs HF (cosine 1.000000); `describeImage()` word-identical to HF; runs on iPad |
-| Text model families (Llama/Mistral/Gemma) | 🟢 cheap — now usually **Tier-1, generator-only, NO new kernels** (kernel library saturated for standard transformers) |
-| Text (Gemma 4 E2B) | ✅ **COHERENT on q4** (`gemma4.ts`): "capital of France"→"Paris", ~83 tok/s, all 35 layers cos≥0.998 vs MLX-LM. PLE, KV-share (20 layers), proportional/dual-theta RoPE, GeGLU, `Softcap`, double-wide MLP, V-norm, per-node `attn_scale`, head_dim-512 attention. **PLE CPU-streamed (0 MB GPU)** — not GPU-sharded |
-| STT | 🟢 **Moonshine native LANDED** (`moonshine.ts`, `moonshine-executor.ts`, `moonshine-stt.ts`): raw-waveform Conv1d front-end (no FFT/log-mel), bit-exact `CrossAttention` kernel (max|err|<2e-4, cos≥0.9999), dual-graph encode-once/frozen-K-V/AR-decode, interleaved RoPE. Encoder cos≈0.990 vs HF; transcript substring-matches HF refs. **Whisper(ONNX) stays as multilingual / no-GPU fallback** (`src/core/stt.ts`) |
-| TTS | 🟡 **Kani-TTS-2 partial** (`kani_tts.ts`): NanoCodec decoder (FSQ + causal HiFi-GAN) **LANDED + validated bit-exact** (`test-nanocodec-decode.mjs`, gate err<1e-3, measured ~4.2e-6). LFM2-350M codec-LM backbone scaffolded; **AR-loop glue remaining** (frame positions + learnable RoPE + 4-token-frame decode). License: kani-tts-2-en = LFM1.0/other; 450m variant = Apache |
-| Memory / RAG | ✅ **SHIPPED** (`src/memory/`, `@tryhamster/gerbil/memory`): vector store (in-memory/IndexedDB/file), token-budgeted `recall()`, chunking, redaction, native EmbeddingGemma adapter. 12/12 tests. No new kernels |
-| No-WebGPU / old devices | not targeted — engine throws a clear error rather than degrading |
-
-## 4. Performance baselines (re-confirm before quoting)
-
-| Platform | Config | Decode tok/s | Note |
-|---|---|---|---|
-| M4 Max, node-dawn | optimized | ~207 | re-confirm on a cooled run; numbers between commit 2f0cabc and the isMetalBackend fix are invalid |
-| iPad (iOS 26.5) native | batch-all | ~41 (cooled, consistent) | NOT thermal — confirmed; Dawn-tuned autoresearch wins did NOT transfer to Metal (was ~51 pre-optimization) |
-| iPad native, submit floor | group=1 awaited | 6–8 | proven-correct floor |
-| iPad transformers.js (same model) | WebGPU | 7–12 | ~5× slower than native |
-
-## 5. Vision — DONE at the encoder level (bit-exact vs HF)
-
-The native Qwen3.5 vision encoder is **built and validated bit-exact** against HF
-transformers 5.12: **per-token cosine = 1.000000, max abs err ~5e-6**. Exposed as
-`engine.encodeImage(patches, gridTHW)` → merged image tokens `[rows, 1024]`. See
-paper §22 and `src/gpu/architectures/qwen3_5_vision.ts`, `src/gpu/vision-executor.ts`,
-`src/gpu/vision-preprocess.ts`.
-
-The earlier strategy panel claimed native vision was blocked on a single-threaded
-attention kernel. **That was wrong** — it read `src/gpu/kernels/wgsl/attention.wgsl`,
-a STALE reference file **not imported anywhere** (kernels are embedded strings in
-`registry.ts`; the dead `.wgsl` files have been deleted). What actually shipped:
-
-1. **The live attention kernel was already parallel** — `WGSL_ATTENTION` is a
-   tiled, online-softmax (flash-attention-style) kernel. No thread-0 serialization,
-   no rewrite needed.
-2. **Non-causal was a one-line flag.** An `is_causal` uniform
-   (`S_eff = is_causal ? min(S, causal_limit) : S`) makes it bidirectional for the
-   ViT; text stays causal by default. Done.
-3. **Patch-embed was NOT a new Conv3d kernel.** Patches arrive pre-flattened to
-   `[N, 1536]` from the host image processor, so the 5-D unfold/Conv3d collapses to
-   a plain `MatMul` + `AddBias`. New ops added were small: `AddBias`, `GeluErf`
-   (exact-erf merger GELU), `ApplyRotaryEmb` (2D rotary), `SliceCols` (fused-QKV
-   split) — plus host-side pos-embed/rotary precompute (grid-only, bit-exact).
-4. **A real bug surfaced during validation:** `WGSL_GELU` returned NaN for large
-   args on Metal/Dawn (`x³` overflow into fast-math `tanh`); fixed by clamping the
-   inner arg to ±15.
-
-**Net: vision is done at the encoder level.** Remaining work is **LM-side
-integration** (M-RoPE, image-token splice into the text stream, pixel→patch
-preprocessing) — phase 2, bounded plumbing over a verified numerical core. The
-ViT prefill runs through the same parallel attention as text, so there is no
-separate mobile attention risk; on-device ViT speed is a measure-when-integrated
-item, not a feasibility gate.
-
-## 6. Build sequence (ordered)
-
-1. ~~**Native embeddings**~~ — **DONE.** Qwen3-Embedding-0.6B (`Qwen3ForCausalLM`): last-token EOS pooling + `L2Norm` tail. Validated: dim 1024, unit norm, cos(similar)=0.81 > cos(unrelated)=0.56. Confirmed causal-LM-pooling. `engine.embed()`.
-2. ~~**OPFS model cache**~~ — **resolved differently: OPFS removed.** Main-thread OPFS `createWritable` is broken on iOS (leaves unclearable junk that fills the quota). The loader is now **Cache-API-only**. Durable iOS caching needs a PWA (§ paper 24, `docs/research/ios-safari-model-caching.md`), not OPFS.
-3. ~~**Native vision encoder**~~ — **DONE, bit-exact vs HF** (§5). Remaining: LM-side integration (M-RoPE, token splice, image preprocessing) — phase 2.
-4. **Vision LM integration (phase 2)** — splice `encodeImage()` tokens into the text stream: M-RoPE position assignment, placeholder-token splice, host pixel→patch preprocessing. Plumbing over a verified encoder.
-5. **Close the mobile WebKit correctness/perf sweep** — `?group=N` + Test-R bisect on real iOS 26.5 for a stable fast-and-correct submit config above the group=1 floor; re-measure desktop.
-6. **Llama/Mistral/Gemma graph generator** — ~90% is the existing qwen2 generator; unlocks most of the HF text zoo. Low risk, high ROI.
-7. **Remove tfjs scaffolding** — delete `chrome-backend.ts`. The engine is native-only; tfjs was temporary dev scaffolding, not a kept lane.
-8. **Audio, native (deferred, not delegated)** — TTS: **OmniVoice** (Qwen3 backbone + codec decoder — mostly the existing text path + a decoder). STT: **Moonshine** (lean encoder-decoder; raw-waveform Conv1d frontend, no log-mel Conv2d; needs a parallel CrossAttention kernel). Ship once a native small model clears the mobile bar.
-
-## 7. What NOT to build
-
-- **A second permanent engine / kept tfjs lane** — the architecture is native-only across text + vision + embeddings. tfjs is temporary scaffolding being removed, not a breadth lane.
-- **A Conv3d/unfold patch-embed kernel for the ViT** — not needed; patches arrive pre-flattened, so patch-embed is a plain MatMul + AddBias (§5).
-- **A heavyweight native TTS pipeline with custom vocoder kernels now** — audio is deferred; when it lands, OmniVoice's Qwen3 backbone reuses the text path. A thin onnxruntime-web bridge stays break-glass-only.
-- **A general Conv2d kernel / Whisper-on-native** — Moonshine's raw-waveform Conv1d avoids the log-mel Conv2d.
-- **An OPFS write path on iOS** — main-thread `createWritable` is broken; durable caching is a PWA concern, not an OPFS one.
-- **Trusting stale tok/s numbers** between 2f0cabc and the isMetalBackend fix.
-
-## 8. Open spikes (measure before committing)
-
-- ~~**Embedder pooling direction**~~ — RESOLVED: causal-LM last-token (EOS) pooling, validated.
-- ~~**Parallel non-causal attention feasibility for the ViT**~~ — RESOLVED: the live attention kernel was already parallel; non-causal is a one-line `is_causal` flag; the encoder is bit-exact. Remaining ViT spike: **on-device ViT prefill speed on a real iPad** (measure when LM-integration lands; not a feasibility gate).
-- **Mobile WebKit submit config** — does `?group=N`/Test-R land a stable fast config above the group=1 floor on iOS 26.5? The critical path.
-- **Vision LM-integration correctness** — M-RoPE + token splice must stay bit-exact end-to-end (image+text), not just at the encoder boundary.
-- **Native audio on mobile** — OmniVoice (codec-decoder) and Moonshine (CrossAttention kernel) latency/correctness once a candidate is validated.
-- **Autoresearch on-device tax** — desktop kernel wins did NOT transfer to mobile (41 vs 51); the loop needs a mobile-validation leg, possibly per-backend tunings.
-
-## 9. Decision log (this cycle)
-
-- Mobile native inference fixed (four-bug diagnosis: jetsam memory, WebKit visibility, attention race, detection predicate). See `docs/mobile-failure-diagnosis.md`, paper §17–19.
-- Desktop 145→~207 via autoresearch (paper §20).
-- **Architecture decision: NATIVE-ONLY** across text + vision + embeddings; no tfjs fallback lane; `chrome-backend.ts` to be deleted (paper §23).
-- **Native embeddings shipped** — Qwen3-Embedding-0.6B, last-token EOS pool + L2Norm, validated (paper §21).
-- **Native vision encoder shipped** — Qwen3.5 ViT, bit-exact vs HF transformers 5.12 (cosine 1.000000); LM-integration is phase 2 (paper §22). Supersedes the earlier "skip the ViT" stance (`docs/research/qwen35-multimodal.md`).
-- **Vision feasibility corrected** — attention was already parallel (panel read a dead `.wgsl` file); non-causal is a one-line flag (paper §22–23, §5 above).
-- **OPFS removed** — main-thread `createWritable` broken on iOS; Cache-API-only loader; durable caching needs a PWA (`docs/research/ios-safari-model-caching.md`, paper §24).
-- Modality model picks (`docs/research/sota-modality-models.md`); chromium decision (`docs/research/native-vs-chromium-decision.md`); site update plan (`docs/site-update-plan.md`).
-
----
-
-## 10. 2026-06-13 evening — multimodal milestone + in-flight work (CAPTURE)
-
-**Major wins this session (all committed to `feat/webgpu-engine-mobile`):**
-
-- **Native VISION end-to-end DONE & validated bit-exact** (commit `5fed12b`). `engine.describeImage({pixels,width,height})` → coherent description. Validated vs HF transformers 5.12 on `examples/skelly.png`: 7/7 checks — encoder cosine 1.000000, M-RoPE 3D position ids EXACT, spliced embeds cosine 1.0, first-token exact, full greedy description WORD-IDENTICAL to HF for 201 chars ("This is a detailed black-and-white line drawing of the skeleton of a fox…"). New ops: `MRoPE`, `EmbedSplice`, host image preprocessing (smart-resize/normalize/patchify in `vision-preprocess.ts`). Text path unchanged (M-RoPE on linear positions reduces to 1D RoPE).
-- **Native EMBEDDINGS** (earlier commit): Qwen3-Embedding-0.6B works on DESKTOP (cosine 0.81>0.56). BUT **not iPad-viable** — only BF16 (~1.2GB, OOMs on-device) or a broken MLX-DWQ. → Pivot below.
-- **`is_causal` flag** on f32 attention (from vision) now unlocks **bidirectional encoders**.
-
-**The 51-vs-41 mobile throughput question — RESOLVED: NOT a regression.** t15 hit 51.7 on a 200-token sustained run; the 38-41 readings were 60-token runs (more warmup overhead/token). A fresh 200-token sustained run on the CURRENT (post-autoresearch-optimization) engine **hit ~51 again**. So the Dawn-tuned wins did NOT regress mobile; sustained mobile decode is ~51 tok/s. Lesson: benchmark with enough tokens (≥200) to be representative.
-
-**IN-FLIGHT / NOT YET MERGED (future-me: merge these):**
-- **LFM2.5-350M** — ✅ **LANDED** on `feat/webgpu-engine-mobile` (commit `3c4bac8`). Tier-1, no new kernels. **~600 tok/s desktop (2.8× Qwen3.5's ~213), ~199MB q4 (half Qwen), coherent, bit-exact vs NumPy ref.** Files: new `src/gpu/architectures/lfm2.ts` (824 lines), registered `Lfm2ForCausalLM`, LFM2 CANONICAL_KEYS, loader key-mapper. TWO GENERAL FIXES worth keeping: (1) effective FF dim is 4608 not config's 6656 (`block_auto_adjust_ff_dim` rounding — `multiple_of(⌊2/3·6656⌋)`); (2) **the "garbage output" was the CHAT TEMPLATE, not the graph** — LFM2.5 ships its template as a `chat_template.jinja` sidecar (absent from tokenizer_config.json), so the engine fell back to Qwen ChatML which auto-injects an empty `<think>` → newline loop. Fix: fetch the `.jinja` sidecar + gate think-injection on the template actually emitting `<think>`. This fix helps ANY model with a jinja sidecar. Verdict: LFM2.5 is a viable faster/smaller text-default alternative to Qwen — user picks.
-- **EmbeddingGemma-300M** — ✅ **LANDED + CONFIRMED ON iPad** (commit `4874d01`, see §11). The iPad-ready embedding model (~173MB q4, MTEB 68.36, standard MLX-4bit). Bidirectional Gemma3 encoder — needs the new generator + a MeanPool kernel + dual-theta RoPE + 2 Dense head layers + the MLX-detection loader fix (research found: detector requires `mode:"affine"` but standard MLX converts omit it → silently fall to F32; and DWQ vs standard MLX are config-indistinguishable → the DWQ-garbage trap). See `docs/research/sota-embedding-models.md`.
-
-**iPad harness:** dashboard at `https://<lan-ip>:8766/` with Text/Vision/transformers.js/Storage-probe/Clear-cache/Stop-queue buttons, live results table, `/results` + `/enqueue` + `/clear-queue` endpoints. Embeddings button removed until EmbeddingGemma lands. TODO: show generated text in the results + default Text benchmark to 200 tokens (representative); wire the Vision button to `describeImage` (needs browser ImageBitmap/Canvas→pixels decode).
-
-**iOS caching:** OPFS removed (main-thread createWritable broken on iOS, left unclearable junk). Durable cache needs a PWA (persist() only granted when installed). See `docs/research/ios-safari-model-caching.md`. Deferred.
-
-**Native modality scorecard (as of §10):** text ✅, vision ✅ (end-to-end bit-exact), embeddings ✅ desktop / iPad-pending (EmbeddingGemma building), LFM2.5 text-alt ✅ (pending merge). Audio (TTS/STT) deferred to native OmniVoice/Moonshine. **The native-only multimodal engine is real.** **→ All "pending/building" items above have since landed; see §11.**
-
----
-
-## 11. 2026-06-14 — EmbeddingGemma on-device, LFM2.5 landed, loader hardening (CAPTURE)
-
-The §10 in-flight items all landed on `feat/webgpu-engine-mobile`. New wins this session
-(paper §25–§30 document each in depth):
-
-- **EmbeddingGemma-300M LANDED and CONFIRMED RUNNING ON iPad Safari** (commit `4874d01`).
-  First **non-Qwen** embedding family — a real **bidirectional Gemma3 encoder**
-  (`src/gpu/architectures/gemma3_encoder.ts`, `generateGemma3EncoderGraph`): 24 pre-norm
-  blocks, GQA 3q/1kv head_dim 256, per-head q/k-norm, **dual-theta RoPE** (sliding θ=10000
-  / full θ=1e6 selected per layer from `layer_types`), GeGLU MLP, Gemma's **four-norm
-  sandwich**, embed ×√768, tail **MeanPool → Dense0(768→3072) → Dense1(3072→768) → L2Norm**.
-  **173 MB at MLX-4bit** (vs the abandoned 1.2 GB Qwen3-Embedding that OOM'd iPad).
-  Two new kernels only: **MeanPool**, **Scale** (`kernels/registry.ts`). Validated
-  **cos=1.00000 vs an independent NumPy reference** (`scripts/engine/test-embedding-gemma-reference.py`),
-  reference gate `cos>0.95`; semantic test asserts a Red-Planet query is closer to two
-  Mars docs than to a bread doc by **>0.1 cosine margin**, unit-norm dim-768, no NaN. The
-  engine **generalizes across embedding families**, not just Qwen.
-
-- **SPM tokenizer fix (load-bearing, cross-family)** (`src/gpu/tokenizer.ts`). Gemma's
-  `tokenizer.json` is `type:"BPE"` but **SentencePiece-flavored** (▁/U+2581 spaces, raw
-  UTF-8 tokens, array-form merges, `<0xHH>` byte-fallback). The byte-level (`Ġ`) BPE path
-  was char-splitting every word → semantically dead embeddings that still passed norm/NaN
-  checks. Auto-detected `spmMode` (structural: `" "→"▁"` Replace normalizer **or**
-  `byte_fallback && "▁the" in vocab`) now drives encode/decode/merges; **Qwen/LFM2 stay
-  byte-level** (no model-name list — they just don't match). Lesson: `type:"BPE"` ≠
-  byte-level BPE; any SentencePiece-lineage family (Gemma/Llama/Mistral) needs this path.
-
-- **MLX-4bit loader hardened** (`src/gpu/model-loader.ts`). (1) Detection broadened to
-  accept mode-less `{bits:4, group_size}` configs (standard mlx-lm omits `mode`). (2) A
-  `VERIFIED_MLX_REPOS` allowlist (currently `mlx-community/embeddinggemma-300m-4bit`) gates
-  mode-less configs. (3) **Explicit DWQ reject** — DWQ repos carry an identical
-  `{bits:4,group_size}` config but pack weights that dequant to garbage, so they're rejected
-  by repo-name substring (`includes("dwq")`). Codifies the MLX-DWQ-garbage trap. (4) Gemma's
-  `(1+weight)` RMSNorm absorption is **baked by the loader even for MLX** (mlx-lm pre-absorbs
-  +1 for Qwen3.5 but NOT for Gemma — the Gemma branch deliberately omits `&& !isMLX`).
-
-- **Progress-reporting fix** (commit `682a09b`, `model-loader.ts`). The bar froze at
-  "10% discovering weight files" because the gap between that emit and the first download
-  chunk (index probe + 2 header range-requests + first-byte latency) emitted nothing. Now
-  emits "Reading {file} header" + "Downloading {file} (0/{N} MB)" up front. **Affected every
-  model**; fixed universally.
-
-- **LFM2.5-350M landed** (commit `3c4bac8`) — Tier-1, no new kernels, ~600 tok/s desktop /
-  ~46 tok/s mobile, ~199 MB q4. The general jinja-sidecar chat-template fix from it helps any
-  model shipping `chat_template.jinja`.
-
-- **Cross-device multi-modal parity.** Text (Qwen3.5 ~51 tok/s, LFM2.5 ~46 tok/s), vision
-  (Qwen3.5 ViT `describeImage`), and embeddings (EmbeddingGemma) **all now run natively on
-  iPad Safari** — the native engine reaches transformers.js-path modality coverage on the
-  modalities that matter, without the mobile crashes and ~5× faster. **Remaining native gap:
-  audio** — TTS via OmniVoice (in progress), STT via Moonshine (not started).
-
-- **Effort-tier shift.** Adding a new TEXT family is now usually **Tier-1: generator only, no
-  new kernels** — the kernel library has saturated for standard transformers (Llama/Mistral/
-  Gemma-text reuse existing ops). New kernels are needed only for genuinely novel ops (SSM,
-  PLE, new norms, cross-attention). See `docs/adding-a-model-family.md`.
-
-- **Site migration assessment** written: `docs/gerbil-site-native-migration.md` (how the
-  marketing/docs site can move its in-browser inference from the transformers.js/ONNX worker
-  to the native engine, modality-by-modality, with the no-fallback device-coverage tradeoff
-  stated plainly). Assessment only — the site repo was not modified.
-
----
-
-## 12. 2026-06-14 — native audio begins, Gemma 4, memory/RAG, autoresearch campaign (CAPTURE)
-
-This session pushed past the §11 "multimodal parity minus audio" milestone. Paper
-**§31–§35** document each in depth. The decided order from §6 (audio last, native not
-delegated) held — and audio is now *under way natively*, not deferred to tfjs.
-
-- **Native STT — Moonshine LANDED** (`src/gpu/architectures/moonshine.ts`,
-  `moonshine-executor.ts`, `moonshine-stt.ts`; paper §31). Chosen over Whisper to avoid
-  a log-mel/Conv2d front-end: it consumes **16 kHz PCM directly** through three strided
-  `Conv1d`s (downsample 384×, ~41.6 frames/s) — new kernels `Conv1dFull`, `GroupNorm`,
-  `Tanh`, `Transpose`. The real new attention primitive is **`CrossAttention`**
-  (`WGSL_CROSS_ATTENTION`, tiled online-softmax), validated **bit-exact vs NumPy**
-  (`test-crossattention.mjs`: max|err|<2e-4, cos≥0.9999). Runtime is a **dual graph**:
-  `MoonshineEncoderExecutor.encode()` runs the front-end + bidirectional encoder once and
-  freezes per-decoder-layer K/V; `MoonshineSTT.transcribe()` runs greedy AR decode with
-  self- + cross-attention into that frozen K/V. **Interleaved RoPE** (`ROPE_INTERLEAVED_SPEC`,
-  adjacent-dim pairing) vs the default split-half. Validation: encoder cos≈0.990 vs HF
-  (source comment); `test-moonshine-transcribe.mjs` asserts transcript contains HF-ref
-  substrings; RTF/4-bit size are computed-and-reported, not hardcoded-asserted.
-  **Whisper(ONNX) stays the multilingual / no-WebGPU fallback** (`src/core/stt.ts`,
-  `WhisperSTT`) — separate, untouched.
-
-- **Native TTS — Kani-TTS-2 (partial)** (`src/gpu/architectures/kani_tts.ts`; paper §32).
-  The hard novel piece — the **NanoCodec decoder** (FSQ 4×4 levels `[9,8,8,7]`, mixed-radix
-  base `[1,9,72,576]` + causal HiFi-GAN, rates `[7,7,6,3,2]`, hop 1764 @ 22050 Hz) — is
-  **implemented and validated bit-exact** (`test-nanocodec-decode.mjs`, gate `err<1e-3`,
-  measured ~4.2e-6 vs MLX). New kernels: `FSQDequant`, `HalfSnake1d`,
-  `ConvTranspose1dDepthwise`. The backbone is **LFM2-350M** (`KaniTTS2ForCausalLM`, audio
-  tokens above the text vocab, 4/frame); `generateKaniTtsGraph` **deliberately throws** —
-  remaining is the frame-position + learnable-RoPE + 4-token-frame AR-decode glue (most
-  block math reused from `lfm2.ts`). License: **kani-tts-2-en = LFM1.0 (other)**, NanoCodec
-  = NVIDIA OML; the **450m variant is Apache** (same arch).
-
-- **Gemma 4 E2B — text decode COHERENT on real q4 weights** (`src/gpu/architectures/gemma4.ts`;
-  paper §33). **Tier-2**, no MatFormer/AltUp/LAuReL. **PLE** (2nd embedding gathered per
-  token, per-layer gate+GELU+multiply+project+norm+residual), **KV-cache sharing** (E2B: 35
-  layers, last **20** shared via graph-rewire, no kernel change), **proportional RoPE**
-  (rotate 0.25·head_dim=64 dims but inv_freq over full head_dim denom; dual-theta 1e6/1e4),
-  **GeGLU**, and a new **`Softcap`** kernel (`cap·tanh(x/cap)`, cap=30) on final logits.
-  Generates coherently ("The capital of France is" → "Paris") at ~83 tok/s; all 35 layers
-  match an MLX-LM reference cos≥0.998 with identical argmax (`test-gemma4-perlayer.mjs`).
-  Structural validation **67/67** (`test-gemma4-graph.mjs`; 942 nodes); softcap kernel
-  separately validated (`test-gemma4-softcap.mjs`, max err<1e-4). **PLE is CPU-streamed, not
-  GPU-sharded**: the ~1.17 GB q4 PLE table stays CPU-resident (**0 MB GPU**) and the executor
-  streams per-token rows each step — Gemma 4's intended flash design, mobile-viable, and it
-  sidesteps the per-binding cap. Coherence required four fixes: per-node `attn_scale`=1.0
-  (HF scaling=1.0; default 1/√head_dim keeps other models byte-identical), parameter-free
-  V-norm, double-wide MLP on the KV-shared layers, and head_dim-512 support in the flash
-  attention kernel (second per-thread accumulator + smem-capped tiling, 16 KB invariant kept).
-
-- **On-device Memory / RAG SHIPPED** (`src/memory/`, `@tryhamster/gerbil/memory`; paper §34).
-  Pluggable vector store (`InMemoryStore` / `IndexedDBStore` / `FileStore`), token-budgeted
-  `recall()` (default 1024-token greedy pack, ~4-chars/token, returns `{context, records,
-  tokensUsed}`), overlapping-window chunking (1000/200), write-time redaction (regex →
-  `[REDACTED]` or fn), and a `createGerbilEmbedder()` adapter over **native EmbeddingGemma**.
-  **12/12 tests** (`src/memory/memory.test.ts`). No new kernels — a clean consumer of the
-  embedding modality.
-
-- **Autoresearch TPS campaign — 3 more batches** (`scripts/engine/results.jsonl`,
-  `scripts/engine/chart.html`; paper §35), M4 Max / node-dawn. Verified peaks:
-  **Qwen3.5-0.8B 219→~234 tok/s**, **LFM2.5-350M 624→~672 tok/s**, **ViT encode
-  581.8→~502 ms (−~14%)**, **`describeImage` 37.0→42.0 tok/s (+13.5%)** — all kept changes
-  bit-exact (merged cos 1.0, e2e 7/7). **The winning lesson (sharpened):** desktop wins come
-  only from eliminating **large wide reads on poorly-occupied kernels** (fused conv+activation,
-  vec4 + register-blocked + f16-mixed ViT matmul, `MatMul+AddBias→MatMulBias`); **pure
-  dispatch-count cuts on already-tuned kernels are noise** (butterfly reduce, subgroup
-  shuffle, bigger N-tiles all reverted). The INT4 matmuls and Mamba SSM sit at the
-  **bandwidth floor** — the remaining headroom is **mobile**, not desktop (several
-  reverted-on-desktop fusions are *predicted mobile wins*; the loop's next leg is a
-  mobile-validation pass).
-
-- **gerbil-site is LIVE on the native engine.** The marketing/docs site's in-browser
-  inference now runs on the native WGSL engine (no longer the transformers.js/ONNX worker)
-  for the migrated modalities — see `docs/gerbil-site-native-migration.md`. (The §11 entry
-  was an assessment-only; this cycle it went live.)
-
-**Native modality scorecard (as of §12):** text ✅ (Qwen3.5, LFM2.5), text-alt families 🟢
-Tier-1, Gemma 4 E2B 🟡 (decode validated, sharding gates real weights), vision ✅ (Qwen3.5
-ViT, on iPad), embeddings ✅ (EmbeddingGemma, on iPad), **STT ✅ native (Moonshine) + Whisper
-fallback**, **TTS 🟡 (Kani NanoCodec decoder validated, backbone AR loop pending)**, memory/RAG
-✅ shipped. **Audio is no longer the deferred gap — STT is native; TTS is one AR-loop away.**

package/docs/adding-a-model-family.md

@@ -1,280 +0,0 @@
-# Adding a Model Family to the Gerbil WebGPU Engine
-
-This is the repeatable process for teaching the native WebGPU engine a new model
-architecture. The engine is **not** Qwen-specific — it's a registry of graph
-generators over a family-agnostic IR and kernel library. Adding a family is
-usually "write one generator + register it," and only occasionally "write a new
-kernel."
-
-> TL;DR effort tiers:
-> - **Tier 1 (hours):** the model is Llama-like (standard transformer). Reuse existing ops; write a config→IR generator. Llama, Mistral, Gemma-text, Phi, Qwen all live here.
-> - **Tier 2 (days):** the model has one novel op (a new norm, sliding-window attention, a gate) OR new per-node params on existing kernels. Write the generator + 1–2 WGSL kernels.
-> - **Tier 3 (weeks):** the model has a fundamentally new computation (SSM/Mamba, MoE routing, dual-graph encoder-decoder, codec decoder). New kernels + executor support. Qwen3.5's Mamba-2 path was Tier 3.
-
-> **Repeatable SOP:** for the step-by-step loop use the `add-model-family` skill
-> (`.claude/skills/add-model-family/SKILL.md`). This doc is the reference; the
-> skill is the procedure. Both encode the lessons in "Lessons from production"
-> below — read that section first.
-
----
-
-## Lessons from production (read first)
-
-A wave of families was added against this framework — Qwen3.5 (Mamba-2 SSM + ViT),
-LFM2.5 (hybrid conv/attn), EmbeddingGemma (bidirectional Gemma3 encoder), Moonshine
-STT (CrossAttention + dual-graph), Kani-TTS (LFM2 codec-LM + NanoCodec decoder),
-Gemma 4 E2B (PLE CPU-streaming + proportional RoPE + double-wide MLP + value-norm +
-head_dim-512). What actually mattered:
-
-| Family | Arch string | Tier | What was new |
-|---|---|---|---|
-| Qwen2/3 | `Qwen2/3ForCausalLM` | 1 | baseline standard transformer (the template) |
-| Qwen3-Embedding | `Qwen3ForCausalLM` (embedding flag) | 1 | `SliceLastRow`→`L2Norm` pooling tail, no new kernels |
-| Qwen3.5 | `Qwen3_5ForConditionalGeneration` | 3 | Mamba-2 SSM kernels + ViT encoder (`VisionExecutor`) |
-| LFM2.5 | `Lfm2ForCausalLM` | 2 | short-conv/attention hybrid; `out_proj`→`o_proj` rename |
-| EmbeddingGemma | `Gemma3TextModel`/`Gemma3Model` | 2 | bidirectional encoder (`causal:false`) + mean-pool + dense tail |
-| Moonshine STT | `MoonshineForConditionalGeneration` | 3 | new `CrossAttention` kernel; dual-graph (encoder+decoder) executor |
-| Kani-TTS | `KaniTTS2ForCausalLM` | 3 | LFM2 codec-LM backbone + NanoCodec decoder (FSQ + causal HiFi-GAN) |
-| Gemma 4 E2B | `Gemma4ForConditionalGeneration` | 2 | PLE CPU-streaming, proportional/partial RoPE, value-norm, head_dim-512, double-wide MLP, per-layer scalar — all via per-node attrs + loader hooks |
-
-1. **VERIFY FIRST against the live checkpoint.** Fetch the live `config.json` and
-   range-read the safetensors header (keys/dtypes/shapes) BEFORE writing anything.
-   Confirm the arch string, dims, quant format, and classify every op. Do NOT
-   trust assumptions from sibling models: Gemma4 had no MatFormer though Gemma3n
-   did; OmniVoice's codec is diffusion not AR; MLX repos ship the ViT under a
-   different key prefix (`vision_tower.*`) than HF bf16 (`model.visual.*`), and the
-   LM under `language_model.model.*` (MLX) vs `model.language_model.*` (GPTQ/HF).
-   Watch for **nested config** — Qwen3.5/Gemma4 put the text tower under
-   `text_config`.
-2. **Prefer per-node params over new kernels.** Most "novel" behavior is a new
-   parameter on an existing kernel. Add it to the WGSL `Params` struct and read it
-   in `buildParams` with a **default that reproduces the old behavior
-   byte-identically** (`attn_scale ?? 1/sqrt(head_dim)`), so every existing caller
-   is unchanged. Gemma4's attn_scale=1.0, partial RoPE
-   (`rope_dim/rope_half/rope_denom/rope_active_pairs`), and per-layer `Scale`
-   (loader-patched from `layer_scalar_key`) all landed this way — zero new attention
-   kernels.
-3. **Validate with a LOAD-ONCE reference harness — the make-or-break lesson.** Dump
-   the HF/MLX reference activations to disk ONCE
-   (`{tokens, per_layer_last[L][hidden], argmax, logits_top}`), then iterate
-   engine-only in SECONDS via a single `engine.create()` + `rawForward()` +
-   per-layer `debugReadBuffer` cosine loop (`test-gemma4-perlayer.mjs`). Reloading
-   the 2.5GB model every cycle made Gemma4 cost an hour per iteration. Localize
-   divergence to the exact layer/op. Set `GERBIL_NO_ACT_POOL=1` when reading early
-   activations (the pool aliases buffers). Gate on coherent generation — bit-exact
-   is not the bar for lossy quants.
-4. **No-regression suite is mandatory for shared-kernel changes.** Keep all
-   existing models bit-exact: `test-q4-generate`, `test-vision-e2e`,
-   `test-crossattention`, `test-nanocodec-decode`, the embedding test, and the fast
-   `test-gemma4-perlayer`. The byte-identical-default pattern (lesson 2) is what
-   keeps these green.
-5. **Big tensors: shard or stream.** Per-buffer caps are ~256MB
-   (`maxBufferSize`) / ~128MB (`maxStorageBufferBindingSize`) on iPad. Shard the
-   embedding if vocab×hidden exceeds the cap; for vocab-scale auxiliary tables keep
-   them CPU-resident and stream per-layer slices. Gemma4's ~1.17GB PLE table has
-   **0 MB GPU residency** — the loader builds a `PleSource`, diverts it from the
-   weights map, and the executor uploads only the needed `[T, width]` rows per step.
-6. **Merge additively, never commit node_modules.** Registry/IR/index conflicts are
-   always additive — keep ALL families/ops from both sides. Use targeted `git add`,
-   never `git add -A`.
-
----
-
-## The architecture
-
-When a model loads, `model-loader.ts` reads `config.json` → `architectures[0]`
-(e.g. `"LlamaForCausalLM"`) and looks it up in the registry:
-
-```
-src/gpu/architectures/index.ts   →  ARCHITECTURES: Record<string, GraphGenerator>
-```
-
-A `GraphGenerator` turns the raw HF config into a `ModelGraph` (the IR):
-
-```ts
-type GraphGenerator = (
-  config: Record<string, unknown>,
-  dtype?: "f32" | "q4",      // q4 = on-the-fly INT4 quantization
-  groupSize?: number,        // INT4 group size (default 128)
-  kvDtype?: "f32" | "f16",   // KV cache precision
-) => ModelGraph;
-```
-
-The `ModelGraph` is `{ tensors, nodes, executionOrder, inputs, outputs, config, capabilities, architecture }`.
-The executor (`executor.ts`) consumes it generically: it allocates buffers
-(liveness-pooled), compiles one pipeline per node from `KERNEL_REGISTRY`, and
-dispatches in `executionOrder`. **The generator never touches WebGPU** — it only
-describes the computation.
-
----
-
-## Step-by-step
-
-### 1. Identify the family and check op coverage
-
-Read the model's `config.json` and its modeling code on HuggingFace. Classify
-every layer's ops and check them against what the engine already has
-(`KERNEL_REGISTRY` in `src/gpu/kernels/registry.ts`):
-
-**Implemented ops (reuse freely):** `Embedding`, `EmbeddingInt4`, `MatMul`,
-`MatMulInt4`, `Add`, `Mul`, `Scale` (per-element, loader-patchable scalar),
-`RMSNorm`, `LayerNorm`, `RoPE` (with optional partial-rotary attrs
-`rope_dim/rope_half/rope_denom/rope_active_pairs`), `Attention` (causal *and*
-bidirectional via the `causal` flag — set `causal: false` for an encoder/ViT;
-takes a per-node `attn_scale` defaulting to `1/sqrt(head_dim)`, plus
-`sliding_window`), `CrossAttention` (encoder-decoder), `Softmax`, `SiLU`,
-`SwiGLU`, `GELU` (tanh approx), `GeluErf` (exact erf), `AddBias` (row-broadcast
-bias), `ApplyRotaryEmb` (precomputed-cos/sin `rotate_half`), `SliceCols`
-(column-range extract, e.g. split a fused QKV), `L2Norm` (row-wise, embedding
-tail), `ResidualRMSNorm`, `KVCacheAppend`, `MambaSSM`, `CausalConv1d`,
-`SigmoidGate`, `ConvStateUpdate`, `SliceLastRow`, and codec ops
-(`Conv1d`/`ConvTranspose1d`/`Snake1d`, FSQ) for audio decoders.
-
-If the model only uses these → **Tier 1**. If it needs something new → note it;
-you'll write a kernel in Step 5.
-
-**Embedding models** are Tier 1: an embedding model (e.g. Qwen3-Embedding-0.6B,
-which is `Qwen3ForCausalLM`) is the causal-LM forward pass with a different tail —
-pass the `embedding` flag to `generateQwen2Graph` to swap `lm_head` for the
-`SliceLastRow` (last-token EOS pool) → `L2Norm` tail. No new kernels (paper §21).
-
-**Vision-capable checkpoints** (e.g. Qwen3.5) ship a ViT in the same weights; the
-encoder is a separate graph (`qwen3_5_vision.ts`) run by `VisionExecutor`, already
-built and bit-exact vs HF (paper §22). Patches arrive pre-flattened, so the
-patch-embed is a plain `MatMul` + `AddBias` — no Conv3d kernel.
-
-**Exotic features that need new work (flag early):**
-- Sliding-window / local attention (needs an attention kernel variant)
-- Mixture-of-Experts routing (`MoERouter`/`ExpertMatMul` are stubbed, not built)
-- Novel normalization (e.g. QK-norm, per-head norm) — new kernel
-- MatFormer / elastic parameters (Gemma 3n) — needs slicing logic + a loader story
-- Attention logit soft-capping (Gemma 2/3) — small kernel tweak
-- Non-RoPE position encodings
-
-### 2. Map config → dimensions
-
-Every generator starts by pulling dimensions from the raw config. Copy the block
-from `src/gpu/architectures/qwen2.ts` and adjust names to the new model's config:
-
-```ts
-const hidden_size = rawConfig.hidden_size as number;
-const num_layers = rawConfig.num_hidden_layers as number;
-const num_heads = rawConfig.num_attention_heads as number;
-const num_kv_heads = (rawConfig.num_key_value_heads as number) ?? num_heads;  // GQA
-const intermediate_size = rawConfig.intermediate_size as number;
-const vocab_size = rawConfig.vocab_size as number;
-const context_length = (rawConfig.max_position_embeddings as number) ?? 32768;
-const rms_norm_eps = (rawConfig.rms_norm_eps as number) ?? 1e-6;
-const rope_base = (rawConfig.rope_theta as number) ?? 1_000_000.0;
-const head_dim = (rawConfig.head_dim as number) ?? Math.floor(hidden_size / num_heads);
-const tieWordEmbeddings = (rawConfig.tie_word_embeddings as boolean) ?? false;
-```
-
-Watch for family differences: **head_dim is often NOT hidden_size/num_heads**
-(Gemma sets it explicitly); GQA means `num_kv_heads < num_heads`; some families
-have **QKV projection bias** (Qwen2 does, Llama/Gemma don't).
-
-### 3. Write the generator
-
-Create `src/gpu/architectures/<family>.ts` exporting
-`generate<Family>Graph(...)`. Use the local `addTensor`/`addNode` helpers (they
-also push to `executionOrder`). The per-layer skeleton, following qwen2.ts:
-
-1. **Embedding** — `Embedding`/`EmbeddingInt4` reading `input_ids` + `CANONICAL_KEYS.EMBED`.
-2. **Per layer** (loop `num_layers`):
-   - input RMSNorm → Q/K/V projections (`MatMulInt4`) → RoPE → `KVCacheAppend` → `Attention` → output projection → residual `Add`
-   - post-attn RMSNorm → gate/up projections → `SwiGLU` → down projection → residual `Add`
-   - (use `ResidualRMSNorm` to fuse the residual+norm where the family allows)
-3. **Final norm** (`RMSNorm`).
-4. **`SliceLastRow`** on the final hidden state → **lm_head** (`MatMulInt4`) → `logits`.
-   The `[1, vocab]` logits + SliceLastRow are mandatory (saves ~485MB at long
-   context and skips the full-vocab prefill matmul — see paper §18). Honor
-   `tieWordEmbeddings` (reuse embed weights for lm_head).
-
-Declare intermediate tensors with `storage: "activation"` — the executor pools
-them automatically; don't manage buffers yourself. Set `capabilities`
-(`text`, `vision`, `moe`) and return the graph.
-
-### 4. Register it
-
-In `src/gpu/architectures/index.ts`:
-
-```ts
-import { generateLlamaGraph } from "./llama.js";
-export const ARCHITECTURES = {
-  // ...existing...
-  LlamaForCausalLM: generateLlamaGraph,
-  MistralForCausalLM: generateLlamaGraph,   // Mistral == Llama arch
-};
-```
-
-One generator can serve a whole family — Llama, Mistral, and many fine-tunes
-share `LlamaForCausalLM`.
-
-### 5. Weight key mapping
-
-The loader maps HF safetensors keys → canonical names via a `HFKeyMapper`
-(`createDefaultHFKeyMapper` strips the `model.` prefix). If the new family names
-weights differently, supply a mapper or set `safetensorsKey` on each `addTensor`.
-`CANONICAL_KEYS` has helpers (`qProj(i)`, `layerInputNorm(i)`, `EMBED`,
-`LM_HEAD`, …) — reuse them so the loader and generator agree.
-
-### 6. Write any new kernel (only if Step 1 flagged one)
-
-Add to `src/gpu/kernels/registry.ts`: a WGSL string + a `KernelSpec`
-(`shaderCode`, `entryPoint`, `bindings`, `getDispatchSize`, `buildParams`), then
-register it in `KERNEL_REGISTRY` under your new `OpType` (add the op to the union
-in `ir.ts`). **Mobile constraints (WebKit/iPad):** workgroup memory ≤ 16 KB
-(`maxComputeWorkgroupStorageSize`); no `enable f16` (use packed `pack2x16float`);
-clamp `exp()` for Metal fast-math; avoid `select()` (use if/else). See the
-attention kernels for the safe patterns and the two-phase reduction.
-
-### 7. Validate correctness (do not skip)
-
-1. **Reference activations:** adapt `scripts/engine/test-reference.py` to dump the
-   new model's logits + key intermediate activations from HF transformers.
-2. **Compare:** run `engine.integrityCheck()` (weights + isolated dispatches +
-   pipeline probes) and diff the engine's logits against the reference. Match the
-   argmax and check sums within tolerance.
-3. **Coherence:** generate from a fixed prompt at temperature 0 and confirm
-   sensible text (the `scripts/engine/test-q4-generate.mjs` pattern).
-4. **Cross-platform:** the desktop (Dawn) and mobile (WebKit) outputs must be
-   byte-identical — queue an iPad run via the harness (`scripts/engine/ipad-queue.json`).
-
-### 8. Mobile / download budget checks
-
-- **Download size at q4** is the headline mobile metric — report it.
-- **Per-buffer limit:** iPad `maxBufferSize` defaults to 256MB and
-  `maxStorageBufferBindingSize` to 128MB. A single weight tensor (often the
-  embedding) must fit — shard it if the model's vocab×hidden exceeds the cap.
-- **maxSeqLen policy:** the engine clamps iOS to 512 by default; large contexts
-  multiply KV-cache memory.
-
----
-
-## Worked example: the smallest possible new family (Llama)
-
-Llama is Qwen2 minus QKV bias. The fastest path:
-1. Copy `qwen2.ts` → `llama.ts`, rename the export.
-2. Remove the QKV-bias tensors/handling (Llama has none).
-3. Confirm head_dim, GQA (`num_key_value_heads`), and `tie_word_embeddings` from config.
-4. Register `LlamaForCausalLM`/`MistralForCausalLM` → `generateLlamaGraph`.
-5. Validate against a reference (Step 7). No new kernels needed — Tier 1.
-
-This single generator unlocks Llama, Mistral, and every model that ships as
-`LlamaForCausalLM` on HuggingFace.
-
----
-
-## Checklist
-
-- [ ] Classified every op; new ops listed (or confirmed none)
-- [ ] `generate<Family>Graph` written, dimensions mapped from config
-- [ ] GQA / head_dim / QKV-bias / tied-embeddings handled
-- [ ] `SliceLastRow` + `[1, vocab]` logits in place
-- [ ] Registered in `ARCHITECTURES`
-- [ ] Key mapping verified (loader finds every weight)
-- [ ] New kernels (if any) respect the 16 KB / no-`enable f16` / clamped-`exp` mobile rules
-- [ ] Validated vs HF reference (logits argmax + coherence)
-- [ ] Desktop and iPad outputs byte-identical
-- [ ] Download size at q4 reported; largest weight fits the 256MB/128MB buffer caps