@tryhamster/gerbil 1.0.0-rc.9 → 1.0.1

package/docs/research/sota-embedding-models.md

@@ -0,0 +1,179 @@
+# SOTA Small Embedding Models for Gerbil on iPad (WebGPU)
+
+> Research date: 2026-06-13. Verified against live HuggingFace `config.json` / file lists / model cards
+> and the MTEB leaderboard, cross-checked with Perplexity (`sonar-pro`). Engine capabilities verified
+> against `src/gpu/` (model-loader, architectures, kernels).
+
+## TL;DR — the ranked pick
+
+| Rank | Model | Arch | q4 size | Compatible q4 format on HF? | Engine work to run it |
+|------|-------|------|---------|------------------------------|------------------------|
+| **1** | **`google/embeddinggemma-300m`** (via `mlx-community/embeddinggemma-300m-4bit`) | **Bidirectional** Gemma3 encoder | **~173 MB** | **YES — standard MLX-4bit** (QAT base, not DWQ) | New **Tier-2 Gemma3 encoder generator** (medium effort) |
+| **2** | **`Qwen/Qwen3-Embedding-0.6B`** | Causal-LM (Qwen3) | ~335 MB (MLX) / ~538 MB (compressed-tensors) | **Risky** — see notes | None for arch (runs today on desktop); but **no clean-verified q4** |
+| **3** | **`Snowflake/snowflake-arctic-embed-s`** or **`bge-small-en-v1.5`** | Bidirectional BERT encoder | ~70–130 MB at q4 (on-the-fly from F32) | No q4 we read (only ONNX q4); use **on-the-fly INT4 from F32** | New **Tier-2 BERT encoder generator** (smaller effort than Gemma3) |
+
+**Recommendation:** ship **EmbeddingGemma-300M** as Gerbil's default iPad embedding model. It is the
+single highest-quality model under 1B params (MTEB English v2 **68.36**, Multilingual v2 **61.15**),
+fits the iPad budget at **~173 MB q4**, and — critically — ships a **standard-affine MLX-4bit** quant
+derived from Google's **quantization-aware-trained (QAT)** base, so q4 quality is near-lossless and the
+format is one our `repackMLX` loader already understands. The cost is one new **bidirectional Gemma3
+encoder generator**. We already have the prerequisite ops (the new `is_causal=false` attention flag,
+RMSNorm, RoPE, GELU-tanh, mean pooling needs adding, L2Norm).
+
+---
+
+## Engine capability ground-truth (verified in `src/gpu/`)
+
+What the loader actually detects (`src/gpu/model-loader.ts` ~L567–584):
+
+```ts
+const quantConfig = rawConfig.quantization_config;
+const isGPTQ = quantConfig?.quant_method === "gptq";
+const isMLX  = !isGPTQ && quantConfig?.bits === 4 && quantConfig?.mode === "affine";
+```
+
+- **GPTQ-Int4**: read via `repackGPTQ` (`src/gpu/gptq-adapter.ts`). Detects `quant_method === "gptq"`.
+- **MLX 4-bit**: read via `repackMLX` (`src/gpu/mlx-adapter.ts`). Tensors `.weight`/`.scales`/`.biases`.
+- **On-the-fly INT4 from F32/BF16**: `quantizeInt4` (`src/gpu/quantize.ts`), group size 128.
+- **NOT supported**: `compressed-tensors` / `pack-quantized`, GGUF, ONNX, MLX **DWQ**.
+
+⚠️ **Two loader caveats this research surfaced (both are quick fixes, but load-bearing):**
+1. **MLX detection currently requires `mode === "affine"`**, but the EmbeddingGemma and Qwen3 MLX
+   converts on HF write only `{"bits":4,"group_size":64}` with **no `mode` field** (and put it under
+   `quantization`, not always `quantization_config`). So the detector must be broadened to recognize
+   `{bits:4, group_size}` MLX configs and read the `quantization` key, else they fall through to F32.
+2. **DWQ vs standard MLX is indistinguishable from config** (both say `{bits:4,group_size:64}`). The
+   difference is in tensor data. This is exactly why the Qwen3 `...-4bit-DWQ` produces garbage — the
+   repack treats it as affine MLX but the weights aren't. We must pin to **known-good standard converts**.
+
+Architecture support today (`src/gpu/architectures/`):
+- `Qwen2ForCausalLM`, `Qwen3ForCausalLM` → `generateQwen2Graph` (causal). **Qwen3-Embedding runs here**
+  with `embedding:true` → `SliceLastRow` (last-token EOS pool) + `L2Norm`.
+- `Qwen3_5ForConditionalGeneration` (hybrid Mamba) + a Qwen3.5 **vision ViT** that already uses
+  **bidirectional attention** (`causal:false`) — proof the non-causal path works end-to-end.
+- **No Gemma3 generator. No BERT/encoder text generator. Only last-token + L2Norm pooling (no mean pooling).**
+
+`is_causal` flag: defined in the Attention kernel (`src/gpu/kernels/registry.ts`, `Params.is_causal: u32`);
+`is_causal=0` → attend to all keys. Consumed by the Qwen3.5 vision tower today.
+
+---
+
+## The survey (live-verified configs + sizes + MTEB)
+
+| Model (HF repo) | model_type / arch | Params | hidden×layers | Pooling | MTEB | F32/BF16 size | q4 on HF (compatible?) |
+|---|---|---|---|---|---|---|---|
+| **google/embeddinggemma-300m** | `gemma3_text` / `Gemma3TextModel`, **`use_bidirectional_attention: true`** | 300M | 768 × 24 | **mean** + 2×Dense + L2Norm | **68.36 En / 61.15 Multi (v2)** | ~1.2 GB f32 | **`mlx-community/embeddinggemma-300m-4bit` = 173 MB, standard MLX-4bit, QAT base ✅** (gated, instant ack) |
+| **Qwen/Qwen3-Embedding-0.6B** | `qwen3` / `Qwen3ForCausalLM` | 0.6B | 1024 × 28 | last-token EOS + L2Norm | ~64 Multi (8B=70.58 #1) | **1192 MB bf16** | DWQ (broken ❌), `kerncore/...-MXL-4bit` 335 MB (MLX, **DWQ-contaminated card, risky**), `boboliu/...-W4A16-G128` 538 MB (compressed-tensors ❌ not read) |
+| **BAAI/bge-base-en-v1.5** | `bert` / `BertModel`, absolute pos | 110M | 768 × 12 | CLS (or mean) + L2Norm | 63.5 En | 438 MB f32 | only ONNX q4 ❌ → on-the-fly INT4 from F32 |
+| **BAAI/bge-small-en-v1.5** | `bert` / `BertModel` | ~33M | 384 × 12 | CLS + L2Norm | 61.9 En | 133 MB f32 | only ONNX q4 ❌ → on-the-fly INT4 |
+| **thenlper/gte-base** | `bert` / `BertModel` | 110M | 768 × 12 | mean + L2Norm | 62.4 En | 219 MB f32 | only ONNX/OpenVINO q8 ❌ → on-the-fly INT4 |
+| **thenlper/gte-small** | `bert` / `BertModel` | ~33M | 384 × 12 | mean + L2Norm | 62.0 En | 67 MB f32 | only ONNX q8 ❌ → on-the-fly INT4 |
+| **Snowflake/snowflake-arctic-embed-s** | `bert` / `BertModel`, absolute pos | ~33M | 384 × 12 | CLS + L2Norm | high mid (En) | 133 MB f32 | only ONNX q4 ❌ → on-the-fly INT4 |
+| **Snowflake/snowflake-arctic-embed-xs** | `bert` / `BertModel` | ~23M | 384 × **6** | CLS + L2Norm | mid (En) | 90 MB f32 | only ONNX q4 ❌ → on-the-fly INT4 |
+| **Snowflake/snowflake-arctic-embed-m-v2.0** | `gte` / `GteModel`, **RoPE**, vocab 250k | 305M | 768 × 12 | CLS + L2Norm, Matryoshka 256 | ~55–56 Multi | 1221 MB f32 | only ONNX q4 ❌ (too big anyway) |
+| **nomic-ai/nomic-embed-text-v2-moe** | `nomic_bert` **MoE** (megablocks, 8 experts) | 475M (305M active) | 768 × 12 | mean + L2Norm | strong Multi | f32 large | ❌ MoE megablocks — not supported, skip |
+| **jinaai/jina-embeddings-v3** | `xlm-roberta` + **5 LoRA adapters**, RoPE, Matryoshka | 572M | 1024 × 24 | mean + task-LoRA | strong Multi | 1.1 GB bf16 | ❌ LoRA-adapter routing not supported, too big, skip |
+| **ibm-granite/granite-embedding-278m-multilingual** | `xlm-roberta` / `XLMRobertaModel`, absolute pos | 278M | 768 × 12 | mean (or CLS) + L2Norm | mid-low Multi | 556 MB bf16 | only standard safetensors → on-the-fly INT4 (~150 MB) |
+| **minishlab/potion-base-8M** | `model2vec` / `StaticModel` (**static**, not a transformer) | 8M | 256 dim, PCA, no layers | token-lookup + mean | **51.08** (much lower) | 30 MB f32 | n/a — trivially fast, but a different (non-graph) code path |
+
+MTEB sources: EmbeddingGemma & Qwen3 figures from official model cards (EmbeddingGemma card:
+68.36 En v2 / 61.15 Multi v2; Qwen3 card: 8B #1 at 70.58, 0.6B is the small-tier sibling). BGE/GTE
+English averages cross-confirmed via the AILog MTEB compilation. Potion from its own card (51.08 MTEB).
+
+---
+
+## Why EmbeddingGemma-300M is the pick (#1)
+
+- **Quality:** best open model under 1B params — MTEB English v2 **68.36**, Multilingual v2 **61.15**.
+  That is materially above every BERT-small option (61–63) and competitive with Qwen3-0.6B while being
+  half the download.
+- **Size:** **~173 MB at q4** — comfortably inside the iPad <300 MB target.
+- **Format we can read:** `mlx-community/embeddinggemma-300m-4bit` is a **standard-affine MLX-4bit**
+  convert (`.scales`/`.biases` tensors, `mlx-lm`) of Google's **QAT-q4_0** base
+  (`google/embeddinggemma-300m-qat-q4_0-unquantized`). Because the base is quantization-aware-trained,
+  q4 is near-lossless. This is the same `repackMLX` path we already use — **not** the broken DWQ variant.
+- **Architecture fits our new capability:** it is a **bidirectional Gemma3 encoder**
+  (`use_bidirectional_attention: true`) — exactly the case the new `is_causal=false` flag unlocked.
+
+### Exact engine work for EmbeddingGemma (Tier-2 generator)
+
+Config (verified): `gemma3_text`, hidden **768**, **24** layers, **3** attention heads (head_dim 256,
+GQA `num_kv_heads:1`), intermediate **1152**, vocab **262144**, `max_position_embeddings` **2048**,
+`gelu_pytorch_tanh`, `rms_norm_eps 1e-6`, `query_pre_attn_scalar 256`. Pooling = **mean** then two
+**Dense** projection layers (the Matryoshka/bottleneck head) then **L2 Normalize**.
+
+New generator `architectures/gemma3_embedding.ts` (or extend a gemma3 generator) needs:
+1. **Bidirectional attention** — set `causal: false` on every Attention op (flag already exists). ✅ op-ready
+2. **Gemma3 attention specifics:** GQA (heads 3 / kv 1), `query_pre_attn_scalar=256` scaling, head_dim 256.
+3. **Mixed `layer_types`** — alternating `sliding_attention` (window 512) and `full_attention` (every 6th
+   layer). For a 2048-max encoder you can **treat all layers as full bidirectional** (window 512 only
+   bounds long contexts; embeddings are short) — simplest correct approximation. Sliding-window masking
+   can be a later optimization.
+4. **Dual RoPE bases:** `rope_local_base_freq=10000` (sliding/local layers) vs `rope_theta=1,000,000`
+   (full/global layers). RoPE op exists; needs per-layer theta selection.
+5. **RMSNorm** (have it), **GELU-tanh** (have it), **AddBias** (have it).
+6. **Mean-pooling tail** — **NEW**: we only have `SliceLastRow` (last-token) + `L2Norm`. Add a
+   `MeanPool` op (sum over valid tokens / count). Small kernel.
+7. **Two Dense head layers** (768→… →768) — plain MatMul(Int4)+AddBias, then **L2Norm** (have it).
+8. **Loader fix:** broaden MLX detection to accept `quantization: {bits:4, group_size:64}` with no
+   `mode` field (see caveat #1 above), and read the `dense.*` head tensors.
+
+Effort: **medium** (one generator + one MeanPool kernel + a small loader tweak). Gated (gemma license),
+but the gate is an instant click-through; we can also point at an ungated standard-MLX mirror or convert
+from the ungated `google/embeddinggemma-300m-qat-q4_0-unquantized` base ourselves.
+
+---
+
+## #2 Qwen3-Embedding-0.6B — runs today, but q4 is the problem
+
+Architecture already works on our engine (causal path, last-token EOS pool + L2Norm) — **zero generator
+work**. The blocker is purely the quant:
+- BF16 is **1192 MB** — too heavy to ship/quantize on-device on iPad.
+- `mlx-community/Qwen3-Embedding-0.6B-4bit-DWQ` → **garbage in our engine** (DWQ).
+- `kerncore/Qwen3-Embedding-0.6B-MXL-4bit` (335 MB) → standard MLX-4bit **in name**, but its model card
+  text is copy-pasted from the **DWQ** repo, so provenance is **unverified/risky**. Would need a runtime
+  correctness check (cosine vs reference) before trusting.
+- `boboliu/Qwen3-Embedding-0.6B-W4A16-G128` (538 MB) → **compressed-tensors / pack-quantized**, which our
+  loader does **not** read. Would need a new compressed-tensors adapter (and 538 MB is over budget).
+
+Net: viable only if we either (a) verify the `kerncore` MLX convert is truly affine (not DWQ), or
+(b) produce our own clean MLX-4bit convert of Qwen3-Embedding-0.6B from BF16. Lower quality-per-MB and
+larger than EmbeddingGemma, so it's the fallback, not the default.
+
+## #3 BERT-small encoders — smallest, but needs a different generator and lower quality
+
+`bge-small` / `gte-small` / `arctic-embed-s` are classic `BertModel` bidirectional encoders
+(hidden 384, 12 layers, absolute position embeddings, ~33M params). At q4 they're **~30–130 MB** — tiny.
+But:
+- They ship q4 **only as ONNX** (`onnx/model_q4.onnx`) which we don't read; we'd quantize **on-the-fly
+  from F32** (no QAT, so slightly more quality loss than EmbeddingGemma's QAT base).
+- They need a **new BERT encoder generator**: token + **learned absolute position** embeddings (+ optional
+  token-type embedding), bidirectional attention (`causal:false`), GELU, post-LN LayerNorm (have it), and
+  **mean or CLS pooling** (CLS = take row 0, easy; mean = new MeanPool op). **No RoPE** → arguably *simpler*
+  than the Gemma3 generator.
+- Quality tops out ~62 MTEB(En) — below EmbeddingGemma's 68.36.
+
+Good "absolute floor" option if we want the smallest possible default or want to validate the Tier-2
+encoder path on a simpler architecture first (`gte-small`/`bge-small` is the easiest first encoder to land).
+
+## Explicitly rejected
+
+- **nomic-embed-text-v2-moe** — megablocks MoE (8 experts, top-2). No MoE support; skip.
+- **jina-embeddings-v3** — XLM-Roberta + 5 task LoRA adapters + custom flash impl; routing not supported,
+  572M too big; skip.
+- **arctic-embed-m-v2.0** — 1221 MB f32, GteModel w/ RoPE + 250k vocab; over budget; skip (the *small*
+  arctic is the keeper).
+- **potion-base-8M** — static Model2Vec (51 MTEB). Not a transformer graph; would be a separate
+  token-lookup+pool code path. Worth a cheap "ultra-light" mode later, but quality is well below the rest.
+
+---
+
+## Suggested path
+
+1. **Land EmbeddingGemma-300M** as the default iPad embedding model:
+   - Add the Gemma3 bidirectional encoder generator + `MeanPool` kernel + Dense head + loader MLX-detect fix.
+   - Use `mlx-community/embeddinggemma-300m-4bit` (or self-convert the QAT base) — ~173 MB, near-lossless q4.
+2. (Optional, faster first win) **Land `gte-small`/`bge-small` BERT encoder generator** first to de-risk
+   the non-causal + mean-pool path on the simplest architecture (~30–67 MB), then reuse the MeanPool for Gemma3.
+3. Keep **Qwen3-Embedding-0.6B** as a desktop/high-RAM option; only ship its q4 on iPad after verifying a
+   clean (non-DWQ) MLX convert against a reference cosine-similarity check.

package/docs/research/sota-mobile-models-2026.md

@@ -0,0 +1,263 @@
+# SOTA Small / Mobile OSS LLMs for Gerbil — Research Report (June 2026)
+
+Research date: 2026-06-13. Engine baseline: Gerbil WebGPU INT4 engine, currently running
+Qwen3.5-0.8B (hybrid SSM + attention) at 207 tok/s desktop / 51 tok/s mobile Safari.
+
+Method: facts below are taken from **primary sources** wherever possible — live HuggingFace
+`config.json` / safetensors headers fetched directly (ground truth for architecture and size),
+backed by model cards and Perplexity web search for release context. Each claim is marked
+**[config]** (verified from the live HF config/safetensors), **[card/docs]** (official model card
+or vendor docs), or **[social/secondary]** (X/Twitter or third-party roundup, treat as softer).
+
+---
+
+## 0. TL;DR for the impatient
+
+- **Gemma 4 is real and shipping.** `google/gemma-4-E2B` and `google/gemma-4-12B` exist on HF
+  *today* with downloadable configs. Gemma 4 E2B is the on-device MatFormer variant (successor to
+  Gemma 3n E2B/E4B). Architecture = Gemma-family with **5:1 sliding-window:full attention** — this
+  is the one genuinely new kernel requirement. **[config]**
+- **The "13x fused LinearAttention" post is mostly a misread.** The real, documented artifact is
+  Alibaba's **FlashQLA** — a *fused linear-attention (Gated DeltaNet) kernel* for Qwen3.5/3.6,
+  claiming ~3x (not 13x), on **TileLang/CUDA GPUs, not WebGPU**. No verifiable WebGPU/Claude 13x
+  kernel exists in public sources. **Gerbil's `qwen3_5.ts` already implements this exact fusion**
+  (causal conv1d + delta-rule scan + gating + RMSNorm in one path). **[card/docs + repo]**
+- **Best next target: the Gemma 4 family**, because one new architecture generator (Gemma4, with
+  sliding-window attention + final-logit softcap + Gemma RMSNorm) unlocks E2B (mobile), 12B, and
+  the 26B-A4B MoE / 31B dense. Qwen3 dense (0.6/1.7/4B) is the cheapest add (it's a config tweak on
+  the existing Qwen2/3 path). SmolLM3 is the easiest "new vendor" win (plain Llama-like + NoPE).
+
+---
+
+## 1. Latest SOTA small / mobile OSS LLMs (mid-2026)
+
+### 1.1 Gemma — what actually exists (the user's "Gemma 4 E2B" question)
+
+The user's instinct was right; "Gemma 4 E2B" is a real model, not a hallucination.
+
+**Verified live from HF configs [config]:**
+
+| Repo | model_type | Notes |
+|---|---|---|
+| `google/gemma-4-E2B` | `gemma4` / `Gemma4ForConditionalGeneration` | On-device MatFormer variant. 35 layers, hidden 1536, MQA (1 KV head), head_dim 256, vocab 262144, sliding_window=512, tied embeddings. |
+| `google/gemma-4-12B` | `gemma4_unified` / `Gemma4UnifiedForConditionalGeneration` | 48 layers, hidden 3840, GQA 16/8, head_dim 256, sliding_window=1024, 262k context. |
+| `google/gemma-3n-E2B-it`, `gemma-3n-E4B-it` | (gated, login required) | The previous-gen MatFormer mobile models. Still exist but gated. |
+| `google/gemma-3-1b-it` etc. | (gated) | Gemma 3 dense family, 270M–27B. |
+
+**The lineage (correcting the secondary-source confusion):**
+- **Gemma 3** (Mar 2025): dense family 270M / 1B / 4B / 12B / 27B. **[card/docs]**
+- **Gemma 3n E2B / E4B**: the *mobile-optimized MatFormer* models — "E2B"/"E4B" = *effective*
+  ~2B / ~4B active params via MatFormer elastic nesting + Per-Layer Embeddings (PLE). Total stored
+  params are larger than the effective label; PLE means real on-device memory ≈ the effective size.
+  These were the Gemma-3-generation mobile models. **[card/docs, social/secondary]**
+- **Gemma 4** (~Apr–Jun 2026): new generation. Edge variants **E2B / E4B** (MatFormer again),
+  dense **12B** (released ~3 Jun 2026), plus larger **26B-A4B MoE** and **31B dense**. So "Gemma 4
+  E2B" = the current-gen on-device model, the direct successor to Gemma 3n E2B. **[config for E2B/12B; card/social for E4B/26B/31B]**
+
+**Sizes (E2B):** the bf16 single-file safetensors is **~10.2 GB on disk** [config: content-length
+10,246,621,918 B]. Official Q4 footprint per Google docs is **~2.9 GB** (Q4_0); Unsloth's QAT int4
+GGUF is **~2.62 GB**. **[card/docs]** E4B Q4 ≈ **4.2–4.5 GB**. The MatFormer "effective 2B" label
+does NOT shrink download — you ship the full static weights.
+
+### 1.2 The rest of the field (verified configs + 4-bit sizes)
+
+4-bit estimate column = group-quantized INT4 (g128) ≈ bf16_size × ~0.30 (weights at 0.5 B/param +
+scales; embeddings often kept higher precision). Treat as ±15%.
+
+| Model | Params | Arch family | model_type [config] | Context | Tied emb | bf16 size | **~INT4 download** | Official mobile/edge? |
+|---|---|---|---|---|---|---|---|---|
+| **Qwen3.5-0.8B** (current) | 0.8B | Hybrid GDN + attn | `qwen3_5` | 262k | yes | 1.62 GB | **~0.49 GB** | yes (edge line 0.8/2/4/9B) [social] |
+| **Qwen3-0.6B** | 0.6B | Llama-like dense | `qwen3` | 32k | yes | 1.40 GB | **~0.42 GB** | de facto; not branded |
+| **Qwen3-1.7B** | 1.7B | Llama-like dense | `qwen3` | 32k | yes | 3.78 GB | **~1.13 GB** | de facto |
+| **Qwen3-4B** | 4B | Llama-like dense | `qwen3` | 40k | yes | 7.49 GB | **~2.25 GB** | de facto |
+| **Phi-4-mini** | 3.8B | Phi3 (packed QKV, partial RoPE) | `phi3` | 128k | yes | 7.14 GB | **~2.14 GB** | local-friendly, no edge SKU |
+| **SmolLM3-3B** | 3B | Llama-like + NoPE | `smollm3` | 64k | yes | 5.72 GB | **~1.72 GB** | yes — built for edge/browser |
+| **Gemma 4 E2B** | ~2B eff | Gemma4 + sliding-window MatFormer | `gemma4` | 131k | yes | (10.2 GB raw) | **~2.6–2.9 GB** (official QAT) | **yes — official on-device** |
+| **Gemma 4 12B** | 12B | Gemma4 unified | `gemma4_unified` | 262k | yes | 22.3 GB | **~6.7 GB** | desktop-class |
+| **Llama 4 small** | 7–8B class | dense (Llama 4) | `llama4`/`llama` | long | — | ~15 GB | ~4.5 GB | no explicit edge SKU |
+
+Notes:
+- **No truly tiny official Llama mobile SKU.** Meta's smallest broadly-used open models remain
+  7–8B-class; phone-class Llama is community-quantized, not an official edge variant. **[social/secondary]**
+- **SmolLM3** is all-`full_attention` (no sliding window, no SSM) with **NoPE** (some layers skip
+  RoPE) and tied embeddings — architecturally the simplest "new vendor" target. **[config]**
+
+---
+
+## 2. Architecture compatibility with Gerbil's WebGPU INT4 engine
+
+Gerbil today (from `src/gpu/architectures/`):
+- `qwen2.ts` — standard transformer (Qwen2/Qwen3 = Llama-like, QKV bias, GQA, RoPE, RMSNorm, SwiGLU).
+- `qwen3_5.ts` — hybrid: full-attention layers (gated, QK-RMSNorm, partial RoPE, attn output gate)
+  **+ linear_attention layers implemented as fused Gated DeltaNet** (RMSNorm → fused QKV proj →
+  A/B/Z proj → causal conv1d+SiLU → gated-delta-net recurrence with L2-normed Q/K + exp decay +
+  delta rule → per-head RMSNorm → SiLU gate → out proj). This is a big asset (see §3).
+
+Existing reusable kernels: INT4 matmul (B^T), GQA attention, RMSNorm, RoPE (incl. partial), SwiGLU,
+causal conv1d, delta-rule scan, gating, tied embeddings.
+
+Per-candidate op delta (effort tiers per Gerbil's `add-model-family` skill: **Tier 1** = hours,
+reuse all ops; **Tier 2** = days, 1 novel op/kernel; **Tier 3** = weeks, new computation class).
+Note Gerbil generators are **not** Qwen-specific — they're config→IR generators over a
+family-agnostic IR + WGSL kernel registry. Adding a family = write a generator, register it in
+`src/gpu/architectures/index.ts`, and only write a kernel if an op is missing. Existing kernels:
+Embedding/MatMul(Int4), Add, Mul, RMSNorm, LayerNorm, RoPE, Attention, Softmax, SiLU, SwiGLU, GELU,
+ResidualRMSNorm, KVCacheAppend, **MambaSSM, CausalConv1d, SigmoidGate, ConvStateUpdate**, SliceLastRow.
+
+### Qwen3 dense (0.6B / 1.7B / 4B) — **Tier 1 (hours)**
+Same Llama-like decoder Gerbil's `qwen2.ts` already runs. `model_type: qwen3`, GQA, RoPE
+(theta 1e6), RMSNorm, SwiGLU, tied embeddings, QK-RMSNorm (already done for Qwen3.5 full-attn
+layers). **New kernels: none.** Mostly a config-mapping / loader change. The Qwen3.5 full-attention
+sublayer is essentially Qwen3, so the code already exists. **[config]**
+
+### SmolLM3-3B — **Tier 1 (hours)**
+Llama-like dense, all full_attention, GQA 16/4, RMSNorm, SwiGLU, tied embeddings. The one quirk:
+**NoPE** — certain layers omit RoPE (config gives `layer_types`/`no_rope_layers`). Need a per-layer
+"skip RoPE" flag in the attention builder. **New kernels: none; one loader/graph flag.** **[config]**
+
+### Phi-4-mini (3.8B) — **Tier 1 (hours, loader-heavy)**
+`model_type: phi3`. Two deviations from the Qwen path: (a) **packed `qkv_proj`** (single fused
+matrix, split into Q/K/V) and (b) **partial RoPE** + a packed `gate_up_proj` for the MLP. Gerbil
+already splits fused QKV (Qwen3.5 splits a fused Q+gate) and already does partial RoPE, so this is
+mostly tensor-slicing config. Sliding_window is set very large (262144) → effectively full
+attention at normal context. **New kernels: none; loader handles packed weights.** **[config]**
+
+### Gemma 4 E2B / 12B — **Tier 2 (days; sliding-window attn is the one novel op)**
+`model_type: gemma4` / `gemma4_unified`. Gemma-family specifics:
+- **Sliding-window attention (SWA)**, interleaved with full attention — E2B is **5:1**
+  (sliding:full, full every 5th layer), 12B is **5:1** with window 1024. **This is the one real new
+  kernel/masking path Gerbil lacks** — a banded/windowed attention mask (and a windowed KV cache to
+  get the memory benefit). At short prompts SWA == full attention, so a correctness-first v1 can
+  treat SWA as full attention and add the banded mask + windowed cache later for long-context wins.
+- **Gemma RMSNorm** uses `(1 + weight)` scaling and norms in more places (pre/post attn, pre/post
+  MLP). Small kernel variant of existing RMSNorm.
+- **Final logit softcapping** (`tanh`-based) and often **attn logit softcapping** — a cheap
+  elementwise op, but must be added or outputs are wrong.
+- **head_dim 256** with **query scaling** (`query_pre_attn_scalar`) — parameter, not new kernel.
+- **GeGLU** activation (Gemma uses gelu-based gating) vs SwiGLU (SiLU). Need a GeGLU variant of the
+  MLP (swap SiLU→GELU). Tiny.
+- **MatFormer / PLE (E2B)**: for a single fixed deployment Gerbil can just load the E2B slice as a
+  normal dense model — no need to implement elastic nesting. PLE means per-layer embedding tables;
+  loader must place them but it's not a new compute kernel.
+- Tied embeddings, vocab 262144 (large — embed/logit matmul is heavier).
+
+**New kernels for Gemma 4: (1) windowed/banded attention mask + windowed KV cache, (2) GeGLU MLP
+variant, (3) Gemma-style `(1+w)` RMSNorm, (4) logit softcap elementwise.** Items 2–4 are small;
+item 1 is the only real engineering. A correctness-first build can defer item 1.
+
+### Gemma 4 26B-A4B (MoE) / Llama 4 — **Tier 3 (weeks); out of scope for mobile**
+MoE needs a router + top-k expert gather + per-expert INT4 matmul (sparse dispatch). New kernel
+family, and 6–7 GB+ at int4 — desktop-only. Skip for the mobile mandate. **[social/secondary]**
+
+Exotic-feature flag summary:
+- **MatFormer/elastic params**: Gemma 3n/4 E-series — ignorable for a fixed deployment.
+- **Sliding-window attention**: Gemma 4 (and Phi3 in principle) — **needs new masking + windowed cache.**
+- **MoE**: Gemma 4 26B-A4B, Qwen3.5 large — new kernel family, desktop-only.
+- **NoPE**: SmolLM3 — one per-layer flag.
+- **Packed QKV / gate_up**: Phi-4-mini — loader-level, already partially handled.
+- **Logit softcap / Gemma (1+w) RMSNorm / GeGLU**: Gemma — small new variants.
+- **Tied embeddings**: every candidate — already supported.
+
+---
+
+## 3. The "fused LinearAttention 13x" claim — verdict
+
+**What's real (confirmed):**
+- Qwen3.5's `linear_attention` layers are **Gated DeltaNet (GDN)** — the Qwen3-Next mechanism:
+  short **causal conv1d**, **delta-rule recurrent/chunked scan**, **exponential gating**, **L2-norm
+  on Q/K** (replacing softmax), and **RMSNorm**. Interleaved 3:1 with gated full attention. Verified
+  in Gerbil's own config (`layer_types`) and NVIDIA Megatron-Bridge / Qwen model-card docs. **[config + card/docs]**
+- Alibaba shipped **FlashQLA** — "high-performance linear attention kernel library built on
+  TileLang, specifically optimized for GDN chunked-prefill, the linear attention mechanism used in
+  Qwen3.5 and Qwen3.6." It **fuses** the GDN ops and reports **up to ~3x** vs prior linear-attention
+  kernels — **on server GPUs (TileLang/CUDA), not WebGPU.** **[card/docs]**
+  (Source: alibabacloud.com/blog/flashqla-cp-bwd-friendly-fused-linear-attention-kernels-for-gdn)
+
+**What's NOT verifiable:**
+- A specific X/Twitter post claiming "Claude Opus 4.7 wrote a WebGPU kernel running Qwen3.5 13x
+  faster via a fused LinearAttention op" — **no traceable primary source.** Two independent
+  Perplexity sweeps (sonar-pro) over WebGPU + Qwen3.5 + GDN + "fused" + "13x" found no repo, no
+  benchmark, no attributable tweet. The closest real artifacts are the FlashQLA GPU kernels (~3x)
+  and the unrelated "Qwen3.6-35B Claude-4.7-Opus reasoning-distilled" model (no kernel content).
+  **Treat the 13x WebGPU claim as social-media rumor / paraphrase of FlashQLA, not a reproduced result.**
+
+**Does it apply to Gerbil?** It already *is* applied. Gerbil's `qwen3_5.ts` describes a single fused
+Gated DeltaNet path (the docstring literally lists conv1d+SiLU → gated-delta-net recurrence →
+per-head RMSNorm → SiLU gate → out proj). The "fused LinearAttention" technique = exactly what
+Gerbil's Qwen3.5 path does. The remaining upside is *autoresearch-style kernel tuning* of that
+fused op (chunked scan tiling, workgroup sizing, reducing dispatch/submit overhead — which the
+recent Safari/Metal commits already touch), not a new architectural fusion. There is no free 13x
+sitting on the table; FlashQLA's ~3x is the realistic ceiling for the GDN op specifically, and only
+if Gerbil's current scan is far from optimal.
+
+---
+
+## 4. Recommendation — ranked targets for Gerbil
+
+Weighting: SOTA quality × mobile download size × implementation effort against the existing engine.
+
+**#1 — Gemma 4 E2B (highest value).** Google's flagship *official on-device* model, multimodal-capable
+text core, ~2.6–2.9 GB at int4 (acceptable mobile download), strong quality. It's the model the
+user actually asked about and it's real and ungated (E2B/12B configs download without auth, unlike
+Gemma 3n). Effort: moderate — needs the Gemma4 generator (sliding-window mask + windowed KV cache,
+GeGLU, Gemma `(1+w)` RMSNorm, logit softcap). A correctness-first v1 can treat SWA as full attention
+and ship fast, then add the windowed cache for long-context/memory wins.
+
+**#2 — Qwen3 dense 0.6B / 1.7B / 4B (cheapest, broadest).** Near-zero new kernel work — it's the
+Qwen3.5 full-attention sublayer Gerbil already runs, minus the SSM layers. Gives a clean size ladder
+(0.42 / 1.13 / 2.25 GB int4) so users pick by device. Best effort-to-coverage ratio; should likely
+ship *before* Gemma to derisk the loader/config plumbing.
+
+**#3 — SmolLM3-3B (easy new-vendor win).** Plain Llama-like + NoPE flag + tied embeddings; ~1.72 GB
+int4; purpose-built for edge/browser. Almost free given the Qwen3 path; adds vendor diversity.
+
+**#4 — Phi-4-mini (3.8B).** Strong reasoning per param, 128k context, ~2.14 GB int4. Effort is
+loader-level (packed QKV / gate_up, partial RoPE — both already partly handled). Good "quality
+small" option but no edge branding and slightly more loader work than SmolLM3.
+
+**#5 — Gemma 4 12B / Llama 4 / MoE variants — desktop-only, defer.** 6.7 GB+ int4 and (for 26B-A4B)
+a whole MoE kernel family. Out of scope for the mobile mandate; revisit for a desktop tier.
+
+### Which single architecture-family generator unlocks the most high-value models?
+
+**The `Gemma4` generator.** One new arch family (sliding-window attention + windowed KV cache +
+GeGLU + Gemma RMSNorm + logit softcap) unlocks **E2B (mobile), E4B, 12B, and — with an added MoE
+path — 26B-A4B / 31B**: the entire current-gen Google open line, anchored by the single most
+requested mobile model. Sliding-window attention is also reusable for other modern models (it's a
+common long-context pattern), so the investment compounds.
+
+**Sequencing recommendation:** ship **Qwen3 dense** first (days, derisks plumbing) → then build the
+**Gemma4 generator** for E2B (the headline mobile model) → fold in **SmolLM3** as a cheap rider on
+the Qwen3 path. That ordering front-loads low-risk wins and lands the high-value Gemma 4 E2B with
+the loader already battle-tested.
+
+---
+
+## Sources
+
+Primary (verified live, 2026-06-13):
+- HF configs fetched directly: `Qwen/Qwen3.5-0.8B`, `Qwen/Qwen3-4B`, `Qwen/Qwen3-1.7B`,
+  `Qwen/Qwen3-0.6B`, `microsoft/Phi-4-mini-instruct`, `HuggingFaceTB/SmolLM3-3B`,
+  `google/gemma-4-E2B`, `google/gemma-4-12B` (config.json + safetensors index/content-length).
+- Gerbil repo: `src/gpu/architectures/qwen3_5.ts`, `src/gpu/architectures/qwen2.ts`,
+  `src/core/model-compat.ts`.
+
+Docs / cards:
+- Gemma 4 overview: https://ai.google.dev/gemma/docs/core
+- Gemma 4 QAT int4 GGUF sizes (Unsloth): https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF ,
+  https://unsloth.ai/docs/models/gemma-4/qat
+- FlashQLA fused GDN linear-attention kernels:
+  https://www.alibabacloud.com/blog/flashqla-cp-bwd-friendly-fused-linear-attention-kernels-for-gdn_603084
+- Qwen 3.5 hybrid GDN arch (NVIDIA): https://docs.nvidia.com/nemo/megatron-bridge/0.4.1/models/vlm/qwen35-vl.html
+- Gated DeltaNet explainer: https://sebastianraschka.com/llms-from-scratch/ch04/08_deltanet/
+- vLLM Qwen3-Next hybrid support: https://vllm.ai/blog/2025-09-11-qwen3-next
+- Qwen3.5 GDN analysis: https://gist.github.com/justinchuby/0213aa253664fb72e9adb0089816de15
+
+Secondary / context (treat softer):
+- BentoML 2026 OSS LLM roundup: https://www.bentoml.com/blog/navigating-the-world-of-open-source-large-language-models
+- Trilogy AI "Qwen 3.6 vs Opus 4.7 vs Gemma 4": https://trilogyai.substack.com/p/qwen-36-open-vs-opus-47-vs-gemma
+- Gemma 4 E2B/E4B edge writeup: https://www.mindstudio.ai/blog/gemma-4-e2b-e4b-edge-models-phone-local
+
+Unverified (flagged):
+- "Claude Opus 4.7 wrote a WebGPU kernel running Qwen3.5 13x faster via fused LinearAttention" — no
+  traceable primary source found; likely a paraphrase/conflation of FlashQLA (GPU, ~3x).