@tryhamster/gerbil 1.0.0-rc.9 → 1.0.1

package/docs/research/qwen35-multimodal.md

@@ -0,0 +1,293 @@
+# Qwen3.5 Multimodality — Native Engine Investigation
+
+**Date:** 2026-06-13
+**Question:** Is `Qwen/Qwen3.5-0.8B` (the model Gerbil runs) actually multimodal? Our native
+loader downloads only its text tensors and skips ~192 MB labeled "vision/MTP". Could one native
+model give us text + vision (+ audio) on our WebGPU engine, dissolving the two-engine problem on mobile?
+
+**Verdict up front:** **YES — Qwen3.5-0.8B is a single, natively multimodal (text + vision)
+checkpoint.** We are deliberately throwing away its vision tower. Re-enabling vision is a **mid-size
+build** (one real new kernel: the patch-embed conv; everything else in the ViT reuses ops we already
+have). **Audio is NOT in Qwen3.5 at all** — omni/audio lives in a separate, much larger family
+(`Qwen3-Omni-30B-A3B`). So one native model covers **text + image/video-in**, not audio.
+
+All facts below tagged `[config-verified]` were read directly from live HuggingFace files on
+2026-06-13 (config.json + safetensors header), not from web prose.
+
+---
+
+## PART A — What Gerbil currently skips
+
+### The filter rule
+
+`src/gpu/model-loader.ts`, `createKeyMapperForArch()` (lines 480–504). For the
+`Qwen3_5ForConditionalGeneration` architecture, the HF→canonical key mapper returns `null` for any
+tensor under three prefixes, which causes the selective downloader to skip those byte ranges entirely:
+
+```ts
+function createKeyMapperForArch(architectureName: string): HFKeyMapper {
+  if (architectureName === "Qwen3_5ForConditionalGeneration") {
+    return (hfKey: string): string | null => {
+      // Skip visual encoder, vision tower, and MTP weights
+      if (
+        hfKey.startsWith("model.visual.") ||
+        hfKey.startsWith("vision_tower.") ||
+        hfKey.startsWith("mtp.")
+      ) {
+        return null;
+      }
+      let key = hfKey;
+      if (key.startsWith("model.language_model.")) {
+        key = key.slice(21);
+      } else if (key.startsWith("language_model.model.")) {
+        key = key.slice(21);
+      } else if (key.startsWith("model.")) {
+        key = key.slice(6);
+      }
+      return key;
+    };
+  }
+  return createDefaultHFKeyMapper();
+}
+```
+
+### Where the skip is realized (the log line)
+
+`loadModel()`, lines 626–640. Any entry whose mapped key is `null` is excluded from the download and
+counted into `skippedBytes`:
+
+```ts
+const neededEntries = file.entries.filter((e) => keyMapper(e.name) !== null);
+const skippedBytes = file.entries
+  .filter((e) => keyMapper(e.name) === null)
+  .reduce((sum, e) => sum + e.dataLength, 0);
+// ...
+onProgress?.(10, 100,
+  `Selective download: ${dlMB} MB needed, skipping ${savedMB} MB (vision/MTP)`);
+```
+
+### What exactly is filtered out — measured from the live safetensors header `[config-verified]`
+
+Single shard `model.safetensors-00001-of-00001.safetensors`, total **1666 MB** (BF16/F32, the
+unquantized base). Broken down by prefix:
+
+| Component | Tensors | Size | Skipped? |
+|---|---|---|---|
+| `model.language_model.*` (text backbone) | 320 | **1435.1 MB** | kept |
+| `model.visual.*` (vision tower / ViT) | 153 | **191.9 MB** | **skipped** |
+| `mtp.*` (multi-token prediction head) | 15 | **39.0 MB** | **skipped** |
+
+So the "skipping 192 MB (vision/MTP)" line is **~192 MB vision + ~39 MB MTP**. (In the q4 path the
+ratios shrink but the same prefixes are dropped; "404 MB needed / 192 MB skipped" is the quantized
+profile.)
+
+### What "MTP" means here `[config-verified]`
+
+**MTP = Multi-Token Prediction**, not a vision module. The text config sets
+`"mtp_num_hidden_layers": 1` and `"mtp_use_dedicated_embeddings": false`. The 15 skipped `mtp.*`
+tensors are a small **one-layer auxiliary transformer head** used for speculative / parallel
+next-token decoding:
+
+```
+mtp.fc.weight, mtp.norm.weight, mtp.pre_fc_norm_embedding.weight, mtp.pre_fc_norm_hidden.weight,
+mtp.layers.0.{input_layernorm,post_attention_layernorm}.weight,
+mtp.layers.0.self_attn.{q_proj,k_proj,v_proj,o_proj,q_norm,k_norm}.weight,
+mtp.layers.0.mlp.{gate_proj,up_proj,down_proj}.weight
+```
+
+It is an inference-speedup head (predicts the +2 token to enable self-speculative decoding); it is
+**correctly skipped** for a basic autoregressive engine and has nothing to do with vision. Dropping
+it costs only decode throughput, not capability.
+
+### The vision tower we're skipping — it's a full ViT `[config-verified]`
+
+`config.json` ships a complete `vision_config` and the multimodal plumbing tokens, proving the base
+checkpoint is multimodal by construction:
+
+```json
+"image_token_id": 248056,
+"video_token_id": 248057,
+"vision_start_token_id": 248053, "vision_end_token_id": 248054,
+"vision_config": {
+  "depth": 12, "hidden_size": 768, "num_heads": 12, "intermediate_size": 3072,
+  "in_channels": 3, "patch_size": 16, "spatial_merge_size": 2, "temporal_patch_size": 2,
+  "num_position_embeddings": 2304, "out_hidden_size": 1024,
+  "hidden_act": "gelu_pytorch_tanh"
+}
+```
+
+Tensor inventory (from the header, leaf names generalized over the 12 blocks):
+
+```
+model.visual.patch_embed.proj.weight   BF16 [768, 3, 2, 16, 16]   ← Conv3d patch embed
+model.visual.patch_embed.proj.bias
+model.visual.pos_embed.weight          BF16 [2304, 768]           ← learned pos embeddings
+model.visual.blocks.N.norm1.{weight,bias}                         ← LayerNorm
+model.visual.blocks.N.attn.qkv.{weight,bias}   [2304, 768]        ← fused QKV (768→3*768)
+model.visual.blocks.N.attn.proj.{weight,bias}
+model.visual.blocks.N.norm2.{weight,bias}                         ← LayerNorm
+model.visual.blocks.N.mlp.linear_fc1.{weight,bias}               ← GELU MLP
+model.visual.blocks.N.mlp.linear_fc2.{weight,bias}
+model.visual.merger.norm.{weight,bias}
+model.visual.merger.linear_fc1.{weight,bias}   [3072, 3072]       ← projector (merger)
+model.visual.merger.linear_fc2.{weight,bias}   [1024, 3072]       → out_hidden_size 1024
+```
+
+The **patch embed is a 5-D Conv3d** kernel `[out=768, in=3, T=2, 16, 16]` (temporal_patch_size=2 ×
+16×16 spatial patches over RGB) — this is the one genuinely new primitive. Everything else (LayerNorm,
+fused QKV matmul, self-attention + softmax, GELU MLP, the 2-layer merger projecting 768→1024 to match
+the LM hidden size) is **standard transformer math the engine already runs for text.**
+
+### What it would take to NOT skip them
+
+For **vision**: change the key mapper to *map* (not null) `model.visual.*` instead of dropping it.
+That's a one-line change to start downloading the 192 MB. The real work is downstream: a Qwen3.5
+vision-tower graph generator + the patch-embed conv kernel + image preprocessing + splicing the
+merged image embeddings into the text sequence at `image_token_id` positions (see Part C).
+
+For **MTP**: leave it skipped unless/until we implement speculative decoding. It is an optimization,
+not a modality.
+
+---
+
+## PART B — Qwen3.5 actual capabilities (live HF, config-verified)
+
+### B1. Is Qwen3.5-0.8B itself multimodal? `[config-verified]`
+
+**Yes, text + vision, in a single checkpoint.** There is no separate VL repo — the base
+`Qwen/Qwen3.5-0.8B` *is* the vision-language model. Evidence (all from the live config/header):
+
+- Architecture `Qwen3_5ForConditionalGeneration` (a `*ForConditionalGeneration` multimodal wrapper),
+  `model_type: "qwen3_5"`.
+- A full `vision_config` (12-layer ViT) ships in the config.
+- `image_token_id` / `video_token_id` / `vision_start/end_token_id` are defined.
+- `rope_parameters.mrope_interleaved: true` with `mrope_section: [11, 11, 10]` — **M-RoPE
+  (multimodal rotary)**, the time/height/width positional scheme used specifically to place image and
+  video tokens. A text-only model would not carry mrope sections.
+- 153 `model.visual.*` weight tensors (191.9 MB) physically present in the safetensors.
+
+"MTP" is the multi-token-prediction head (Part A), **not** a vision module — they are two separate
+skipped groups.
+
+### B2. The Qwen3.5 family and its VL / Omni variants `[verified via HF API]`
+
+Live `huggingface.co/api/models?author=Qwen&search=Qwen3.5` returns the whole family. **Every dense
+member is a unified text+vision checkpoint; there are NO `Qwen3.5-VL` and NO `Qwen3.5-Omni` repos** —
+vision is built into the base models:
+
+| Model | Type | Notes |
+|---|---|---|
+| `Qwen/Qwen3.5-0.8B` (+ `-Base`) | dense | **smallest multimodal**; ~1.0 B params (0.8 B label) |
+| `Qwen/Qwen3.5-2B` (+ `-Base`) | dense | |
+| `Qwen/Qwen3.5-4B` (+ `-Base`) | dense | |
+| `Qwen/Qwen3.5-9B` (+ `-Base`) | dense | |
+| `Qwen/Qwen3.5-27B` | dense | also `-FP8`, `-GPTQ-Int4` |
+| `Qwen/Qwen3.5-35B-A3B` | MoE (3B active) | also `-Base/-FP8/-GPTQ-Int4` |
+| `Qwen/Qwen3.5-122B-A10B` | MoE | also `-FP8/-GPTQ-Int4` |
+| `Qwen/Qwen3.5-397B-A17B` | MoE | also `-FP8/-GPTQ-Int4` |
+
+**Pre-quantized 4-bit:** official `-GPTQ-Int4` exists for 27B and the MoE tiers only. For 0.8B there
+is no official Int4 — Gerbil quantizes on the fly (our loader supports GPTQ and MLX repack paths).
+
+**4-bit download size for the 0.8B (the one we run):** text backbone ~1435 MB BF16 → roughly
+**~400 MB at Int4** (matches the "404 MB needed" log). Adding the vision tower (191.9 MB BF16 →
+~50–60 MB Int4, or keep it F16 at ~96 MB) keeps a vision-capable native model well under ~500 MB
+total — viable on mobile.
+
+**Modalities, per family:**
+
+- **Qwen3.5 (all sizes incl. 0.8B):** text in/out, **image-in, video-in** (vision tokens + temporal
+  patch + M-RoPE). **No audio, no speech-out.** `[config-verified]`
+- **Audio / omni is a different family:** `Qwen/Qwen3-Omni-30B-A3B-{Instruct,Thinking,Captioner}` —
+  **30B MoE (~3B active)**, far too big for our mobile target, and it's Qwen3-Omni, not Qwen3.5.
+  Smallest omni anywhere is `Qwen/Qwen2.5-Omni-3B` (~5.5 B params) `[verified via HF API]`.
+
+So: **vision is free inside the model we already run; audio is not available in any small Qwen3.5/omni
+checkpoint we could realistically run on a phone.**
+
+### B3. Architecture of the smallest multimodal variant (Qwen3.5-0.8B vision tower) `[config-verified]`
+
+- **Vision encoder:** a 12-layer **ViT** (`depth 12`, `hidden 768`, `12 heads`, `mlp 3072`,
+  `gelu_pytorch_tanh`), pre-norm blocks (norm1→attn→norm2→MLP), fused QKV.
+- **Patch embed:** **Conv3d** `[768, 3, 2, 16, 16]` — `temporal_patch_size 2` × `16×16` patches over 3
+  RGB channels. (For single images the temporal dim is duplicated to 2; for video it spans frame
+  pairs.)
+- **Positional:** learned `pos_embed [2304, 768]` plus M-RoPE on the LM side.
+- **Merger / projector:** `spatial_merge_size 2` → concatenates a 2×2 patch block (768×4 = 3072),
+  LayerNorm, then `linear_fc1 [3072,3072]` → GELU → `linear_fc2 [1024,3072]` to produce one
+  `out_hidden_size = 1024` token per merged block, matching the LM `hidden_size 1024`.
+- **No audio encoder/decoder** in Qwen3.5.
+
+**Ops the engine needs vs. what it already has.** Engine implemented ops (from `src/gpu/ir.ts` +
+`src/gpu/kernels/registry.ts`): Embedding(+Int4), MatMul(+Int4), Add, Mul, RMSNorm, **LayerNorm**,
+RoPE, **Attention**, **Softmax**, SiLU, SwiGLU, **GELU**, plus the Mamba-SSM stack (MambaSSM,
+CausalConv1d, etc.). Stubbed (declared in IR, **no WGSL kernel**): `Conv2d`, `AvgPool2d`,
+`CrossAttention`, and the MoE ops.
+
+Mapping the ViT onto that:
+
+| ViT component | Engine status |
+|---|---|
+| LayerNorm (norm1/norm2/merger.norm) | ✅ have `LayerNorm` |
+| Fused QKV / proj / MLP matmuls | ✅ have `MatMul` / `MatMulInt4` |
+| Self-attention + softmax (bidirectional, non-causal) | ✅ have `Attention` + `Softmax` — needs a **non-causal mask flag** (text attention is causal) |
+| GELU (`gelu_pytorch_tanh`) | ✅ have `GELU` (verify tanh approximation variant) |
+| Merger projector (concat + 2×MLP) | ✅ matmul + a Concat/reshape (Concat is currently stubbed but trivial) |
+| **Patch embed Conv3d `[768,3,2,16,16]`** | ❌ **new** — only genuinely missing primitive |
+
+The patch-embed conv is a strided non-overlapping conv = an **im2col/unfold + MatMul**: reshape each
+non-overlapping 2×16×16×3 patch into a 1536-vector and multiply by the reshaped
+`[768, 1536]` weight. **No general Conv2d/Conv3d kernel needed** — a small unfold (gather/reshape)
+shader feeding the existing MatMul covers it. `CrossAttention` and `AvgPool2d` are **not** required by
+this ViT (it uses standard self-attention and a learned merger, not pooling/cross-attn).
+
+---
+
+## PART C — Verdict & build plan
+
+### Can one native model deliver text + vision (+ audio) on our WebGPU engine?
+
+- **Text + vision (image/video-in): YES, realistically.** It's the *same checkpoint we already run* —
+  we are choosing to discard its vision tower. The ViT reuses our existing matmul / attention /
+  LayerNorm / GELU / softmax kernels. Only one new primitive (the patch-embed conv, implementable as
+  unfold+matmul) plus host-side image preprocessing and token splicing are required.
+- **Audio: NO.** Qwen3.5 has no audio path at all. The only omni checkpoints are `Qwen3-Omni-30B-A3B`
+  (30B MoE) and `Qwen2.5-Omni-3B` (~5.5B) — both too large for mobile, both a different architecture
+  requiring a Whisper-style audio encoder, a codec/talker, and `code2wav` vocoder (RVQ codec decode)
+  we don't remotely have. Audio stays out of scope for the native engine.
+
+**So it dissolves the two-engine problem *partially*: vision yes, audio no.** If the second
+engine (transformers.js) on mobile was there for *vision*, a vision-enabled Gerbil can replace it and
+we ship one native model for text + image/video. If audio was also required, that still needs a
+separate path — but no small Qwen model gives us audio either, so that's a model-availability wall,
+not a Gerbil limitation.
+
+### Build tier and specific work to enable native Qwen3.5 vision
+
+**Tier: mid-size (1 new kernel, 1 new graph generator, host glue). Not a rewrite.**
+
+1. **Loader (trivial):** stop nulling `model.visual.*` in `createKeyMapperForArch` (keep skipping
+   `mtp.*`). Map vision keys to canonical names; download grows by ~192 MB BF16 (~50–96 MB if we
+   quant/keep-F16 the tower).
+2. **Vision graph generator (new):** add a `generateQwen3_5VisionGraph()` reading `vision_config`,
+   emitting the 12 ViT blocks + merger, reusing existing LayerNorm/MatMul/Attention/Softmax/GELU
+   nodes. Add a **non-causal flag** to the Attention op (vision attention is bidirectional).
+3. **Patch-embed kernel (new, small):** unfold/im2col shader that reshapes 2×16×16×3 patches →
+   `[num_patches, 1536]`, then MatMul by `[768,1536]` weight + bias. (Reuses MatMul; the only new WGSL
+   is the gather/unfold.) `Concat`/`Reshape` for the spatial-merge step need real kernels (currently
+   stubbed) but are simple memory shuffles.
+4. **Image preprocessing (host):** resize/normalize to the patch grid, build the temporal pair, emit
+   patch tensors — JS/Canvas/`ImageBitmap`, no GPU.
+5. **Sequence splicing (executor):** replace `image_token_id` (248056) placeholder positions in the
+   text embedding stream with merged vision tokens, and apply **M-RoPE** (`mrope_section [11,11,10]`,
+   `mrope_interleaved`) for the multimodal position ids. This M-RoPE variant is new vs. our current
+   text RoPE and is the second-most-involved piece after the patch conv.
+
+**New kernels needed:** patch-embed unfold (small), real `Concat`/`Reshape` (trivial), non-causal
+attention flag, M-RoPE position handling. **Not needed:** general `Conv2d`/`Conv3d`, `AvgPool2d`,
+`CrossAttention`, MoE ops.
+
+**Bottom line:** Qwen3.5-0.8B is one native multimodal model that, with a mid-size vision build
+(~1 real new kernel + a vision graph + M-RoPE + host image prep), gives Gerbil **text + image/video**
+on WebGPU and lets us drop a second vision engine on mobile. **Audio is genuinely unavailable** in any
+small Qwen3.5/omni checkpoint, so it remains a separate problem regardless of engine work.

package/docs/research/qwen36-gemma4-targets.md

@@ -0,0 +1,337 @@
+# Qwen3.6 & Gemma 4 — Smallest On-Device Targets (Verification Follow-up)
+
+Research date: **2026-06-13**. Follow-up to `docs/research/sota-mobile-models-2026.md`.
+Goal: pick the two smallest on-device models to support next. SmolLM dropped per instruction.
+
+Fact tags: **[config-verified]** = fetched the actual `config.json` / HF API live today;
+**[card/docs]** = official model card / vendor docs; **[social/secondary]** = third-party.
+
+---
+
+## 0. TL;DR (the two picks)
+
+| Pick | Exact repo ID | Arch | 4-bit download | Dense or hybrid | Gerbil tier |
+|---|---|---|---|---|---|
+| **Smallest "Qwen"** | **`Qwen/Qwen3-0.6B`** (+ `Qwen/Qwen3-0.6B-MLX-4bit`) | dense full-attention transformer | **0.32 GB** (MLX 4-bit) | **DENSE** (not hybrid) | **Tier 1** |
+| **Smallest "Gemma 4"** | **`google/gemma-4-E2B`** (+ `google/gemma-4-E2B-it-qat-q4_0-gguf`) | dense transformer w/ sliding-window attn | **3.35 GB** (official QAT q4_0, text-only) | dense (interleaved SWA) | **Tier 2** |
+
+**Headline correction:** There is **no small Qwen3.6**. Qwen3.6 exists but the smallest official
+variant is **27B dense** (also a 35B-A3B MoE). No 0.6B / 1.7B / 4B Qwen3.6 exists or is announced.
+The smallest deployable Qwen on-device model remains **Qwen3-0.6B** (the prior Qwen3 dense generation).
+
+**Gemma 4 correction:** No `gemma-4-270m`, no `gemma-4-1B`. The 270M only exists for **Gemma 3**.
+The smallest Gemma 4 is **E2B** (effective ~2B via Per-Layer Embeddings; *not* MatFormer — see
+§2.5). Confirmed against the full `author=google&search=gemma-4` listing.
+
+---
+
+## 1. Target 1 — "smallest Qwen3.6" → does not exist; use Qwen3-0.6B
+
+### 1a. Qwen3.6 existence check [config-verified]
+
+Full official Qwen org listing for `Qwen3.6` (HF API `author=Qwen&search=Qwen3.6`, fetched today):
+
+```
+Qwen/Qwen3.6-35B-A3B        (MoE, 35B total / 3B active)
+Qwen/Qwen3.6-27B            (dense)
+Qwen/Qwen3.6-35B-A3B-FP8
+Qwen/Qwen3.6-27B-FP8
+```
+
+That is the **entire** official family. Hub-wide search for `Qwen3.6 0.6B / 1.7B / 4B / mini / nano / edge`
+returns only community quants/finetunes **of the 27B and 35B-A3B** — no small base model anywhere.
+Probes for `Qwen/Qwen3.6-0.6B`, `-0.5B`, `-1.7B`, `-0.8B`, `-0.6B-Instruct` all return HF API **404-equivalent
+401** (nonexistent repo, not gated). **[config-verified]**
+
+Independent confirmation: Perplexity (`sonar-pro`, today) — *"No official ~0.6B/1.7B/≤5B-class Qwen3.6
+model has been released; the documented Qwen3.6 family is 27B dense and 35B-A3B MoE only."* Sources:
+Unsloth Qwen3.6 docs (https://unsloth.ai/docs/models/qwen3.6), Qwen blog
+(https://qwen.ai/blog?id=qwen3.6-35b-a3b), GitHub QwenLM/Qwen3.6. **[social/secondary]**
+
+> The user's belief that "Qwen released a 3.6 generation with a ~0.6B model" is **incorrect** as of
+> 2026-06-13. Qwen3.6 is a large-model release (27B/35B-A3B). For a smallest on-device Qwen, the
+> correct target is **Qwen3-0.6B** from the Qwen3 dense generation.
+
+### 1b. `Qwen/Qwen3-0.6B` — verified config [config-verified]
+
+Fetched `https://huggingface.co/Qwen/Qwen3-0.6B/resolve/main/config.json`:
+
+| Field | Value |
+|---|---|
+| `architectures[0]` | **`Qwen3ForCausalLM`** |
+| `model_type` | **`qwen3`** |
+| `hidden_size` | 1024 |
+| `num_hidden_layers` | 28 |
+| `num_attention_heads` | 16 |
+| `num_key_value_heads` | 8 (GQA 2:1) |
+| `head_dim` | **128** (note: ≠ hidden/heads = 64; set explicitly) |
+| `intermediate_size` | 3072 |
+| `vocab_size` | 151936 |
+| `tie_word_embeddings` | **true** |
+| `rope_theta` | 1000000 |
+| `max_position_embeddings` | 40960 |
+| `rms_norm_eps` | 1e-6 |
+| `attention_bias` | **false** |
+| `sliding_window` / `use_sliding_window` | null / false |
+| `hidden_act` | silu (→ SwiGLU MLP) |
+
+**Dense vs hybrid: 100% DENSE full-attention transformer.** No SSM, no linear/Gated-DeltaNet
+layers, no sliding window. It uses the **simple dense path**, NOT Gerbil's Qwen3.5 hybrid path.
+**[config-verified]**
+
+**Non-standard features:**
+- **No QKV bias** (`attention_bias: false`) — opposite of Qwen2, which has QKV bias. **[config-verified]**
+- **QK-norm: YES** — Qwen3 applies per-head RMSNorm to Q and K (`self_attn.q_norm.weight`,
+  `self_attn.k_norm.weight`). This is the defining Qwen3 architectural change vs Qwen2. Not visible
+  as a scalar in config.json but present in the weights and modeling code; confirmed by Gerbil's own
+  loader already listing `layers.{i}.self_attn.q_norm.weight` / `k_norm.weight`. **[card/docs]**
+- Tied embeddings (saves shipping a separate lm_head). **[config-verified]**
+
+**Sizes:**
+- Raw bf16 `model.safetensors` = **1,503,300,328 B ≈ 1.50 GB** (HTTP content-length). **[config-verified]**
+- **Official `Qwen/Qwen3-0.6B-MLX-4bit` = 0.317 GB total** (4-bit incl. quantized tied embeddings). **[config-verified]**
+- Official `Qwen/Qwen3-0.6B-GGUF` exists (Q4 ≈ 0.4–0.5 GB; Gerbil quantizes on-the-fly from bf16
+  anyway, so the ~0.42 GB g128 INT4 estimate from the prior report stands). **[config-verified for repo existence]**
+
+Official quant repos confirmed live: `Qwen/Qwen3-0.6B-MLX-4bit`, `Qwen/Qwen3-0.6B-GGUF`,
+`unsloth/Qwen3-0.6B-GGUF`, `unsloth/Qwen3-0.6B-bnb-4bit`. **[config-verified]**
+
+---
+
+## 2. Target 2 — smallest Gemma 4 = `google/gemma-4-E2B`
+
+### 2a. Smallest-variant check [config-verified]
+
+Full `author=google&search=gemma-4` listing (fetched today) contains, by size:
+**E2B, E4B, 12B, 26B-A4B (MoE), 31B** — plus `-it`, `-qat-q4_0-gguf`, `-qat-mobile-transformers`,
+`-qat-w4a16-ct` variants of each. **No `gemma-4-270m`, no `gemma-4-1B`, no `gemma-4-E1B`.** The only
+270M / 1B Google models are **Gemma 3** (`google/gemma-3-270m-*`, `gemma-3-1b-*`). So **E2B is the
+smallest Gemma 4.** Perplexity concurs: *"Gemma 4-2B (~2B dense) is the smallest public Gemma 4; no
+~270M or ~1B Gemma 4 as of June 2026."* **[config-verified + social/secondary]**
+
+`google/gemma-4-E2B` HF API returns HTTP 200 (ungated, downloadable without auth). **[config-verified]**
+
+### 2b. `google/gemma-4-E2B` — verified config (`text_config`) [config-verified]
+
+Fetched `https://huggingface.co/google/gemma-4-E2B/resolve/main/config.json`. It is a multimodal
+container (`Gemma4ForConditionalGeneration`, `model_type: gemma4`) with `text_config` + `vision_config`
++ `audio_config`. For a text-only Gerbil deployment, only `text_config` matters:
+
+| Field (`text_config`) | Value |
+|---|---|
+| `architectures[0]` (top) | `Gemma4ForConditionalGeneration` |
+| `model_type` | `gemma4_text` (container `gemma4`) |
+| `hidden_size` | 1536 |
+| `num_hidden_layers` | 35 |
+| `num_attention_heads` | 8 |
+| `num_key_value_heads` | **1 (MQA)** |
+| `head_dim` | **256** (≠ hidden/heads) |
+| `intermediate_size` | 6144 |
+| `vocab_size` | **262144** (large embed/logit matmul) |
+| `tie_word_embeddings` | **true** |
+| `rms_norm_eps` | 1e-6 |
+| `attention_bias` | false |
+| `hidden_activation` | **`gelu_pytorch_tanh`** (→ **GeGLU** MLP, not SwiGLU) |
+| `sliding_window` | **512** |
+| `final_logit_softcapping` | **30.0** |
+| `max_position_embeddings` | 131072 |
+
+**Sliding-window attention layout (`layer_types`)** [config-verified]:
+35 layers, pattern = 4× sliding then 1× full, repeating → **full attention at layers
+4,9,14,19,24,29,34 (7 full / 28 sliding) = 4:1 sliding:full ratio.** (Prior report said 5:1 — the
+live config shows full every 5th layer, i.e. a **4:1** sliding-to-full ratio.) `sliding_window: 512`.
+
+**Dual RoPE** [config-verified]:
+- full_attention layers: `rope_theta: 1e6`, `rope_type: proportional`, **`partial_rotary_factor: 0.25`** (partial RoPE).
+- sliding_attention layers: `rope_theta: 1e4`, `rope_type: default` (full RoPE).
+
+**Other Gemma-4 specifics** [config-verified]:
+- `query_pre_attn_scalar`: not present as a separate field in this config — query scaling is folded
+  via `head_dim`/`global_head_dim` (`global_head_dim: 512`). Gemma uses head_dim-based scaling; treat
+  as a parameter, not a kernel.
+- `hidden_size_per_layer_input: 256` + `vocab_size_per_layer_input: 262144` → **Per-Layer Embeddings
+  (PLE)**: extra per-layer embedding tables the loader must place. Not a new compute kernel.
+- `num_kv_shared_layers: 20` → **cross-layer KV sharing** (20 layers reuse KV from a paired layer).
+  New loader/graph wiring; reduces KV cache but adds plumbing.
+- `use_double_wide_mlp: true`, `attention_k_eq_v: false`, `enable_moe_block: false` (E2B is dense, not MoE).
+- **Effective "~2B" via PLE (NOT MatFormer):** the effective-vs-total split comes from Per-Layer
+  Embeddings, not elastic nesting (see §2.5). Load E2B as a normal fixed dense checkpoint — no
+  elastic-nesting code needed. **[card/docs]**
+- Gemma RMSNorm uses `(1 + weight)` scaling — Gerbil's loader already bakes `+1` into Gemma-style
+  norm weights (see `model-loader.ts` comment), and norms appear pre/post attn + pre/post MLP.
+
+**Sizes:**
+- Raw bf16 single-file `model.safetensors` = **10,246,621,918 B ≈ 10.25 GB** (full multimodal weights). **[config-verified]**
+- **Official QAT INT4 (`google/gemma-4-E2B-it-qat-q4_0-gguf`):** text weights
+  `gemma-4-E2B_q4_0-it.gguf` = **3.349 GB**; multimodal projector `gemma-4-E2B-it-mmproj.gguf` =
+  0.987 GB (skip for text-only). So **text-only 4-bit download ≈ 3.35 GB.** **[config-verified]**
+- (Prior report's "~2.6–2.9 GB" figure was the Unsloth/Google headline; the official Google QAT
+  q4_0 text GGUF measures **3.35 GB** live — use this.)
+
+---
+
+## 2.5 E2B vs E4B — separate checkpoints, NOT MatFormer slices [config-verified]
+
+The addendum asked whether E2B is an elastic MatFormer sub-network nested inside E4B, or a
+genuinely separate checkpoint. **Answer: genuinely separate checkpoints.** Both `config.json`
+files were fetched live today and the dims differ structurally — E2B is not a slice of E4B.
+
+**What "E" actually means.** The official E2B model card states verbatim: *"The 'E' in E2B and
+E4B stands for 'effective' parameters. The smaller models incorporate **Per-Layer Embeddings
+(PLE)** to maximize parameter efficiency… These embedding tables are large but are only used for
+quick lookups, which is why the effective parameter count is much smaller than the total."* The
+card lists Gemma 4 as **four distinct sizes — E2B, E4B, 26B-A4B, 31B** — i.e. separate models,
+not nested elastic variants. **The card never mentions "MatFormer" or "Mix-n-Match" (grep count
+= 0).** That is a Gemma-3n-generation mechanism; Gemma 4's edge effective/total split is driven
+by **PLE**, not MatFormer nesting. So the addendum's hypothesis ("E4B is the full model with E2B
+nested inside it as an elastic sub-network") is **incorrect for Gemma 4** — they are two distinct
+checkpoints, each with its own `model.safetensors`. **[card/docs + config-verified]**
+
+> Correction to earlier notes in this repo (and `sota-mobile-models-2026.md`): the "MatFormer"
+> label applied to Gemma 4 E2B is wrong. Gemma 4 uses **PLE** for the effective-vs-total split.
+> Treat E2B and E4B as ordinary fixed dense checkpoints — no elastic-nesting code is needed for
+> either, which was the practical conclusion anyway.
+
+### Per-checkpoint facts (both `text_config`, fetched live 2026-06-13)
+
+| | **E2B** | **E4B** |
+|---|---|---|
+| Exact HF repo ID | **`google/gemma-4-E2B`** (+ `-it`) | **`google/gemma-4-E4B`** (+ `-it`) |
+| Effective params (card) | **2.3B effective** | **4.5B effective** |
+| Total params incl. embeddings (card) | **5.1B** | **8B** |
+| Raw bf16 `model.safetensors` (content-length) | **10,246,621,918 B ≈ 10.25 GB** | **15,992,595,884 B ≈ 15.99 GB** |
+| Implied 4-bit download (full multimodal, ≈ raw/3.6) | **≈ 2.85 GB** | **≈ 4.45 GB** |
+| Official QAT q4_0 GGUF (text-only) | **3.35 GB** (`gemma-4-E2B_q4_0-it.gguf`, measured) | **≈ 5.0–5.2 GB** (`-E4B-it-qat-q4_0-gguf`, gated; scales with raw) |
+| `hidden_size` | **1536** | **2560** |
+| `num_hidden_layers` | **35** | **42** |
+| `num_attention_heads` | 8 | 8 |
+| `num_key_value_heads` | **1 (MQA)** | **2 (GQA)** |
+| `head_dim` / `global_head_dim` | 256 / 512 | 256 / 512 |
+| `intermediate_size` (MLP) | **6144** | **10240** |
+| `use_double_wide_mlp` | **true** | **false** |
+| `num_kv_shared_layers` | **20** | **18** |
+| `sliding_window` | 512 | 512 |
+| SWA layout (`layer_types`) | 4:1 sliding:full (full @ 4,9,14,19,24,29,34) | 5:1 sliding:full (full @ 5,11,17,23,29,35,41) |
+| `vocab_size` | 262144 | 262144 |
+| `final_logit_softcapping` | 30.0 | 30.0 |
+| `hidden_activation` | `gelu_pytorch_tanh` (GeGLU) | `gelu_pytorch_tanh` (GeGLU) |
+| `tie_word_embeddings` | true | true |
+| `max_position_embeddings` | 131072 | 131072 |
+| `model_type` (container / text) | `gemma4` / `gemma4_text` | `gemma4` / `gemma4_text` |
+| `architectures[0]` | `Gemma4ForConditionalGeneration` | `Gemma4ForConditionalGeneration` |
+| HF gating | ungated (HTTP 200, no auth) | ungated (HTTP 200, no auth) |
+
+The structural differences (hidden 1536 vs 2560, 35 vs 42 layers, 1 vs 2 KV heads, MLP 6144 vs
+10240, double-wide MLP only on E2B, 20 vs 18 KV-shared layers) prove these are **independently
+trained checkpoints**, not one model sliced two ways. **[config-verified]**
+
+### Smaller download
+
+**E2B is the smaller download** — by every measure: raw bf16 **10.25 GB vs 15.99 GB**, and
+official text-only 4-bit **≈ 3.35 GB vs ≈ 5.0+ GB**. E2B is ~36% smaller on disk.
+
+### Does the choice change Gerbil's implementation work? **No — same generator, same kernels.**
+
+Both share `model_type: gemma4` / `Gemma4ForConditionalGeneration` and the **identical op set**:
+sliding-window attention, GeGLU MLP, Gemma `(1+w)` RMSNorm, final-logit softcap, dual RoPE
+(partial on full-attn layers, full on sliding layers), PLE tables, cross-layer KV sharing, tied
+embeddings. Gerbil's generators read **every dimension from `config.json` at runtime** (verified
+in `qwen3_5.ts`: `hidden_size`, `num_hidden_layers`, `num_key_value_heads`, `intermediate_size`,
+`head_dim`, `layer_types`, … are all pulled from the config object). Therefore:
+
+- **Same `gemma4.ts` generator** handles both — no E4B-specific code.
+- **Same kernels** — the only deltas are config values (1536→2560, 35→42, MQA→GQA, MLP width,
+  `use_double_wide_mlp`, KV-shared count, SWA stride). All of these are already
+  config-parameterized inputs to the graph, not branches.
+- The only practical difference is **which weights you load and how much memory/VRAM they need.**
+
+One nuance to verify when implementing: `use_double_wide_mlp` is `true` for E2B and `false` for
+E4B. This is a config flag the generator must honor (likely a 2× factor on the gate/up
+projection), but it is a parameterized branch inside the single generator — not a separate kernel.
+
+### Verdict: is E4B worth the extra size on-device?
+
+**For Gerbil's on-device / mobile target: pick E2B.** E4B is ~57% larger (15.99 vs 10.25 GB raw;
+~5.0 vs 3.35 GB at 4-bit). On an iPad/phone-class budget the extra ~1.7 GB of 4-bit weights plus
+the larger KV/activation footprint (hidden 2560, 42 layers, 2 KV heads) materially raises peak
+memory and lowers tokens/sec, for a quality bump (2.3B→4.5B effective) that is real but not
+transformative at the edge. **E4B is "worth it" only on a laptop/desktop-class device** with
+headroom to spare and a quality-over-latency preference. Since implementation cost is identical
+(same generator, same kernels), Gerbil can support both from one code path and simply let the
+user pick the weight set — but the **default on-device pick should be E2B**.
+
+---
+
+## 3. Gerbil implementation tiers (per `docs/adding-a-model-family.md`)
+
+### Target 1 — `Qwen/Qwen3-0.6B` → **Tier 1 (hours)**
+
+`Qwen3ForCausalLM` is **already registered** in `src/gpu/architectures/index.ts` →
+`generateQwen2Graph`. But `qwen2.ts` models the **Qwen2** layer (QKV bias, **no QK-norm**), while
+Qwen3 is the inverse: **no QKV bias + adds per-head QK-RMSNorm**. So a vanilla load would produce
+wrong outputs (missing q_norm/k_norm).
+
+**What must be built (all reuse, zero new kernels):**
+1. Add **QK-norm** (per-head RMSNorm on Q and K) to the dense attention path. This is a
+   **copy-paste from `qwen3_5.ts`** (lines ~514–555 already wire `CANONICAL_KEYS.qNorm(i)` /
+   `kNorm(i)` + two `RMSNorm` nodes), and the loader already handles `self_attn.q_norm.weight` /
+   `k_norm.weight` (with the `+1` bake). Cleanest fix: branch on `model_type === "qwen3"` to emit the
+   QK-norm nodes and skip QKV bias.
+2. Confirm `attention_bias: false` path (skip the Qwen2 QKV-bias tensors).
+3. `head_dim` is read from config (128) — already handled. GQA 16/8, tied embeddings, RoPE θ=1e6 —
+   all already supported.
+4. Validate vs HF reference (Step 7 of the guide).
+
+**New kernels: none.** All ops (RMSNorm, QK-norm RMSNorm, RoPE, GQA Attention, SwiGLU, tied
+embeddings, SliceLastRow) already exist. Effort: hours.
+
+### Target 2 — `google/gemma-4-E2B` → **Tier 2 (days; 1 real new kernel)**
+
+New `gemma4.ts` generator + register `Gemma4ForConditionalGeneration` (read `config.text_config`).
+
+**What must be built:**
+- **Tier-2 new kernel (the one real engineering item): sliding-window / banded attention** +
+  (for the memory win) a **windowed KV cache** — `sliding_window: 512`, interleaved 4:1 with full
+  attention. *Correctness-first v1 can treat SWA as full attention* (identical results at prompts
+  ≤512 tokens) and ship without the new kernel; add the banded mask + windowed cache later for
+  long-context memory wins.
+- **Small new variants (each tiny, not a full kernel class):**
+  - **GeGLU MLP** (swap SiLU→GELU `gelu_pytorch_tanh` in the gated MLP).
+  - **Final logit softcapping** (`tanh`-based, `final_logit_softcapping: 30.0`) — elementwise op on logits.
+  - **Gemma `(1+w)` RMSNorm** — loader already bakes `+1`; norms placed pre/post attn + pre/post MLP.
+  - **Dual RoPE** — full layers use partial RoPE (`partial_rotary_factor: 0.25`, θ=1e6); sliding
+    layers use full RoPE (θ=1e4). Gerbil already does partial RoPE; this is per-layer parameterization.
+- **Loader work (no new compute kernels):**
+  - **Per-Layer Embeddings (PLE)** — place per-layer embedding tables (`hidden_size_per_layer_input: 256`).
+  - **Cross-layer KV sharing** (`num_kv_shared_layers: 20`) — graph wiring so 20 layers reuse a peer's KV.
+  - No MatFormer/elastic handling needed — E2B (and E4B) are fixed dense checkpoints; effective
+    param count is a PLE artifact, not a slice (§2.5).
+- **Mobile budget note:** vocab 262144 × hidden 1536 tied embedding is large; at INT4 the embed
+  tensor is ~0.2 GB (fits the iPad 128 MB `maxStorageBufferBindingSize` only if sharded — **the
+  embed/logit weight will need sharding** per the guide's buffer-cap rule).
+
+**Net:** Gemma 4 E2B is **Tier 2** — one genuinely new kernel (windowed attention, deferrable) plus
+several small variants/loader features. Matches the prior report's Gemma4 assessment; the live config
+refines it (4:1 not 5:1, MQA `num_kv_heads=1`, head_dim 256, dual RoPE, PLE, 20 shared-KV layers).
+
+---
+
+## Sources
+
+Primary, fetched live 2026-06-13:
+- `https://huggingface.co/Qwen/Qwen3-0.6B/resolve/main/config.json` [config-verified]
+- `https://huggingface.co/google/gemma-4-E2B/resolve/main/config.json` and `.../gemma-4-E2B-it/...` [config-verified]
+- `https://huggingface.co/google/gemma-4-E4B/resolve/main/config.json` and `.../gemma-4-E4B-it/...` [config-verified]
+- `https://huggingface.co/google/gemma-4-E2B/raw/main/README.md` (official card: "E" = effective
+  params via Per-Layer Embeddings; four distinct sizes E2B/E4B/26B-A4B/31B; no MatFormer mention) [card/docs]
+- HF API: `api/models?author=Qwen&search=Qwen3.6`, `api/models?author=google&search=gemma-4`,
+  `api/models/{repo}?blobs=true`, content-length headers [config-verified]
+- `Qwen/Qwen3-0.6B-MLX-4bit` (0.317 GB), `google/gemma-4-E2B-it-qat-q4_0-gguf`
+  (`gemma-4-E2B_q4_0-it.gguf` 3.349 GB, mmproj 0.987 GB) [config-verified]
+- Gerbil repo: `src/gpu/architectures/index.ts` (Qwen3ForCausalLM→qwen2.ts already registered),
+  `qwen2.ts` (no QK-norm), `qwen3_5.ts` (QK-norm reference wiring), `model-loader.ts`
+  (q_norm/k_norm keys + Gemma `+1` bake).
+
+Secondary:
+- Perplexity sonar-pro (2026-06-13): no small Qwen3.6; smallest Gemma 4 ≈ 2B. Citing unsloth.ai
+  Qwen3.6 docs, qwen.ai blog, github.com/QwenLM/Qwen3.6. [social/secondary]