@tryhamster/gerbil 1.0.8 → 1.0.9

package/docs/kernel-research-queue.md

@@ -1,59 +1,85 @@
-# Kernel optimization research queue — derived from the Gemma-4 WebGPU Space
-
-Source reference: `webml-community/gemma-4-webgpu-kernels` (custom WGSL, no ONNX —
-same approach as Gerbil). Running `google/gemma-4-E2B-it-qat`. Kernel grep of its
-`gemma-4-e2b.js`: **24 `enable subgroups`, 44 `subgroupAdd`, 36 `enable f16`, flash
-attention, `unpack4xU8` int4, mixed 2/4-bit packing**. Gerbil today: only 4 `enable
-subgroups` / 7 `subgroupAdd` / 15 f16, no flash-attn WGSL.
-
-Each test = an A/B on Gerbil's existing harness (`scripts/engine/benchmark.mjs`,
-`optimize.mjs`, `results.jsonl`, the iPad runner for on-device tok/s). Metric:
-decode tok/s + prefill ms + peak GPU/heap MB, on M-series desktop AND iPad/iPhone
-(mobile is the target). Gate: keep only if it beats baseline on-device without
-quality regression.
-
-## Generalizable to the whole engine (validate before shipping)
-
-1. **Subgroup-reduced GEMV (int4 decode matvec)** — HIGHEST PRIORITY.
-   Hypothesis: replacing Gerbil's `workgroupBarrier` reduction in `MatVecInt4`
-   with a `subgroupAdd` reduction (the Space's GEMV pattern) cuts the decode
-   reduction cost — decode is Gerbil's documented mobile bottleneck.
-   NOTE: Gerbil's autoresearch tried "subgroups-matvec" before and it was
-   inconclusive/reverted — re-run with the Space's *exact* pattern (it's a
-   shipped proof it works on broad WebGPU incl. Apple). Test wg sizes 64/128/256.
-
-2. **f16 scales/zeros in the int4 dequant** — already flagged as "THE documented
-   top desktop lever" in prior research; the Space confirms pervasive f16.
-   Hypothesis: f16 scales/zeros halve the dequant bandwidth. A/B f16 vs f32
-   scales in `MatVecInt4` / `MatMulInt4`.
-
-3. **`unpack4xU8` for int4 nibble unpack.** The Space uses the WGSL builtin
-   (`packed & 0x0F0F0F0F`, `>>4`). Hypothesis: faster/cleaner than manual shifts.
-   A/B Gerbil's current unpack vs `unpack4xU8`.
-
-4. **Flash-attention WGSL.** The Space ships one; Gerbil has none.
-   Hypothesis: lower attention peak memory + fewer dispatches (helps the iOS OOM
-   *and* tok/s). Port a tiled flash-attn kernel, A/B vs current attention on
-   prefill/decode + peak memory.
-
-5. **Mixed 2/4-bit quantization** (the Space's `0x03`/`>>6` masks ⇒ 2-bit layers).
-   Hypothesis: 2-bit for tolerant layers cuts memory (directly relevant to the
-   iOS OOM) at acceptable quality. Bigger change (quant pipeline) — defer behind
-   1–4, needs a mixed-bit checkpoint + per-layer bit assignment.
-
-## Gemma-4 specific — prep now (QAT is obviously the way)
-
-- **Repack `google/gemma-4-E2B-it-qat` into Gerbil's INT4 safetensors layout.**
-  The checkpoint is a custom/transformers-mobile format — NOT directly loadable by
-  Gerbil's MLX/GPTQ loader. Write a converter (their group-size scale+zeroPoint
-  int4 → Gerbil's packed-uint32 + scales/biases layout). QAT int4 holds
-  near-bf16 quality at int4 size — strictly better than the PTQ MLX-4bit shipped
-  today, same memory.
-- **Add `gemma-4-E2B-it-qat` as a supported model** once repacked (Gemma-4 e2b
-  is already a supported architecture; this is a new quantized checkpoint).
-- Context: Qwen 3.5 0.8B/2B remain the launch defaults; Gemma-4-QAT is a
-  quality option + the kernel reference, not a default switch.
+# Kernel optimization research queue — data-backed from the Gemma-4 challenge corpus
+
+Source of truth: `scripts/research/out/gemma-results.jsonl` (532 experiments harvested
+from the HF `gemma-challenge` bucket; `harvest-gemma-bucket.mjs` re-runs it). 241
+legit rows (ppl 1.5–2.31). Analyzed by a multi-agent workflow that separated
+WebGPU-portable wins from vLLM/CUDA server noise.
+
+**Headline reframe:** the 412-TPS leaderboard top is largely **server-only** —
+its biggest delta is `feopt2` (orjson + HTTP/JSON serialization; no browser
+analog) on a trained vLLM speculative drafter. Stripping serving noise, the
+portable wins attack Gerbil's two confirmed bottlenecks exactly: **dispatch-bound
+decode** and the **270 MB vocab OOM**. IGNORE the 412/506/511 headlines as porting
+targets (server serialization + FA3 + ppl spent past Gerbil's 2.31 guardrail).
+
+## THE CONVERGENCE (do these first — they fix the iOS OOM *and* boost TPS)
+
+**#1 · vocabprune (lm_head row-prune + logits scatter)** — directly-portable, gemma4.
+Prune the LM head 262144 → 32k–128k emittable rows, run the INT4 GEMV/argmax over
+the small head, scatter logits back to full vocab (−inf complement) before sampling.
+*Corpus proof:* `braiam-fable onegraph-pck04-vocabprune` 262144→32768 = **+21.3 TPS
+(311→332) AND ppl improved 2.0266→2.0091**; `dixie pck04b 16k` verify bytes
+**~335MB→~21MB/step**. (16k fails MMLU-Pro 0.596; 128k clears 0.688 → keep ~128k
+for a general assistant; still halves the head.) Pure load-time data transform,
+**no new WGSL**: slice lm_head q/scales/zeros to keep-rows, set `config.vocab_size`
+(argmax `executor.ts:1016` + matmul_int4 N already read it), add scatter-to-full.
+*Test:* vocab ∈ {full, 131072, 32768} × {decode TPS, resident MB, ppl, MMLU-Pro/GSM8K} on M-series + iPad.
+
+**#2 · Extend the prune to embed_tokens + the Gemma-4 PLE table** — directly-portable, gemma4.
+NOT in the corpus (the competition ran on servers with no load-time memory limit;
+they only pruned the head for *speed*). **Gerbil's OOM is at model LOAD**, and the
+Gemma-4 `embed_tokens_per_layer` (PLE) table is **~1.17 GB at 4-bit — the single
+largest resident tensor** (`model-loader.ts:87, ~1916`). Apply the same keep-set
+row-slice to `embed_tokens` + the PLE table so the *resident* weight actually
+shrinks. **Highest-leverage iOS-OOM action.** *Test:* peak resident MB at load with
+embed/PLE pruned to {full,131072,32768} on iPad; confirm no OOM at the risky tier.
+
+**#4 · Untie + INT4 the lm_head** — directly-portable, gemma4. Prerequisite for #1
+(a tied head can't be row-pruned independently). `model-loader.ts:1844-1847`
+currently does the tied fallback (lm_head→embed_tokens). Untie + `quantizeInt4`
+(`quantize.ts:32`, DEFAULT_GROUP_SIZE=128 already matches the corpus g128 sweet spot).
+*Corpus proof:* `foffee int4-lmhead` **118.3 TPS / ppl 2.0067 — 2.68× over bf16**
+(kills the ~1.34GB bf16 head read/step); `senpai int4_g128_lmhead` 126.4 / 2.0057,
+MMLU-Pro 0.668 / GSM8K 0.850.
+
+## Whole-engine, portable (after the OOM convergence)
+
+**#3 · onegraph-analog (fewer submits, no per-token readback on WebKit)** —
+maps-to-webgpu-analog. Gerbil already built the Dawn version: `forwardArgmax`
+(`executor.ts:971-1024`, argmax stays on GPU), `submitGreedyDecodeStep`
+(`executor.ts:1109`, PIPELINE_DEPTH=2, submits without awaiting). **Port that
+no-readback shape to the Safari/iOS path**; raise `webkitGroupSize` toward
+llama.cpp's ~64/CB; no host-JS branch in the hot loop (the `lffn` trap: a host
+`os.path.exists` in the graph cost 420→368 TPS). *Test:* sweep `?group=N` ∈ {1,8,32,64} for correctness + TPS on iPad.
+
+**#5 · More decode-op fusion** — directly-portable. Gerbil already fuses
+(`fuseDualMatVecDecode`/`fuseDualKVCacheAppend`/`fuseGatedOProj`/`fuseDualRMSNorm`,
+`executor.ts:507-517`). Next: fuse final RMSNorm→lm_head, and argmax into the
+lm_head tail. On WebKit each removed dispatch = a removed submit+drain. *Test:*
+diff `[executor] decode: N dispatches/token` + parity per fusion.
+
+**#6 · W4A16 QAT checkpoint** — directly-portable, gemma4. QAT int4 is a quality
+GAIN, not a cost: stock bf16 44.0 / 2.302 → W4A16 QAT **~95.6 / 2.0056** (ppl-guard,
+steve, speed-demon-ne, infinityoptimizer all reproduce). **PREFERRED:** requant
+`google/gemma-4-E4B-it-qat-q4_0-unquantized` through the **existing** `quantizeInt4`
+(g128) — zero new loader code, reuses the safetensors path. **ALT:** add a
+compressed-tensors/pack-quantized adapter like `repackGPTQ` (`gptq-adapter.ts:41`)
+to consume the official `…-qat-w4a16-ct` directly (`model-loader.ts:1074` only knows
+`quant_method=='gptq'` today). *Test:* current post-hoc int4 vs QAT-requant int4 at
+identical g128: ppl + MMLU-Pro/GSM8K (expect QAT lower ppl), TPS unchanged.
+
+**#7 · splitkv (KV layout f16/packed/head-split)** — maps-to-webgpu-analog.
+`deja-vu splitkv-honest-16k` 377.8 / 2.002 (best ppl-2.00 run) but n=1, terse body.
+`executor.ts allocateKVCacheBuffers ~1652` + `DualKVCacheAppend ~1936` + `softmax.wgsl`.
+
+**#8 · Speculative decoding** — DEFER until #1–#3 land. Biggest raw multiplier
+(~2–3.5×) but needs a **second model in memory** (fights the OOM #1/#2 just relieved)
+and a batched verify that depends on the #3 onegraph-analog (per-token submit+drain
+eats the acceptance win). No draft/verify infra exists in `src/gpu` today. Add
+relaxed/typical acceptance (eps=0.5, +~6%) only as an OPT-IN sampler flag (it changes
+emitted tokens; off by default for an honest on-device assistant).
 
 ## Sequence
-P0 (iOS OOM crash) first — do NOT start kernel work until the playground loads.
-Then 1 → 2 → 3 → 4 (each gated on-device), Gemma repack in parallel (independent).
+1 → 4 → 2 (the OOM convergence — load-time data transforms, no new WGSL, fixes iOS
+OOM + boosts TPS) → 6 (QAT requant, parallel) → 3 (WebKit onegraph-analog) → 5
+(fusion) → 7 (KV) → 8 (spec decode, last). Each gated on-device (ppl + MMLU-Pro/GSM8K).

package/docs/research/autoresearch-portable.md

@@ -902,3 +902,32 @@ From Karpathy:
 - [trevin-creator/autoresearch-mlx](https://github.com/trevin-creator/autoresearch-mlx) — Apple Silicon port
 - [DeepWiki: autoresearch](https://deepwiki.com/karpathy/autoresearch) — detailed system documentation
 - [DeepWiki: autoresearch-mlx](https://deepwiki.com/trevin-creator/autoresearch-mlx) — MLX port documentation
+
+---
+
+## Gerbil application: lever taxonomy + dual-track gating (from the Gemma-4 corpus, 2026-06-21)
+
+We mined a 532-experiment Gemma-4 optimization competition (`scripts/research/out/gemma-results.jsonl`, harvested by `harvest-gemma-bucket.mjs`; analysis in `docs/kernel-research-queue.md`). It validated the loop above AND taught us how to point Gerbil's own autoresearch. Five methodology learnings:
+
+### 1. The metric is TPS *under a quality guardrail* — never raw TPS
+The competition maximized decode TPS gated by perplexity ≤ ~2.31 (and downstream MMLU-Pro / GSM8K / GPQA at the extremes). Raw TPS gets gamed instantly (we found rows at `tps=1.79e308`, `1000000`, `ppl=0`). **Gerbil's autoresearch metric must be `TPS subject to ppl ≤ guard AND eval-suite ≥ floor`.** A speedup that moves ppl past the guard or fails MMLU-Pro is a REVERT, not a keep. (E.g. vocabprune to 16k rows hit 330 TPS but failed MMLU-Pro 0.596; 128k cleared 0.688 — only the 128k variant is a "keep".)
+
+### 2. Device-class gating — a win on one class can CRASH another
+The same kernel change has three regimes: **server/Dawn** (no memory limit, batching), **iPad WebKit** (tolerates `webkitGroupSize` up to ~128 → ~3.4× decode), **iPhone WebKit** (group>1 HARD-CRASHES the GPU process — the "third device class"). An experiment is only "kept" once validated on the **target device class**. Autoresearch must run with a device-class tag and never promote a change that regresses/crashes a class it's meant to support. (This is why the decided "Production submit strategy" must be a *crash-surviving auto-probe*, not a global default.)
+
+### 3. Two tracks — same engine, different lever sets
+Gerbil runs both **client WebGPU (browser, iOS/desktop)** and **server (Node + Dawn)**. The corpus's biggest headline wins were **server-only** (vLLM serving infra: HTTP/JSON serialization `feopt2`, continuous batching, FA3 CUDA tiling) — irrelevant to a single-user browser but **directly relevant to Gerbil-on-server**. So run autoresearch as TWO tracks with different `program.md` constraints: WebGPU-mobile (dispatch-bound + memory-tight) and server (batching, larger models, spec-decode with a small draft). Don't discard server wins — route them to the server track.
+
+### 4. Filter portable-from-infra
+Most corpus "improvements" were serving/infrastructure, not engine. Before queueing an experiment, classify it: directly-portable (a WGSL kernel / decode-loop change), maps-to-WebGPU-analog (e.g. CUDA-graph → fewer `queue.submit()`s; the `onegraph` family → Gerbil's `forwardArgmax`/PIPELINE_DEPTH path), or infra-only (skip for the mobile track). Only the first two belong in the WebGPU autoresearch loop.
+
+### 5. Per-model AND whole-engine levers
+Some levers are family-specific (Gemma-4's PLE table; a model's QAT checkpoint; vocab size); most are whole-engine (dispatch/launch reduction, op fusion, KV layout, int4 packing group-size). Run autoresearch **per-model** to catch arch-specific wins (e.g. Gemma PLE pruning) AND **overall** on a shared kernel set. Note Gerbil already captures several corpus levers (int4-*tied* head, CPU-streamed PLE, extensive decode fusion, the `forwardArgmax` Dawn path) — so the baseline is higher than the competition's; target the *unshipped* levers (the `webkitGroupSize` auto-probe, vocab prune, QAT requant) not the already-done ones.
+
+### The lever taxonomy to seed Gerbil's `program.md`
+Ranked, with bottleneck attacked (full detail + corpus evidence + file:line in `docs/kernel-research-queue.md`):
+- **Launch/dispatch reduction** (whole-engine; mobile decode is dispatch-bound): the `webkitGroupSize` crash-surviving auto-probe (decided in paper §19, never shipped — default still 1), more op fusion, no host-JS in the hot loop.
+- **Vocab/memory** (per-model, large-vocab; fixes iOS load OOM): prune lm_head/embed/PLE to a freq keepset + logits scatter; relevant to Qwen3.5 (248K vocab) too, not just Gemma.
+- **Quantization** (per-checkpoint): QAT int4 (quality *gain*), int4 packing group-size sweet spot (g128), mixed 2/4-bit.
+- **KV** (whole-engine): f16/packed/head-split layout.
+- **Speculative decoding** (server track first): small draft + batched verify; ~2–3.5× but needs a 2nd model in memory (fights mobile OOM) → server/large-model regime.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tryhamster/gerbil",
-  "version": "1.0.8",
+  "version": "1.0.9",
   "description": "Local LLM inference for Node.js. GPU-accelerated. Zero config. Works standalone or with Vercel AI SDK.",
   "type": "module",
   "main": "dist/index.mjs",
@@ -216,6 +216,6 @@
     "test:watch": "vitest",
     "changeset": "changeset",
     "version": "changeset version",
-    "release": "pnpm publish && changeset tag"
+    "release": "changeset publish"
   }
 }

package/dist/gerbil-BOsuusm3.mjs

@@ -1,4 +0,0 @@
-import { t as Gerbil } from "./gerbil-D9fzoBGM.mjs";
-import "./utils-DKO55ZmZ.mjs";
-
-export { Gerbil };

package/dist/moonshine-stt-D0EXyr4g.mjs

@@ -1,4 +0,0 @@
-import "./architectures-C1I5V3Dt.mjs";
-import { t as MoonshineSTT } from "./moonshine-stt-C2PR8fM4.mjs";
-
-export { MoonshineSTT };