@tryhamster/gerbil 1.1.2 → 1.1.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (62) hide show
  1. package/README.md +18 -16
  2. package/dist/cli.mjs +1 -1
  3. package/dist/cli.mjs.map +1 -1
  4. package/dist/gpu/hooks.d.mts +3 -3
  5. package/dist/gpu/hooks.mjs +5 -5
  6. package/dist/gpu/hooks.mjs.map +1 -1
  7. package/dist/index-Dgmb2kE3.d.mts.map +1 -1
  8. package/package.json +1 -2
  9. package/docs/PROJECT-STATE.md +0 -321
  10. package/docs/adding-a-model-family.md +0 -280
  11. package/docs/ai-sdk.md +0 -205
  12. package/docs/architecture/README.md +0 -84
  13. package/docs/architecture/caching.md +0 -227
  14. package/docs/architecture/inference.md +0 -176
  15. package/docs/architecture/overview.md +0 -189
  16. package/docs/architecture/streaming.md +0 -261
  17. package/docs/architecture/webgpu.md +0 -213
  18. package/docs/browser.md +0 -762
  19. package/docs/cli.md +0 -155
  20. package/docs/embeddings.md +0 -156
  21. package/docs/frameworks.md +0 -90
  22. package/docs/gerbil-site-native-migration.md +0 -217
  23. package/docs/gpu-engine/architectures.md +0 -398
  24. package/docs/gpu-engine/ir.md +0 -372
  25. package/docs/gpu-engine/kernels.md +0 -718
  26. package/docs/gpu-engine/paper.html +0 -1759
  27. package/docs/gpu-engine/paper.md +0 -2109
  28. package/docs/gpu-engine/safetensors.md +0 -312
  29. package/docs/gpu-engine/tokenizer.md +0 -302
  30. package/docs/kernel-research-queue.md +0 -85
  31. package/docs/mcp-client.md +0 -224
  32. package/docs/mcp.md +0 -109
  33. package/docs/memory-rag.md +0 -91
  34. package/docs/memory.md +0 -301
  35. package/docs/metal-safari-intel.md +0 -190
  36. package/docs/mobile-failure-diagnosis.md +0 -124
  37. package/docs/mobile.md +0 -99
  38. package/docs/observability.md +0 -230
  39. package/docs/onnx-removal-plan.md +0 -339
  40. package/docs/repl.md +0 -473
  41. package/docs/research/autoresearch-portable.md +0 -933
  42. package/docs/research/dispatch-reduction-hivemind.md +0 -84
  43. package/docs/research/ios-safari-model-caching.md +0 -117
  44. package/docs/research/mobile-webgpu-speed-fusion.md +0 -135
  45. package/docs/research/native-stt-model-selection.md +0 -49
  46. package/docs/research/native-tts-model-selection.md +0 -90
  47. package/docs/research/native-vs-chromium-decision.md +0 -152
  48. package/docs/research/nemotron-mamba2-inference.md +0 -910
  49. package/docs/research/qwen35-multimodal.md +0 -293
  50. package/docs/research/qwen36-gemma4-targets.md +0 -337
  51. package/docs/research/sota-embedding-models.md +0 -179
  52. package/docs/research/sota-mobile-models-2026.md +0 -263
  53. package/docs/research/sota-modality-models.md +0 -202
  54. package/docs/research/tps-baselines.md +0 -71
  55. package/docs/research/webgpu-m4-reference.md +0 -104
  56. package/docs/site-update-plan.md +0 -155
  57. package/docs/skills.md +0 -261
  58. package/docs/structured-output.md +0 -123
  59. package/docs/stt.md +0 -111
  60. package/docs/tools.md +0 -304
  61. package/docs/tts.md +0 -147
  62. package/docs/vision.md +0 -158
package/docs/cli.md DELETED
@@ -1,155 +0,0 @@
1
- # Gerbil CLI
2
-
3
- ## Quick Start
4
-
5
- ```bash
6
- # Without installing (one-off usage)
7
- npx @tryhamster/gerbil # Opens interactive REPL (default)
8
- npx @tryhamster/gerbil "Your prompt" # Quick text generation
9
-
10
- # After installing globally
11
- npm install -g @tryhamster/gerbil
12
- gerbil # Opens interactive REPL
13
- gerbil "Your prompt" # Quick text generation
14
- ```
15
-
16
- > All examples below use `gerbil` (assumes global install). If using `npx`, replace `gerbil` with `npx @tryhamster/gerbil`.
17
-
18
- ## Generate Text
19
-
20
- ```bash
21
- gerbil "Write a haiku about code"
22
- gerbil "Explain monads" --thinking
23
- gerbil "Hello" -m qwen3-0.6b --stream
24
- ```
25
-
26
- Options:
27
- - `-m, --model <id>` - Model to use (default: qwen3-0.6b)
28
- - `-n, --max-tokens <n>` - Max tokens (default: 256)
29
- - `-t, --temperature <t>` - Temperature (default: 0.7)
30
- - `-s, --system <text>` - System prompt
31
- - `--thinking` - Enable thinking mode
32
- - `--stream` - Stream output
33
- - `--json` - Output as JSON
34
-
35
- ## Skills
36
-
37
- ```bash
38
- gerbil commit # Commit message from staged changes
39
- gerbil commit --type conventional # Conventional commit format
40
- gerbil commit --write # Write to .git/COMMIT_EDITMSG
41
-
42
- gerbil summarize README.md # Summarize file
43
- gerbil summarize README.md -l short
44
-
45
- gerbil explain src/index.ts # Explain code
46
- gerbil explain src/index.ts -l beginner
47
-
48
- gerbil review src/ # Code review
49
- gerbil review src/ -f security,performance
50
- ```
51
-
52
- ## Interactive REPL
53
-
54
- ```bash
55
- gerbil # Opens REPL (or just run 'gerbil' with no args)
56
- gerbil repl # Same as above
57
- gerbil repl --gpu # Force WebGPU mode
58
- gerbil repl --cpu # Force CPU mode
59
- ```
60
-
61
- Launches the interactive terminal dashboard with:
62
- - Chat with multiple modes (Assistant, Coder, Teacher, Agent)
63
- - Skills execution and creation
64
- - Model management
65
- - Benchmark testing (compare CPU vs GPU performance)
66
- - Server management
67
-
68
- ### REPL Shortcuts
69
-
70
- | Key | Action |
71
- |-----|--------|
72
- | `t` | Toggle thinking mode (Qwen3) |
73
- | `a` | Toggle agent mode (tool calling) |
74
- | `m` | Switch between CPU and GPU mode |
75
- | `q` | Quit |
76
- | `1-8` | Jump to menu item |
77
-
78
- ## Chat
79
-
80
- Opens the REPL directly in chat view:
81
-
82
- ```bash
83
- gerbil chat
84
- gerbil chat --gpu # Chat with WebGPU acceleration
85
- ```
86
-
87
- This is equivalent to `gerbil` (or `gerbil repl`) and then pressing `1` for Chat.
88
-
89
- ## Server
90
-
91
- ```bash
92
- gerbil serve # HTTP server on :3000
93
- gerbil serve -p 8080 # Custom port
94
- gerbil serve --mcp # MCP server (stdio)
95
- ```
96
-
97
- HTTP endpoints:
98
- - `POST /generate` - Generate text
99
- - `GET /info` - Server info
100
-
101
- ## Model Management
102
-
103
- ```bash
104
- gerbil models # List built-in models
105
- gerbil models --search "code" # Search models
106
- gerbil info # System info
107
- gerbil info -m qwen3-0.6b # Model info
108
- ```
109
-
110
- ## Updates
111
-
112
- Gerbil checks for updates automatically but **never installs without your permission**.
113
-
114
- ```bash
115
- gerbil update # Check and install updates
116
- ```
117
-
118
- ### Update Notifications
119
-
120
- After running any command, you'll see:
121
-
122
- ```
123
- 💡 Update available: v1.0.0 → v1.0.1
124
- Run 'gerbil update' to install
125
- ```
126
-
127
- ### In REPL
128
-
129
- When using the REPL, updates appear in the header:
130
-
131
- ```
132
- gerbil 1.0.0 / Chat | Update v1.0.1 → Press u to update
133
- ```
134
-
135
- Press **`u`** to install the update.
136
-
137
- ## Use Any Model
138
-
139
- ```bash
140
- gerbil -m hf:microsoft/Phi-3-mini "Hello"
141
- gerbil -m hf:Qwen/Qwen2.5-Coder-0.5B "Write a function"
142
- ```
143
-
144
- ## CI/CD Integration
145
-
146
- ```yaml
147
- # .github/workflows/commit.yml
148
- - name: Generate Commit Message
149
- run: npx @tryhamster/gerbil commit --write
150
- ```
151
-
152
- Pre-commit hook:
153
- ```bash
154
- npx husky add .husky/prepare-commit-msg "npx @tryhamster/gerbil commit --write"
155
- ```
@@ -1,156 +0,0 @@
1
- # Embeddings
2
-
3
- Gerbil generates text embeddings natively on the WebGPU engine using **EmbeddingGemma-300M**
4
- — a bidirectional Gemma3 encoder with mean pooling and a 2-layer Dense head, producing
5
- 768-dim, L2-normalized vectors. Runs on-device (including iPad Safari), no ONNX, no API keys.
6
-
7
- > **Pre-1.0.** `engine.embed()` is the native path. The old ONNX/transformers.js embedding
8
- > lane (MiniLM/BGE/GTE) has been removed. The `Gerbil`-class helpers (`embed`, `similarity`,
9
- > `search`, `findNearest`) still work but now run native EmbeddingGemma under the hood (see
10
- > [below](#gerbil-class-embeddings-native-wrapper)).
11
-
12
- ## Quick Start
13
-
14
- ### Node
15
-
16
- ```typescript
17
- import { WebGPUEngine } from "@tryhamster/gerbil/gpu";
18
-
19
- const engine = await WebGPUEngine.create({
20
- repo: "mlx-community/embeddinggemma-300m-4bit",
21
- embedding: true,
22
- });
23
-
24
- // EmbeddingGemma is asymmetric — queries and documents use different prefixes.
25
- const query = await engine.embed("capital of France", { taskType: "query" });
26
- const doc = await engine.embed("Paris is the capital of France.", { taskType: "document" });
27
-
28
- // Vectors are unit-norm, so cosine similarity is just a dot product.
29
- const dot = query.reduce((s, v, i) => s + v * doc[i], 0);
30
- console.log(dot); // ~0.7+
31
-
32
- engine.destroy();
33
- ```
34
-
35
- `embed()` returns a `Float32Array` of length 768 (EmbeddingGemma) with unit L2 norm.
36
-
37
- ### React (Browser)
38
-
39
- ```tsx
40
- import { useEngine } from "@tryhamster/gerbil/gpu/hooks";
41
-
42
- function SemanticSearch() {
43
- const { embed, similarity, isLoading } = useEngine({
44
- model: "mlx-community/embeddinggemma-300m-4bit",
45
- embedding: true,
46
- autoLoad: true,
47
- });
48
-
49
- if (isLoading) return <div>Loading embedding model…</div>;
50
-
51
- const compare = async () => {
52
- // similarity() embeds a as a query and b as a document, returns cosine.
53
- const score = await similarity("Hello world", "Hi there");
54
- console.log(score);
55
- };
56
-
57
- return <button onClick={compare}>Compare</button>;
58
- }
59
- ```
60
-
61
- The hook exposes `embed(text, { taskType })` (defaults to `"query"`) and
62
- `similarity(a, b)`.
63
-
64
- ## Asymmetric tasks
65
-
66
- EmbeddingGemma uses different task prefixes for queries vs documents. Pass `taskType`, or a
67
- raw `taskPrompt` for non-retrieval tasks (clustering / classification / STS):
68
-
69
- ```typescript
70
- await engine.embed(text, { taskType: "query" }); // "task: search result | query: "
71
- await engine.embed(text, { taskType: "document" }); // "title: none | text: "
72
- await engine.embed(text, { taskPrompt: "task: clustering | query: " });
73
- ```
74
-
75
- ## API
76
-
77
- ```typescript
78
- interface EmbedOptions {
79
- /** EmbeddingGemma: "query" (default) or "document". */
80
- taskType?: "query" | "document";
81
- /** EmbeddingGemma: raw task prefix, overrides taskType. */
82
- taskPrompt?: string;
83
- /** Qwen3-Embedding: instruction prefix for query embeddings. */
84
- instruction?: string;
85
- /** Max tokens to encode (longer inputs are truncated). */
86
- maxTokens?: number;
87
- }
88
-
89
- // async embed(text: string, options?: EmbedOptions): Promise<Float32Array>
90
- ```
91
-
92
- `embed()` requires an engine loaded with `{ embedding: true }`. The pooling strategy is
93
- chosen by architecture: EmbeddingGemma mean-pools over all tokens; Qwen3-Embedding uses
94
- last-token (EOS-position) pooling.
95
-
96
- ## RAG
97
-
98
- EmbeddingGemma pairs with `@tryhamster/gerbil/memory` for token-budgeted retrieval, or you
99
- can build a simple pipeline by hand:
100
-
101
- ```typescript
102
- const engine = await WebGPUEngine.create({
103
- repo: "mlx-community/embeddinggemma-300m-4bit",
104
- embedding: true,
105
- });
106
-
107
- // Index documents.
108
- const docs = await loadDocuments();
109
- const index = [];
110
- for (const text of docs) {
111
- index.push({ text, vector: await engine.embed(text, { taskType: "document" }) });
112
- }
113
-
114
- // Retrieve.
115
- const q = await engine.embed(question, { taskType: "query" });
116
- const ranked = index
117
- .map((d) => ({ text: d.text, score: d.vector.reduce((s, v, i) => s + v * q[i], 0) }))
118
- .sort((a, b) => b.score - a.score)
119
- .slice(0, 3);
120
- ```
121
-
122
- ## Other native embedders
123
-
124
- - **Qwen3-Embedding-0.6B** (`Qwen/Qwen3-Embedding-0.6B`) — also supported natively
125
- (`{ embedding: true }`); uses last-token pooling and an optional `instruction` prefix.
126
- Larger (BF16 OOMs iPad); EmbeddingGemma is the recommended default.
127
-
128
- ## Models
129
-
130
- | Model | Repo | Dim | Notes |
131
- |-------|------|-----|-------|
132
- | **EmbeddingGemma-300M** | `mlx-community/embeddinggemma-300m-4bit` | 768 | Default; asymmetric; runs on iPad |
133
- | Qwen3-Embedding-0.6B | `Qwen/Qwen3-Embedding-0.6B` | 1024 | Last-token pooling; desktop |
134
-
135
- ---
136
-
137
- ## `Gerbil`-class embeddings (native wrapper)
138
-
139
- > The ONNX/transformers.js embedding lane (MiniLM/BGE/GTE) has been removed. The `Gerbil`-class
140
- > helpers below still work but now run native EmbeddingGemma under the hood (768-dim,
141
- > WebGPU-required). The old browser `useEmbedding` worker hook is gone — use `useEmbedding`
142
- > from `@tryhamster/gerbil/gpu/hooks`.
143
-
144
- ```typescript
145
- import { Gerbil } from "@tryhamster/gerbil";
146
-
147
- const g = new Gerbil();
148
- const { vector } = await g.embed("Hello world"); // number[768] (EmbeddingGemma)
149
- const { score } = await g.similarity("Hello world", "Hi there");
150
- const results = await g.search("capital of France", ["Paris…", "London…"]);
151
- ```
152
-
153
- ## See Also
154
-
155
- - [Browser Hooks](./browser.md) — React hooks
156
- - [Vision](./vision.md), [TTS](./tts.md), [STT](./stt.md)
@@ -1,90 +0,0 @@
1
- # Framework Integrations
2
-
3
- ## Next.js
4
-
5
- ```typescript
6
- // app/api/chat/route.ts
7
- import { gerbil } from "@tryhamster/gerbil/next";
8
-
9
- export const POST = gerbil.handler({ model: "qwen3-0.6b" });
10
- ```
11
-
12
- ## Express
13
-
14
- ```typescript
15
- import express from "express";
16
- import { gerbil } from "@tryhamster/gerbil/express";
17
-
18
- const app = express();
19
- app.use("/ai", gerbil());
20
-
21
- // Endpoints:
22
- // POST /ai/generate
23
- // POST /ai/stream
24
- // POST /ai/json
25
- // POST /ai/embed
26
- ```
27
-
28
- ## React Hooks
29
-
30
- ```typescript
31
- import { useGerbil, useChat } from "@tryhamster/gerbil/react";
32
-
33
- function Chat() {
34
- const { generate, isLoading } = useGerbil();
35
-
36
- const handleSubmit = async (prompt: string) => {
37
- const result = await generate(prompt);
38
- };
39
- }
40
- ```
41
-
42
- ## Fastify
43
-
44
- ```typescript
45
- import Fastify from "fastify";
46
- import { gerbilPlugin } from "@tryhamster/gerbil/fastify";
47
-
48
- const app = Fastify();
49
- app.register(gerbilPlugin, { prefix: "/ai" });
50
- ```
51
-
52
- ## Hono
53
-
54
- ```typescript
55
- import { Hono } from "hono";
56
- import { gerbil } from "@tryhamster/gerbil/hono";
57
-
58
- const app = new Hono();
59
- app.route("/ai", gerbil());
60
- ```
61
-
62
- ## tRPC
63
-
64
- ```typescript
65
- import { gerbilRouter } from "@tryhamster/gerbil/trpc";
66
-
67
- const appRouter = t.router({
68
- ai: gerbilRouter(t),
69
- });
70
- ```
71
-
72
- ## LangChain
73
-
74
- ```typescript
75
- import { GerbilLLM, GerbilEmbeddings } from "@tryhamster/gerbil/langchain";
76
-
77
- const llm = new GerbilLLM({ model: "qwen3-0.6b" });
78
- const embeddings = new GerbilEmbeddings();
79
-
80
- // Use with chains
81
- const chain = new LLMChain({ llm, prompt: template });
82
- ```
83
-
84
- ## LlamaIndex
85
-
86
- ```typescript
87
- import { GerbilLLM } from "@tryhamster/gerbil/llamaindex";
88
-
89
- const llm = new GerbilLLM({ model: "qwen3-0.6b" });
90
- ```
@@ -1,217 +0,0 @@
1
- # gerbil-site → Native-Engine Migration Assessment
2
-
3
- **As of 2026-06-14. Assessment only — no code in `gerbil-site` was changed.**
4
- The marketing/docs site lives in a **separate repo** at `/Users/shenron/code/gerbil-site`.
5
- This document maps how that site does in-browser inference today onto the native
6
- WebGPU engine's browser API (`@tryhamster/gerbil` `src/browser/*` + `src/gpu/index.ts`),
7
- says which features can switch **today** vs which must **wait**, states the device-coverage
8
- consequence of dropping the ONNX fallback **plainly**, and gives a phased plan with the
9
- smallest first step.
10
-
11
- It pairs with the engine paper §29 (cross-device multi-modal parity) and §23 (the
12
- native-only architecture decision). The owner decision is native-only with **no kept
13
- transformers.js/ONNX fallback lane**; this doc is the concrete site-side consequence.
14
-
15
- ---
16
-
17
- ## 0. The one fact that reframes everything
18
-
19
- The site does **not** call `@huggingface/transformers` or `onnxruntime-web` directly
20
- anywhere in runtime code. Every grep hit for `pipeline(`, `AutoModel`, `AutoTokenizer`,
21
- `KokoroTTS`, `feature-extraction`, etc. is **docs prose / code samples** under
22
- `app/docs/**`. All real inference goes through **gerbil**, by two paths:
23
-
24
- - **Path A — gerbil browser hooks** (`@tryhamster/gerbil/browser`): `useCompletion`,
25
- `useSpeech`, `useVoiceInput`. These wrap a **transformers.js + onnxruntime-web Web
26
- Worker** that lives *inside the gerbil tgz*. transformers.js@3.8.1 and the pinned
27
- `onnxruntime-web@1.21.0-dev…` are declared deps but consumed only transitively here.
28
- - **Path B — gerbil native engine** (`@tryhamster/gerbil/gpu`): the `WebGPUEngine`
29
- (pure WGSL, no worker, no ONNX), dynamically imported in `hooks/useNativeEngine.ts`.
30
-
31
- So the migration is **not** "rip ONNX out of the site." It is: **move each gerbil
32
- browser hook from its transformers.js/ONNX worker backend to the native engine** — and
33
- the site *already has a working native path* for chat. The work is (a) extend the native
34
- hook coverage to embeddings + vision, (b) wire audio when native audio lands, and (c)
35
- decide when to flip the default backend.
36
-
37
- > The site depends on gerbil as a **local tgz at rc.26**
38
- > (`@tryhamster/gerbil: file:/Users/shenron/Code/gerbil/tryhamster-gerbil-1.0.0-rc.26.tgz`).
39
- > Any new native browser hooks must be published in a new rc and the tgz re-pinned before
40
- > the site can consume them.
41
-
42
- ---
43
-
44
- ## (a) What the site uses for inference today, and where
45
-
46
- Framework: **Next.js 14.2.0** (App Router, React 18), `next.config.js` aliases the gerbil
47
- *browser* bundle to an empty module server-side and loads ORT WASM/MJS from CDN, with
48
- COOP/COEP headers for SharedArrayBuffer threading.
49
-
50
- | Call site (file) | Modality | Path | Library actually used | Model id(s) |
51
- |---|---|---|---|---|
52
- | `components/PlaygroundFull.tsx:309` `useCompletion` | Chat/completion | A (hooks) | transformers.js worker | `qwen3-0.6b` default + `smollm2-*`, `LFM2-*-ONNX`, `qwen3*` |
53
- | `components/PlaygroundFull.tsx:318` `useCompletion({model:"ministral-3b"})` | Vision (image→text) | A (hooks) | transformers.js worker (`AutoModelForImageTextToText`) | `ministral-3b` |
54
- | `components/PlaygroundFull.tsx:366` `useSpeech` | TTS | A (hooks) | `kokoro-js` / transformers.js | `kokoro-82m` (def), `supertonic-66m` |
55
- | `components/PlaygroundFull.tsx:383` `useVoiceInput` | STT | A (hooks) | transformers.js (Whisper) | `whisper-tiny.en` … `whisper-large-v3-turbo` |
56
- | `components/PlaygroundFull.tsx` Embed tab (~1700) `similarity()` | Embeddings | — | **REMOVED / dead** (`// useEmbedding removed in this gerbil version`) | n/a |
57
- | `components/AISDKPlayground.tsx:159` `useCompletion` | Chat | A (hooks) | transformers.js worker | `qwen3-0.6b` |
58
- | `components/AISDKPlayground.tsx:174` `useCompletion({model:"ministral-3b"})` | Vision | A (hooks) | transformers.js worker | `ministral-3b` |
59
- | `components/AISDKPlayground.tsx:187` `useSpeech` | TTS | A (hooks) | kokoro-js | `kokoro-82m` |
60
- | `hooks/useNativeEngine.ts:204` `import("@tryhamster/gerbil/gpu")` → `WebGPUEngine.create/generate` | Chat **only** | **B (native)** | **native WGSL** | `mlx-community/Qwen3.5-0.8B-4bit` (def), `Qwen/Qwen3.5-0.8B`, `Qwen/Qwen3-0.6B`, GPTQ variants |
61
-
62
- Wiring/render sites: `components/Playground.tsx` chooses `PlaygroundNative` (native-only
63
- chat; other tabs disabled) vs `PlaygroundFull` (all modalities, hooks) off
64
- `localStorage["gerbil-backend"]`. Both `<Playground />` and `<AISDKPlayground />` render on
65
- `app/page.tsx` and `app/playground/page.tsx`, all `dynamic(..., { ssr:false })`.
66
-
67
- Two states worth flagging now:
68
- - **Embeddings are already broken** in rc.26: `useEmbedding` was removed; both playgrounds
69
- null out `similarity`/`embed` but keep live Embed UI that would throw if clicked. So the
70
- embeddings migration is also a *bug fix*.
71
- - The native path (`hooks/useNativeEngine.ts`) is the site's **own** hook calling
72
- `WebGPUEngine` directly — it does **not** use a gerbil-published `useNativeEngine`
73
- (there is no `@tryhamster/gerbil/gpu/hooks` export subpath; see §(b) caveat).
74
-
75
- ---
76
-
77
- ## (b) Which native browser hook/API replaces each ONNX/transformers.js call site
78
-
79
- The native engine surface (`@tryhamster/gerbil/gpu`, `src/gpu/index.ts`) is the class
80
- `WebGPUEngine`, constructed via `static create(options)` and exposing
81
- `generate()`, `embed()`, `describeImage()`, `encodeImage()`. The browser hooks
82
- (`@tryhamster/gerbil/browser`, `src/browser/*`) currently target the transformers.js
83
- worker; native hooks either exist privately (`src/browser/use-native-engine.ts`,
84
- intentionally **not** re-exported because the GPU engine drags in `@huggingface/hub`
85
- Node-only `node:fs` paths) or must be added.
86
-
87
- | Site call site | Today (ONNX/tfjs) | Native replacement | Native symbol(s) | Status |
88
- |---|---|---|---|---|
89
- | `useCompletion` (chat) | tfjs worker `generate` | `WebGPUEngine.create({repo, dtype, maxSeqLen, onProgress})` → `engine.generate(prompt, {maxTokens, sampling, systemPrompt, stopSequences, onToken})` | `WebGPUEngine.create`, `generate` (`src/gpu/index.ts`) | ✅ **today** — already wired in `hooks/useNativeEngine.ts` |
90
- | Embed tab (removed) | (was `useEmbedding`) | load with `{ embedding: true }`, then `engine.embed(text, {taskType:"query"|"document"})` → unit-L2 `Float32Array` (dim 768 for EmbeddingGemma) | `embed`, guard `isEmbedding` | ✅ **today** — model `mlx-community/embeddinggemma-300m-4bit` (173 MB, runs on iPad, paper §25) |
91
- | Vision (`ministral-3b`) | tfjs `AutoModelForImageTextToText` | load with `{ enableVision: true }` (Qwen3.5), then `engine.describeImage({pixels,width,height}, prompt, opts)` → `GenerateResult` | `describeImage`, guard `hasVision`; lower-level `encodeImage(patches, gridTHW)` | ✅ **today** — but **model changes** to a Qwen3.5 ViT checkpoint (the native ViT is Qwen3.5's own tower, not Ministral); paper §22 / §10 |
92
- | `useSpeech` (TTS) | kokoro-js / tfjs | **none yet** — OmniVoice native TTS in progress | — (`src/browser/use-speech.ts` stays tfjs) | ❌ **wait** (audio) |
93
- | `useVoiceInput` (STT) | tfjs Whisper | **none yet** — Moonshine native STT not started | — (`src/browser/use-voice-input.ts` stays tfjs) | ❌ **wait** (audio) |
94
-
95
- Native `WebGPUEngine` option/method reference (from `src/gpu/index.ts` + `model-loader.ts`,
96
- exact signatures):
97
- - `create(options)`: `options extends LoadModelOptions` (`repo` required HF id/URL,
98
- `onProgress(loaded,total,message)`, `dtype?: "f32"|"q4"`, `revision?`, `hfToken?`) plus
99
- `maxSeqLen?` (capped 4096), `kvMode?`, `enableVision?` (downloads ~192 MB ViT, Qwen3.5
100
- only), `embedding?: boolean` (last-token pool + L2; on the Gemma encoder path it builds
101
- the encoder graph instead). Flags: `get isEmbedding`, `get hasVision`.
102
- - `generate(prompt|ChatMessage[], {maxTokens?, stopSequences?, sampling?, systemPrompt?, onToken?})`
103
- → `{text, tokensGenerated, tokensPerSecond, totalTime, finishReason, thinking?}`.
104
- - `embed(text, {instruction?, taskType?:"query"|"document", taskPrompt?, maxTokens?})` →
105
- unit-L2 `Float32Array`. Throws if not loaded with `{embedding:true}`.
106
- - `describeImage(image, prompt?, options?)` where `image` is `{pixels,width,height}` **or**
107
- `{patches, gridTHW}` → `GenerateResult`. Throws if not loaded with `{enableVision:true}`.
108
-
109
- **Caveat (publishing gap to fix first):** the native engine is exposed at
110
- `@tryhamster/gerbil/gpu`, but there is **no published React hook** wrapping it. The site
111
- solved this itself by writing `hooks/useNativeEngine.ts`. To migrate the other modalities
112
- cleanly, gerbil should publish proper native hooks (e.g. `useNativeChat`, `useNativeEmbedding`,
113
- `useNativeVision`) under a real subpath (today `src/browser/use-native-engine.ts` exists but
114
- is excluded from the `browser` barrel, and `@tryhamster/gerbil/gpu/hooks` is referenced in a
115
- comment but **not** declared in `package.json` `exports`). Until then, the site keeps wrapping
116
- `WebGPUEngine` directly, modality by modality, as it already does for chat.
117
-
118
- ---
119
-
120
- ## (c) Modality coverage map — switch TODAY vs WAIT
121
-
122
- | Site feature | Native today? | Native model | Notes |
123
- |---|---|---|---|
124
- | **Chat / completion** | ✅ **yes — already live** | Qwen3.5-0.8B (4bit), LFM2.5-350M | `useNativeEngine` already ships; LFM2.5 is the faster/smaller alt (paper §30) |
125
- | **Embeddings** | ✅ **yes** | EmbeddingGemma-300M (173 MB) | Runs on iPad (paper §25). Also fixes the currently-broken Embed tab |
126
- | **Vision (image→text)** | ✅ **yes** | Qwen3.5 ViT (`describeImage`) | Bit-exact vs HF, word-identical greedy output (paper §22). **Model swaps off `ministral-3b`** |
127
- | **TTS** | ❌ **wait** | OmniVoice (in progress) | Keep `useSpeech` on kokoro-js/tfjs until native audio validates |
128
- | **STT** | ❌ **wait** | Moonshine (not started) | Keep `useVoiceInput` on tfjs Whisper |
129
-
130
- Net: **chat, embeddings, and vision can all move to native today**; **audio must wait**.
131
- That maps exactly to the engine's "multi-modal parity minus audio" status (paper §29).
132
-
133
- ---
134
-
135
- ## (d) Device/browser coverage of WebGPU-only — and the no-fallback consequence, stated plainly
136
-
137
- The native engine is **WebGPU-only**. There is **no WASM/CPU fallback** in the native path
138
- (`src/browser/backend-selector.ts`'s WASM tiers belong to the *transformers.js* path, not
139
- the native engine; native `WebGPUEngine.create` simply requires a WebGPU adapter).
140
-
141
- **Devices that gain native (faster, no mobile crashes):**
142
- - **iPad / iPhone Safari (iPadOS/iOS 26.5+, WebKit)** — the headline win; previously crashed,
143
- now runs text + vision + embeddings (paper §17–§29). On older WebKit, the grouped-submit
144
- `?group=N` dial is the compatibility lever (paper §18.3).
145
- - **Desktop Chrome / Edge 113+**, **desktop Safari 18+**, **Firefox 141+**.
146
- - **Android Chrome 113+**, Samsung Internet 25+ (per paper Appendix B).
147
-
148
- **The plain consequence of dropping the ONNX fallback:** **any device or browser without
149
- WebGPU loses in-browser inference entirely.** There is no graceful degradation to WASM/CPU
150
- in the native path — the engine **throws a clear error rather than degrading** (PROJECT-STATE
151
- §3, "No-WebGPU / old devices: not targeted"). Concretely, the users who lose support are:
152
- older iOS/iPadOS (pre-26 WebKit where WebGPU is absent or buggy), older desktop browsers,
153
- locked-down enterprise browsers with WebGPU disabled, and low-end Android without a WebGPU
154
- adapter. Today those users fall back to the slow-but-working transformers.js WASM path; a
155
- hard native-only cutover **removes that safety net**. This is a deliberate owner decision
156
- (paper §23: a permanent fallback "assumes defeat to begin with") — but the site must own the
157
- UX of it: feature-detect WebGPU up front (`isWebGPUSupported` is already imported in the
158
- playgrounds) and show an explicit "this demo needs WebGPU" state instead of a silent failure.
159
-
160
- ---
161
-
162
- ## (e) Phased migration plan (smallest first step first)
163
-
164
- **Phase 0 — publish native hooks + re-pin the tgz (prerequisite, gerbil-side).**
165
- In gerbil, expose browser-safe native React hooks (`useNativeChat`/`useNativeEmbedding`/
166
- `useNativeVision`) under a declared `exports` subpath, fixing the `@huggingface/hub`
167
- `node:fs` leak that currently keeps `use-native-engine.ts` out of the barrel. Cut a new rc,
168
- rebuild the tgz, re-pin `@tryhamster/gerbil` in the site. (If this slips, the site can keep
169
- hand-wrapping `WebGPUEngine` as it does for chat — but publishing is the clean path.)
170
-
171
- **Phase 1 — SMALLEST FIRST STEP: make native chat the default behind WebGPU detection.**
172
- The native chat path **already exists and works** (`hooks/useNativeEngine.ts` →
173
- `WebGPUEngine.generate`). The minimal change is in `components/Playground.tsx`: when
174
- `isWebGPUSupported()` is true, default `localStorage["gerbil-backend"]` to native
175
- (`PlaygroundNative`) instead of requiring a manual toggle; keep `PlaygroundFull` (tfjs) as the
176
- explicit opt-out and the no-WebGPU path. Zero new gerbil API needed, fully reversible, and it
177
- flips the highest-traffic modality to native first.
178
-
179
- **Phase 2 — fix + migrate embeddings to native (also un-breaks the dead tab).**
180
- Replace the removed `useEmbedding` in `PlaygroundFull`/`AISDKPlayground` with a native
181
- embedding hook loading `mlx-community/embeddinggemma-300m-4bit` + `engine.embed(text,
182
- {taskType})`. This both ships native embeddings and repairs the currently-throwing Embed tab.
183
-
184
- **Phase 3 — migrate vision to native, swapping the model.**
185
- Replace the `ministral-3b` `useCompletion` vision instances with a native vision hook:
186
- `WebGPUEngine.create({repo: <Qwen3.5 ViT checkpoint>, enableVision:true})` +
187
- `describeImage({pixels,width,height})`. Requires host pixel→patch handling (the engine's
188
- `describeImage` accepts decoded `{pixels,width,height}` and preprocesses internally) and a
189
- copy/UI change because the model id and capabilities differ from Ministral.
190
-
191
- **Phase 4 — keep audio on tfjs; flip when native audio lands.**
192
- Leave `useSpeech` (TTS) and `useVoiceInput` (STT) on the transformers.js path. Swap TTS to
193
- OmniVoice when it validates, then STT to Moonshine. This is the only phase gated on engine work
194
- not yet done.
195
-
196
- **Phase 5 — retire the tfjs path for non-audio (optional, end-state).**
197
- Once Phases 1–3 are stable and audio is native (Phase 4), the transformers.js/ONNX worker can
198
- be removed for all but the explicit no-WebGPU fallback decision — matching paper §23
199
- (`chrome-backend.ts` slated for deletion engine-side). Whether to keep *any* tfjs fallback at
200
- all is the owner call in §(d).
201
-
202
- ---
203
-
204
- ## (f) The single biggest risk
205
-
206
- **Losing the WebGPU-less audience with no graceful fallback, on a demo that is the product's
207
- shop window.** The site is gerbil's marketing front door: a visitor on an older iPhone, a
208
- locked-down work laptop, or any browser without WebGPU currently still gets a working (if slow)
209
- WASM demo. A native-only cutover turns that into a hard "unsupported" wall. The mitigation is
210
- non-negotiable and cheap: **feature-detect WebGPU at the top of every playground** (the hooks
211
- are already imported), default non-WebGPU visitors to either the retained tfjs `PlaygroundFull`
212
- or an explicit, friendly "needs WebGPU" state — and **never** flip the default to native without
213
- that guard in place. Secondary risks, in order: the **tgz/publishing coupling** (no native hooks
214
- are published yet, so the site is hand-wrapping `WebGPUEngine` — a versioning and maintenance
215
- liability until Phase 0 lands), the **vision model swap** (Ministral → Qwen3.5 ViT changes
216
- behavior and copy, not just an import), and **iPad re-download cost** (no durable cache without a
217
- PWA, paper §24 — a UX, not correctness, issue).