diskformer 0.4.0 → 0.4.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/CHANGELOG.md +4 -0
  2. package/README.md +22 -23
  3. package/package.json +1 -1
package/CHANGELOG.md CHANGED
@@ -1,5 +1,9 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.4.1 (2026-10-13)
4
+ - Docs only: the README now leads with the llama.cpp path (Qwen3-Next 80B-A3B, a 48 GB file, in under 5 GB of GPU
5
+ memory; OLMoE and LFM2 in real time) and the recording. No code changes.
6
+
3
7
  ## 0.4.0 (2026-10-13)
4
8
  - Changed: errors are now `DiskformerError`s with a `code` (see below). A caller that checked `e.name === 'AbortError'`
5
9
  after an aborted ingest should check `isDiskformerError(e, 'LOAD_ABORTED')`; messages are unchanged.
package/README.md CHANGED
@@ -17,19 +17,27 @@ Plain ES modules, no dependencies, no build step. Chromium with WebGPU. Weights
17
17
  | How | Command |
18
18
  |---|---|
19
19
  | npm | `npm install diskformer` |
20
- | CDN | `import { Gemma4MoeSsd } from 'https://cdn.jsdelivr.net/npm/diskformer@0.3/engines/gemma4_moe_ssd.js'` |
20
+ | CDN | `import { Gemma4MoeSsd } from 'https://cdn.jsdelivr.net/npm/diskformer@0.4/engines/gemma4_moe_ssd.js'` |
21
21
  | Copy | `src/`, `engines/` and `index.js`; nothing else is needed |
22
22
 
23
- ## Try it: a 14.4 GB model in 2.5 GB of GPU memory
23
+ ## Try it: an 80B model (48 GB file) in under 5 GB of GPU memory
24
24
 
25
- [diskformer.naklitechie.com](https://diskformer.naklitechie.com/) (also a [Hugging Face Space](https://huggingface.co/spaces/naklitechie/diskformer-chat)) runs llama.cpp's WebGPU backend with paged experts: LFM2 8B-A1B, OLMoE, Qwen3 30B-A3B, Qwen3-Next 80B (source in `ggml/`; references in `examples/chat/refs/`).
26
- The hand-written engines measured below are at [/classic/](https://diskformer.naklitechie.com/classic/); serve `examples/chat` to run them locally. Each page downloads a model once into OPFS, resumably.
25
+ https://github.com/user-attachments/assets/4a23b43e-995a-4ba8-b1df-5ffae0222b4e
27
26
 
28
- ```bash
29
- python3 -m http.server 8000 # then open http://localhost:8000/examples/chat/
30
- ```
27
+ [diskformer.naklitechie.com](https://diskformer.naklitechie.com/) (also a [Hugging Face Space](https://huggingface.co/spaces/naklitechie/diskformer-chat))
28
+ runs llama.cpp's WebGPU backend with the experts paged in from OPFS (source in `ggml/`). Pick a model; it downloads
29
+ once. Measured in Chrome on a MacBook Pro M4 Pro, 24 GB, 2026-10-13 (the recording above):
30
+
31
+ | Model (GGUF) | On disk | GPU memory | Decode | Paged vs unpaged replies |
32
+ |---|---|---|---|---|
33
+ | Qwen3-Next 80B-A3B, Q4_K_M | 48.4 GB | 4.89 GB | 2.9 tok/s, 359 MB read per token | 9/9 identical (16 vs 64 slots) |
34
+ | OLMoE 1B-7B, Q4_K_M | 4.21 GB | 3.92 GB | 22.4 tok/s | 9/9 identical |
35
+ | LFM2 8B-A1B, Q4_K_M | 5.04 GB | 3.89 GB | 17.1 tok/s | 9/9 identical |
36
+
37
+ Qwen3 30B-A3B runs too. References: `examples/chat/refs/`. The two hand-written engines below run at
38
+ [/classic/](https://diskformer.naklitechie.com/classic/), or locally with `python3 -m http.server` and `examples/chat/`.
31
39
 
32
- Measured on a MacBook M4 Pro with 24 GB, Chrome 154, on 2026-10-05:
40
+ The hand-written engines, measured on a MacBook M4 Pro with 24 GB, Chrome 154, on 2026-10-05:
33
41
 
34
42
  | Model | On disk | GPU budget | GPU memory used | Decode | Output |
35
43
  |---|---|---|---|---|---|
@@ -38,14 +46,11 @@ Measured on a MacBook M4 Pro with 24 GB, Chrome 154, on 2026-10-05:
38
46
  | Gemma 4 26B-A4B, QAT Q4_0 | 14.4 GB | 7 GB | 6.63 GB | 17.9 tok/s | 9/9 |
39
47
  | Qwen3.6 35B-A3B, Q8_0 | 36.9 GB | 4 GB | 3.52 GB | 9.2 tok/s | not compared |
40
48
 
41
- "Identical" means the reply text matches llama.cpp b9830's Metal path character for character on the same GGUF. The
42
- gate first drops the empty thought-channel marker llama-server echoes before each Gemma reply. 9 conversations,
43
- greedy, up to 64 tokens each. Qwen3.6 is not compared: llama.cpp's Metal path cannot hold 36.9 GB here.
49
+ "Identical": the reply matches llama.cpp b9830 Metal character for character on the same GGUF (9 conversations, greedy,
50
+ 64 tokens; Qwen3.6 does not fit Metal here). Each row is `node examples/run-demo.mjs chat --budget <GB>`, decode the
51
+ median. Only Apple silicon is tested; discrete GPUs and 8–16 GB machines are not.
44
52
 
45
- Each row is `node examples/run-demo.mjs chat --budget <GB>`; decode is the median over the 9 replies. A larger budget
46
- caches more experts. Only Apple silicon is tested; discrete GPUs and 8–16 GB machines are not.
47
-
48
- In your own page:
53
+ In your own page (the first load downloads the GGUF into OPFS in the engine's layout; later loads take seconds):
49
54
 
50
55
  ```js
51
56
  import { Gemma4MoeSsd } from 'diskformer/engines/gemma4_moe_ssd.js';
@@ -54,8 +59,6 @@ const engine = await Gemma4MoeSsd.load(null, { gpuBudgetBytes: 3e9, onProgress:
54
59
  for await (const { text } of engine.generate([{ role: 'user', content: 'Hello' }], { maxNewTokens: 128 })) show(text);
55
60
  ```
56
61
 
57
- The first load downloads the GGUF from Hugging Face into OPFS, in the engine's layout. Later loads take seconds.
58
-
59
62
  ## Why
60
63
 
61
64
  Your model does not fit on the GPU, or fits only by crowding out everything else. Browser engines load every weight
@@ -102,7 +105,6 @@ node examples/run-demo.mjs rows # headless Chrome: a 64 MB table on
102
105
  node examples/run-demo.mjs embedding # lookup → slots → a WGSL gather kernel, every output value checked
103
106
  node examples/run-demo.mjs chat --budget 2.5 # downloads Gemma once (14.4 GB) to a fixed profile, replays 9 llama.cpp replies, fails on any difference
104
107
  node examples/run-demo.mjs test/browser/ingest-resume.html --models <dir> --file <gguf> # abort, resume, compare bytes
105
- node examples/run-demo.mjs test/browser/server.html --models <dir> --file <any file> # the runner's file server rejects bad requests
106
108
  node examples/run-demo.mjs test/browser/greedy.html --models <dir> --file <gguf> --ref test/refs/<ref>.json # token by token vs llama.cpp
107
109
  node examples/run-demo.mjs test/browser/states.html --models <dir> --file g4-6l.gguf # per-layer states; then test/state-check.py
108
110
  npm run sync # the files shared with LocalMind match their sources
@@ -111,10 +113,7 @@ npm run sync # the files shared with LocalMind m
111
113
  The chat gate replays `examples/chat/refs/gemma.json` (llama.cpp's replies) under a GPU budget; any differing reply
112
114
  exits 1. `--models <dir>` serves a local GGUF; `--site <url>` gates a deployed copy, such as the Space.
113
115
 
114
- The code runs in [LocalMind](https://github.com/NakliTechie/LocalMind); `src/` and `engines/` are its copies.
115
-
116
116
  ## License
117
117
 
118
- MIT. `engines/` contains kernels ported from Apache-2.0 code: see `engines/NOTICE`. Model weights are not
119
- distributed; each model is downloaded under its own terms. Pointers:
120
- [LocalMind](https://github.com/NakliTechie/LocalMind) · `CHANGELOG.md`.
118
+ MIT. `engines/` contains kernels ported from Apache-2.0 code: see `engines/NOTICE`. Model weights are not distributed;
119
+ each model downloads under its own terms. The code runs in [LocalMind](https://github.com/NakliTechie/LocalMind) (`src/` and `engines/` are its copies) · `CHANGELOG.md`.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "diskformer",
3
- "version": "0.4.0",
3
+ "version": "0.4.1",
4
4
  "description": "Run a model larger than your GPU memory in a browser tab: weights stay in OPFS, WebGPU holds only what each step needs. A disk tier plus two reference MoE engines.",
5
5
  "type": "module",
6
6
  "license": "MIT",