anyformer 0.3.2 → 0.4.2

package/CHANGELOG.md

@@ -1,5 +1,51 @@
 # Changelog
 
+## 0.4.2 — 2026-10-23
+
+### Added
+- Flint 40B-A2.5B (RecursiveSI/flint-int4-gguf, Q4_1, 26.0 GB) and LFM2.5 24B-A2B (LiquidAI/LFM2.5-24B-A2B-GGUF,
+  QAD-Q4_0, 13.6 GB) on the `diskformer` back end, through diskformer's `FlintMoeSsd` and `Lfm2MoeSsd` hand engines.
+  Before, both routed to the llama.cpp path (Flint 1.8–3.4 tok/s there against 8.66 on its engine). Numbers from
+  LocalMind's gates: Flint ~7.3 GB on the GPU with a 4 GiB pool, 8.66 tok/s; LFM2.5 24B ~16 tok/s at a 4 GiB pool.
+  Floors of 4 GB are unmeasured. Not yet run through anyformer in a browser; LocalMind's live check is that check.
+
+### Changed
+- README: the `localmind` back end's kernels ship in diskformer.
+
+## 0.4.1 — 2026-10-23
+
+### Fixed
+- `session.speculative` and `session.pleMode` were copied once when `load()` returned, because the session object was
+  built with spreads, which read getters. A Bonsai session's `speculative.status` stayed `'loading'` after DFlash or
+  MTP attached. They are now read live. A test through `load()` failed on 0.4.0 and passes now.
+
+### Added
+- `session.engine`: the engine instance the `diskformer` and `localmind` back ends built, read-only, for token-level
+  work a caller already does with it (Bonsai 2's tokenizer and think-marker ids in LocalMind's worker). Calls on it
+  bypass the session's one-call-at-a-time queue.
+
+## 0.4.0 — 2026-10-23
+
+What LocalMind's own workers do with its last three engine families (LFM2.5, Gemma 4 E2B/E4B, Ternary Bonsai 2 27B),
+on the `localmind` back end, so those models can move onto anyformer with nothing lost.
+
+### Added
+- `load(repo, { fetch })`: the engines' fetch, for a caller's retrying fetch on multi-GB range reads.
+- `load(repo, { ple: true })` for Gemma 4 E2B/E4B: the per-layer-embedding table streams from OPFS (diskformer's
+  `ple-opfs.js`); `session.pleMode` reports the mode.
+- `load(repo, { speculative })` for Ternary Bonsai 2 27B: `{ dflash: { block } }` (DFlash 2) or
+  `{ mtp: { depth, adaptive } }`, attached after the model is ready so the first reply never waits for it. The drafter
+  and the MTP head download once into Cache Storage, under LocalMind's names and keys, so a browser that holds them
+  reuses them. `session.speculative` gives `{ status, error, stats }`. Output is greedy-identical either way.
+- `generate(messages, { thinking, assistantPrefix })` on the `localmind` back end: thinking per request (Bonsai's
+  template, `reasoning_effort: 'medium'`) and a raw prefix to continue from; chunks carry `token`.
+- `session.reset()` where the engine has it, queued with `generate()`.
+
+### Measured
+- LFM2.5 230M through the `localmind` back end: the same reply as before these changes. Node tests cover each option
+  with stand-in engines (83/83). Gemma's PLE table and Bonsai's DFlash and MTP have not run through anyformer in a
+  browser; LocalMind's A/B per model is that check.
+
 ## 0.3.2 — 2026-10-23
 
 ### Changed

package/README.md

@@ -56,9 +56,9 @@ back end's answer: yes with an estimate (GPU bytes, disk bytes, tok/s when measu
 |---|---|---|
 | `transformers` | transformers.js 4.3.1: text generation, image and audio chat (Gemma 4, Qwen3.5), embeddings, speech to text, detection, classification, depth, segmentation | ONNX repos in the onnx-community / Xenova layout |
 | `kohra` | kohra, text-diffusion over onnxruntime-web | MDLM and BD3LM repos tagged `text-diffusion` |
-| `localmind` | LocalMind's hand-written WebGPU kernels, the whole model on the GPU | any LFM2 / LFM2.5 GGUF; Gemma 4 E2B / E4B QAT-mobile; Ternary Bonsai 2 |
+| `localmind` | LocalMind's hand-written WebGPU kernels (shipped in diskformer), the whole model on the GPU | any LFM2 / LFM2.5 GGUF; Gemma 4 E2B / E4B QAT-mobile; Ternary Bonsai 2 |
 | `wllama` | llama.cpp in WebAssembly: WebGPU when the budget holds the model, the CPU otherwise (no WebGPU needed) | any GGUF up to about 3.6 GB that no engine above covers; files over 2 GB are split into OPFS shards once |
-| `diskformer` | diskformer engines: weights in OPFS, only what a step needs on the GPU | Gemma 4 26B-A4B (from 2.5 GB), Qwen3.6 35B-A3B (from 4 GB), Ling 3.0 Flash 124B (from 6.8 GB), MiniCPM-V 4.7 with images (from 4 GB) |
+| `diskformer` | diskformer engines: weights in OPFS, only what a step needs on the GPU | Gemma 4 26B-A4B (from 2.5 GB), Qwen3.6 35B-A3B (from 4 GB), Ling 3.0 Flash 124B (from 6.8 GB), MiniCPM-V 4.7 with images, Flint 40B, LFM2.5 24B (from 4 GB) |
 | `diskformer-llama` | diskformer's llama.cpp build with WebGPU, mixture-of-experts layers paged in from OPFS | GGUFs of the 17 MoE architectures diskformer has checked (Granite 4.0 H, gpt-oss, GLM-4.7-Flash, LFM2 MoE…); a hand engine above wins where it covers the model |
 
 Each runtime loads from an exact version or commit in [src/sources.js](src/sources.js); set a back end's `.source`
@@ -76,7 +76,7 @@ and `threw`. Every other failure is an `AnyformerError` too, with a code from `E
 
 ```js
 plan(repo, { budget, task })      // what would run it, and why the others would not; loads nothing
-load(repo, { budget, task, file, maxCtx, poolBytes, localFile, onProgress, signal })  // → { backend, task, plan, run(input), generate?, think?, prime?, dispose() }
+load(repo, { budget, task, file, maxCtx, poolBytes, localFile, fetch, ple, speculative, onProgress, signal })  // → { backend, task, plan, run(input), generate?, think?, prime?, reset?, speculative?, dispose() }
 pipeline(task, repo, { budget })  // transformers.js's call shape: gen(messages) → [{ generated_text }]
 probe()                           // WebGPU, shader-f16, subgroups, limits, OPFS quota, memory64
 describe(repo)                    // Hub repo → formats, architecture, task, tags, files

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "anyformer",
-  "version": "0.3.2",
+  "version": "0.4.2",
   "description": "One call to run a model in a browser tab: anyformer reads the Hub repo, probes the browser, and picks the back end (transformers.js, LocalMind's WebGPU kernels, diskformer, kohra) that fits the model's format, architecture and your GPU budget.",
   "type": "module",
   "main": "./index.js",

package/src/backends/diskformer.js

@@ -27,6 +27,21 @@ export const DISKFORMER_MODELS = [
     engine: 'qwen35_moe_ssd.js', Engine: 'MiniCpmV47Ssd', minBudget: 4e9,
     speeds: [[6.1e9, 14.1]],
   },
+  // From LocalMind's gates (M4 Pro, 4 GiB pool; plan/2026-10-22-flint-harness): Flint ~7.3 GB on the GPU at maxCtx
+  // 8192 (dense 2.33, pool 4.29, KV 0.41, batch 0.20), 8.66 tok/s at lookahead 1. LFM2.5 24B ~16 tok/s at a 4 GiB
+  // pool; its total GPU use was not recorded, so it carries no speed estimate. The smallest working budget of either
+  // is unmeasured: 4 GB is a floor, and the engine refuses a smaller fit with the need. Both files are also on the
+  // llama.cpp path's checked list; this back end ranks first (Flint on llama.cpp: 1.8–3.4 tok/s).
+  {
+    repo: 'RecursiveSI/flint-int4-gguf', file: 'flint-int4-q4_1.gguf', size: 25992966976,
+    engine: 'flint_moe_ssd.js', Engine: 'FlintMoeSsd', minBudget: 4e9,
+    speeds: [[7.3e9, 8.66]],
+  },
+  {
+    repo: 'LiquidAI/LFM2.5-24B-A2B-GGUF', file: 'LFM2.5-24B-A2B-QAD-Q4_0.gguf', size: 13578190880,
+    engine: 'lfm2_moe_ssd.js', Engine: 'Lfm2MoeSsd', minBudget: 4e9,
+    speeds: [],
+  },
   // Two parts, ingested as one file by the engine's splitFetch (engines/ling_moe_ssd.js LING_FLASH, rev 29edce5).
   // Budget and speed from LocalMind's gate (plan/2026-10-22-ling-harness/results.md): 2.51 GB of dense weights on
   // the GPU plus a 4.29 GB expert pool, 5.53 tok/s decode; the OPFS store is 80.7 GB.

package/src/backends/localmind.js

@@ -72,23 +72,33 @@ export const localmindBackend = {
     return { ok: true, estimate: { gpuBytes: m.bytes, diskBytes: m.bytes, tokPerSec: MEASURED[model.repo] ?? null } };
   },
 
-  async load(model, { maxCtx, onProgress, signal }) {
+  // Options beyond the shared ones, as LocalMind's workers use the engines:
+  //   fetch        the engines' fetch (a caller's retrying fetch for multi-GB range reads)
+  //   ple          Gemma 4: stream the per-layer-embedding table from OPFS (ple-opfs.js) instead of holding it
+  //   speculative  Ternary Bonsai 2: { dflash: { block } } or { mtp: { depth, adaptive } }, attached after the model is
+  //                ready, so the first reply never waits for the drafter's download (src/backends/speculative.js)
+  async load(model, { maxCtx, fetch, ple, speculative, onProgress, signal }) {
     const m = match(model);
     const e = ENGINES[m.engine];
-    const mod = await import(`${localmindBackend.source}/${e.file}`);
+    const engines = localmindBackend.source;
+    const mod = await import(`${engines}/${e.file}`);
+    const say = onProgress && ((p) => onProgress({ backend: 'localmind', engine: m.engine, ...p }));
+    let pleTable = null;
+    if (ple && m.engine === 'gemma4') {
+      const { createGemmaPle } = await import(`${engines}/ple-opfs.js`);
+      pleTable = await createGemmaPle({ source: { repo: model.repo }, onStatus: (s) => say?.({ status: 'ple', ...s }) });
+    }
     const opts = {
-      revision: model.revision || 'main', signal,
-      onProgress: onProgress && ((p) => onProgress({ backend: 'localmind', engine: m.engine, ...p })),
+      revision: model.revision || 'main', signal, onProgress: say,
+      ...(fetch && { fetch }),
+      ...(pleTable && { ple: pleTable }),
       ...(m.engine !== 'gemma4' && { file: m.file }),
       ...(maxCtx && m.engine !== 'gemma4' && { maxLength: maxCtx }),
     };
     const engine = await mod[e.Engine].load(model.repo, opts);
     signal?.throwIfAborted();
-    // Thinking off, as the diskformer back end does: a chat reply, not the model's reasoning. (The qwen35 engine
-    // takes template arguments as a field; LocalMind sets the same one.)
-    if ('chatTemplateArgs' in engine) engine.chatTemplateArgs = { ...engine.chatTemplateArgs, enable_thinking: false };
     // Compile the kernels now, as LocalMind's workers do, so the first reply is not paying for it.
-    onProgress?.({ backend: 'localmind', engine: m.engine, status: 'warmup' });
+    say?.({ status: 'warmup' });
     await engine.warmup?.();
     // Gemma's template has no system turn; LocalMind folds the system prompt into the first user turn the same way.
     const prepare = m.engine !== 'gemma4' ? (msgs) => msgs : (msgs) => {
@@ -97,18 +107,43 @@ export const localmindBackend = {
       const i = rest.findIndex((x) => x.role === 'user');
       return sys && i >= 0 ? rest.map((x, k) => (k === i ? { ...x, content: `${sys}\n\n${x.content}` } : x)) : rest;
     };
+    // Speculative decoding, Bonsai only, in the background: a reply that starts before it lands decodes plainly.
+    const spec = { status: speculative && m.engine === 'qwen35' ? 'loading' : 'off', runner: null, error: null, disposed: false };
+    if (spec.status === 'loading') {
+      import('./speculative.js')
+        .then(({ attachSpeculative }) => attachSpeculative(engine, speculative, { engines, fetch, signal, onProgress: (p) => say?.({ ...p, speculative: true }) }))
+        .then((runner) => {
+          // Disposed while the drafter downloaded: detach and free it rather than attach to a dead engine.
+          if (spec.disposed) { engine.model.dflashRunner = null; runner.dispose?.(); return; }
+          spec.runner = runner; spec.status = 'ready';
+        }, (err) => { spec.error = err.message; spec.status = 'error'; });
+    }
     let tokens = 0;
     return {
       task: 'text-generation',
       engine,
-      async *generate(messages, { maxNewTokens = 512, signal } = {}) {
+      // Gemma 4 with the OPFS table: 'resident' or the streamed mode the engine reports.
+      ...(m.engine === 'gemma4' && { get pleMode() { return engine.pleMode ?? null; } }),
+      // { status: 'off' | 'loading' | 'ready' | 'error', error, stats }; stats are the runner's counters.
+      get speculative() { return { status: spec.status, error: spec.error, stats: spec.runner?.stats ?? null }; },
+      // thinking: per request, through the qwen35 engine's template arguments (reasoning_effort 'medium', LocalMind's
+      // serving recipe); the other two engines have no thinking switch. assistantPrefix: raw text the reply continues
+      // from (Bonsai's think-budget continuation). Chunks carry the engine's token id.
+      async *generate(messages, { maxNewTokens = 512, signal, thinking = false, assistantPrefix } = {}) {
+        if ('chatTemplateArgs' in engine) engine.chatTemplateArgs = { ...engine.chatTemplateArgs, enable_thinking: thinking === true, reasoning_effort: 'medium' };
         tokens = 0;
-        for await (const c of engine.generate(prepare(messages), { maxNewTokens, signal })) {
+        for await (const c of engine.generate(prepare(messages), { maxNewTokens, signal, ...(assistantPrefix != null && { assistantPrefix }) })) {
           if (c.token != null) tokens++;
-          yield { text: c.text, tokens };
+          yield { text: c.text, token: c.token, tokens };
         }
       },
-      dispose: () => engine.dispose?.(),
+      // Drops the KV and recurrent state, for a new conversation or a shorter history.
+      ...(typeof engine.reset === 'function' && { reset: () => engine.reset() }),
+      dispose: () => {
+        spec.disposed = true;
+        if (spec.runner) { engine.model.dflashRunner = null; spec.runner.dispose?.(); }
+        return engine.dispose?.();
+      },
     };
   },
 };

package/src/backends/speculative.js

@@ -0,0 +1,96 @@
+// Speculative decoding for Ternary Bonsai 2 27B: DFlash 2 (a 5-layer drafter) or MTP (one extra decoder layer that
+// drafts ahead). Both are greedy-identical to the engine's own decode; they only change how many tokens one verify
+// pass accepts. The runners live in diskformer (engines/ternary_bonsai_2_dflash.js and ternary_bonsai_2_mtp.js) and
+// attach to the engine through its specDecodeRunner() seam (engine.model.dflashRunner).
+//
+// Their weights are separate downloads, kept in Cache Storage under the names and keys LocalMind uses, so a browser
+// that already holds them from LocalMind reuses them:
+//   'dflash-drafter-v1'  the drafter GGUF (1.14 GB), keyed by DRAFTER_URL
+//   'mtp-head-v1'        the MTP head's byte ranges (451 MB) of a grafted GGUF, keyed by MTP_HEAD.url
+
+// LocalMind's drafter: naklitechie/Qwen3.8-27B-DFlash2-ternary-bonsai2 (index.html DRAFTER_URL, main febbc93).
+export const DRAFTER_URL = 'https://huggingface.co/naklitechie/Qwen3.8-27B-DFlash2-ternary-bonsai2/resolve/main/Qwen3.8-27B-DFlash2-r3-Q4_K_M.gguf';
+export const DRAFTER_CACHE = 'dflash-drafter-v1';
+export const MTP_CACHE = 'mtp-head-v1';
+
+// One file's byte ranges ([[start, length], …], or the whole file) as a Blob, from Cache Storage or downloaded in 32 MB
+// pieces, 8 in flight (Hugging Face's CDN throttles each connection), then stored. A Blob keeps a 1 GB file on disk
+// rather than in the JS heap.
+export async function cachedBlob({ cacheName, url, ranges = null, fileSize = null, fetch = globalThis.fetch, onProgress, signal }) {
+  const cache = await caches.open(cacheName);
+  const want = ranges ? ranges.reduce((n, [, len]) => n + len, 0) : null;
+  const hit = await cache.match(url);
+  if (hit) { const b = await hit.blob(); if (want === null || b.size === want) return b; }
+  if (!ranges) {
+    const probe = await fetch(url, { headers: { Range: 'bytes=0-0' }, signal });
+    if (probe.status !== 206) throw new Error(`${url}: HTTP ${probe.status}; the server must support Range`);
+    fileSize = Number((probe.headers.get('content-range') || '').split('/')[1]) || 0;
+    try { await probe.body?.cancel(); } catch { /* nothing to cancel */ }
+    if (!fileSize) throw new Error(`${url}: unknown size`);
+    ranges = [[0, fileSize]];
+  }
+  const PIECE = 32 << 20, STREAMS = 8;
+  const jobs = [];
+  for (const [start, len] of ranges) for (let o = 0; o < len; o += PIECE) jobs.push([start + o, Math.min(PIECE, len - o)]);
+  const total = jobs.reduce((n, [, len]) => n + len, 0);
+  const pieces = new Array(jobs.length);
+  let next = 0, loaded = 0;
+  const worker = async () => {
+    for (;;) {
+      const i = next++;
+      if (i >= jobs.length) return;
+      const [start, len] = jobs[i];
+      const r = await fetch(url, { headers: { Range: `bytes=${start}-${start + len - 1}` }, signal });
+      if (r.status !== 206) throw new Error(`${url}: HTTP ${r.status} on bytes ${start}+${len}`);
+      if (fileSize && Number((r.headers.get('content-range') || '').split('/')[1]) !== fileSize) throw new Error(`${url}: the file is not ${fileSize} bytes`);
+      const buf = new Uint8Array(await r.arrayBuffer());
+      if (buf.byteLength !== len) throw new Error(`${url}: short read on bytes ${start}+${len}`);
+      pieces[i] = buf;
+      loaded += len;
+      onProgress?.({ loaded, total });
+    }
+  };
+  await Promise.all(Array.from({ length: STREAMS }, worker));
+  const blob = new Blob(pieces, { type: 'application/octet-stream' });
+  // A failed write only costs a download next time.
+  await cache.put(url, new Response(blob, { headers: { 'content-type': 'application/octet-stream', 'content-length': String(blob.size) } })).catch(() => {});
+  return blob;
+}
+
+const blobSource = (blob) => async (off, len) => new Uint8Array(await blob.slice(off, off + len).arrayBuffer());
+
+// Attaches a runner to a loaded Ternary Bonsai 2 engine. spec: { dflash: { block = 5 } } or
+// { mtp: { depth = 2, adaptive = false } }. engines: the diskformer engines/ URL. Returns the runner; its .stats
+// counts cycles, tokens and accepted drafts (MTP adds drafted and plainTokens).
+export async function attachSpeculative(engine, spec, { engines, fetch, onProgress, signal } = {}) {
+  if (spec.dflash) {
+    const { DFlashRunner } = await import(`${engines}/ternary_bonsai_2_dflash.js`);
+    const blob = await cachedBlob({ cacheName: DRAFTER_CACHE, url: DRAFTER_URL, fetch, signal,
+      onProgress: (p) => onProgress?.({ status: 'download', kind: 'dflash', ...p }) });
+    signal?.throwIfAborted();
+    const runner = await DFlashRunner.create(engine, { drafter: blobSource(blob), block: spec.dflash.block ?? 5,
+      onProgress: (p) => onProgress?.({ status: 'load', kind: 'dflash', loaded: p.done, total: p.total }) });
+    engine.model.dflashRunner = runner;
+    return runner;
+  }
+  if (spec.mtp) {
+    const { MTPRunner, MTP_HEAD } = await import(`${engines}/ternary_bonsai_2_mtp.js`);
+    const blob = await cachedBlob({ cacheName: MTP_CACHE, url: MTP_HEAD.url, ranges: MTP_HEAD.ranges, fileSize: MTP_HEAD.fileSize, fetch, signal,
+      onProgress: (p) => onProgress?.({ status: 'download', kind: 'mtp', ...p }) });
+    signal?.throwIfAborted();
+    // Absolute file offsets → offsets in the blob, which holds the ranges back to back.
+    const at = [];
+    let acc = 0;
+    for (const [start, len] of MTP_HEAD.ranges) { at.push([start, len, acc]); acc += len; }
+    const head = async (off, len) => {
+      const r = at.find(([s, l]) => off >= s && off + len <= s + l);
+      if (!r) throw new Error(`MTP head: bytes ${off}+${len} are outside the downloaded ranges`);
+      return blobSource(blob)(r[2] + off - r[0], len);
+    };
+    const runner = await MTPRunner.create(engine, { head, depth: spec.mtp.depth ?? 2, adaptive: !!spec.mtp.adaptive,
+      onProgress: (p) => onProgress?.({ status: 'load', kind: 'mtp', loaded: p.done, total: p.total }) });
+    engine.model.dflashRunner = runner;
+    return runner;
+  }
+  throw new Error('speculative: pass { dflash: { block } } or { mtp: { depth, adaptive } }');
+}

package/src/load.js

@@ -41,16 +41,17 @@ export async function plan(repoOrEntry, opts = {}) {
 
 export async function load(repoOrEntry, options = {}) {
   const [repo, opts] = entry(repoOrEntry, options);
-  const { backends = BACKENDS, maxCtx, dtype, poolBytes, localFile, onProgress, signal } = opts;
+  const { backends = BACKENDS, maxCtx, dtype, poolBytes, localFile, fetch: engineFetch, ple, speculative, onProgress, signal } = opts;
   let p;
   try { p = await plan(repo, opts); } catch (e) { throw wrap(e, 'LOAD_FAILED', signal, `${repo}: `); }
   if (!p.chosen) throw noBackend(p);
   const backend = backends.find((b) => b.id === p.chosen);
   let session;
   try {
-    session = await backend.load(p.model, { budget: p.budget, task: p.task, file: p.file, caps: p.caps, maxCtx, dtype, poolBytes, localFile, onProgress, signal });
+    session = await backend.load(p.model, { budget: p.budget, task: p.task, file: p.file, caps: p.caps, maxCtx, dtype, poolBytes, localFile, fetch: engineFetch, ple, speculative, onProgress, signal });
   } catch (e) { throw wrap(e, 'LOAD_FAILED', signal, `${backend.id}: `); }
-  return { backend: backend.id, task: session.task || p.task || p.model.task || null, ...(session.device && { device: session.device }), plan: p, ...serialize(session) };
+  // Added onto serialize()'s object, not spread: a spread would read its getters once and freeze them.
+  return Object.assign(serialize(session), { backend: backend.id, task: session.task || p.task || p.model.task || null, ...(session.device && { device: session.device }), plan: p });
 }
 
 // content: a string, or an array of { type, … } blocks (text, image, audio) for multimodal models.
@@ -97,12 +98,13 @@ function serialize(session) {
       release();
     }
   };
-  return {
+  const out = {
     ...(generate && { generate }),
     // Where reasoning starts and ends as token ids ({ openId, closeId, inPrompt, preamble, always }), or null.
     ...('think' in session && { think: session.think }),
     ...(session.prime && { prime: queued(session.prime) }),
     ...(session.ensureVision && { ensureVision: queued(session.ensureVision) }),
+    ...(session.reset && { reset: queued(session.reset) }),
     // The one call every task has. For text generation it is the last chunk of generate().
     async run(input, o = {}) {
       if (generate) { let last; for await (const c of generate(input, o)) last = c; return last; }
@@ -124,4 +126,9 @@ function serialize(session) {
       await session.dispose?.();
     },
   };
+  // Read live from the back end's session, never copied: the engine instance it built (diskformer and localmind back
+  // ends; read-only, for token-level work a caller already does with the engine; calls on it bypass the queue),
+  // Gemma 4's PLE mode, and Bonsai's speculative decoding state, which changes after load.
+  for (const k of ['engine', 'pleMode', 'speculative']) if (k in session) Object.defineProperty(out, k, { get: () => session[k], enumerable: true });
+  return out;
 }