anyformer 0.4.11 → 0.4.13

package/CHANGELOG.md

@@ -1,5 +1,43 @@
 # Changelog
 
+## 0.4.13 — 2026-10-30
+
+### Added
+- `diskformer-onnx` back end: ONNX exports that transformers.js has no model class for, run by a diskformer hand
+  engine on onnxruntime-web WebGPU (the engine pins onnxruntime-web 1.30.0). Ranks second, after `transformers`, so
+  transformers.js takes a model over once it adds the class. Streams text, passes `signal` (an abort ends the reply
+  quietly), `thinking` (off by default), `tools` (tool calls stay raw text) and `repetitionPenalty`; image blocks take
+  ImageData, ImageBitmap, a URL or `{ data, width, height }`, audio blocks a 16 kHz Float32Array of at most 30 s.
+  `session.baseUrl` names the URLs its Cache Storage entries ('diskformer-models', listed by `stored()`) are keyed by.
+- `MODELS.chat.LFM3_OMNI_2B`: `LiquidAI/LFM3-Omni-2B-ONNX` (task `any-to-any`, 2.14 GB: decoder_q4f16,
+  vision_encoder_q8, audio_encoder) on `diskformer-onnx`, asked for by LocalMind for Chirag. Browser test `omni`
+  (Chrome, this Mac, 2026-10-30), from local files and from Hugging Face: diskformer's three gate cases through
+  `session.generate()` equal the text Liquid's runner gives on onnxruntime's CPU EP (text 7 tokens, image 32, speech
+  32); the image reply streams in 32 chunks; an abort after 3 chunks ends the reply; a `get_weather` tool list gives a
+  Qwen3-Coder XML tool call.
+
+### Changed
+- diskformer pin 53c976e (v0.6.9) → 48f2c2a (`engines/lfm3_omni.js`; adds files only, no other engine changed).
+- `test/browser/run.mjs --models` serves subfolders (a repo layout), still only inside the named folders.
+
+## 0.4.12 — 2026-10-29
+
+### Changed
+- transformers.js pin 4.3.1 → 4.3.2 (in step with LocalMind), which adds `model_type` `lfm3` and `lfm3_vl`.
+- transformers back end: a graph whose last-position input is named `logits_to_keep` (LiquidAI's LFM3 ONNX export)
+  gets it fed at the session's run; transformers.js 4.3.2 feeds only `num_logits_to_keep` and stopped with "Missing
+  the following inputs". `LiquidAI/LFM3-1.7B-ONNX` now loads and generates through `chat.html` (64 tokens at
+  67 tok/s, Chrome, this Mac, 2026-10-28).
+- transformers back end, text pipeline path: `thinking: false` by default, as on every other back end. The template
+  renders with `enable_thinking` off (through `tokenizer_encode_kwargs`), so LFM3 answers directly; `thinking: true`
+  per request turns it on. Templates without the switch ignore it.
+- `test/browser/chat.html` takes `?dtype=` and fails a reply that contains `<think>` text.
+
+### Added
+- `MODELS.chat.LFM3_1_7B`: `LiquidAI/LFM3-1.7B-ONNX` on `transformers` (asked for by LocalMind for Chirag). Browser
+  chat check on 2026-10-29, Chrome, this Mac: on topic, no `<think>` text, on q4 (anyformer's pick, 1.63 GB; 82.4
+  tok/s) and on q4f16 (`dtype: 'q4f16'`, LocalMind's precision, 1.38 GB; 76.6 tok/s), the same 61-token reply.
+
 ## 0.4.11 — 2026-10-24
 
 ### Added

package/README.md

@@ -54,7 +54,8 @@ back end's answer: yes with an estimate (GPU bytes, disk bytes, tok/s when measu
 
 | Back end | Runs | Models |
 |---|---|---|
-| `transformers` | transformers.js 4.3.1: text generation (Nemotron 3.5 Bolt E4B among them), image and audio chat (Gemma 4, Qwen3.5), embeddings, speech to text, detection, classification, depth, segmentation | ONNX repos in the onnx-community / Xenova layout |
+| `transformers` | transformers.js 4.3.2: text generation (Nemotron 3.5 Bolt E4B and LFM3 1.7B among them), image and audio chat (Gemma 4, Qwen3.5), embeddings, speech to text, detection, classification, depth, segmentation | ONNX repos in the onnx-community / Xenova layout |
+| `diskformer-onnx` | diskformer's hand engines on onnxruntime-web WebGPU, for ONNX models transformers.js has no class for | LFM3-Omni-2B: text, image and speech in, text out |
 | `kohra` | kohra, text-diffusion over onnxruntime-web | MDLM and BD3LM repos tagged `text-diffusion`; Nemotron-Labs-Diffusion-3B |
 | `localmind` | LocalMind's hand-written WebGPU kernels (shipped in diskformer), the whole model on the GPU | any LFM2 / LFM2.5 GGUF; Gemma 4 E2B / E4B QAT-mobile; Ternary Bonsai 2 |
 | `wllama` | llama.cpp in WebAssembly: WebGPU when the budget holds the model, the CPU otherwise (no WebGPU needed) | any GGUF up to about 3.6 GB that no engine above covers; files over 2 GB are split into OPFS shards once |

package/index.js

@@ -14,4 +14,5 @@ export { stored, remove } from './src/store.js';
 export { pipeline } from './src/pipeline.js';
 export { localmindBackend } from './src/backends/localmind.js';
 export { wllamaBackend } from './src/backends/wllama.js';
+export { diskformerOnnxBackend, ONNX_ENGINES } from './src/backends/diskformer-onnx.js';
 export { PINS, SOURCES } from './src/sources.js';

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "anyformer",
-  "version": "0.4.11",
+  "version": "0.4.13",
   "description": "One call to run a model in a browser tab: anyformer reads the Hub repo, probes the browser, and picks the back end (transformers.js, LocalMind's WebGPU kernels, diskformer, kohra) that fits the model's format, architecture and your GPU budget.",
   "type": "module",
   "main": "./index.js",

package/src/backends/diskformer-onnx.js

@@ -0,0 +1,96 @@
+// diskformer-onnx: ONNX exports that transformers.js has no model class for, run by a diskformer hand engine on
+// onnxruntime-web's WebGPU EP. The engine loads from the CDN at load() time, at the commit src/sources.js pins (it
+// pins its own onnxruntime-web and transformers.js tokenizer versions); set diskformerOnnxBackend.source to load
+// another copy. Model files go to Cache Storage 'diskformer-models' on the first load and come from there later.
+//
+// When transformers.js adds a model class for one of these architectures, its entry moves to the transformers back
+// end (which ranks first in BACKENDS) and leaves this table.
+
+import { graphBytes } from './transformers.js';
+import { SOURCES } from '../sources.js';
+
+// Keyed by config.model_type. graphs: the ONNX files the engine reads (each with its _data file); json: the small files.
+export const ONNX_ENGINES = {
+  // LiquidAI LFM3-Omni-2B: text, image and speech in (16 kHz Float32Array, at most 30 s a clip), text out. transformers.js
+  // 4.3.2 has no lfm3_omni class; swap to transformers.js when it adds the model. diskformer's gate
+  // (test/browser/lfm3-omni.html, onnxruntime-web 1.30.0): prompt ids and 32 greedy tokens equal Liquid's runner on
+  // onnxruntime's CPU EP for text, image and speech, 3/3.
+  lfm3_omni: {
+    engine: 'lfm3_omni.js', Engine: 'Lfm3Omni',
+    graphs: ['onnx/decoder_q4f16.onnx', 'onnx/vision_encoder_q8.onnx', 'onnx/audio_encoder.onnx'],
+    json: ['config.json', 'generation_config.json', 'processor_config.json', 'tokenizer.json', 'tokenizer_config.json', 'chat_template.jinja'],
+  },
+};
+
+const missing = (model, e) => {
+  const have = new Set(model.files.all.map((f) => f.name));
+  return [...e.graphs.flatMap((g) => [g, `${g}_data`]), ...e.json].filter((f) => !have.has(f));
+};
+const bytesOf = (model, e) => e.graphs.reduce((a, g) => a + graphBytes(model, g), 0) +
+  model.files.all.filter((f) => e.json.includes(f.name)).reduce((a, f) => a + (f.size || 0), 0);
+
+// An image URL becomes an ImageBitmap (the engine takes ImageData, { data, width, height } or anything drawImage takes).
+const toImage = async (x) => (typeof x === 'string' ? createImageBitmap(await (await fetch(x)).blob()) : x);
+const media = (messages) => Promise.all(messages.map(async (m) => (typeof m.content === 'string' ? m : {
+  ...m, content: await Promise.all(m.content.map(async (b) => (b.type === 'image' ? { ...b, image: await toImage(b.image) } : b))),
+})));
+
+export const diskformerOnnxBackend = {
+  id: 'diskformer-onnx',
+  // LFM3-Omni's repo is tagged any-to-any; the engine writes text only.
+  tasks: ['any-to-any', 'text-generation', 'image-text-to-text'],
+  source: SOURCES.diskformer,
+  // The folder the engine reads the repo layout from; null is Hugging Face at the model's revision. A test points it at
+  // a local server.
+  url: null,
+
+  canRun(model, caps, { task } = {}) {
+    const e = ONNX_ENGINES[model.architecture];
+    if (!e) return { ok: false, code: 'model', reason: `no diskformer ONNX engine for ${model.architecture || 'this architecture'}; engines cover ${Object.keys(ONNX_ENGINES).join(', ')}` };
+    if (!model.formats.includes('onnx')) return { ok: false, code: 'format', reason: 'no ONNX graphs in the repo' };
+    if (task && !diskformerOnnxBackend.tasks.includes(task)) return { ok: false, code: 'task', reason: `task ${task}; this engine writes text (${diskformerOnnxBackend.tasks.join(', ')})` };
+    const gone = missing(model, e);
+    if (gone.length) return { ok: false, code: 'format', reason: `the engine reads ${gone.join(', ')}, which the repo lacks` };
+    if (!caps.webgpu) return { ok: false, code: 'webgpu', reason: 'no WebGPU adapter' };
+    if (!caps.f16) return { ok: false, code: 'f16', reason: 'the GPU lacks shader-f16 (the q4f16 decoder runs fp16 activations)' };
+    const bytes = bytesOf(model, e);
+    return { ok: true, estimate: { gpuBytes: bytes, diskBytes: bytes, tokPerSec: null } };
+  },
+
+  async load(model, { onProgress, signal }) {
+    const e = ONNX_ENGINES[model.architecture];
+    const mod = await import(`${diskformerOnnxBackend.source}/${e.engine}`);
+    const baseUrl = diskformerOnnxBackend.url ? diskformerOnnxBackend.url(model) : `https://huggingface.co/${model.repo}/resolve/${model.revision || 'main'}/`;
+    const engine = await mod[e.Engine].load({ baseUrl, onProgress: onProgress && ((p) => onProgress({ backend: 'diskformer-onnx', ...p })) });
+    if (signal?.aborted) { await engine.dispose(); signal.throwIfAborted(); }
+    return {
+      task: 'text-generation',
+      engine,
+      // Where the files came from: the engine's Cache Storage entries are keyed by these URLs
+      // (s.engine.constructor.cached({ baseUrl }) / deleteCache({ baseUrl })).
+      baseUrl,
+      // Content is a string or blocks: { type: 'text', text }, { type: 'image', image } (ImageData, ImageBitmap, a URL,
+      // or { data, width, height }) and { type: 'audio', audio } (16 kHz mono Float32Array). thinking: false (the
+      // default, as on every back end here) answers directly. tools: the chat template's tool list; tool calls stay
+      // raw text in the reply (LFM3 writes Qwen3-Coder XML), for the caller to parse. An abort ends the reply quietly.
+      async *generate(messages, { maxNewTokens = 512, signal, thinking = false, tools, repetitionPenalty } = {}) {
+        const queue = [];
+        let wake = () => {}, done = false, failed = null, text = '', tokens = 0;
+        const msgs = await media(messages);
+        engine.generate(msgs, {
+          max_new_tokens: maxNewTokens, do_sample: false, enable_thinking: thinking === true, signal,
+          ...(tools && { tools }), ...(repetitionPenalty && { repetition_penalty: repetitionPenalty }),
+          onToken: () => { tokens++; },
+          onText: (delta) => { text += delta; queue.push({ text, tokens }); wake(); },
+        }).catch((err) => { failed = err; }).finally(() => { done = true; wake(); });
+        while (true) {
+          if (queue.length) { yield queue.shift(); continue; }
+          if (done) break;
+          await new Promise((r) => { wake = r; });
+        }
+        if (failed) throw failed;
+      },
+      dispose: () => engine.dispose(),
+    };
+  },
+};

package/src/backends/transformers.js

@@ -120,16 +120,33 @@ export const transformersBackend = {
     const gen = await lib.pipeline('text-generation', model.repo, {
       device: 'webgpu', dtype, revision: model.revision || 'main', progress_callback,
     });
+    feedLogitsToKeep(lib, gen.model);
     signal?.throwIfAborted();
     return {
       task: 'text-generation',
-      generate: (messages, { maxNewTokens = 256, repetitionPenalty, signal } = {}) => stream(lib, gen.tokenizer, signal,
-        (streamer, stopping_criteria) => gen(messages, { max_new_tokens: maxNewTokens, do_sample: false, ...(repetitionPenalty && { repetition_penalty: repetitionPenalty }), streamer, stopping_criteria })),
+      // thinking: false (the default, as on every back end here) renders the template with enable_thinking off, so a
+      // model that reasons by default (LFM3) answers directly; templates without the switch ignore it.
+      generate: (messages, { maxNewTokens = 256, repetitionPenalty, signal, thinking = false } = {}) => stream(lib, gen.tokenizer, signal,
+        (streamer, stopping_criteria) => gen(messages, { max_new_tokens: maxNewTokens, do_sample: false, ...(repetitionPenalty && { repetition_penalty: repetitionPenalty }), tokenizer_encode_kwargs: { enable_thinking: thinking === true }, streamer, stopping_criteria })),
       dispose: () => gen.dispose?.(),
     };
   },
 };
 
+// LiquidAI's LFM3 ONNX export names its last-position input logits_to_keep (shape [1]); transformers.js 4.3.2 feeds
+// only num_logits_to_keep and stops with "Missing the following inputs: logits_to_keep". Hide the input from
+// transformers.js's check and add it at the session's run, set to 1: generate() reads only the last position's logits.
+// LocalMind carries the same shim (index.html feedLogitsToKeep).
+function feedLogitsToKeep(lib, net) {
+  const sess = net?.sessions?.model;
+  if (!sess?.inputNames.includes('logits_to_keep')) return;
+  const names = sess.inputNames.filter((n) => n !== 'logits_to_keep');
+  const one = new lib.Tensor('int64', [1n], [1]).ort_tensor;
+  const run = sess.run.bind(sess);
+  Object.defineProperty(sess, 'inputNames', { value: names });
+  sess.run = (feeds, opts) => run({ ...feeds, logits_to_keep: one }, opts);
+}
+
 // A non-generation task: one pipeline, run(input, opts) calls it. ImageData (from a canvas) becomes a RawImage; URLs,
 // RawImage and, for speech, 16 kHz Float32Array samples pass through as transformers.js takes them.
 async function loadPipeline(lib, model, task, { progress_callback, signal }) {

package/src/load.js

@@ -7,12 +7,14 @@ import { diskformerBackend } from './backends/diskformer.js';
 import { diskformerLlamaBackend } from './backends/diskformer-llama.js';
 import { localmindBackend } from './backends/localmind.js';
 import { wllamaBackend } from './backends/wllama.js';
+import { diskformerOnnxBackend } from './backends/diskformer-onnx.js';
 import { AnyformerError, wrap } from './errors.js';
 
 // Back ends land here as each one passes its browser test (test/browser/). The list order breaks ties: a
 // diskformer hand engine carries a measured speed and outranks diskformer-llama for the same model; diskformer-llama
-// comes before wllama because it pages experts instead of holding the whole model on the GPU.
-export const BACKENDS = [transformersBackend, kohraBackend, diskformerBackend, localmindBackend, diskformerLlamaBackend, wllamaBackend];
+// comes before wllama because it pages experts instead of holding the whole model on the GPU. diskformer-onnx runs only
+// the ONNX architectures transformers.js has no class for, so transformers.js wins once it adds one.
+export const BACKENDS = [transformersBackend, diskformerOnnxBackend, kohraBackend, diskformerBackend, localmindBackend, diskformerLlamaBackend, wllamaBackend];
 
 // 2.5 GB: the smallest budget diskformer's chat gate passes at. Browsers do not report GPU memory, so a caller that
 // knows more passes { budget }.
@@ -128,7 +130,8 @@ function serialize(session) {
   };
   // Read live from the back end's session, never copied: the engine instance it built (diskformer and localmind back
   // ends; read-only, for token-level work a caller already does with the engine; calls on it bypass the queue),
-  // Gemma 4's PLE mode, and Bonsai's speculative decoding state, which changes after load.
-  for (const k of ['engine', 'pleMode', 'speculative']) if (k in session) Object.defineProperty(out, k, { get: () => session[k], enumerable: true });
+  // Gemma 4's PLE mode, Bonsai's speculative decoding state, which changes after load, and diskformer-onnx's baseUrl
+  // (its Cache Storage keys).
+  for (const k of ['engine', 'pleMode', 'speculative', 'baseUrl']) if (k in session) Object.defineProperty(out, k, { get: () => session[k], enumerable: true });
   return out;
 }

package/src/models.js

@@ -20,6 +20,10 @@ const MINICPM5_2B = M('openbmb/MiniCPM5-2B-GGUF', 'wllama', 'text-generation', '
 const QWEN3_0_6B_MDLM = M('naklitechie/Qwen3-0.6B-diffusion-mdlm-ONNX', 'kohra', 'text-generation', 'kohra', { downloadBytes: 1.54e9, diffusion: true });
 const NEMOTRON_DIFFUSION_3B = M('naklitechie/Nemotron-Labs-Diffusion-3B-ONNX', 'kohra', 'text-generation', 'kohra', { downloadBytes: 2.77e9, diffusion: true, experimental: true });
 const GEMMA4_26B_A4B = M('google/gemma-4-26B-A4B-it-qat-q4_0-gguf', 'diskformer', 'text-generation', 'diskformer-chat', { budget: 2.5e9, downloadBytes: 14.44e9 });
+const LFM3_1_7B = M('LiquidAI/LFM3-1.7B-ONNX', 'transformers', 'text-generation', 'chat', { downloadBytes: 1.63e9 });
+// transformers.js 4.3.2 has no lfm3_omni class, so diskformer's hand engine runs it; swap the entry to transformers.js
+// when it adds the model.
+const LFM3_OMNI_2B = M('LiquidAI/LFM3-Omni-2B-ONNX', 'diskformer-onnx', 'any-to-any', 'omni', { downloadBytes: 2.14e9 });
 const NEMOTRON_3_5_BOLT_E4B = M('danyalmalik/NVIDIA-Nemotron-3.5-Bolt-E4B-ONNX-WebGPU', 'transformers', 'text-generation', 'transformers', { budget: 5e9, downloadBytes: 4.94e9 });
 // A tech demo, not a gated model: 231 GB on disk (4 IQ1_M parts plus the 110.6 GB Engram side file), about 0.4 tok/s.
 // diskformer checked one hand-written prompt through complete(); generate() renders the GGUF's chat template, which no
@@ -54,6 +58,10 @@ export const MODELS = {
     GEMMA4_26B_A4B,
     /** NVIDIA's Nemotron 3.5 Bolt E4B (dense, text only), a community q4f16 ONNX export in Gemma 4 E4B's graph. */
     NEMOTRON_3_5_BOLT_E4B,
+    /** LiquidAI's LFM3 1.7B (dense), LiquidAI's own ONNX export; thinking off unless a request asks for it. */
+    LFM3_1_7B,
+    /** LiquidAI's LFM3-Omni-2B: text, image and speech in (16 kHz, at most 30 s a clip), text out (diskformer's engine). */
+    LFM3_OMNI_2B,
     /** DeepSeek V4.1 Flash (763B MoE), a tech demo on diskformer's llama.cpp path: 231 GB on disk, very slow. */
     DEEPSEEK_V4_1_FLASH,
   },

package/src/sources.js

@@ -4,17 +4,19 @@
 // transformers.js and wllama move only by hand, in step with LocalMind's pins (plan/history.md, 2026-10-08).
 
 export const PINS = {
-  transformers: '4.3.1',
+  transformers: '4.3.2', // 4.3.2 adds model_type lfm3 (LFM3 1.7B)
   wllama: '3.4.1',      // LocalMind's pin; its stream-drain workaround (src/backends/wllama.js) is for this version
   // v0.3.1: seeded block-KV graphs (Nemotron-Labs-Diffusion-3B), chatFormat, split external data; download progress
   // (progress_callback) and the graph files cached in Cache Storage 'kohra-models'.
   kohra: 'bf95d44c3e3cfc001201baa03fa121c2bf9c7d73',
+  // 48f2c2a (main after v0.6.9): engines/lfm3_omni.js, LFM3-Omni-2B on onnxruntime-web 1.30.0 (text, image and speech
+  // in; text out; files in Cache Storage 'diskformer-models'). Adds files only: no other engine or ggml/ changed.
   // v0.6.9: llama.load({ extra }) mounts side files beside the GGUF (DeepSeek V4.1 Flash's Engram tables).
   // v0.6.8: every engine (the disk-tier MoE engines, Ling 3.0 Flash, and LocalMind's whole-model engines LFM2.5,
   // Gemma 4 E2B/E4B and Ternary Bonsai 2) under engines/, and the llama.cpp path under ggml/, which loads split GGUFs
   // and drafters, and reads past 2 GiB (Nemotron 3.5 Bolt E4B needs v0.6.8 or later). The hand engines' ingest
   // streams the GGUF header with progress and resumes a dropped range.
-  diskformer: '53c976e43d34ac2d2601a731124de81cb07fdab4',
+  diskformer: '48f2c2a2f5bf59b16db8f3d57653e7c44705ce6c',
 };
 
 const gh = (repo, sha, path) => `https://cdn.jsdelivr.net/gh/NakliTechie/${repo}@${sha}/${path}`;

package/src/store.js

@@ -3,7 +3,8 @@
 // inventoryModelCaches() (index.html), which covers the same engines' stores.
 //   Cache Storage   'transformers-cache' (transformers.js; kohra's tokenizer too); 'gguf-v1', 'gguf-v1-headers' and
 //                   'bonsai-pipeline-v1' (LocalMind's engines; 'gguf-v1' holds the LFM2
-//                   engine), 'kohra-models' (kohra v0.3.1+: its ONNX graph and data files): one entry per file URL
+//                   engine), 'kohra-models' (kohra v0.3.1+: its ONNX graph and data files), 'diskformer-models' (diskformer's
+//                   ONNX engines, LFM3-Omni): one entry per file URL
 //                   (…/<repo>/resolve/<rev>/<file>). Older kohra left its graph in the HTTP cache, which a page cannot list.
 //   IndexedDB       'gguf-cache-v1' (Bonsai 2) and 'safetensors-cache-v1' (Gemma 4 QAT-mobile): a store keyed by URL
 //                   with { size } and a chunk store keyed [url, start, end]; 'webgpu-prefix-snapshots' (Bonsai 2's
@@ -18,7 +19,7 @@ import { DISKFORMER_MODELS } from './backends/diskformer.js';
 import { SPLIT_ROOT } from './gguf-split.js';
 import { readLlamaIndex, LLAMA_INDEX } from './backends/diskformer-llama.js';
 
-const CACHE_BACKEND = { 'transformers-cache': 'transformers', 'gguf-v1': 'localmind', 'gguf-v1-headers': 'localmind', 'bonsai-pipeline-v1': 'localmind', 'kohra-models': 'kohra' };
+const CACHE_BACKEND = { 'transformers-cache': 'transformers', 'gguf-v1': 'localmind', 'gguf-v1-headers': 'localmind', 'bonsai-pipeline-v1': 'localmind', 'kohra-models': 'kohra', 'diskformer-models': 'diskformer-onnx' };
 const IDB_STORES = ['gguf-cache-v1', 'safetensors-cache-v1'];
 const BONSAI2 = 'prism-ml/Ternary-Bonsai-2-27B-gguf';