anyformer 0.4.13 → 0.4.15

package/CHANGELOG.md

@@ -1,5 +1,39 @@
 # Changelog
 
+## 0.4.15 — 2026-10-30
+
+### Fixed
+- A caller that stops reading a reply early (a `break` out of `for await`, as after a tool call) without aborting
+  now stops the reply on `diskformer-onnx` and `transformers`: the back end aborts the engine call (a linked
+  AbortController for diskformer-onnx, the stopping criteria's interrupt for transformers.js) and waits for it before
+  the next generate() starts. Before, the engine kept generating, and the next reply ran on the same onnxruntime
+  sessions beside it: "failed to call OrtRun() … Failed to download data from buffer" or a hang of over 10 minutes in
+  Chrome (reported by NakliOS, root-caused by LocalMind). `diskformer-llama` already did this.
+
+### Changed
+- diskformer pin 1fe19f8 → 7ffbea5 (v0.6.13: LFM3-Omni serves one reply at a time; changes only
+  `engines/lfm3_omni.js`).
+- Browser tests `omni` and `omni-speech` break out of a reply (text, and spoken at its first speech chunk) without
+  aborting, then require the next reply to equal the text reference within 10 s. Passed 2026-10-30 (Chrome, this Mac,
+  local files), with `transformers` and the LFM3 1.7B `chat` check.
+
+## 0.4.14 — 2026-10-30
+
+### Added
+- `diskformer-onnx`: LFM3-Omni speech out. `load(MODELS.chat.LFM3_OMNI_2B, { speech: true })` also fetches the depth
+  and codec decoders (515 MB, Cache Storage 'diskformer-models'); `generate(messages, { speech: true })` answers in
+  interleaved text and speech, and a chunk with new speech carries `audio` (24 kHz Float32 PCM since the last chunk)
+  and `sampleRate`. `seed`, `audioTemperature`, `audioRepetitionPenalty` and `audioRepetitionWindow` pass through.
+  Catalog flag `audioOut: true` on `MODELS.chat.LFM3_OMNI_2B`.
+- Browser test `omni-speech` (Chrome, this Mac, 2026-10-30, local files): the reference speech clip in, a spoken reply
+  that names Mount Everest, 13.5 s of 24 kHz speech in 15 chunks over 9.3 s (first sound after 1.1 s), RMS 0.037
+  (the reference's 0.048); an abort from the first speech chunk ended the reply after 0.9 s of speech; a text reply
+  after it still equals the text reference. Code-level exactness is diskformer's own gate.
+
+### Changed
+- diskformer pin 48f2c2a → 1fe19f8 (v0.6.12, LFM3-Omni speech out; changes only `engines/lfm3_omni.js`). The `omni`
+  test passes again on it (3/3 equal to the reference, streaming, abort, tool call).
+
 ## 0.4.13 — 2026-10-30
 
 ### Added

package/README.md

@@ -55,7 +55,7 @@ back end's answer: yes with an estimate (GPU bytes, disk bytes, tok/s when measu
 | Back end | Runs | Models |
 |---|---|---|
 | `transformers` | transformers.js 4.3.2: text generation (Nemotron 3.5 Bolt E4B and LFM3 1.7B among them), image and audio chat (Gemma 4, Qwen3.5), embeddings, speech to text, detection, classification, depth, segmentation | ONNX repos in the onnx-community / Xenova layout |
-| `diskformer-onnx` | diskformer's hand engines on onnxruntime-web WebGPU, for ONNX models transformers.js has no class for | LFM3-Omni-2B: text, image and speech in, text out |
+| `diskformer-onnx` | diskformer's hand engines on onnxruntime-web WebGPU, for ONNX models transformers.js has no class for | LFM3-Omni-2B: text, image and speech in; text or text and speech out |
 | `kohra` | kohra, text-diffusion over onnxruntime-web | MDLM and BD3LM repos tagged `text-diffusion`; Nemotron-Labs-Diffusion-3B |
 | `localmind` | LocalMind's hand-written WebGPU kernels (shipped in diskformer), the whole model on the GPU | any LFM2 / LFM2.5 GGUF; Gemma 4 E2B / E4B QAT-mobile; Ternary Bonsai 2 |
 | `wllama` | llama.cpp in WebAssembly: WebGPU when the budget holds the model, the CPU otherwise (no WebGPU needed) | any GGUF up to about 3.6 GB that no engine above covers; files over 2 GB are split into OPFS shards once |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "anyformer",
-  "version": "0.4.13",
+  "version": "0.4.15",
   "description": "One call to run a model in a browser tab: anyformer reads the Hub repo, probes the browser, and picks the back end (transformers.js, LocalMind's WebGPU kernels, diskformer, kohra) that fits the model's format, architecture and your GPU budget.",
   "type": "module",
   "main": "./index.js",

package/src/backends/diskformer-onnx.js

@@ -11,7 +11,9 @@ import { SOURCES } from '../sources.js';
 
 // Keyed by config.model_type. graphs: the ONNX files the engine reads (each with its _data file); json: the small files.
 export const ONNX_ENGINES = {
-  // LiquidAI LFM3-Omni-2B: text, image and speech in (16 kHz Float32Array, at most 30 s a clip), text out. transformers.js
+  // LiquidAI LFM3-Omni-2B: text, image and speech in (16 kHz Float32Array, at most 30 s a clip); text out, or text and
+  // speech interleaved (24 kHz, from the depth and codec decoders, 515 MB more, fetched on the first spoken reply or by
+  // load({ speech: true })). transformers.js
   // 4.3.2 has no lfm3_omni class; swap to transformers.js when it adds the model. diskformer's gate
   // (test/browser/lfm3-omni.html, onnxruntime-web 1.30.0): prompt ids and 32 greedy tokens equal Liquid's runner on
   // onnxruntime's CPU EP for text, image and speech, 3/3.
@@ -57,11 +59,12 @@ export const diskformerOnnxBackend = {
     return { ok: true, estimate: { gpuBytes: bytes, diskBytes: bytes, tokPerSec: null } };
   },
 
-  async load(model, { onProgress, signal }) {
+  // speech: true also fetches the speech-out graphs now, so the first spoken reply does not wait for them.
+  async load(model, { speech = false, onProgress, signal }) {
     const e = ONNX_ENGINES[model.architecture];
     const mod = await import(`${diskformerOnnxBackend.source}/${e.engine}`);
     const baseUrl = diskformerOnnxBackend.url ? diskformerOnnxBackend.url(model) : `https://huggingface.co/${model.repo}/resolve/${model.revision || 'main'}/`;
-    const engine = await mod[e.Engine].load({ baseUrl, onProgress: onProgress && ((p) => onProgress({ backend: 'diskformer-onnx', ...p })) });
+    const engine = await mod[e.Engine].load({ baseUrl, ...(speech && { speech: true }), onProgress: onProgress && ((p) => onProgress({ backend: 'diskformer-onnx', ...p })) });
     if (signal?.aborted) { await engine.dispose(); signal.throwIfAborted(); }
     return {
       task: 'text-generation',
@@ -73,22 +76,47 @@ export const diskformerOnnxBackend = {
       // or { data, width, height }) and { type: 'audio', audio } (16 kHz mono Float32Array). thinking: false (the
       // default, as on every back end here) answers directly. tools: the chat template's tool list; tool calls stay
       // raw text in the reply (LFM3 writes Qwen3-Coder XML), for the caller to parse. An abort ends the reply quietly.
-      async *generate(messages, { maxNewTokens = 512, signal, thinking = false, tools, repetitionPenalty } = {}) {
+      // speech: true answers in interleaved text and speech (the engine adds its speech system turn when the messages
+      // have none, and thinks never): a chunk that carries new speech has `audio`, the 24 kHz Float32 PCM since the last
+      // chunk, and `sampleRate`; maxNewTokens then counts text tokens and speech frames (80 ms each) together. Speech is
+      // sampled (generation_config.json: temperature 0.4, repetition penalty 1.2 over 64 frames); audioTemperature,
+      // audioRepetitionPenalty, audioRepetitionWindow and seed override it.
+      async *generate(messages, { maxNewTokens = 512, signal, thinking = false, tools, repetitionPenalty, speech = false, seed,
+        audioTemperature, audioRepetitionPenalty, audioRepetitionWindow } = {}) {
         const queue = [];
         let wake = () => {}, done = false, failed = null, text = '', tokens = 0;
         const msgs = await media(messages);
-        engine.generate(msgs, {
-          max_new_tokens: maxNewTokens, do_sample: false, enable_thinking: thinking === true, signal,
-          ...(tools && { tools }), ...(repetitionPenalty && { repetition_penalty: repetitionPenalty }),
+        const sampleRate = engine.speech?.sampling_rate ?? 24000;
+        // The engine reads this signal, linked to the caller's, so a caller that stops reading can end the reply too.
+        const stop = new AbortController();
+        const abort = () => stop.abort();
+        if (signal?.aborted) abort(); else signal?.addEventListener('abort', abort, { once: true });
+        const job = engine.generate(msgs, {
+          max_new_tokens: maxNewTokens, do_sample: false, enable_thinking: thinking === true, signal: stop.signal,
+          ...(tools && { tools }), ...(repetitionPenalty && { repetition_penalty: repetitionPenalty }), ...(seed !== undefined && { seed }),
+          ...(speech && {
+            output_modalities: ['text', 'audio'],
+            onAudio: (pcm) => { queue.push({ text, tokens, audio: pcm, sampleRate }); wake(); },
+            ...(audioTemperature !== undefined && { audio_temperature: audioTemperature }),
+            ...(audioRepetitionPenalty !== undefined && { audio_repetition_penalty: audioRepetitionPenalty }),
+            ...(audioRepetitionWindow !== undefined && { audio_repetition_window: audioRepetitionWindow }),
+          }),
           onToken: () => { tokens++; },
           onText: (delta) => { text += delta; queue.push({ text, tokens }); wake(); },
         }).catch((err) => { failed = err; }).finally(() => { done = true; wake(); });
-        while (true) {
-          if (queue.length) { yield queue.shift(); continue; }
-          if (done) break;
-          await new Promise((r) => { wake = r; });
+        try {
+          while (true) {
+            if (queue.length) { yield queue.shift(); continue; }
+            if (done) break;
+            await new Promise((r) => { wake = r; });
+          }
+          if (failed) throw failed;
+        } finally {
+          signal?.removeEventListener('abort', abort);
+          // The caller stopped reading early (a break after a tool call): stop the reply, so the next call on this
+          // engine's sessions does not run beside it or queue behind it.
+          if (!done) { stop.abort(); await job; }
         }
-        if (failed) throw failed;
       },
       dispose: () => engine.dispose(),
     };

package/src/backends/transformers.js

@@ -177,14 +177,21 @@ async function* stream(lib, tokenizer, signal, run) {
     token_callback_function: () => { tokens++; },
   });
   const stop = new lib.InterruptableStoppingCriteria();
-  signal?.addEventListener('abort', () => stop.interrupt(), { once: true });
-  run(streamer, stop).catch((e) => { failed = e; }).finally(() => { done = true; wake(); });
-  while (true) {
-    if (queue.length) { yield queue.shift(); continue; }
-    if (done) break;
-    await new Promise((r) => { wake = r; });
+  const interrupt = () => stop.interrupt();
+  signal?.addEventListener('abort', interrupt, { once: true });
+  const job = run(streamer, stop).catch((e) => { failed = e; }).finally(() => { done = true; wake(); });
+  try {
+    while (true) {
+      if (queue.length) { yield queue.shift(); continue; }
+      if (done) break;
+      await new Promise((r) => { wake = r; });
+    }
+    if (failed) throw failed;
+  } finally {
+    signal?.removeEventListener('abort', interrupt);
+    // The caller stopped reading early: stop the reply so the next call on this model does not run beside it.
+    if (!done) { stop.interrupt(); await job; }
   }
-  if (failed) throw failed;
 }
 
 // A text-only model in the multimodal layout: the causal-LM class over embed_tokens + decoder_model_merged.

package/src/load.js

@@ -43,14 +43,14 @@ export async function plan(repoOrEntry, opts = {}) {
 
 export async function load(repoOrEntry, options = {}) {
   const [repo, opts] = entry(repoOrEntry, options);
-  const { backends = BACKENDS, maxCtx, dtype, poolBytes, localFile, fetch: engineFetch, ple, speculative, onProgress, signal } = opts;
+  const { backends = BACKENDS, maxCtx, dtype, poolBytes, localFile, fetch: engineFetch, ple, speculative, speech, onProgress, signal } = opts;
   let p;
   try { p = await plan(repo, opts); } catch (e) { throw wrap(e, 'LOAD_FAILED', signal, `${repo}: `); }
   if (!p.chosen) throw noBackend(p);
   const backend = backends.find((b) => b.id === p.chosen);
   let session;
   try {
-    session = await backend.load(p.model, { budget: p.budget, task: p.task, file: p.file, caps: p.caps, maxCtx, dtype, poolBytes, localFile, fetch: engineFetch, ple, speculative, onProgress, signal });
+    session = await backend.load(p.model, { budget: p.budget, task: p.task, file: p.file, caps: p.caps, maxCtx, dtype, poolBytes, localFile, fetch: engineFetch, ple, speculative, speech, onProgress, signal });
   } catch (e) { throw wrap(e, 'LOAD_FAILED', signal, `${backend.id}: `); }
   // Added onto serialize()'s object, not spread: a spread would read its getters once and freeze them.
   return Object.assign(serialize(session), { backend: backend.id, task: session.task || p.task || p.model.task || null, ...(session.device && { device: session.device }), plan: p });

package/src/models.js

@@ -2,7 +2,8 @@
 // repo id; it carries the task and, where the back end needs one, the budget. DEFAULT is the smallest download that
 // passed. `test` names the page under test/browser/ that proves the entry; test/models.test.js checks it exists.
 // `diffusion: true` marks a text-diffusion model (the reply denoises in blocks); `experimental: true` marks an entry its
-// app shows as opt-in until that app's own checks pass.
+// app shows as opt-in until that app's own checks pass; `audioOut: true` marks a model that can also answer in speech
+// (generate({ speech: true })).
 
 const M = (repo, backend, task, test, extra = {}) => ({ repo, backend, task, test, ...extra });
 
@@ -23,7 +24,7 @@ const GEMMA4_26B_A4B = M('google/gemma-4-26B-A4B-it-qat-q4_0-gguf', 'diskformer'
 const LFM3_1_7B = M('LiquidAI/LFM3-1.7B-ONNX', 'transformers', 'text-generation', 'chat', { downloadBytes: 1.63e9 });
 // transformers.js 4.3.2 has no lfm3_omni class, so diskformer's hand engine runs it; swap the entry to transformers.js
 // when it adds the model.
-const LFM3_OMNI_2B = M('LiquidAI/LFM3-Omni-2B-ONNX', 'diskformer-onnx', 'any-to-any', 'omni', { downloadBytes: 2.14e9 });
+const LFM3_OMNI_2B = M('LiquidAI/LFM3-Omni-2B-ONNX', 'diskformer-onnx', 'any-to-any', 'omni', { downloadBytes: 2.14e9, audioOut: true });
 const NEMOTRON_3_5_BOLT_E4B = M('danyalmalik/NVIDIA-Nemotron-3.5-Bolt-E4B-ONNX-WebGPU', 'transformers', 'text-generation', 'transformers', { budget: 5e9, downloadBytes: 4.94e9 });
 // A tech demo, not a gated model: 231 GB on disk (4 IQ1_M parts plus the 110.6 GB Engram side file), about 0.4 tok/s.
 // diskformer checked one hand-written prompt through complete(); generate() renders the GGUF's chat template, which no
@@ -60,7 +61,7 @@ export const MODELS = {
     NEMOTRON_3_5_BOLT_E4B,
     /** LiquidAI's LFM3 1.7B (dense), LiquidAI's own ONNX export; thinking off unless a request asks for it. */
     LFM3_1_7B,
-    /** LiquidAI's LFM3-Omni-2B: text, image and speech in (16 kHz, at most 30 s a clip), text out (diskformer's engine). */
+    /** LiquidAI's LFM3-Omni-2B: text, image and speech in (16 kHz, at most 30 s a clip); text or text and speech out (diskformer's engine). */
     LFM3_OMNI_2B,
     /** DeepSeek V4.1 Flash (763B MoE), a tech demo on diskformer's llama.cpp path: 231 GB on disk, very slow. */
     DEEPSEEK_V4_1_FLASH,

package/src/sources.js

@@ -9,6 +9,9 @@ export const PINS = {
   // v0.3.1: seeded block-KV graphs (Nemotron-Labs-Diffusion-3B), chatFormat, split external data; download progress
   // (progress_callback) and the graph files cached in Cache Storage 'kohra-models'.
   kohra: 'bf95d44c3e3cfc001201baa03fa121c2bf9c7d73',
+  // v0.6.13 (7ffbea5): LFM3-Omni serves one reply at a time; overlapping runs broke onnxruntime-web WebGPU.
+  // v0.6.12 (1fe19f8): LFM3-Omni speech out (interleaved text and 24 kHz speech; depth and codec decoders). Changes
+  // only engines/lfm3_omni.js; its text, image and speech-in gate (3/3) is unchanged.
   // 48f2c2a (main after v0.6.9): engines/lfm3_omni.js, LFM3-Omni-2B on onnxruntime-web 1.30.0 (text, image and speech
   // in; text out; files in Cache Storage 'diskformer-models'). Adds files only: no other engine or ggml/ changed.
   // v0.6.9: llama.load({ extra }) mounts side files beside the GGUF (DeepSeek V4.1 Flash's Engram tables).
@@ -16,7 +19,7 @@ export const PINS = {
   // Gemma 4 E2B/E4B and Ternary Bonsai 2) under engines/, and the llama.cpp path under ggml/, which loads split GGUFs
   // and drafters, and reads past 2 GiB (Nemotron 3.5 Bolt E4B needs v0.6.8 or later). The hand engines' ingest
   // streams the GGUF header with progress and resumes a dropped range.
-  diskformer: '48f2c2a2f5bf59b16db8f3d57653e7c44705ce6c',
+  diskformer: '7ffbea54a5f09494e1f6c970cce7c4c7b1b8460d',
 };
 
 const gh = (repo, sha, path) => `https://cdn.jsdelivr.net/gh/NakliTechie/${repo}@${sha}/${path}`;