@tanstack-ai-browser/wllama 0.0.0-stage → 0.2.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 daveycodez
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,152 @@
-# Temporary Holding Version
+# @tanstack-ai-browser/wllama
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+A [TanStack AI](https://tanstack.com/ai) adapter for GGUF models that run in the browser, with [wllama](https://github.com/ngxson/wllama): llama.cpp compiled to WebAssembly, in a Web Worker. It runs on WebGPU where the browser has it, and on the CPU elsewhere, so it runs in any browser with WebAssembly SIMD: Firefox on Android and iPhone Safari included. It runs the official [`@wllama/wllama`](https://www.npmjs.com/package/@wllama/wllama) package (3.13.0) as published, through its public API and options, with its own WebAssembly builds: no patched, vendored or rebuilt copy of wllama or llama.cpp. A wllama or llama.cpp bug is fixed here when a wllama release with the fix is out.
+
+```sh
+npm install @tanstack-ai-browser/wllama @tanstack/ai
+```
+
+```ts
+import { chat } from "@tanstack/ai"
+import { wllamaText } from "@tanstack-ai-browser/wllama"
+
+const stream = chat({
+  adapter: wllamaText("LiquidAI/LFM2.5-1.2B-Instruct-GGUF"),
+  messages,
+  threadId
+})
+```
+
+## Setup
+
+- **Any bundler:** install and go. wllama's WebAssembly file loads from jsDelivr by default, at the installed wllama's version, and Safari's compat build too (wllama's own default); nothing is emitted into your build. To host them yourself, see [Self-hosting the WebAssembly](#self-hosting-the-webassembly).
+- **Storage:** wllama downloads models to the browser's Origin Private File System. Nothing to set up.
+- **Threads (optional):** serve the site cross-origin isolated (`Cross-Origin-Opener-Policy: same-origin` and `Cross-Origin-Embedder-Policy: require-corp`) and wllama runs on half the cores; otherwise on one. Set the headers on your host, and in development (below). With `require-corp`, cross-origin images and other resources load only with CORS (`<img crossOrigin="anonymous">`) or a `Cross-Origin-Resource-Policy` header (jsDelivr sends both).
+
+```ts
+// vite.config.ts
+const headers = {
+  "Cross-Origin-Opener-Policy": "same-origin",
+  "Cross-Origin-Embedder-Policy": "require-corp"
+}
+export default defineConfig({ server: { headers }, preview: { headers } })
+```
+
+```ts
+// next.config.ts
+const nextConfig: NextConfig = {
+  async headers() {
+    return [
+      {
+        source: "/(.*)",
+        headers: [
+          { key: "Cross-Origin-Opener-Policy", value: "same-origin" },
+          { key: "Cross-Origin-Embedder-Policy", value: "require-corp" }
+        ]
+      }
+    ]
+  }
+}
+```
+
+## Exports
+
+The package is shaped like `@tanstack/ai-openai`'s text adapter:
+
+- `@tanstack-ai-browser/wllama`: `wllamaText(model, config?)`, `WllamaTextAdapter` (with the runtime methods `availability`, `create` and `destroy`) and the types `WllamaConfig`, `WllamaModel`, `WllamaTextModelOptions`, `Availability` and `DownloadProgressCallback`.
+- The catalog, from the same root: `WLLAMA_CHAT_MODELS`, `WLLAMA_MODEL_META` and `WLLAMA_MODEL_REASONING`, with the types `WllamaChatModel`, `WllamaModelMeta` and `WllamaModelReasoningByName`.
+
+## Models
+
+`WLLAMA_CHAT_MODELS` lists the models this package knows, and `WLLAMA_MODEL_META` has each one's details: Hugging Face repository, GGUF file, capabilities, reasoning levels and the sampling from its card. They're Liquid AI's official GGUF builds, and Unsloth's build of Qwen3.5 2B.
+
+Any other GGUF works too: name its file, as `"owner/repo/file.gguf"` on Hugging Face (its latest version) or as a URL. An unlisted model gets llama.cpp's default sampling and the chat template's `enable_thinking` switch for `chat({ reasoning })`.
+
+| Model                                | Reasoning                       | File                                    |
+| ------------------------------------ | ------------------------------- | --------------------------------------- |
+| `LiquidAI/LFM3-1.7B-GGUF`            | on or off (`"high"` or `"off"`) | Q4_K_M, 1.10 GB                         |
+| `LiquidAI/LFM3-VL-2B-GGUF`           | on or off (`"high"` or `"off"`) | Q4_K_M, 1.10 GB, and a 203 MB projector |
+| `LiquidAI/LFM2.5-VL-3B-GGUF`         | none                            | Q4_K_M, 1.67 GB, and a 583 MB projector |
+| `LiquidAI/LFM2.5-VL-1.6B-GGUF`       | none                            | Q4_K_M, 731 MB, and a 583 MB projector  |
+| `LiquidAI/LFM2.5-VL-450M-GGUF`       | none                            | Q4_K_M, 229 MB, and a 103 MB projector  |
+| `LiquidAI/LFM3-Omni-2B-GGUF`         | none                            | Q4_K_M, 1.10 GB, and a 507 MB projector |
+| `LiquidAI/LFM3-Audio-2B-GGUF`        | none                            | Q4_K_M, 1.10 GB, and a 304 MB projector |
+| `LiquidAI/LFM2.5-Audio-1.5B-GGUF`    | none                            | Q4_0, 696 MB, and a 220 MB projector    |
+| `LiquidAI/LFM2.5-1.2B-Instruct-GGUF` | none                            | Q4_K_M, 731 MB                          |
+| `LiquidAI/LFM2.5-1.2B-Thinking-GGUF` | always (`"high"`)               | Q4_K_M, 731 MB                          |
+| `LiquidAI/LFM2.5-350M-GGUF`          | none                            | Q4_K_M, 229 MB                          |
+| `unsloth/Qwen3.5-2B-GGUF`            | on or off (`"high"` or `"off"`) | Q4_K_M, 1.28 GB                         |
+
+Liquid's models are Q4_K_M: on a Pixel 10 Pro in Firefox, LFM2.5 350M read prompts at 55 tokens/s in Q4_K_M, 40 in Q8_0 and 15 in Q4_0 with wllama's own build. In Chrome on an M3 Ultra, Qwen3.5 2B on WebGPU read about 850 prompt tokens/s and wrote about 60 tokens/s (with other work on the machine).
+
+wllama answers llama.cpp's OpenAI-compatible Chat Completions API in the page, so the adapter is TanStack's Chat Completions adapter (`@tanstack/openai-base`) with the loaded model as its client: messages, tools, structured output, the stream and usage are TanStack's and llama.cpp's. It adds loading the model, the chat template's thinking switch and llama.cpp's option names.
+
+**Tools** work as with any adapter: pass them to `chat({ tools })`, MCP tools included. llama.cpp writes them into the prompt with the model's chat template and parses the model's calls back.
+
+LFM3 needs wllama 3.13 or later: its llama.cpp is the first published with LFM3's architecture.
+
+**Images:** Liquid's vision models (LFM3-VL 2B and LFM2.5-VL 3B, 1.6B and 450M) read images: send image parts in a user message, as data (base64) or URLs the page can fetch. Each loads with its vision projector, a second GGUF file in the same repository (the Q8_0 one), and the adapter's `inputModalities` has images for them. Another GGUF loads as text only.
+
+**Audio:** LFM3-Omni 2B (which reads images too), LFM3-Audio 2B and LFM2.5-Audio 1.5B read audio: send audio parts in a user message, as data (base64) or URLs the page can fetch, and the adapter's `inputModalities` has audio for them. They transcribe a recording (when asked, or with the system prompt `Perform ASR.`) and answer questions about it, in text: their speech output isn't used. llama.cpp decodes WAV, MP3 and FLAC itself; other audio (Opus in WebM or Ogg, AAC in MP4) is decoded in the page with Web Audio and goes as WAV, so it needs a browser. Each loads with its projector, the audio encoder (Q8_0, or Q4_0 for LFM2.5-Audio, whose repository has no Q4_K_M model either). The LFM3 ones call tools; LFM2.5-Audio doesn't, and told of tools it answers that none can hear a recording, so send it none. LFM3-Audio reasons when it chooses to: its chat template has no switch.
+
+**Reasoning:** thinking (llama.cpp's `reasoning_content`) streams as reasoning events. LFM3 thinks by default; `chat({ reasoning: "off" })` turns it off, through its chat template's `enable_thinking` (llama.cpp's `chat_template_kwargs`). Qwen3.5 2B doesn't think unless asked; `chat({ reasoning: "high" })` turns it on the same way. LFM2.5 1.2B Thinking always thinks. `chat({ reasoning })` is a type error for the models that don't reason.
+
+**Options** go in `modelOptions` per call: llama.cpp's own request fields, sent as given, as TanStack's adapters do with their provider's fields. wllama answers with llama.cpp's OpenAI-compatible server code, so these are OpenAI's names and llama.cpp's: `max_tokens`, `temperature`, `top_p`, `top_k`, `min_p`, `repeat_penalty`, `frequency_penalty`, `presence_penalty`, `seed`, `stop`, `logit_bias`, `parallel_tool_calls` (false constrains the model to one tool call; left out, llama.cpp allows several where the chat template supports them) and `chat_template_kwargs`; llama.cpp's other fields go through too. Sampling left out comes from the model card (`WLLAMA_MODEL_META[model].sampling`, in the same names). Without `max_tokens`, a reply may run to the context limit, as llama.cpp's default.
+
+**Structured output:** `chat({ outputSchema })` sends the schema as given in `response_format`, and llama.cpp constrains the answer to it with its JSON Schema grammar. With `stream: true` it streams as TanStack's standard events.
+
+**Context:** `wllamaText(model, { contextLength })` sets how many tokens a chat may hold, prompt and reply together. By default it's the model's whole context window, read from its GGUF (262,144 for LFM3 and Qwen3.5 2B, 128,000 for LFM2.5). Unlike Transformers.js and the WebGL engine, which grow their attention memory as a chat fills it, llama.cpp reserves it for the whole context when the model loads, which can be GBs for a full window. Fine on a desktop; on an iPhone set 8,192. On the CPU, where that memory is WebAssembly's (4 GB at most), set it too: a model whose whole window doesn't fit there (Qwen3.5 2B's 262,144 tokens take 3 GB) fails to load with an error. Changing it loads the model again.
+
+**Usage** is in each call's `RUN_FINISHED` event, as llama.cpp reports it: `promptTokens`, `completionTokens`, `totalTokens`, and `promptTokensDetails.cachedTokens` for the start of the prompt llama.cpp reused from an earlier call. Timings aren't in it: measure them from the stream, as for any adapter.
+
+**Prompt reuse:** use the standard TanStack AI option, for example
+`chat({ adapter, messages, threadId, promptCache: "none" })` or
+`promptCache: { retention: "short", key: "conversation-key" }`. No adapter-specific
+cache option is needed. Structured output passes its exact schema through native
+`response_format.json_schema`, separately from the conversation; schemas are not
+inserted into caller/system messages.
+
+`"none"` maps internally to llama.cpp's public `cache_prompt: false`. Omitted, `"short"`,
+and `"long"` policies preserve native runtime defaults and introduce no TTL. The standard
+key has no advertised native slot/session equivalent, so the adapter does not invent one.
+Actual reuse comes from llama.cpp's reported `cached_tokens`, forwarded as
+`promptTokensDetails.cachedTokens`; enabling caching alone does not prove a hit.
+
+Reuse lasts as long as the loaded model: llama.cpp keeps what it has read in memory, and
+a reload of the page or the model reads the prompt again.
+
+**Downloads:** each adapter has the runtime methods of this project's adapters, with the Prompt API's values:
+
+- `adapter.availability()` resolves to `"unavailable"` where wllama can't run (it needs WebAssembly, Web Workers and the Origin Private File System; not on a server), `"downloading"` while this page downloads the model, `"available"` when the model is loaded or its whole GGUF file is in wllama's storage, else `"downloadable"`. It lists wllama's storage only (its `ModelManager`, which reads the Origin Private File System): no worker, no WebAssembly, no network.
+- `adapter.create({ onDownloadProgress?, signal }?)` downloads and loads the model ahead of time, with the download's progress, 0 to 1, then 1 once it's ready (at once if it already is). Aborting `signal` stops that call's wait (it rejects at once); the load goes on, and `destroy()` stops it. It resolves to the adapter, and rejects on failure; it's fine to leave unawaited. The config's `onDownloadProgress` hears a download the first `chat()` starts too.
+- `adapter.destroy()` unloads the model (stopping its download), if it's the one loaded; its file stays in storage.
+
+The page holds one wllama model at a time, loaded with the settings of the adapter that asked for it (`contextLength`, `gpu`, `threads`, the WebAssembly URLs): adapters with the same settings share it, another adapter's settings load it again, and `destroy()` on one unloads it for all of them.
+
+## How it runs
+
+`chat()` runs tools and the agent loop; each model call is one chat completion from llama.cpp's OpenAI-compatible API (`createChatCompletion`), in wllama's worker. llama.cpp writes the prompt with the model's chat template, reads thinking and tool calls back, and reuses whatever start of the prompt it has already read. wllama's defaults apply, except:
+
+- **Context:** the model's own training context (see above), not wllama's 1,024, which a system prompt with tools outgrows.
+- **WebGPU or CPU:** wllama's default: on WebGPU where the browser has it and wllama can drive it (Chrome and Edge with JSPI, Safari through wllama's compat build), on the CPU elsewhere (Firefox, which has no JSPI, and browsers without WebGPU). `wllamaText(model, { gpu: false })` runs on the CPU only. llama.cpp's WebGPU and CPU code are wllama 3.13.0's, unchanged.
+- **Full sliding-window cache:** `swa_full: true`, so a model with sliding-window layers reuses what it has read after a tool call instead of rereading the whole prompt. It costs a little memory, and nothing for models without such layers.
+- **Threads:** wllama's default, half the cores, where the page is cross-origin isolated (`Cross-Origin-Opener-Policy: same-origin` with `Cross-Origin-Embedder-Policy: require-corp`; Safari and Firefox on Android ignore `credentialless`); one elsewhere. `wllamaText(model, { threads })` sets a number.
+
+The GGUF file is saved to the Origin Private File System by wllama, and loads from there afterwards, offline too.
+
+### Self-hosting the WebAssembly
+
+By default wllama loads its WebAssembly file from jsDelivr, at the version this package depends on (`https://cdn.jsdelivr.net/npm/@wllama/wllama@3.13.0/esm/wasm/wllama.wasm`), as wllama loads its compat build for Safari. To serve them with your site (offline use, no third-party request), copy `@wllama/wllama/esm/wasm/wllama.wasm` and, for Safari, `@wllama/wllama-compat`'s `wasm/wllama.wasm` and `wasm/wllama.js` (same version), and pass their URLs. The file must match the installed wllama's JavaScript, so copy it from the same version.
+
+```ts
+wllamaText("LiquidAI/LFM2.5-350M-GGUF", {
+  wasmUrl: "/wllama.wasm",
+  compat: { wasm: "/wllama-compat.wasm", worker: "/wllama-compat.js" }
+})
+```
+
+`compat` goes to wllama's `setCompat`.
+
+**Status:** early (0.x); the API may change. See the [repository README](https://github.com/daveycodez/tanstack-ai-browser#readme).
+
+Thinking-capable models accept `reasoning: { level: "high", budgetTokens: 512 }` through TanStack AI. The adapter forwards an explicit budget through wllama's public `reasoning_budget_tokens` option, alongside the template's thinking switch. Omitting the budget preserves the runtime default; `level: "off"` ignores a valid budget on models that support switching thinking off. Zero is forwarded unchanged. Negative, fractional, non-finite, and unsafe budgets fail before inference.

package/dist/adapter.d.ts

@@ -0,0 +1,166 @@
+/**
+ * A TanStack AI text adapter for GGUF models that run in the browser with
+ * wllama (llama.cpp in WebAssembly). wllama answers llama.cpp's
+ * OpenAI-compatible Chat Completions API in the page, so this is TanStack's
+ * Chat Completions adapter with the loaded model as its client: messages,
+ * tools, structured output and the stream (text, thinking, tool calls) are
+ * TanStack's, and llama.cpp writes the prompt with the model's chat template
+ * and reads its thinking and tool calls back. This file adds what's wllama's
+ * own: loading the model with the adapter's settings, the chat template's
+ * thinking switch, the model card's sampling, and audio parts for the
+ * models that read them.
+ */
+import type { TextOptions } from "@tanstack/ai";
+import { OpenAIBaseChatCompletionsTextAdapter } from "@tanstack/openai-base";
+import type { OpenAIClientLike } from "@tanstack/openai-base";
+import type { WllamaCompat } from "@wllama/wllama/esm/index.js";
+import type { Availability, DownloadProgressCallback } from "./client.js";
+import type { WllamaChatModel, WllamaModel, WllamaModelReasoningByName } from "./catalog.js";
+export type WllamaConfig = {
+    /**
+     * Most tokens a chat may hold, prompt and reply together: the model's
+     * whole context window by default (its GGUF's training context), never
+     * more. llama.cpp reserves the attention cache for all of it when the
+     * model loads (LFM3: ~12 KB a token, so 3 GB for its 262,144), so set
+     * less where memory is short: 8,192 on an iPhone. Changing it loads the
+     * model again.
+     */
+    contextLength?: number;
+    /**
+     * Where wllama's WebAssembly file is served. By default, the installed
+     * `@wllama/wllama`'s own file on jsDelivr, at its version; serve
+     * `@wllama/wllama/esm/wasm/wllama.wasm` with the site and pass its URL to
+     * host it yourself.
+     */
+    wasmUrl?: string;
+    /**
+     * Where wllama's compat build (for Safari: `@wllama/wllama-compat`'s
+     * `wasm/wllama.wasm` and `wasm/wllama.js`) is served, passed to
+     * `Wllama.setCompat`. By default, wllama's: the same version on jsDelivr.
+     */
+    compat?: WllamaCompat;
+    /** CPU threads. wllama's default otherwise: half the cores. */
+    threads?: number;
+    /**
+     * `false` to run on the CPU only. By default wllama runs on WebGPU where
+     * it can, and on the CPU elsewhere.
+     */
+    gpu?: boolean;
+    /** wllama's and llama.cpp's logs, in the console. */
+    debug?: boolean;
+    /**
+     * The model's download progress, 0 to 1, when a call (`chat()`, say)
+     * starts its download; then 1 once it's ready. To load it ahead of time
+     * with progress, see `create`.
+     */
+    onDownloadProgress?: DownloadProgressCallback;
+};
+/**
+ * Per-call `modelOptions`: llama.cpp's own request fields, sent as given
+ * (wllama answers with llama.cpp's OpenAI-compatible server code), over the
+ * model card's sampling. These are the common ones; llama.cpp's others go
+ * through too.
+ */
+export type WllamaTextModelOptions = {
+    /** Most tokens to generate, thinking included. llama.cpp's default otherwise: up to the context limit. */
+    max_tokens?: number;
+    /** Randomness of sampling: 0 is greedy. */
+    temperature?: number;
+    /** Nucleus sampling: the likeliest tokens up to this probability, in (0, 1]. */
+    top_p?: number;
+    /** Sampling from the likeliest `top_k` tokens only (0: all of them). */
+    top_k?: number;
+    /** Drops tokens less likely than this share of the likeliest one's odds. */
+    min_p?: number;
+    /** Penalizes tokens already in the chat (1: off). */
+    repeat_penalty?: number;
+    /** Lowers the odds of tokens by how often they've appeared. */
+    frequency_penalty?: number;
+    /** Lowers the odds of tokens that have appeared at all. */
+    presence_penalty?: number;
+    /** Seeds the sampler, for the same samples from the same request. */
+    seed?: number;
+    /** Where to stop: the reply ends before the first of these it would write. */
+    stop?: string | Array<string>;
+    /** Adds to the odds of tokens, by token id. */
+    logit_bias?: Record<string, number>;
+    /**
+     * False constrains the model to one tool call. True or left out: as many
+     * as the model writes, when its chat template supports several.
+     */
+    parallel_tool_calls?: boolean;
+    /** The chat template's own variables; `chat({ reasoning })` sets its `enable_thinking`. */
+    chat_template_kwargs?: Record<string, unknown>;
+};
+/** A model's reasoning levels: a listed model's, none for a listed one that doesn't reason, else the switch. */
+type ResolveReasoning<TModel extends string> = TModel extends keyof WllamaModelReasoningByName ? WllamaModelReasoningByName[TModel] : TModel extends WllamaChatModel ? never : {
+    levels: "off" | "high";
+    budget: true;
+};
+type MetadataByModality = {
+    text: unknown;
+    image: unknown;
+    audio: unknown;
+    video: unknown;
+    document: unknown;
+};
+/** The base adapter's conversion of a message part to a Chat Completions one. */
+type ConvertContentPart = OpenAIBaseChatCompletionsTextAdapter<string>["convertContentPart"];
+/** A Chat Completions request, as the base adapter writes it. */
+type CreateParams = Parameters<OpenAIClientLike["chat"]["completions"]["create"]>[0];
+export declare class WllamaTextAdapter<TModel extends WllamaModel> extends OpenAIBaseChatCompletionsTextAdapter<TModel, WllamaTextModelOptions, ReadonlyArray<"text" | "image" | "audio">, MetadataByModality, ReadonlyArray<string>, ResolveReasoning<TModel>> {
+    readonly name = "wllama";
+    readonly api = "wllama";
+    /** Text, and images or audio for a listed model whose projector reads them. */
+    readonly inputModalities: ReadonlyArray<"text" | "image" | "audio">;
+    private readonly settings;
+    private readonly onDownloadProgress;
+    constructor(config: WllamaConfig, model: TModel);
+    /**
+     * Where the model stands in this browser, with the Prompt API's values:
+     * "available" once it's in wllama's storage (or loaded), "downloading"
+     * while this page downloads it, else "downloadable"; "unavailable" where
+     * wllama can't run.
+     */
+    availability(): Promise<Availability>;
+    /**
+     * Downloads the model if it isn't stored and loads it with this adapter's
+     * settings, with the download's progress, 0 to 1, then 1 once it's ready.
+     * Resolves to this adapter; rejects if it can't load (fine to leave
+     * unawaited). `chat()` doesn't need it: its first call loads the model.
+     */
+    create(options?: {
+        onDownloadProgress?: DownloadProgressCallback;
+        signal?: AbortSignal;
+    }): Promise<this>;
+    /** Unloads the model, freeing its memory. It stays in wllama's storage. */
+    destroy(): Promise<void>;
+    /** The request, with `modelOptions` as given over the model card's sampling, and the chat template's thinking switch. */
+    protected mapOptionsToRequest(options: TextOptions): CreateParams & {
+        stream: true;
+    };
+    /** llama.cpp streams thinking as `reasoning_content`. */
+    protected extractReasoning(chunk: unknown): {
+        text: string;
+    } | undefined;
+    /**
+     * An audio part by its URL, as an image goes: Chat Completions takes
+     * audio only as base64 in two formats, and the client reads the bytes of
+     * either kind from a URL (a data URL for base64).
+     */
+    protected convertContentPart(part: Parameters<ConvertContentPart>[0]): ReturnType<ConvertContentPart>;
+    /** Schemas as given: llama.cpp's grammar takes any JSON Schema, without OpenAI's strict-mode rewrite. */
+    protected makeStructuredOutputCompatible(schema: Record<string, any>): Record<string, any>;
+}
+/**
+ * A TanStack AI text adapter for a GGUF model that wllama runs in the
+ * browser.
+ *
+ * @example
+ * ```ts
+ * chat({ adapter: wllamaText("unsloth/Qwen3.5-2B-GGUF"), messages, threadId })
+ * ```
+ */
+export declare function wllamaText<TModel extends WllamaModel>(model: TModel, config?: WllamaConfig): WllamaTextAdapter<TModel>;
+export {};
+//# sourceMappingURL=adapter.d.ts.map

package/dist/adapter.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"adapter.d.ts","sourceRoot":"","sources":["../src/adapter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AACH,OAAO,KAAK,EAAoB,WAAW,EAAE,MAAM,cAAc,CAAA;AAEjE,OAAO,EAAE,oCAAoC,EAAE,MAAM,uBAAuB,CAAA;AAC5E,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,uBAAuB,CAAA;AAC7D,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,6BAA6B,CAAA;AAI/D,OAAO,KAAK,EAAE,YAAY,EAAE,wBAAwB,EAAgB,MAAM,aAAa,CAAA;AAEvF,OAAO,KAAK,EAAY,eAAe,EAAE,WAAW,EAAE,0BAA0B,EAAE,MAAM,cAAc,CAAA;AAEtG,MAAM,MAAM,YAAY,GAAG;IACzB;;;;;;;OAOG;IACH,aAAa,CAAC,EAAE,MAAM,CAAA;IACtB;;;;;OAKG;IACH,OAAO,CAAC,EAAE,MAAM,CAAA;IAChB;;;;OAIG;IACH,MAAM,CAAC,EAAE,YAAY,CAAA;IACrB,+DAA+D;IAC/D,OAAO,CAAC,EAAE,MAAM,CAAA;IAChB;;;OAGG;IACH,GAAG,CAAC,EAAE,OAAO,CAAA;IACb,qDAAqD;IACrD,KAAK,CAAC,EAAE,OAAO,CAAA;IACf;;;;OAIG;IACH,kBAAkB,CAAC,EAAE,wBAAwB,CAAA;CAC9C,CAAA;AAED;;;;;GAKG;AACH,MAAM,MAAM,sBAAsB,GAAG;IACnC,0GAA0G;IAC1G,UAAU,CAAC,EAAE,MAAM,CAAA;IACnB,2CAA2C;IAC3C,WAAW,CAAC,EAAE,MAAM,CAAA;IACpB,gFAAgF;IAChF,KAAK,CAAC,EAAE,MAAM,CAAA;IACd,wEAAwE;IACxE,KAAK,CAAC,EAAE,MAAM,CAAA;IACd,4EAA4E;IAC5E,KAAK,CAAC,EAAE,MAAM,CAAA;IACd,qDAAqD;IACrD,cAAc,CAAC,EAAE,MAAM,CAAA;IACvB,+DAA+D;IAC/D,iBAAiB,CAAC,EAAE,MAAM,CAAA;IAC1B,2DAA2D;IAC3D,gBAAgB,CAAC,EAAE,MAAM,CAAA;IACzB,qEAAqE;IACrE,IAAI,CAAC,EAAE,MAAM,CAAA;IACb,8EAA8E;IAC9E,IAAI,CAAC,EAAE,MAAM,GAAG,KAAK,CAAC,MAAM,CAAC,CAAA;IAC7B,+CAA+C;IAC/C,UAAU,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAA;IACnC;;;OAGG;IACH,mBAAmB,CAAC,EAAE,OAAO,CAAA;IAC7B,2FAA2F;IAC3F,oBAAoB,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAA;CAC/C,CAAA;AAED,gHAAgH;AAChH,KAAK,gBAAgB,CAAC,MAAM,SAAS,MAAM,IAAI,MAAM,SAAS,MAAM,0BAA0B,GAC1F,0BAA0B,CAAC,MAAM,CAAC,GAClC,MAAM,SAAS,eAAe,GAC5B,KAAK,GACL;IAAE,MAAM,EAAE,KAAK,GAAG,MAAM,CAAC;IAAC,MAAM,EAAE,IAAI,CAAA;CAAE,CAAA;AAQ9C,KAAK,kBAAkB,GAAG;IAAE,IAAI,EAAE,OAAO,CAAC;IAAC,KAAK,EAAE,OAAO,CAAC;IAAC,KAAK,EAAE,OAAO,CAAC;IAAC,KAAK,EAAE,OAAO,CAAC;IAAC,QAAQ,EAAE,OAAO,CAAA;CAAE,CAAA;AAE9G,iFAAiF;AACjF,KAAK,kBAAkB,GAAG,oCAAoC,CAAC,MAAM,CAAC,CAAC,oBAAoB,CAAC,CAAA;AAE5F,iEAAiE;AACjE,KAAK,YAAY,GAAG,UAAU,CAAC,gBAAgB,CAAC,MAAM,CAAC,CAAC,aAAa,CAAC,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAA;AA6FpF,qBAAa,iBAAiB,CAAC,MAAM,SAAS,WAAW,CAAE,SAAQ,oCAAoC,CACrG,MAAM,EACN,sBAAsB,EACtB,aAAa,CAAC,MAAM,GAAG,OAAO,GAAG,OAAO,CAAC,EACzC,kBAAkB,EAClB,aAAa,CAAC,MAAM,CAAC,EACrB,gBAAgB,CAAC,MAAM,CAAC,CACzB;IACC,SAAkB,IAAI,YAAW;IACjC,SAAkB,GAAG,YAAW;IAChC,+EAA+E;IAC/E,SAAkB,eAAe,EAAE,aAAa,CAAC,MAAM,GAAG,OAAO,GAAG,OAAO,CAAC,CAAA;IAE5E,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAc;IACvC,OAAO,CAAC,QAAQ,CAAC,kBAAkB,CAAsC;IAEzE,YAAY,MAAM,EAAE,YAAY,EAAE,KAAK,EAAE,MAAM,EAM9C;IAED;;;;;OAKG;IACH,YAAY,IAAI,OAAO,CAAC,YAAY,CAAC,CAEpC;IAED;;;;;OAKG;IACH,MAAM,CAAC,OAAO,GAAE;QAAE,kBAAkB,CAAC,EAAE,wBAAwB,CAAC;QAAC,MAAM,CAAC,EAAE,WAAW,CAAA;KAAO,GAAG,OAAO,CAAC,IAAI,CAAC,CAY3G;IAED,2EAA2E;IACrE,OAAO,IAAI,OAAO,CAAC,IAAI,CAAC,CAE7B;IAED,yHAAyH;IACzH,UAAmB,mBAAmB,CAAC,OAAO,EAAE,WAAW,GAAG,YAAY,GAAG;QAAE,MAAM,EAAE,IAAI,CAAA;KAAE,CAS5F;IAED,yDAAyD;IACzD,UAAmB,gBAAgB,CAAC,KAAK,EAAE,OAAO;;kBAGjD;IAED;;;;OAIG;IACH,UAAmB,kBAAkB,CAAC,IAAI,EAAE,UAAU,CAAC,kBAAkB,CAAC,CAAC,CAAC,CAAC,GAAG,UAAU,CAAC,kBAAkB,CAAC,CAW7G;IAED,yGAAyG;IACzG,UAAmB,8BAA8B,CAAC,MAAM,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,uBAE5E;CACF;AAED;;;;;;;;GAQG;AACH,wBAAgB,UAAU,CAAC,MAAM,SAAS,WAAW,EACnD,KAAK,EAAE,MAAM,EACb,MAAM,GAAE,YAAiB,GACxB,iBAAiB,CAAC,MAAM,CAAC,CAE3B"}

package/dist/adapter.js

@@ -0,0 +1,183 @@
+import { resolveReasoning } from "@tanstack/ai/adapter-internals";
+import { OpenAIBaseChatCompletionsTextAdapter } from "@tanstack/openai-base";
+import { llamaAudio } from "./audio.js";
+import { availability, loadModel, unloadModel } from "./client.js";
+import { listed, modelMeta } from "./catalog.js";
+/** The switch for a model the catalog doesn't list: the chat template's `enable_thinking`. */
+const UNLISTED_REASONING = { levels: { off: "false", high: "true" }, budget: true };
+/** A model's reasoning levels: the catalog's, or the switch for one it doesn't list. */
+const reasoningOf = (model) => (listed(model) ? modelMeta(model).reasoning : UNLISTED_REASONING);
+/**
+ * The chat template's `enable_thinking` for a request, through llama.cpp's
+ * `chat_template_kwargs`, and a thinking budget: left out when the model has
+ * no levels or the request doesn't ask, so the template's default applies.
+ */
+function thinkingKwargs(request, model, kwargs) {
+    const value = resolveReasoning(request, reasoningOf(model))?.value;
+    if (value === undefined)
+        return {};
+    const thinking = value === "true";
+    const budget = request?.budgetTokens;
+    return {
+        chat_template_kwargs: { ...kwargs, enable_thinking: thinking },
+        ...(thinking && budget !== undefined && { reasoning_budget_tokens: budget })
+    };
+}
+/**
+ * The card's sampling for a request: its thinking settings while the chat
+ * template thinks (asked for, or a model that always thinks), else its
+ * own. None for a model the catalog doesn't list: llama.cpp's defaults.
+ */
+function cardSampling(request, model) {
+    if (!listed(model))
+        return {};
+    const meta = modelMeta(model);
+    const value = resolveReasoning(request, meta.reasoning)?.value;
+    const thinking = value !== undefined ? value === "true" : !!meta.reasoning && !("off" in meta.reasoning.levels);
+    return (thinking && meta.thinkingSampling) || meta.sampling;
+}
+/** What a URL holds: a data URL's bytes, or those the page fetches. */
+const bytesAt = async (url, signal) => (await fetch(url, { ...(signal && { signal }) })).arrayBuffer();
+/**
+ * A request's messages with each image and audio clip as its bytes: wllama
+ * reads them as `{ type: "image", data }` and `{ type: "audio", data }`,
+ * where the request has an `image_url` or an `audio_url` (a data URL here,
+ * or one the page can fetch). Audio llama.cpp can't decode goes as WAV.
+ */
+async function withMediaBytes(messages, signal) {
+    if (!Array.isArray(messages))
+        return messages;
+    return Promise.all(messages.map(async (message) => {
+        if (!Array.isArray(message.content))
+            return message;
+        const content = await Promise.all(message.content.map(async (part) => {
+            if (part.type === "image_url" && part.image_url) {
+                return { type: "image", data: await bytesAt(part.image_url.url, signal) };
+            }
+            if (part.type === "audio_url" && part.audio_url) {
+                return { type: "audio", data: await llamaAudio(await bytesAt(part.audio_url.url, signal)) };
+            }
+            return part;
+        }));
+        return { ...message, content };
+    }));
+}
+/**
+ * The model as an OpenAI client: each request loads it with the adapter's
+ * settings (at once when it's loaded with them) and goes to wllama's
+ * `createChatCompletion`.
+ */
+function clientFor(model, settings, onDownloadProgress) {
+    const create = async (body, options) => {
+        const signal = options?.signal ?? undefined;
+        const wllama = await loadModel(model, settings, { ...(onDownloadProgress && { onDownloadProgress }), ...(signal && { signal }) });
+        const messages = await withMediaBytes(body.messages, signal);
+        return wllama.createChatCompletion({ ...body, messages, ...(signal && { abortSignal: signal }) });
+    };
+    return { baseURL: "wllama:", chat: { completions: { create } } };
+}
+/** The settings a model loads with, from the adapter's config. */
+const loadSettings = ({ contextLength, wasmUrl, compat, threads, gpu, debug }) => ({
+    ...(contextLength !== undefined && { contextLength }),
+    ...(wasmUrl && { wasmUrl }),
+    ...(compat && { compat }),
+    ...(threads && { threads }),
+    ...(gpu !== undefined && { gpu }),
+    ...(debug && { debug })
+});
+export class WllamaTextAdapter extends OpenAIBaseChatCompletionsTextAdapter {
+    name = "wllama";
+    api = "wllama";
+    /** Text, and images or audio for a listed model whose projector reads them. */
+    inputModalities;
+    settings;
+    onDownloadProgress;
+    constructor(config, model) {
+        const settings = loadSettings(config);
+        super(model, "wllama", clientFor(model, settings, config.onDownloadProgress), { strictFallbackWarning: false });
+        this.settings = settings;
+        this.onDownloadProgress = config.onDownloadProgress;
+        this.inputModalities = listed(model) ? modelMeta(model).supports.input : ["text"];
+    }
+    /**
+     * Where the model stands in this browser, with the Prompt API's values:
+     * "available" once it's in wllama's storage (or loaded), "downloading"
+     * while this page downloads it, else "downloadable"; "unavailable" where
+     * wllama can't run.
+     */
+    availability() {
+        return availability(this.model);
+    }
+    /**
+     * Downloads the model if it isn't stored and loads it with this adapter's
+     * settings, with the download's progress, 0 to 1, then 1 once it's ready.
+     * Resolves to this adapter; rejects if it can't load (fine to leave
+     * unawaited). `chat()` doesn't need it: its first call loads the model.
+     */
+    create(options = {}) {
+        const own = this.onDownloadProgress;
+        const theirs = options.onDownloadProgress;
+        const ready = loadModel(this.model, this.settings, {
+            onDownloadProgress: (progress) => {
+                own?.(progress);
+                theirs?.(progress);
+            },
+            ...(options.signal && { signal: options.signal })
+        }).then(() => this);
+        ready.catch(() => undefined);
+        return ready;
+    }
+    /** Unloads the model, freeing its memory. It stays in wllama's storage. */
+    async destroy() {
+        await unloadModel(this.model);
+    }
+    /** The request, with `modelOptions` as given over the model card's sampling, and the chat template's thinking switch. */
+    mapOptionsToRequest(options) {
+        const modelOptions = options.modelOptions;
+        const settings = {
+            ...cardSampling(options.reasoning, this.model),
+            ...modelOptions,
+            ...(options.promptCache?.retention === "none" && { cache_prompt: false })
+        };
+        const request = super.mapOptionsToRequest({ ...options, modelOptions: settings, reasoning: undefined });
+        return Object.assign(request, thinkingKwargs(options.reasoning, this.model, modelOptions?.chat_template_kwargs));
+    }
+    /** llama.cpp streams thinking as `reasoning_content`. */
+    extractReasoning(chunk) {
+        const text = chunk.choices?.[0]?.delta?.reasoning_content;
+        return text ? { text } : undefined;
+    }
+    /**
+     * An audio part by its URL, as an image goes: Chat Completions takes
+     * audio only as base64 in two formats, and the client reads the bytes of
+     * either kind from a URL (a data URL for base64).
+     */
+    convertContentPart(part) {
+        if (part.type !== "audio")
+            return super.convertContentPart(part);
+        const { source } = part;
+        if (source.type === "file") {
+            throw new Error("wllama takes audio as data or URLs, not as uploaded files.");
+        }
+        const url = source.type === "data" && !source.value.startsWith("data:")
+            ? `data:${source.mimeType || "application/octet-stream"};base64,${source.value}`
+            : source.value;
+        return { type: "audio_url", audio_url: { url } };
+    }
+    /** Schemas as given: llama.cpp's grammar takes any JSON Schema, without OpenAI's strict-mode rewrite. */
+    makeStructuredOutputCompatible(schema) {
+        return schema;
+    }
+}
+/**
+ * A TanStack AI text adapter for a GGUF model that wllama runs in the
+ * browser.
+ *
+ * @example
+ * ```ts
+ * chat({ adapter: wllamaText("unsloth/Qwen3.5-2B-GGUF"), messages, threadId })
+ * ```
+ */
+export function wllamaText(model, config = {}) {
+    return new WllamaTextAdapter(config, model);
+}

package/dist/audio.d.ts

@@ -0,0 +1,16 @@
+/**
+ * Audio as llama.cpp reads it. Its multimodal library decodes WAV, MP3 and
+ * FLAC files itself, and resamples them to the model's rate. Anything else
+ * a browser records or plays (Opus in WebM or Ogg, AAC in MP4) is decoded
+ * here, with Web Audio, and handed over as WAV.
+ */
+/** Whether `data` is a WAV, FLAC or MP3 file, by how it starts: the files llama.cpp decodes. */
+export declare function decodesInLlama(data: ArrayBuffer): boolean;
+/** Mono samples, each from -1 to 1, as a 16-bit PCM WAV file. */
+export declare function wavFile(samples: Float32Array, sampleRate: number): ArrayBuffer;
+/**
+ * An audio file's bytes as llama.cpp can decode them: as they are for WAV,
+ * MP3 and FLAC, else decoded by the browser and written as mono WAV.
+ */
+export declare function llamaAudio(data: ArrayBuffer): Promise<ArrayBuffer>;
+//# sourceMappingURL=audio.d.ts.map

package/dist/audio.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"audio.d.ts","sourceRoot":"","sources":["../src/audio.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AASH,gGAAgG;AAChG,wBAAgB,cAAc,CAAC,IAAI,EAAE,WAAW,GAAG,OAAO,CAUzD;AAED,iEAAiE;AACjE,wBAAgB,OAAO,CAAC,OAAO,EAAE,YAAY,EAAE,UAAU,EAAE,MAAM,GAAG,WAAW,CAyB9E;AAED;;;GAGG;AACH,wBAAsB,UAAU,CAAC,IAAI,EAAE,WAAW,GAAG,OAAO,CAAC,WAAW,CAAC,CAmBxE"}

package/dist/audio.js

@@ -0,0 +1,80 @@
+/**
+ * Audio as llama.cpp reads it. Its multimodal library decodes WAV, MP3 and
+ * FLAC files itself, and resamples them to the model's rate. Anything else
+ * a browser records or plays (Opus in WebM or Ogg, AAC in MP4) is decoded
+ * here, with Web Audio, and handed over as WAV.
+ */
+/** The rate other formats are decoded at: the audio encoders' own, so llama.cpp resamples nothing. */
+const SAMPLE_RATE = 16000;
+/** Whether `bytes` holds `text` (ASCII) at `at`. */
+const holds = (bytes, at, text) => [...text].every((character, i) => bytes[at + i] === character.charCodeAt(0));
+/** Whether `data` is a WAV, FLAC or MP3 file, by how it starts: the files llama.cpp decodes. */
+export function decodesInLlama(data) {
+    const bytes = new Uint8Array(data, 0, Math.min(data.byteLength, 12));
+    if (bytes.length < 4)
+        return false;
+    if (holds(bytes, 0, "RIFF") && holds(bytes, 8, "WAVE"))
+        return true;
+    if (holds(bytes, 0, "fLaC"))
+        return true;
+    // An MP3 that opens with an ID3 tag, or with a frame: 11 sync bits, then
+    // Layer III (AAC's ADTS frames open the same way, as layer 0).
+    if (holds(bytes, 0, "ID3"))
+        return true;
+    const [first = 0, second = 0] = bytes;
+    return first === 0xff && (second & 0xe0) === 0xe0 && (second & 0x06) === 0x02;
+}
+/** Mono samples, each from -1 to 1, as a 16-bit PCM WAV file. */
+export function wavFile(samples, sampleRate) {
+    const size = samples.length * 2;
+    const file = new DataView(new ArrayBuffer(44 + size));
+    const text = (at, value) => {
+        for (let i = 0; i < value.length; i++)
+            file.setUint8(at + i, value.charCodeAt(i));
+    };
+    text(0, "RIFF");
+    file.setUint32(4, 36 + size, true);
+    text(8, "WAVE");
+    text(12, "fmt ");
+    file.setUint32(16, 16, true);
+    // PCM, one channel, 2 bytes a sample.
+    file.setUint16(20, 1, true);
+    file.setUint16(22, 1, true);
+    file.setUint32(24, sampleRate, true);
+    file.setUint32(28, sampleRate * 2, true);
+    file.setUint16(32, 2, true);
+    file.setUint16(34, 16, true);
+    text(36, "data");
+    file.setUint32(40, size, true);
+    samples.forEach((value, i) => {
+        const sample = Math.max(-1, Math.min(1, value));
+        file.setInt16(44 + i * 2, Math.round(sample < 0 ? sample * 0x8000 : sample * 0x7fff), true);
+    });
+    return file.buffer;
+}
+/**
+ * An audio file's bytes as llama.cpp can decode them: as they are for WAV,
+ * MP3 and FLAC, else decoded by the browser and written as mono WAV.
+ */
+export async function llamaAudio(data) {
+    if (decodesInLlama(data))
+        return data;
+    if (typeof OfflineAudioContext === "undefined") {
+        throw new Error("This audio isn't WAV, MP3 or FLAC, and there's no Web Audio here to decode it.");
+    }
+    let decoded;
+    try {
+        // A copy: decoding detaches the buffer it's given.
+        decoded = await new OfflineAudioContext(1, 1, SAMPLE_RATE).decodeAudioData(data.slice(0));
+    }
+    catch {
+        throw new Error("The browser couldn't decode this audio. WAV, MP3 and FLAC always work.");
+    }
+    const mono = new Float32Array(decoded.length);
+    for (let channel = 0; channel < decoded.numberOfChannels; channel++) {
+        decoded.getChannelData(channel).forEach((sample, i) => {
+            mono[i] = (mono[i] ?? 0) + sample / decoded.numberOfChannels;
+        });
+    }
+    return wavFile(mono, decoded.sampleRate);
+}