mlx-bun 0.2.0 → 0.4.0

package/docs/reference/troubleshooting.md

@@ -0,0 +1,158 @@
+# Troubleshooting
+
+Symptom → cause → fix, for the problems users actually hit. Every path and
+flag below is taken from the code (`src/`) or the linked reference doc; when a
+sentence here disagrees with the code, the code wins — fix the doc.
+
+Related: [cli.md](./cli.md) (commands), [server-config.md](./server-config.md)
+(serve flags, admission), [distribution.md](./distribution.md) (install paths,
+signing), [docs/reference/distribution.md](./distribution.md) (sidecar layout, self-signing),
+[environment.md](./environment.md) (contributor/agent platform facts).
+
+## Runtime and install
+
+### `dlopen` fails / `libmlxc.dylib` not found
+
+**Cause.** The `mlx-bun` executable ships without the MLX native runtime
+(`libmlxc.dylib`, `libmlx.dylib`, `libjaccl.dylib`, `mlx.metallib`, plus the
+expert-IO dylib and frame-extract helper — the list is `NATIVE_PACK_FILES` in
+`src/native-pack.ts`). The runtime is a versioned tarball downloaded on first
+run into `~/Library/Caches/mlx-bun/native-v<ver>-<arch>/`. The download is
+resumable and sha256-verified through the same machinery as model downloads
+(`src/download.ts`), so an interrupted first run leaves a partial file, not a
+broken install.
+
+**Fix.** Rerun `mlx-bun serve` (or any command that loads the runtime); it
+resumes and verifies. The library is resolved in this order
+(`resolveLibmlxc()` in `src/mlx/ffi.ts`, mirrored by `nativeRuntimeDir()` in
+`src/native-pack.ts`):
+
+1. `MLX_BUN_LIBMLXC=/path/to/libmlxc.dylib` — explicit override; the other
+   libs are expected in the same directory.
+2. Beside the executable (the sidecar layout from `docs/reference/distribution.md`).
+3. The native-pack cache dir above.
+4. Homebrew: `/opt/homebrew/lib`, then `/usr/local/lib`.
+
+To use your own build, set `MLX_BUN_LIBMLXC`. If you see
+`no native pack published for <arch> — set MLX_BUN_LIBMLXC or install mlx via
+homebrew`, there is no published pack for that CPU architecture (only `arm64`
+has a baked checksum in `src/native-pack.ts`); use one of the two escapes in
+the message. `MLX_BUN_NATIVE_PACK_URL` overrides the download URL.
+
+### Gatekeeper: "unidentified developer" / app can't be opened
+
+**Cause.** Released binaries are Developer-ID signed and notarized. macOS only
+checks the notarization ticket when a file carries the `com.apple.quarantine`
+xattr, which browser downloads and `curl -O`-style saves get. Homebrew installs
+and the documented `curl ... | tar -xz` one-liner never write a quarantined
+file, so they never prompt (`distribution.md`, "Direct download").
+
+**Fix.**
+- Online: the notarization ticket resolves the check; allow it once.
+- Offline: `xattr -d com.apple.quarantine mlx-bun` (the manual escape hatch
+  from `distribution.md`).
+- Self-built binary: the bundle's dylibs must be (re)signed with the hardened
+  runtime and entitlements — see "Signing & notarization" in
+  [docs/reference/distribution.md](./distribution.md).
+
+### Wrong Bun version / `Bun.Image` missing / FFI crashes in a source checkout
+
+**Cause.** `package.json` pins `"bun": ">=1.4.0"`. Two things depend on it:
+`Bun.Image` is the vision decode path (`src/vision/preprocess.ts`), and Bun
+1.4 fixed a `bun:ffi` stack-argument ABI bug on macOS arm64 that made
+multi-argument mlx-c bindings hand the callee shifted garbage
+(`lab/repro/bun-ffi-stack-args/ISSUE.md`; the code now uses natural mlx-c
+signatures with no workaround).
+
+**Fix.** `bun upgrade`, then `bun --version` ≥ 1.4.0. The compiled release
+binary embeds its own Bun, so this only affects running from source.
+
+### `port 8080 is already serving …`
+
+**Cause.** `serve` probes the port before loading weights and refuses to start
+a second server on a port that already answers `/v1` (`src/cli.ts`, serve
+branch; `--port` row in `server-config.md`).
+
+**Fix.** Reuse the running server (`mlx-bun pi` attaches automatically), stop
+it, or pass `--port <other>`. A second server is a second model in memory;
+check `mlx-bun fit` first.
+
+## Models and downloads
+
+### `no models match (try \`mlx-bun scan\`)`
+
+**Cause.** `ls`/`serve`/`fit` resolve models from a registry built by scanning
+the Hugging Face hub cache. The registry is stale or empty, or the model is
+not in the cache directory the registry reads.
+
+**Fix.** Run `mlx-bun scan` after downloading. The cache path is resolved
+exactly as `huggingface_hub` does (`DEFAULT_HUB` in `src/registry.ts`):
+`HF_HUB_CACHE`, else `HF_HOME/hub`, else `~/.cache/huggingface/hub`. If you
+downloaded to a custom location, set the same env var for `mlx-bun`. The
+registry database lives at `~/.cache/mlx-bun/registry.sqlite`.
+
+### Hugging Face download stalls at 0%
+
+**Cause.** The Python `hf` downloader uses the Xet transfer path, which stalls
+on some networks.
+
+**Fix.** Use `mlx-bun get <org/repo>` — it speaks plain HTTPS resolve/CDN with
+no Xet, Range-resumes partial blobs, and verifies every file (sha256 for LFS
+blobs, git-blob sha1 for small files) into the standard hub cache layout
+(`src/download.ts`). Rerunning the same command resumes verified
+`.incomplete` prefixes. If you must use the Python CLI, set
+`HF_HUB_DISABLE_XET=1` before the download command.
+
+### `get` refuses for lack of disk space
+
+**Cause.** Before transferring, `get` credits complete blobs and partial
+prefixes, then requires free space for the exact remainder plus a fixed 1 GiB
+reserve (`cli.md`, `get`).
+
+**Fix.** Free space, or point `HF_HUB_CACHE` at a larger volume before
+starting.
+
+### Gated repo: 401/403 from Hugging Face
+
+**Cause.** No token. `get` reads `HF_TOKEN`, else
+`~/.cache/huggingface/token` (`hfToken()` in `src/download.ts`).
+
+**Fix.** `hf auth login`, or export `HF_TOKEN`.
+
+## Memory and performance
+
+### Will it fit? / decode is very slow on a model near the RAM ceiling
+
+Memory pressure can reduce speed or cause allocation failures. The default
+`fit` estimate uses machine RAM × 0.75 (`WIRED_FRACTION` in `src/fit.ts`).
+Serving treats that estimate as advisory; `--memory-budget` opts into a limit.
+
+**Fix.** Check first:
+
+```sh
+mlx-bun fit <model> --ctx 8192          # this machine
+mlx-bun fit <model> --ctx 8192 --skus   # across the Apple Silicon lineup
+mlx-bun fit <model> --kv-quant 4        # window with a 4-bit KV cache
+```
+
+`fit` reports weights bytes (from safetensors headers), KV bytes/token, the
+calibrated prefill transient, and the wired ceiling. Then close memory-heavy
+apps, pick a smaller quantization, or serve with `--kv-quant 4|8` for a larger
+context window in the same budget. GLM-5.2 uses its own streamed-runtime plan
+(`fit GLM-5.2`; `--context-length`), not the resident-weights estimator.
+
+### Completion stops before my `max_tokens` (no error)
+
+Check `finish_reason`: `stop` means EOS or a stop condition; `length` means
+an output/context cap was reached. Default memory estimates do not shorten
+requests. Explicit `--memory-budget`, context settings and GLM-5.2's fixed
+layout can cap a completion. `/stats.admission.enforced_context_tokens` shows
+that cap, or null when none applies. `max_safe_context` is only an estimate.
+
+### `400` with `"type": "memory_admission"`, `"code": "context_over_budget"`
+
+This error applies to an explicitly configured budget/context cap or a fixed
+runtime layout. The prompt leaves no room to generate under that limit.
+Remove the optional budget/context cap to attempt the request without an
+estimated restriction, or adjust the limit you selected. See
+[server configuration](server-config.md#memory-admission-and-caches).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mlx-bun",
-  "version": "0.2.0",
+  "version": "0.4.0",
   "description": "Native MLX inference for Bun on Apple Silicon — local LLM server (OpenAI/Anthropic-compatible) and TypeScript library. No Python.",
   "type": "module",
   "license": "MIT",
@@ -26,7 +26,9 @@
   },
   "exports": {
     ".": "./src/index.ts",
-    "./package.json": "./package.json"
+    "./package.json": "./package.json",
+    "./engine": "./src/library.ts",
+    "./client": "./src/client.ts"
   },
   "files": [
     "bin",
@@ -47,16 +49,19 @@
   ],
   "scripts": {
     "test": "bash scripts/test.sh",
+    "typecheck": "tsc --noEmit && tsc -p tsconfig.web.json --noEmit && tsc -p src/contracts/tsconfig.json --noEmit",
     "link-cli": "chmod +x src/cli.ts && ln -sf $PWD/src/cli.ts $HOME/.bun/bin/mlx-bun && mlx-bun --version",
     "build:binary": "sh scripts/build-binary.sh",
     "release": "PUBLISH=1 sh scripts/release-binary.sh",
-    "publish:npm": "bun publish"
+    "publish:npm": "bun publish",
+    "clean": "bun scripts/clean.ts"
   },
   "devDependencies": {
     "@anthropic-ai/sdk": "^0.104.1",
     "@types/bun": "^1.3.14",
     "happy-dom": "^20.10.6",
-    "openai": "^6.42.0"
+    "openai": "^6.42.0",
+    "typescript": "6.0.3"
   },
   "dependencies": {
     "@earendil-works/pi-ai": "0.80.3",

package/src/audio/decode.ts

@@ -1,9 +1,9 @@
-// WAV decode for audio input (A1 of docs/design/audio-input-plan.md).
+// WAV decode for audio input (A1 of docs/design/generic-model-support.md).
 //
 // Scope: RIFF/WAVE containers — PCM 16/24/32-bit and IEEE float32, any
 // channel count (mean mixdown), any sample rate (linear resample to 16 kHz).
 // PCM16 scales by 1/32768 to match the oracle golden generator
-// (scripts/gen-e4b-audio-golden.py) exactly. Non-WAV containers (mp3/m4a/
+// (scripts/oracle/gen-e4b-audio-golden.py) exactly. Non-WAV containers (mp3/m4a/
 // flac/ogg) are transcoded to WAV by the serve layer via macOS `afconvert`
 // before reaching this parser.
 //

package/src/audio/features.ts

@@ -1,9 +1,9 @@
 // USM log-mel feature extraction for gemma-4 audio — a verbatim-semantics
 // port of the oracle's Gemma4AudioFeatureExtractor (optiq/vlm/_mlxvlm/
 // models/gemma4/audio_feature_extractor.py, constructor DEFAULTS — §3.3 Q3
-// of docs/design/audio-input-plan.md: no per-bin normalization, params are
+// of docs/design/generic-model-support.md: no per-bin normalization, params are
 // fixed for all gemma-4 models). Gated bit-close against the T0 mel goldens
-// (goldens/e4b-audio-*-mel.bin) in tests/audio-features.test.ts.
+// (goldens/e4b-audio-*-mel.bin) in tests/parity/audio-features.test.ts.
 //
 // Pipeline (single waveform, 16 kHz mono float32):
 //   pad to a multiple of 128 samples (validity mask over the original part)

package/src/backends/mlx/assistant-target.ts

@@ -0,0 +1,46 @@
+import type { MlxArray } from "../../mlx/array";
+import { Dtype } from "../../mlx/ffi";
+import * as ops from "../../mlx/ops";
+import type { Cache, KvDonorAttention } from "../../model/gemma4-base";
+import { captureKvDonorAttention } from "../../model/kv-attention-view";
+import type { AssistantAttention } from "../../spec/drafter";
+import { disposeResources } from "../../engine/resources";
+
+/** Bind captured cache validity to Q-only attention. Encoded layouts can
+ * supply this same numerical port without changing the draft graph. */
+export function readAssistantDonors(sliding: Cache, full: Cache) {
+  const views: KvDonorAttention[] = [], masks: MlxArray[] = [];
+  const dispose = () => disposeResources([...masks, ...views]);
+  try {
+    for (const cache of [sliding, full]) views.push(captureKvDonorAttention(cache));
+    const bind = (view: KvDonorAttention): AssistantAttention => {
+      // All draft steps borrow the same target snapshot. Build each validity
+      // mask once, then reuse it across matching layers and the entire chain.
+      const byWindow = new Map<number | null, MlxArray | null>();
+      return { attend(query, scale, window) {
+        if (!byWindow.has(window)) {
+          const width = view.width, B = view.offsets.length;
+          const starts = view.starts.map((start, row) => Math.max(start,
+            window === null ? start : view.ends[row]! - window));
+          let mask: MlxArray | null = null;
+          if (starts.some(start => start !== 0) || view.ends.some(end => end !== width)) {
+            using columns = ops.arange(0, width, 1, Dtype.int32);
+            using lower = ops.fromInt32(starts, [B, 1, 1, 1]);
+            using upper = ops.fromInt32([...view.ends], [B, 1, 1, 1]);
+            using before = ops.less(columns, lower);
+            using after = ops.greaterEqual(columns, upper);
+            using invalid = ops.logicalOr(before, after);
+            using floats = invalid.astype(view.dtype);
+            mask = ops.mulScalar(floats, -1e9);
+            masks.push(mask);
+          }
+          byWindow.set(window, mask);
+        }
+        const mask = byWindow.get(window)!;
+        return view.attend(query, scale, { mode: mask ? "array" : "", arr: mask });
+      } };
+    };
+    return { positions: views[0]!.offsets.map(offset => offset - 1),
+      sliding: bind(views[0]!), full: bind(views[1]!), dispose };
+  } catch (error) { dispose(); throw error; }
+}

package/src/backends/mlx/autoregressive-draft-rows.ts

@@ -0,0 +1,120 @@
+import type { MlxArray } from "../../mlx/array";
+import * as ops from "../../mlx/ops";
+import type { AutoregressiveGraph } from "../../inference/graph";
+import type { Cache, BatchableCache, PaddedPrefillCache } from "../../model/gemma4-base";
+import type { DraftRowSampling } from "../../spec/source";
+import { toLogprobs } from "../../sampler";
+import { disposeResources } from "../../engine/resources";
+import { MlxStateRows } from "./state-rows";
+import type { PreparedStateChange } from "../../contracts/resources";
+import { evalCacheState } from "./prefill";
+
+export interface MlxDraftLayout extends BatchableCache, PaddedPrefillCache {
+  makeEmptyBatch(): MlxDraftLayout;
+}
+
+/** The draft graph and sampler operate at B. State snapshots own rewind and
+ * membership independently of the graph, target verifier and scheduler. */
+export class MlxAutoregressiveDraftRows {
+  #history: MlxStateRows[] = [];
+  #drafts: number[][] = [];
+  #depth = 0;
+  #pendingLast: Array<number | null>;
+  constructor(readonly graph: AutoregressiveGraph<MlxArray, Cache[], MlxArray>,
+    readonly state: MlxStateRows<MlxDraftLayout>, readonly sampling: DraftRowSampling | null) {
+    this.#pendingLast = Array.from({ length: state.rowCount }, () => null);
+  }
+  get pendingLast(): readonly (number | null)[] { return this.#pendingLast; }
+  prepareAppend(rows: readonly (readonly Cache[])[], pending: readonly (number | null)[]): PreparedStateChange {
+    const state = this.state.prepareAppendMany(rows), next = [...this.#pendingLast, ...pending];
+    return { commit: () => { state.commit(); this.#pendingLast = next; }, dispose: () => state.dispose() };
+  }
+  filterRows(keep: readonly number[]): void {
+    this.state.filterRows(keep); this.#pendingLast = keep.map(row => this.#pendingLast[row]!);
+  }
+
+  async #forward(ids: MlxArray, lengths?: readonly number[]): Promise<MlxArray> {
+    const width = ids.shape[1]!, padded = lengths?.some(length => length !== width);
+    if (padded) for (const cache of this.state.caches) cache.preparePrefill({
+      lengths: [...lengths!], rightPadding: lengths!.map(length => width - length),
+    });
+    const hidden = await this.graph.forwardHidden(ids, this.state.caches);
+    try {
+      if (padded) for (const cache of this.state.caches) cache.finalizePrefill();
+      return hidden;
+    } catch (error) { hidden.dispose(); throw error; }
+  }
+
+  /** A restored full-accept boundary may have one unprocessed draft token.
+   * Combine it with the incoming block on device, retaining per-row lengths. */
+  async prefill(tokens: MlxArray): Promise<void> {
+    const [B, N] = tokens.shape as [number, number];
+    if (this.#pendingLast.some(token => token !== null)) {
+      using previous = ops.fromInt32(this.#pendingLast.map(token => token ?? 0), [B, 1]);
+      using joined = ops.concatAxis([previous, tokens], 1);
+      const lengths = this.#pendingLast.map(token => N + Number(token !== null));
+      using indices = ops.fromInt32(lengths.flatMap(length => Array.from({ length: N + 1 },
+        (_, position) => length === N + 1 ? position : Math.min(position + 1, N))), [B, N + 1]);
+      using inputs = ops.takeAlongAxis(joined, indices, 1);
+      (await this.#forward(inputs, lengths)).dispose();
+    } else (await this.#forward(tokens)).dispose();
+    this.#pendingLast.fill(null); this.materialize();
+  }
+  materialize(): void { evalCacheState(this.state.caches); }
+
+  async draft(pending: readonly number[], depth: number, steps: readonly number[]): Promise<number[][]> {
+    this.#releaseHistory();
+    this.#depth = depth;
+    const tokens: MlxArray[] = [];
+    const feeds = pending.map((token, row) => this.#pendingLast[row] == null ? [token] : [this.#pendingLast[row]!, token]);
+    const lengths = feeds.map(feed => feed.length), width = Math.max(...lengths);
+    let ids = ops.fromInt32(feeds.flatMap(feed => [...feed, ...Array(width - feed.length).fill(0)]), [pending.length, width]);
+    try {
+      // With zero proposals, consume the pending input to keep companion
+      // coverage aligned with the target's one-token verification.
+      for (let position = 0; position < Math.max(1, depth); position++) {
+        const padded = position === 0 && lengths.some(length => length !== width);
+        using hidden = await this.#forward(ids, position === 0 ? lengths : undefined);
+        this.#history.push(this.state.clone());
+        if (depth === 0) break;
+        using selection = padded ? ops.fromInt32(lengths.map(length => length - 1), [pending.length, 1, 1]) : null;
+        using selected = selection ? ops.takeAlongAxis(hidden, selection, 1) : null;
+        using logits = this.graph.projectLogits(selected ?? hidden, { type: "last" });
+        using flat = ops.reshape(logits, [pending.length, logits.shape.at(-1)!]);
+        using logprobs = toLogprobs(flat);
+        const token = this.sampling!.sample(logprobs, steps.map(step => step + position));
+        tokens.push(token);
+        ids.dispose(); ids = ops.reshape(token, [pending.length, 1]);
+      }
+      if (depth === 0) this.#drafts = pending.map(() => []);
+      else {
+        using packed = ops.concatAxis(tokens, 0);
+        const values = packed.toIntTokens();
+        this.#drafts = pending.map((_, row) => Array.from({ length: depth },
+          (_, position) => values[position * pending.length + row]!));
+      }
+      return this.#drafts;
+    } catch (error) { this.#releaseHistory(); throw error; }
+    finally { ids.dispose(); disposeResources(tokens); }
+  }
+
+  /** Preserve mlx-lm's full-accept re-feed shape: the final draft remains
+   * pending and joins the next target token in one forward. Recurrent state
+   * uses saved boundaries, never trimming. The lag is companion state. */
+  async commit(accepted: readonly number[]): Promise<void> {
+    const selected: Cache[][] = [];
+    try {
+      if (!accepted.every(count => count >= this.#depth - 1)) {
+        for (const [row, count] of accepted.entries())
+          selected.push(this.#history[Math.min(count, Math.max(0, this.#depth - 1))]!.extractRow(row));
+        this.state.mergeRows(selected);
+      }
+    } finally { disposeResources(selected.flat()); this.#releaseHistory(); }
+
+    this.#pendingLast = accepted.map((count, row) => this.#depth > 0 && count === this.#depth ? this.#drafts[row]!.at(-1)! : null);
+    this.#drafts = []; this.#depth = 0;
+  }
+
+  #releaseHistory(): void { disposeResources(this.#history.splice(0)); }
+  dispose(): void { this.#releaseHistory(); this.#drafts = []; this.state.dispose(); }
+}

package/src/backends/mlx/autoregressive.ts

@@ -0,0 +1,108 @@
+import type { AutoregressiveGraph } from "../../inference/graph";
+import type { MlxArray } from "../../mlx/array";
+import type { Cache, Gemma4Model } from "../../model/gemma4";
+import type { RuntimeModel } from "../../model/factory";
+import { CompiledDecode } from "../../model/compiled-decode";
+import { runtimeConfig, type RuntimeConfig } from "../../runtime-config";
+import { bindMlxGraph } from "./graph";
+
+/** One declaration shared by planning and both native execution lanes. */
+export function legacyCompiledDecodeAvailable(model: RuntimeModel): boolean {
+  return model.config?.modelType?.startsWith("gemma4") === true && !model.config.text.enableMoeBlock;
+}
+
+export interface MlxModelMemory {
+  readonly weightsBytes: number;
+  readonly expertRuntime?: {
+    readonly plan: { readonly plannedBytes: number };
+    flushUsage?: () => void;
+    finishUsage?: () => Promise<void>;
+  } | null;
+}
+
+export interface MlxDecodeStep {
+  /** Consume one pending token and advance state once. Returned arrays belong
+   * to the caller. null permits ordinary graph execution and MUST leave state
+   * unchanged. Errors propagate unless this implementation can roll back. */
+  tryStep(token: MlxArray, state: Cache[]): { logits: MlxArray; evalWith: MlxArray[] } | null;
+  /** Release per-run workspaces after pending outputs are dropped and before
+   * cache disposal. Wait for native work if releasing requires completion. */
+  close(): void | Promise<void>;
+}
+
+/** Model-owned execution for committed tokens. Recheck the chunk limit after
+ * each forward: native arithmetic can change at a cache-length boundary. */
+export interface MlxTokenAppend {
+  maxChunkSize(state: readonly Cache[]): number;
+  forwardHidden(ids: MlxArray, state: Cache[]): MlxArray | Promise<MlxArray>;
+}
+
+/** One replaceable binding owns all graph-specific operations used by the AR
+ * loop. The current method requires MLX tensors and the legacy Cache[] ABI.
+ * Weights are borrowed for the binding's lifetime; generated caches are owned
+ * by the run. Caller-provided caches and media remain borrowed. */
+export interface MlxAutoregressiveBinding {
+  readonly runtime?: RuntimeConfig;
+  readonly graph: AutoregressiveGraph<MlxArray, Cache[], MlxArray>;
+  readonly eosTokenIds: readonly number[];
+  readonly memory: MlxModelMemory;
+  readonly adapters?: { active: string[] };
+  makeCache(): Cache[];
+  forwardEmbeddings?(
+    embeddings: MlxArray, state: Cache[], imageMask: MlxArray | null,
+    ids: MlxArray, multimodalMask: MlxArray | null,
+  ): MlxArray;
+  /** Called once after prefill. A compiled/fused decoder belongs to THIS graph,
+   * never to a model inferred from a name or inherited by a replacement. */
+  createDecode?(policy: { hasAdapters: boolean; pagedKv: boolean }): MlxDecodeStep | null;
+  createAppend?(policy: { hasAdapters: boolean; pagedKv: boolean }): MlxTokenAppend | null;
+}
+
+/** Keep concrete model and compiled-decode decisions at the legacy boundary. */
+export function bindLegacyAutoregressiveModel(model: RuntimeModel): MlxAutoregressiveBinding {
+  const runtime = runtimeConfig();
+  return {
+    runtime,
+    graph: bindMlxGraph<Cache[]>(model, {
+      id: `legacy:${model.config.modelType}`, artifact: "legacy-resident-model",
+      stateAbi: "legacy-cache-array-v1", // not a persistence identity
+    }),
+    eosTokenIds: model.config.eosTokenIds,
+    memory: model,
+    adapters: model.loraState,
+    makeCache: model.makeCache.bind(model),
+    forwardEmbeddings: model.forwardEmbeddings?.bind(model),
+    createAppend: "createAppend" in model ? model.createAppend.bind(model) : undefined,
+    createDecode(policy) {
+      // Preserve the existing Gemma path and exclusions. MoE shapeless replay
+      // retraces growing windows; adapters would bake residuals into the tape.
+      if (!runtime.flag("MLX_BUN_COMPILED_DECODE", true) || policy.hasAdapters || policy.pagedKv ||
+          !legacyCompiledDecodeAvailable(model))
+        return null;
+      let compiled: CompiledDecode | null = CompiledDecode.for(model as Gemma4Model);
+      return {
+        tryStep(token, state) {
+          if (!compiled || !CompiledDecode.supports(state)) return null;
+          try { return compiled.step(token, state); }
+          catch (error) {
+            // CompiledDecode restores committed writes on failure, so retrying
+            // this token through the ordinary graph is safe. Other decoders
+            // must establish that guarantee themselves before returning null.
+            compiled = null;
+            console.warn(`compiled decode disabled for this generation: ${error}`);
+            return null;
+          }
+        },
+        // Compiled closures belong to the model's existing unload lifecycle.
+        close() { compiled = null; },
+      };
+    },
+  };
+}
+
+export function assertMlxAutoregressiveBinding(binding: MlxAutoregressiveBinding): void {
+  const descriptor = binding.graph.descriptor;
+  if (descriptor.backend !== "mlx" || descriptor.graphAbi !== "mlx-hidden-bsh-v1" ||
+      descriptor.stateAbi !== "legacy-cache-array-v1")
+    throw new Error(`AR binding ${descriptor.id} has an incompatible backend, graph, or state ABI`);
+}