mlx-bun 0.0.13 → 0.3.0

package/docs/reference/troubleshooting.md

@@ -0,0 +1,166 @@
+# Troubleshooting
+
+Symptom → cause → fix, for the problems users actually hit. Every path and
+flag below is taken from the code (`src/`) or the linked reference doc; when a
+sentence here disagrees with the code, the code wins — fix the doc.
+
+Related: [cli.md](./cli.md) (commands), [server-config.md](./server-config.md)
+(serve flags, admission), [distribution.md](./distribution.md) (install paths,
+signing), [docs/reference/distribution.md](./distribution.md) (sidecar layout, self-signing),
+[environment.md](./environment.md) (contributor/agent platform facts).
+
+## Runtime and install
+
+### `dlopen` fails / `libmlxc.dylib` not found
+
+**Cause.** The `mlx-bun` executable ships without the MLX native runtime
+(`libmlxc.dylib`, `libmlx.dylib`, `libjaccl.dylib`, `mlx.metallib`, plus the
+expert-IO dylib and frame-extract helper — the list is `NATIVE_PACK_FILES` in
+`src/native-pack.ts`). The runtime is a versioned tarball downloaded on first
+run into `~/Library/Caches/mlx-bun/native-v<ver>-<arch>/`. The download is
+resumable and sha256-verified through the same machinery as model downloads
+(`src/download.ts`), so an interrupted first run leaves a partial file, not a
+broken install.
+
+**Fix.** Rerun `mlx-bun serve` (or any command that loads the runtime); it
+resumes and verifies. The library is resolved in this order
+(`resolveLibmlxc()` in `src/mlx/ffi.ts`, mirrored by `nativeRuntimeDir()` in
+`src/native-pack.ts`):
+
+1. `MLX_BUN_LIBMLXC=/path/to/libmlxc.dylib` — explicit override; the other
+   libs are expected in the same directory.
+2. Beside the executable (the sidecar layout from `docs/reference/distribution.md`).
+3. The native-pack cache dir above.
+4. Homebrew: `/opt/homebrew/lib`, then `/usr/local/lib`.
+
+To use your own build, set `MLX_BUN_LIBMLXC`. If you see
+`no native pack published for <arch> — set MLX_BUN_LIBMLXC or install mlx via
+homebrew`, there is no published pack for that CPU architecture (only `arm64`
+has a baked checksum in `src/native-pack.ts`); use one of the two escapes in
+the message. `MLX_BUN_NATIVE_PACK_URL` overrides the download URL.
+
+### Gatekeeper: "unidentified developer" / app can't be opened
+
+**Cause.** Released binaries are Developer-ID signed and notarized. macOS only
+checks the notarization ticket when a file carries the `com.apple.quarantine`
+xattr, which browser downloads and `curl -O`-style saves get. Homebrew installs
+and the documented `curl ... | tar -xz` one-liner never write a quarantined
+file, so they never prompt (`distribution.md`, "Direct download").
+
+**Fix.**
+- Online: the notarization ticket resolves the check; allow it once.
+- Offline: `xattr -d com.apple.quarantine mlx-bun` (the manual escape hatch
+  from `distribution.md`).
+- Self-built binary: the bundle's dylibs must be (re)signed with the hardened
+  runtime and entitlements — see "Signing & notarization" in
+  [docs/reference/distribution.md](./distribution.md).
+
+### Wrong Bun version / `Bun.Image` missing / FFI crashes in a source checkout
+
+**Cause.** `package.json` pins `"bun": ">=1.4.0"`. Two things depend on it:
+`Bun.Image` is the vision decode path (`src/vision/preprocess.ts`), and Bun
+1.4 fixed a `bun:ffi` stack-argument ABI bug on macOS arm64 that made
+multi-argument mlx-c bindings hand the callee shifted garbage
+(`lab/repro/bun-ffi-stack-args/ISSUE.md`; the code now uses natural mlx-c
+signatures with no workaround).
+
+**Fix.** `bun upgrade`, then `bun --version` ≥ 1.4.0. The compiled release
+binary embeds its own Bun, so this only affects running from source.
+
+### `port 8080 is already serving …`
+
+**Cause.** `serve` probes the port before loading weights and refuses to start
+a second server on a port that already answers `/v1` (`src/cli.ts`, serve
+branch; `--port` row in `server-config.md`).
+
+**Fix.** Reuse the running server (`mlx-bun pi` attaches automatically), stop
+it, or pass `--port <other>`. A second server is a second model in memory;
+check `mlx-bun fit` first.
+
+## Models and downloads
+
+### `no models match (try \`mlx-bun scan\`)`
+
+**Cause.** `ls`/`serve`/`fit` resolve models from a registry built by scanning
+the Hugging Face hub cache. The registry is stale or empty, or the model is
+not in the cache directory the registry reads.
+
+**Fix.** Run `mlx-bun scan` after downloading. The cache path is resolved
+exactly as `huggingface_hub` does (`DEFAULT_HUB` in `src/registry.ts`):
+`HF_HUB_CACHE`, else `HF_HOME/hub`, else `~/.cache/huggingface/hub`. If you
+downloaded to a custom location, set the same env var for `mlx-bun`. The
+registry database lives at `~/.cache/mlx-bun/registry.sqlite`.
+
+### Hugging Face download stalls at 0%
+
+**Cause.** The Python `hf` downloader uses the Xet transfer path, which stalls
+on some networks.
+
+**Fix.** Use `mlx-bun get <org/repo>` — it speaks plain HTTPS resolve/CDN with
+no Xet, Range-resumes partial blobs, and verifies every file (sha256 for LFS
+blobs, git-blob sha1 for small files) into the standard hub cache layout
+(`src/download.ts`). Rerunning the same command resumes verified
+`.incomplete` prefixes. If you must use the Python CLI, set
+`HF_HUB_DISABLE_XET=1` before the download command.
+
+### `get` refuses for lack of disk space
+
+**Cause.** Before transferring, `get` credits complete blobs and partial
+prefixes, then requires free space for the exact remainder plus a fixed 1 GiB
+reserve (`cli.md`, `get`).
+
+**Fix.** Free space, or point `HF_HUB_CACHE` at a larger volume before
+starting.
+
+### Gated repo: 401/403 from Hugging Face
+
+**Cause.** No token. `get` reads `HF_TOKEN`, else
+`~/.cache/huggingface/token` (`hfToken()` in `src/download.ts`).
+
+**Fix.** `hf auth login`, or export `HF_TOKEN`.
+
+## Memory and performance
+
+### Will it fit? / decode is very slow on a model near the RAM ceiling
+
+**Cause.** A model whose weights plus KV cache exceed the wired-memory budget
+pages weights every token. The default budget is machine RAM × 0.75
+(`WIRED_FRACTION` in `src/fit.ts`; `--memory-budget` in `server-config.md`).
+
+**Fix.** Check first:
+
+```sh
+mlx-bun fit <model> --ctx 8192          # this machine
+mlx-bun fit <model> --ctx 8192 --skus   # across the Apple Silicon lineup
+mlx-bun fit <model> --kv-quant 4        # window with a 4-bit KV cache
+```
+
+`fit` reports weights bytes (from safetensors headers), KV bytes/token, the
+calibrated prefill transient, and the wired ceiling. Then close memory-heavy
+apps, pick a smaller quantization, or serve with `--kv-quant 4|8` for a larger
+context window in the same budget. GLM-5.2 uses its own streamed-runtime plan
+(`fit GLM-5.2`; `--context-length`), not the resident-weights estimator.
+
+### Completion stops before my `max_tokens` (no error)
+
+**Cause.** This is memory admission clamping, not a bug. When the prompt fits
+but the client's `max_tokens` would push the reservation past the safe
+context, the server caps `max_tokens` to the remaining room and proceeds
+(`admitRequestContext()` in `src/server.ts`: `max_tokens` is a ceiling, not a
+promise). Earlier versions returned 400 for this case; the clamp shipped for
+GLM-5.2 fixed-context serving in v0.0.13 and for all memory admission in
+v0.2.0 (`docs/archive/planning/release-notes-v0.2.0.md`).
+
+**Fix.** Nothing is required. To get more room: shorten the prompt, raise
+`--memory-budget` if the machine allows, or serve with `--kv-quant 4|8`. The
+current ceiling is visible at `/stats` (`context_tokens`).
+
+### `400` with `"type": "memory_admission"`, `"code": "context_over_budget"`
+
+**Cause.** The prompt alone leaves no generation slot inside the safe context.
+Only this case is rejected; the alternative is an uncatchable GPU OOM
+(`server-api.md`, error table; `--memory-budget` in `server-config.md`).
+
+**Fix.** Same levers as above: shorter prompt, larger budget, quantized KV.
+The server never refuses to *start* over budget — a budget too small for any
+context logs a warning and serves, and requests then get this 400.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mlx-bun",
-  "version": "0.0.13",
+  "version": "0.3.0",
   "description": "Native MLX inference for Bun on Apple Silicon — local LLM server (OpenAI/Anthropic-compatible) and TypeScript library. No Python.",
   "type": "module",
   "license": "MIT",
@@ -26,7 +26,9 @@
   },
   "exports": {
     ".": "./src/index.ts",
-    "./package.json": "./package.json"
+    "./package.json": "./package.json",
+    "./engine": "./src/library.ts",
+    "./client": "./src/client.ts"
   },
   "files": [
     "bin",
@@ -37,7 +39,7 @@
     "THIRD_PARTY_LICENSES.md"
   ],
   "engines": {
-    "bun": ">=1.3.14"
+    "bun": ">=1.4.0"
   },
   "os": [
     "darwin"
@@ -47,16 +49,19 @@
   ],
   "scripts": {
     "test": "bash scripts/test.sh",
+    "typecheck": "tsc --noEmit && tsc -p tsconfig.web.json --noEmit && tsc -p src/contracts/tsconfig.json --noEmit",
     "link-cli": "chmod +x src/cli.ts && ln -sf $PWD/src/cli.ts $HOME/.bun/bin/mlx-bun && mlx-bun --version",
     "build:binary": "sh scripts/build-binary.sh",
     "release": "PUBLISH=1 sh scripts/release-binary.sh",
-    "publish:npm": "bun publish"
+    "publish:npm": "bun publish",
+    "clean": "bun scripts/clean.ts"
   },
   "devDependencies": {
     "@anthropic-ai/sdk": "^0.104.1",
     "@types/bun": "^1.3.14",
     "happy-dom": "^20.10.6",
-    "openai": "^6.42.0"
+    "openai": "^6.42.0",
+    "typescript": "6.0.3"
   },
   "dependencies": {
     "@earendil-works/pi-ai": "0.80.3",

package/src/admission-plan.ts

@@ -0,0 +1,18 @@
+/** Common admission contract for every runtime. Representation-specific
+ * planners may expose richer fields, but serving consumes only this shape. */
+export interface MemoryPlan {
+  readonly schemaVersion: 1;
+  readonly strategy: "generic-kv" | "glm52-colibri";
+  readonly fits: boolean;
+  readonly contextTokens: number;
+  readonly maxSafeContext: number;
+  readonly weightsBytes: number;
+  readonly kvBytes: number;
+  readonly transientBytes: number;
+  readonly reserveBytes: number;
+  readonly totalBytes: number;
+  readonly usableBytes: number;
+  readonly predictedDecodeTps: number | null;
+  /** mlx allocator-cache limit, distinct from the process admission limit. */
+  readonly allocatorLimitBytes?: number;
+}

package/src/anthropic.ts

@@ -23,6 +23,10 @@
 //     semantics) instead of the oracle's json.dumps fallback.
 
 import type { ChatMessage, ToolDefinition } from "./chat-template";
+import type {
+  CompletionEvent,
+  CompletionStreamProtocol,
+} from "./serve/completion-sink";
 
 // ---------------------------------------------------------------------
 // Request translation: Anthropic → OpenAI (our ChatRequest shape)
@@ -128,7 +132,15 @@ export function anthropicToChatBody(body: AnthropicRequest): Record<string, unkn
           break; // prior-turn thinking is not re-fed (Anthropic semantics)
         case "image": {
           const part = imagePartFromBlock(block);
-          if (part) imageParts.push(part);
+          if (!part)
+            // e.g. a Files-API {type:"file"} source we cannot fetch —
+            // reject loudly, never answer about an image the model can't
+            // see (2026-08-18 review; the audio-doctrine below).
+            throw new Error(
+              `unsupported image source "${block.source?.type ?? "missing"}" — ` +
+                `send base64 or url image sources`,
+            );
+          imageParts.push(part);
           break;
         }
         case "tool_use":
@@ -141,15 +153,52 @@ export function anthropicToChatBody(body: AnthropicRequest): Record<string, unkn
             },
           });
           break;
-        case "tool_result":
+        case "tool_result": {
           // Anthropic packs tool results into the next user message;
           // OpenAI (and the gemma template) want role:"tool" messages.
+          // Nested image blocks (screenshot-returning tools — the
+          // computer-use shape) are PRESERVED as image parts: flattening
+          // them away had the model answering about screenshots it never
+          // saw (2026-08-18 review). An unrenderable pairing downstream
+          // fails loudly on the media-count check, never silently.
+          const trImages: Record<string, unknown>[] = [];
+          if (Array.isArray(block.content)) {
+            for (const b of block.content as AnthropicContentBlock[]) {
+              if (b && typeof b === "object" && b.type === "image") {
+                const part = imagePartFromBlock(b);
+                if (!part)
+                  throw new Error(
+                    `unsupported image source "${b.source?.type ?? "missing"}" ` +
+                      `in tool_result — send base64 or url image sources`,
+                  );
+                trImages.push(part);
+              }
+            }
+          }
+          const txt = flattenText(block.content);
           messages.push({
             role: "tool",
             tool_call_id: block.tool_use_id,
-            content: flattenText(block.content),
+            content: trImages.length
+              ? [
+                  ...(txt ? [{ type: "text", text: txt }] : []),
+                  ...trImages,
+                ] as never
+              : txt,
           });
           break;
+        }
+        case "video":
+        case "video_url":
+          // Not an Anthropic Messages block type — reject loudly instead of
+          // silently answering about a video the model never saw (same
+          // doctrine as audio below; the OpenAI surface serves video on
+          // Qwen3.5-family models).
+          throw new Error(
+            `"${block.type}" content blocks are not part of the Anthropic ` +
+              `Messages protocol — send video to the OpenAI endpoint ` +
+              `/v1/chat/completions as a video_url content part`,
+          );
         case "audio":
         case "input_audio":
         case "audio_url":
@@ -168,6 +217,11 @@ export function anthropicToChatBody(body: AnthropicRequest): Record<string, unkn
       }
     }
     if (role === "assistant") {
+      if (imageParts.length)
+        // Assistants don't produce images in the Anthropic protocol; a
+        // client that puts one here would previously have it silently
+        // discarded (2026-08-18 review — loud beats silent).
+        throw new Error("image content blocks are not supported in assistant messages");
       const msg: ChatMessage = { role, content: textParts.join("\n") || null };
       if (toolCalls.length) msg.tool_calls = toolCalls;
       if (msg.content || toolCalls.length) messages.push(msg);
@@ -457,6 +511,41 @@ export class AnthropicStreamTranslator {
   }
 }
 
+/** Direct semantic-event adapter used by the server. This avoids encoding
+ * OpenAI SSE only to parse those bytes back into protocol events. */
+export function createAnthropicStreamProtocol(model: string): CompletionStreamProtocol {
+  const translator = new AnthropicStreamTranslator(model);
+  const chunk = (delta: Record<string, unknown>) => ({ choices: [{ delta }] });
+  return {
+    start: () => translator.addChunk(chunk({ role: "assistant", content: "" })),
+    addEvents(events: CompletionEvent[]) {
+      return events.flatMap((event) => {
+        if (event.type === "reasoning") {
+          return translator.addChunk(chunk({ reasoning: event.text }));
+        }
+        if (event.type === "content") {
+          return translator.addChunk(chunk({ content: event.text }));
+        }
+        return translator.addChunk(chunk({
+          tool_calls: event.calls.map((call, index) => ({ index, ...call })),
+        }));
+      });
+    },
+    finish(reason, usage) {
+      return [
+        ...translator.addChunk({ choices: [{ delta: {}, finish_reason: reason }], usage }),
+        ...translator.finalize(),
+      ];
+    },
+    error(message) {
+      return [sse("error", {
+        type: "error",
+        error: { type: "api_error", message },
+      })];
+    },
+  };
+}
+
 /** Wrap our OpenAI SSE byte stream as an Anthropic SSE byte stream.
  *  Frames translate as they arrive (per-chunk flush preserved — the
  *  upstream already rate-limits socket flushes). */

package/src/audio/decode.ts

@@ -1,9 +1,9 @@
-// WAV decode for audio input (A1 of docs/design/audio-input-plan.md).
+// WAV decode for audio input (A1 of docs/design/generic-model-support.md).
 //
 // Scope: RIFF/WAVE containers — PCM 16/24/32-bit and IEEE float32, any
 // channel count (mean mixdown), any sample rate (linear resample to 16 kHz).
 // PCM16 scales by 1/32768 to match the oracle golden generator
-// (scripts/gen-e4b-audio-golden.py) exactly. Non-WAV containers (mp3/m4a/
+// (scripts/oracle/gen-e4b-audio-golden.py) exactly. Non-WAV containers (mp3/m4a/
 // flac/ogg) are transcoded to WAV by the serve layer via macOS `afconvert`
 // before reaching this parser.
 //

package/src/audio/features.ts

@@ -1,9 +1,9 @@
 // USM log-mel feature extraction for gemma-4 audio — a verbatim-semantics
 // port of the oracle's Gemma4AudioFeatureExtractor (optiq/vlm/_mlxvlm/
 // models/gemma4/audio_feature_extractor.py, constructor DEFAULTS — §3.3 Q3
-// of docs/design/audio-input-plan.md: no per-bin normalization, params are
+// of docs/design/generic-model-support.md: no per-bin normalization, params are
 // fixed for all gemma-4 models). Gated bit-close against the T0 mel goldens
-// (goldens/e4b-audio-*-mel.bin) in tests/audio-features.test.ts.
+// (goldens/e4b-audio-*-mel.bin) in tests/parity/audio-features.test.ts.
 //
 // Pipeline (single waveform, 16 kHz mono float32):
 //   pad to a multiple of 128 samples (validity mask over the original part)

package/src/backends/mlx/autoregressive.ts

@@ -0,0 +1,108 @@
+import type { AutoregressiveGraph } from "../../inference/graph";
+import type { MlxArray } from "../../mlx/array";
+import type { Cache, Gemma4Model } from "../../model/gemma4";
+import type { RuntimeModel } from "../../model/factory";
+import { CompiledDecode } from "../../model/compiled-decode";
+import { runtimeConfig, type RuntimeConfig } from "../../runtime-config";
+import { bindMlxGraph } from "./graph";
+
+/** One declaration shared by planning and both native execution lanes. */
+export function legacyCompiledDecodeAvailable(model: RuntimeModel): boolean {
+  return model.config?.modelType?.startsWith("gemma4") === true && !model.config.text.enableMoeBlock;
+}
+
+export interface MlxModelMemory {
+  readonly weightsBytes: number;
+  readonly expertRuntime?: {
+    readonly plan: { readonly plannedBytes: number };
+    flushUsage?: () => void;
+    finishUsage?: () => Promise<void>;
+  } | null;
+}
+
+export interface MlxDecodeStep {
+  /** Consume one pending token and advance state once. Returned arrays belong
+   * to the caller. null permits ordinary graph execution and MUST leave state
+   * unchanged. Errors propagate unless this implementation can roll back. */
+  tryStep(token: MlxArray, state: Cache[]): { logits: MlxArray; evalWith: MlxArray[] } | null;
+  /** Release per-run workspaces after pending outputs are dropped and before
+   * cache disposal. Wait for native work if releasing requires completion. */
+  close(): void | Promise<void>;
+}
+
+/** Model-owned execution for committed tokens. Recheck the chunk limit after
+ * each forward: native arithmetic can change at a cache-length boundary. */
+export interface MlxTokenAppend {
+  maxChunkSize(state: readonly Cache[]): number;
+  forwardHidden(ids: MlxArray, state: Cache[]): MlxArray | Promise<MlxArray>;
+}
+
+/** One replaceable binding owns all graph-specific operations used by the AR
+ * loop. The current method requires MLX tensors and the legacy Cache[] ABI.
+ * Weights are borrowed for the binding's lifetime; generated caches are owned
+ * by the run. Caller-provided caches and media remain borrowed. */
+export interface MlxAutoregressiveBinding {
+  readonly runtime?: RuntimeConfig;
+  readonly graph: AutoregressiveGraph<MlxArray, Cache[], MlxArray>;
+  readonly eosTokenIds: readonly number[];
+  readonly memory: MlxModelMemory;
+  readonly adapters?: { active: string[] };
+  makeCache(): Cache[];
+  forwardEmbeddings?(
+    embeddings: MlxArray, state: Cache[], imageMask: MlxArray | null,
+    ids: MlxArray, multimodalMask: MlxArray | null,
+  ): MlxArray;
+  /** Called once after prefill. A compiled/fused decoder belongs to THIS graph,
+   * never to a model inferred from a name or inherited by a replacement. */
+  createDecode?(policy: { hasAdapters: boolean; pagedKv: boolean }): MlxDecodeStep | null;
+  createAppend?(policy: { hasAdapters: boolean; pagedKv: boolean }): MlxTokenAppend | null;
+}
+
+/** Keep concrete model and compiled-decode decisions at the legacy boundary. */
+export function bindLegacyAutoregressiveModel(model: RuntimeModel): MlxAutoregressiveBinding {
+  const runtime = runtimeConfig();
+  return {
+    runtime,
+    graph: bindMlxGraph<Cache[]>(model, {
+      id: `legacy:${model.config.modelType}`, artifact: "legacy-resident-model",
+      stateAbi: "legacy-cache-array-v1", // not a persistence identity
+    }),
+    eosTokenIds: model.config.eosTokenIds,
+    memory: model,
+    adapters: model.loraState,
+    makeCache: model.makeCache.bind(model),
+    forwardEmbeddings: model.forwardEmbeddings?.bind(model),
+    createAppend: "createAppend" in model ? model.createAppend.bind(model) : undefined,
+    createDecode(policy) {
+      // Preserve the existing Gemma path and exclusions. MoE shapeless replay
+      // retraces growing windows; adapters would bake residuals into the tape.
+      if (!runtime.flag("MLX_BUN_COMPILED_DECODE", true) || policy.hasAdapters || policy.pagedKv ||
+          !legacyCompiledDecodeAvailable(model))
+        return null;
+      let compiled: CompiledDecode | null = CompiledDecode.for(model as Gemma4Model);
+      return {
+        tryStep(token, state) {
+          if (!compiled || !CompiledDecode.supports(state)) return null;
+          try { return compiled.step(token, state); }
+          catch (error) {
+            // CompiledDecode restores committed writes on failure, so retrying
+            // this token through the ordinary graph is safe. Other decoders
+            // must establish that guarantee themselves before returning null.
+            compiled = null;
+            console.warn(`compiled decode disabled for this generation: ${error}`);
+            return null;
+          }
+        },
+        // Compiled closures belong to the model's existing unload lifecycle.
+        close() { compiled = null; },
+      };
+    },
+  };
+}
+
+export function assertMlxAutoregressiveBinding(binding: MlxAutoregressiveBinding): void {
+  const descriptor = binding.graph.descriptor;
+  if (descriptor.backend !== "mlx" || descriptor.graphAbi !== "mlx-hidden-bsh-v1" ||
+      descriptor.stateAbi !== "legacy-cache-array-v1")
+    throw new Error(`AR binding ${descriptor.id} has an incompatible backend, graph, or state ABI`);
+}