mlx-bun 0.0.9 → 0.0.11

package/docs/reference/library-api.md

@@ -3,16 +3,19 @@
 The HTTP server is one consumer of a library-first API. Everything it
 does — generation, sampling, KV schemes, LoRA, vision, caching, memory
 admission — is importable directly into a Bun process. Published as
-`mlx-bun` on npm (0.0.4); `bunx mlx-bun` works out of the box, or
+`mlx-bun` on npm; `bunx mlx-bun` works out of the box, or
 `import … from "mlx-bun"` for programmatic use.
 
-One rule everywhere: **one generation at a time per process** (one GPU,
-batch=1). The server serializes through a queue; do the same.
+One rule everywhere: **one generation at a time per process** (one GPU).
+The server serializes through a queue (or, under `--batch N`, its
+continuous-batching scheduler — a server-lane feature, not a library
+surface); direct library callers must do the same.
 
 ## Quick start
 
 ```ts
 import {
+  ensureNativeRuntime,
   loadModelConfig,
   Weights,
   createModel,       // dispatches to Gemma4Model / MiniCPM5Model / Qwen35Model
@@ -21,6 +24,7 @@ import {
   generate,
 } from "mlx-bun";   // or "./src/index" in-repo
 
+await ensureNativeRuntime();
 const dir = "/path/to/hf-snapshot";          // mlx-bun ls prints these
 const config = await loadModelConfig(dir);
 const model = createModel(await Weights.open(dir), config);   // returns RuntimeModel
@@ -29,7 +33,7 @@ const template = await ChatTemplate.load(dir);
 // Gemma4Model and MiniCPM5Model can still be imported directly from "mlx-bun"
 // if you need the concrete type. Qwen35Model is NOT exported from the public
 // package — use createModel/RuntimeModel (preferred dispatch), or import from
-// "./src/model/qwen35" in-repo.
+// "./src/model/qwen3_5" in-repo.
 
 const ids = tok.encode(template.render([{ role: "user", content: "hi" }]));
 const promptIds = ids[0] === ids[1] && ids[0] === tok.bosTokenId ? ids.slice(1) : ids;
@@ -44,6 +48,14 @@ console.log(gen.stats);                          // set once iteration ends
 Weight loading is lazy (mmap + mlx native loader): construction is
 milliseconds; weights materialize on first forward.
 
+> **First run:** library consumers bypass the CLI's first-run step, so
+> call `await ensureNativeRuntime()` once before constructing a model on
+> a machine that may not have the MLX native runtime yet — it downloads
+> the sha256-verified native pack to `~/Library/Caches/mlx-bun/` and is
+> a no-op when the runtime is already present (beside the executable,
+> already cached, or installed via homebrew). `nativeRuntimeDir()`
+> returns the resolved directory, or `null` on a fresh machine.
+
 ## generate(model, promptTokens, options) → Generation
 
 `Generation` is an `AsyncIterable<GeneratedToken>` with a `stats`
@@ -64,13 +76,20 @@ after an early `break`, which cleanly cancels the in-flight step).
 | `kvConfig` | off | per-layer mixed precision from `kv_config.json` (`config.kvQuant`); overrides `kvBits`, start 0 — optiq serve semantics |
 | `adapters` | none | mounted LoRA adapter ids, applied for exactly this generation |
 | `promptEmbeddings` / `imageMask` | — | vision path (see `src/vision/`) |
+| `logprobs` / `topLogprobs` | off | per-token logprob capture (mlx_lm.server parity): `logprobs` captures the emitted token's log-probability, `topLogprobs` (0/unset = off) captures the top-k (id, logprob) pairs — same full-vocab log-softmax distribution mlx-lm's generate_step samples from, taken after logits processors and before the sampler |
+| `grammar` | off | a compiled `GrammarController` (`src/grammar.ts`) for structured output — masks invalid tokens to `-inf` each step (L2-class, oMLX oracle) |
+| `turboQuant` | off | TurboQuant KV scheme (docs/design/turboquant-kv.md), a rotation-based alternative to uniform `kvBits`; mutually exclusive with `kvBits`/`kvConfig` |
+| `pagedKv` | off | `{ blockSize? }` — OPTIONAL paged KV cache (docs/design/paged-kv-cache.md); v1 scope is serial batch=1 Gemma4-family bf16, mutually exclusive with `kvBits`/`kvConfig`/`turboQuant`/draft/compiled decode |
 
 `GenerateStats`: `promptTokens`, `cachedTokens`, `generatedTokens`,
 `prefillTps` / `prefillMs`, `decodeTps` / `decodeMs`, `cacheTokens`
 (the exact token sequence whose KV is in the cache — feed it to
 `PromptCache.put`). Timing semantics match mlx-lm: the prompt clock
 runs until the first token arrives (the prefill→decode boundary is
-prompt time), the decode clock from there.
+prompt time), the decode clock from there. An optional `spec` field
+carries speculative-decoding telemetry (serve `--draft-model` path
+only): `drafted`, `accepted`, `targetCalls`, and per-draft-position
+`draftedByPos` / `acceptedByPos` counts.
 
 ## Serving pieces
 
@@ -80,7 +99,7 @@ import { loadContext, createServer } from "mlx-bun";
 const ctx = await loadContext(dir, "my-model", { memoryBudgetBytes: 12e9 });
 const server = createServer(ctx, 8080, {
   promptCacheBytes: 2e9,        // byte-capped LRU (never count-capped)
-  kvQuant: undefined,           // default: kv_config.json; "off" | bits
+  kvQuant: undefined,           // default: kv_config.json; "off" | "config" | bits
   memoryBudgetBytes: 12e9,      // admission control — the only OOM defense
 });
 ```
@@ -135,11 +154,13 @@ under one adapter must not seed another's prefill).
 ```ts
 import { saveKvCache, loadKvCache } from "./src/kv-store";
 saveKvCache("/tmp/prefix.kv", tokens, caches);       // page-aligned file
-const { tokens, caches, mmap } = loadKvCache("/tmp/prefix.kv", model as Gemma4Model);
+const { tokens, caches, mmap } = loadKvCache("/tmp/prefix.kv", model);
 // reload is a zero-copy MAP_PRIVATE mmap straight to the GPU (~1 ms);
 // keep `mmap` referenced as long as the caches live
-// Note: loadKvCache is typed to accept Gemma4Model, but only uses model.layers.length
-// at runtime — a cast from RuntimeModel is safe for any model that has a layers array.
+// Note: loadKvCache accepts anything with makeCache(): Cache[] — it validates
+// the entry count against the cache list makeCache() returns (not
+// model.layers.length, which is wrong for KV-shared models like e4b, whose
+// makeCache() returns donors only). RuntimeModel qualifies directly, no cast.
 ```
 
 Quantized caches are not persistable yet (documented gap).

package/docs/reference/memory.md

@@ -42,18 +42,38 @@ mlx-bun memory unschedule
 
 ## Agent tools
 
-Both `mlx-bun pi` and the web chat expose these read-only tools:
+Both `mlx-bun pi` and the web chat expose these read-only tools, in the recommended call order — FIND the article, READ it small, follow the graph, fall back to reference docs and utilities, search only as a last resort:
 
-- `memory_search`
-- `memory_read`
-- `memory_toc`
-- `memory_section`
-- `memory_links`
+**FIND** (deterministic lookup, never a vector search):
+- `memory_resolve` — name/alias → the article
+- `memory_category` — category/type/series → members
+
+**READ** (TOC → one section, not a full-article dump):
+- `memory_read` — TOC + lead by default; `force=true` for the full article
+- `memory_section` — one section's body (the default granularity)
+
+**Follow the graph**:
+- `memory_links` — outbound/inbound wikilinks
+- `memory_infobox` — read an article's infobox fields (there is no infobox *query* tool — the infobox is content the model reads, not a filter)
+
+**Reference docs** (mlx-bun's own docs, mirrored read-only into the vault):
+- `reference_search`
+- `reference_read`
+- `reference_list`
+
+**Utility**:
 - `memory_list` — user articles plus read-only `Reference/*` docs
 - `memory_status` — vault path, setup state, article count, git state, last synthesis run (from the vault's git log), schedule state, recent changed articles
 
+**Last resort**:
+- `memory_search` — substring fallback when FIND failed
+
 The tools are always registered. If no vault exists, they return a setup message instead of disappearing. Search/read results prefixed with `Reference/` are mlx-bun docs, not user memory articles.
 
+## Web chat REST surface
+
+The web chat's Memory panel talks to a separate, HTTP-only `/api/memory/*` route surface (status/list/search/article/links/history/diff + a `POST /api/memory/init` consent-card endpoint) — thin wrappers over the same vault read path above, documented in [server-api.md](server-api.md#get-apimemory--post-apimemoryinit). The agent-tool surface above is unaffected by it.
+
 ## Obsidian
 
 ```bash

package/docs/reference/models.md

@@ -79,6 +79,8 @@ Schema (table `models`, one row **per snapshot**, keyed on `path`):
 | `quant_bits`, `quant_group_size`, `quant_mode` | quantization |
 | `has_vision_sidecar` | sidecar file present |
 | `vision_config_type` | `config.json`'s `vision_config.model_type` when it names a vision tower (`*_vision`) |
+| `has_audio_config` | `config.json` declares an `audio_config` block |
+| `has_audio_tower` | sidecar header names `audio_tower.*` tensors |
 | `has_kv_config`, `has_tool_template`, `license` | capabilities/terms |
 | `scanned_at` | freshness (canonical tie-break) |
 
@@ -89,6 +91,14 @@ Schema (table `models`, one row **per snapshot**, keyed on `path`):
 Presence of a `vision_config` key alone is **not** a signal — Qwen3.5
 nests a copy of its own text config there.
 
+**Audio capability** is `has_audio_config AND has_audio_tower`: both
+legs are required because the 12B OptiQ snapshot carries an
+`audio_config` but a stub sidecar whose only audio entry is
+`embed_audio.embedding_projection.weight` (no `audio_tower.*` Conformer
+tensors) — config presence alone would advertise audio that fails at
+request time. The check reads only the sidecar's safetensors header,
+never tensor bytes.
+
 ## Query resolution rules (per verb)
 
 Every verb resolves fuzzy queries the same way, with revision collapsing
@@ -98,7 +108,7 @@ so a stale snapshot never makes a repo "ambiguous":
   row per repo** (the canonical revision). `--all-revisions` shows the
   per-snapshot truth with the canonical marked `*`. Capabilities column
   labels the support tier: `supported (targeted)` (dedicated forward,
-  L2/L3 paths) vs `supported (generic)` (Tier-0 universal module).
+  L2/opt-in paths) vs `supported (generic)` (Tier-0 universal module).
 - **`fit` / `serve` / `train` / … (single-model verbs)** — `resolve()`:
   the query must match exactly one **repo**; multiple cached revisions of
   that repo collapse to the canonical snapshot (refs/main, else most

package/docs/reference/orpo-quickstart.md

@@ -14,9 +14,9 @@ mlx-bun train <model> --data <dir>
 - `<model>` is a query or snapshot path (e.g. `e4b`, `MiniCPM5`); omit it to auto-pick.
 - `--data <dir>` is a directory with `train.jsonl` (+ optional `valid.jsonl`); rows are
   `{"prompt": "...", "chosen": "...", "rejected": "..."}`.
-- The command **auto-detects e4b/Gemma** and sets its required training env flags
-  (`MLX_BUN_PERF_KERNEL=0`, `MLX_BUN_FUSED_GELU=0`) before loading the model — you don't
-  set them yourself. The full stack (flash head + prefix-share + segmented) is on by default.
+- The full stack (flash head + prefix-share + segmented) is on by default. No env-flag
+  setup is needed for any model (the old Gemma training-flag requirement is gone —
+  the no-vjp kernels were deleted 2026-07-05).
 - It prints a plan box (dataset counts, format, the resolved knobs), streams per-step loss
   + peak memory, and saves a **mountable** adapter (`mlx-bun serve <model> --adapter <dir>`).
 - `--dry-run` inspects the dataset and prints the plan without training. `mlx-bun help train`
@@ -59,7 +59,6 @@ mlx-bun train e4b --data ./prefs --resume ~/.cache/mlx-bun/mlx-bun-finetunes/orp
 MODEL=/path/to/snapshot DATA=/path/to/datadir bun scripts/train-orpo.ts
 ```
 - `DATA` is a directory with `train.jsonl` (+ optional `valid.jsonl`).
-- Auto-detects e4b/Gemma and sets the training env flags, same as the CLI.
 
 ### Preconfigured defaults (override via env)
 | env | default | meaning |