mlx-bun 0.0.4 → 0.0.5

package/docs/planning/PRODUCT_ROADMAP.md

@@ -19,9 +19,34 @@ provider config before they can try local AI. They should install one
 binary, run one command, wait for one resumable model download, and get a
 working local chat UI plus a local AI server.
 
+The intended first-run shape is product, not demo: start with the tiny
+welcoming model quickly, begin the stronger recommended model download in the
+background, and graduate the user into the best default profile for their
+actual Mac without requiring them to learn the model zoo first.
+
 The runtime can still expose every advanced knob. The default path should
 not require them.
 
+## Four product faces
+
+The same runtime has to support four related but distinct jobs:
+
+1. **Local AI product** — the one-command experience for someone who just wants
+   useful local AI. This path should be opinionated, benchmarked, and boringly
+   reliable.
+2. **Library for app developers** — the embeddable TypeScript/MLX layer for
+   Bun, Tauri, Electron, and desktop tools that want local inference without a
+   Python sidecar.
+3. **AI lab for researchers** — the place where parity, evals, kernels,
+   compiled graphs, ORPO, LoRA adapters, and model experiments can happen in
+   the same stack that serves the result.
+4. **Autonomous background agent runtime** — the always-available local server
+   for scripts, recurring tasks, personal tools, and per-request adapter
+   routing.
+
+These are not separate products. They are one runtime with progressive
+disclosure: appliance defaults at the front, sharp instruments underneath.
+
 ## The three modes (the organizing design law)
 
 mlx-bun has not published anything: the entire CLI, command, and flag
@@ -72,7 +97,7 @@ verbs and one flag vocabulary, not three separate CLIs:
   serving, latency prediction, the Lab UI) graft onto the mlx-lm spine
   as additive flags and a few new verbs.
 
-### Status (2026-06-12)
+### Status (2026-06-12; compat surface design remains open as of 2026-06-17)
 
 The compat layer does not exist yet: today's commands (`serve`, `fit`,
 `benchmark`, `evals`, `get`/`scan`/`ls`, `pi`, `harness`) are ad-hoc,
@@ -83,27 +108,42 @@ reconcile the existing commands into it, and fill the capability matrix
 behind it (convert/quantize, LoRA training, eval/perplexity, fuse,
 cache_prompt, upload/share). Decomposed into phases in PLAN.md.
 
-## Core idea
+## Core idea: finite matrix runtime
 
-The local-AI problem is not continuous for Mac users. People own a small
-set of real Apple Silicon SKUs: 16 GB, 24 GB, 32 GB, 36 GB, 48 GB,
-64 GB, and larger workstation tiers. For each profile, we can research
-and preselect the best known model artifact, context target, KV mode,
-memory budget, and runtime flags.
+The local-AI problem is not continuous for Mac users. MLX means Apple Silicon,
+and Apple ships a finite set of real chip/RAM/bandwidth combinations. The
+supported model universe is also curated. For each physical device profile and
+model snapshot, we can research and preselect the best known model artifact,
+context target, KV mode, memory budget, kernel route, graph route, and runtime
+flags.
 
-That means first-run behavior can be a lookup table, not a wizard or a
-benchmark:
+That means first-run behavior can be a measured lookup table, not a wizard or
+an exploratory benchmark:
 
 ```text
-detect Mac profile
-look up recommended runtime profile
+detect exact Mac profile
+fingerprint the model snapshot
+choose the task mode: chat, vision, tool loop, adapter, training, eval
+look up the measured execution plan
 download/resume the exact model artifact
 start the tuned local server
 open the built-in UI
 show how to connect external tools and harnesses
 ```
 
-The research work feeds the table. The table feeds the user.
+The current `fit` endpoint is the nucleus of this idea: it already reasons over
+real Apple Silicon profiles, model bytes, KV bytes, context, and predicted
+decode. The product version should evolve from "does this fit?" into:
+
+```text
+device profile + model checksum + task mode + live memory pressure
+  -> recommended execution plan
+```
+
+Execution plans can eventually include specialized graph variants, fused kernel
+routes, prompt-cache policy, adapter policy, safe context limits, expected
+decode/TTFT, and user-facing rationale. The research work feeds the table. The
+table feeds the user.
 
 ## Recommended profiles
 
@@ -143,6 +183,31 @@ changes a device-tier decision, unlocks a larger useful context window,
 keeps decode above a user-visible threshold, improves quality at the same
 budget, or prevents failure. Saving memory for its own sake is secondary.
 
+## Specialization strategy
+
+For the curated product path, it is acceptable to trade more work up front for a
+better hot path later. The Mac is known, the model snapshot is known, and the
+task mode is known. That lets us precompute or cache exact plans instead of
+shipping every decision to runtime.
+
+Candidate plan fields:
+
+- Model checksum and architecture fingerprint.
+- Apple Silicon profile: chip, unified memory, bandwidth tier, GPU/CPU shape.
+- Context buckets and fixed-shape cache layouts.
+- Kernel choices: MLX built-ins, MLX fast paths, custom Metal kernels, or
+  eager fallback.
+- Graph choices: eager, compiled exact-shape graph, compiled bucketed graph,
+  or prebuilt graph table.
+- Adapter choices: none, single LoRA, stacked LoRA, task-routed LoRA.
+- Measured results: TTFT, decode tok/s, peak memory, failure envelope.
+
+This is where loop unrolling, branch deletion, static DAG extraction, fused
+kernels, and shape-specialized compiled graphs become product features rather
+than abstract optimizations. They matter when they let the default profile be
+faster, larger, more stable, or better aligned without asking the user to tune
+anything.
+
 ## First-run UX
 
 The no-args command should be the friendly path:
@@ -153,17 +218,40 @@ mlx-bun
 
 Likely behavior:
 
-1. Detect the hardware profile.
-2. Pick the recommended runtime profile.
-3. Show the selected model and why it was chosen.
-4. Start or resume the background download.
-5. Start the local server when ready.
-6. Open or serve the built-in chat UI.
-7. Show connection details for OpenAI/Anthropic-compatible clients.
-8. Offer harness setup actions for Pi, Hermes, OpenClaw, and others.
+1. Detect the hardware profile. *(not yet: `recommendedRepoId()` at `src/fit.ts:86` returns `DEFAULT_REPO_ID` unconditionally — device-profile lookup table is unimplemented)*
+2. Pick the recommended runtime profile. *(same caveat as above)*
+3. Show the selected model and why it was chosen. *(not yet: model is printed, rationale is not)*
+4. Start or resume the background download. **Shipped** — auto-download on first run; resumable.
+5. Start the local server when ready. **Shipped.**
+6. Open or serve the built-in chat UI. **Shipped** — browser tab opened automatically; URL printed either way; `--no-open` skips the tab.
+7. Show connection details for OpenAI/Anthropic-compatible clients. **Shipped** — printed at startup.
+8. Offer harness setup actions for Pi, Hermes, OpenClaw, and others. *Partially shipped: `mlx-bun harness pi` configures an existing Pi install; Hermes and OpenClaw adapters do not exist.*
 
 This should feel like an appliance, not a framework tutorial.
 
+## Adapter catalog vision
+
+The average user should not have to train adapters to benefit from them. The
+product path can ship or offer a curated catalog of lightweight LoRA adapters:
+assistant, coding, writing, research, planning, personal-ops, and other
+well-scoped modes. A user can install one small adapter and get a better local
+assistant without changing the base model.
+
+Advanced users can still train their own adapters with mlx-bun. The broader
+product loop is:
+
+```text
+curate conversations and task data
+train / ORPO task adapters
+publish small adapter artifacts
+route requests through the right adapter per task
+keep the base model resident
+```
+
+This makes ORPO performance directly product-relevant. If training is cheap and
+repeatable enough, the project can keep improving small local models through
+adapter updates instead of asking every user to download a larger base model.
+
 ## Built-in UI
 
 Bun can serve the UI directly from the same executable. The default UI
@@ -213,6 +301,27 @@ The UI should not hide that this is local. It should make "local" feel
 easy: predictable memory, transparent downloads, offline-capable serving,
 and clear integration instructions.
 
+### SigLIP vision sidecar — image input for gemma e4b (shipped, 2026-06-17)
+
+The Gemma 4 e4b vision encoder (`src/vision/siglip.ts`) is a faithful port
+of the optiq SigLIP tower. It auto-loads on first image input and falls back
+to text-only mode when no image is present, so the chat UI gains image support
+with no startup cost. Features are within ~1% rel-RMSE of the optiq oracle;
+every MLX primitive (rms_norm, gelu, matmul, SDPA, RoPE, pooling) is bit-exact.
+The residual is sub-bf16 floating-point accumulation across the deep encoder,
+not a porting gap. L1/L2/L3 paths all work; vision falls back to the monolith
+path on L1 and L2.
+
+### Adapter web selector and CLI /v1/adapters (shipped)
+
+Trained LoRA adapters can be selected from the chat UI via a per-conversation
+picker (`#chat-adapter` in `src/web/app.html`). The picker lists adapters
+discovered by `GET /v1/adapters/available`, mounts the chosen one via
+`POST /v1/adapters`, and sends a `set_adapter` message to Pi-web so that
+every subsequent turn is routed through the right adapter — including the
+`before_provider_request` hook in `src/pi-web.ts`. Adapters are base-model-scoped
+and dropped on model swap.
+
 ### The UI must never lag while the AI is busy (candidate, 2026-06-16)
 
 A product guarantee: **the AI may crash, the UI never may** — and the UI
@@ -413,14 +522,25 @@ reuse across pipeline stages.
 
 ## Open questions
 
-- What exact profile table should ship first?
-- Should no-args open the UI automatically, print a local URL, or both?
-- Which UI stack should be embedded?
-- How much of Pi should be bundled, embedded, or configured externally?
-- What is the minimum first-run UI that feels good enough?
-- What is the right install path: Homebrew, curl installer, GitHub
-  release, or all of the above?
+- What exact profile table should ship first? *(The device-profile lookup
+  table is unimplemented — `recommendedRepoId()` returns `DEFAULT_REPO_ID`
+  for every SKU. The profile shape in this doc is the spec; the research
+  work and the fill remain open.)*
 - How should background downloads interact with server startup and UI
   readiness?
 - How should profile recommendations be versioned as benchmarks improve?
 - When Qwen lands, what evals decide whether it replaces a Gemma default?
+
+### Resolved
+
+- **Should no-args open the UI automatically, print a local URL, or both?**
+  Both: the browser tab is opened automatically on start; the URL is always
+  printed; `--no-open` suppresses the tab without hiding the URL.
+- **Which UI stack should be embedded? / How much of Pi should be bundled?**
+  Resolved by harness-pi + pi-web shipping: the web UI is the bundled
+  `src/web/app.html` chat page talking to Pi via `pi-web.ts`; Pi is
+  configured externally via `mlx-bun harness pi`, or launched embedded via
+  `mlx-bun pi`.
+- **What is the right install path?** All of the above — shipped: npm
+  publish (v0.0.4, `bunx mlx-bun`), Homebrew release pipeline (sign,
+  notarize, `joshuarossi/homebrew-tap`), and GitHub release binary.

package/docs/planning/chunk-task-roadmap.md

@@ -1,7 +1,7 @@
 # Chunk-task quality roadmap (2026-06-17)
 
 Where the MiniCPM5 chunking adapter stands and what raises real segmentation
-quality. Companion to [[chunk-eval-metric-vs-accuracy]] and
+quality. Companion to the chunk-eval metric vs. accuracy finding (score = well-formedness, not gold accuracy) and
 [`docs/investigations/chunk-finetune-experiment.md`](../investigations/chunk-finetune-experiment.md).
 
 ## Where we are

package/docs/planning/journal.md

@@ -153,3 +153,164 @@ the what/when (phases, exit criteria); this is the narrative behind it.
   3. **Tool param values were JSON-coerced regardless of schema.** `<param name="path">2025</param>` became the number 2025. Oracle (`_decode_tool_parameter_value` + `_tool_parameter_schema`) keeps string-typed params as raw text. Ported; also made the `<param>` regex CDATA-safe (a `</param>` inside CDATA no longer truncates the value). Unit tests added.
   4. **Sampling ignored the model's `generation_config.json`.** The server hardcoded temp 0.7 / no top-p; optiq serve injects the author-recommended sampling (`optiq/runtime/gen_config.py`) as defaults. Ported: `loadContext` reads `generation_config.json` (MiniCPM5: temp 0.9, top_p 0.95; Gemma: temp 1.0, top_k 64, top_p 0.95) and `toOptions` uses it when the request omits the field. Explicit request values still win. This was the likely "quality" gap on a 1B model.
   - Verification: MiniCPM5 parity (bf16 + mixed-KV) still 100/100 bit-exact; minicpm5-tools, tool-call, chat-template, tools-template, parity (Gemma 12B), server-tools, and server suites all green. Live two-turn agent round-trip (tool call → tool result → final answer) confirmed working in-process.
+
+## 2026-06-16 — Segmented-backward LoRA training (PRs #9–10)
+
+Long-context LoRA training was OOMing: MiniCPM5 @4096 spiked to ~25.5 GB full
+backward because all activations had to stay live simultaneously.  The
+segmented-backward design threads the backward segment-by-segment so only one
+segment's activations are resident at a time.
+
+The key implementation decision was to drive the per-segment backward with
+`mlx_vjp` (a new `Vjp` binding in `autograd.ts`) rather than a surrogate
+`value_and_grad` over a reconstructed loss.  The surrogate path leaked roughly
+one activation buffer per segment per step because mlx `eval` does not detach;
+`mlx_vjp` is explicit about what it differentiates and stops the leak.
+Segment-boundary tensors are materialized as graph-free leaves via
+`fromBytesCopy` for the same reason.
+
+Phase A (MiniCPM5, PR #9) landed first: grads bit-exact vs the full backward
+(relNorm 0.000%), peak 10.91 → 3.29 GB @2048, no leak, trains end-to-end via
+`TrainConfig.segmentSize`.  The default training attention is `ops.sdpa`
+(correct gradients, one fused kernel); `MLX_BUN_TRAIN_ATTN=flash` is an
+opt-in for bit-exact grad comparison against the full backward only, because
+`ops.sdpa` has a pre-existing eager-vs-autograd asymmetry in bf16 (~6%) that
+is unrelated to the segmentation mechanism — the segmented path works
+correctly under either setting.
+
+Phase B (e4b / Gemma4, PR #10) was harder: e4b's KV-shared donor pattern
+means multiple sharer segments reference the same K/V tensors produced by a
+single producer segment, requiring cotangent accumulation across sharers before
+the producer vjp can run.  Implemented `SegmentedBackwardGemma4` with a
+two-boundary stream (per-layer-input + donor K/V) and a multi-output producer
+vjp `[h, donorK, donorV]` with accumulated `[dh, dKV.k, dKV.v]` cotangents.
+Grads are bit-exact for single-consumer donor reuse and 0.97% bf16-class for
+multi-consumer accumulation (confirmed the full backward also sums in bf16, so
+this is order-of-operations noise, not a logic bug).  e4b trains at 8 K context
+(~17.5 GB, segSize=2) where the full backward would OOM (~70 GB).
+
+A contiguity fix was also required: donor K/V are transposed views, so
+`detachLeaf` must row-major-copy them before `rawBytes` reads linearly.
+
+## 2026-06-16 — Qwen3.5 / MiniCPM5 reasoning-channel fix (PR #12)
+
+Two serving-layer bugs left reasoning-capable models unusable in chat:
+
+The first was a role-name mismatch.  pi-ai renames the system role to
+`developer` for reasoning models.  Our chat templates only accepted
+system/user/assistant/tool and returned 400.  The turn completed with
+`stopReason: "error"` without throwing, so the browser saw an empty turn with no
+error shown.  Fix: `normalizeMessages` maps developer→system (genuine OpenAI
+compat — OpenAI itself uses the developer role), and `mapEventToFrames` surfaces
+a `stopReason:"error"` turn as a visible error frame instead of silently
+discarding it.
+
+The second was reasoning leaking into content.  Qwen3.5 and MiniCPM5 templates
+prime an *open* `<think>` tag in the generation prompt, so the model's first
+token is the closing `</think>` — the opening tag never appears in the decoded
+stream.  `ThinkingTagSplitter` was waiting for the open tag before routing
+reasoning content into the OpenAI `reasoning_content` channel, so all of it
+leaked into `content` instead.  Fix: `promptEndsInOpenThink` detects this
+condition and seeds `ThinkingTagSplitter` via `startInThinking` so the splitter
+begins in the reasoning state.
+
+## 2026-06-16 — Adapters end-to-end (PR #13)
+
+The goal was to make LoRA adapter switching a first-class operation in both the
+web chat UI and the Pi CLI, without building a custom protocol — lean on Pi's
+`before_provider_request` hook instead.
+
+Server-side: `GET /v1/adapters/available` scans the two adapter stores
+(`~/.cache/mlx-bun-finetunes`, `~/.cache/mlx-bun/adapters`) and returns
+`{id, path, rank, scale, mounted}` for weight-bearing checkpoints (dataset-only
+dirs are skipped).  `lora.ts` gained `listAvailableAdapters` for the scan and
+base-model detection so the web selector can filter to compatible adapters only.
+
+Pi hook: `pi-web.ts` registers `pi.on("before_provider_request")` to inject the
+per-connection selected adapter into every outgoing provider payload using the
+existing inline-extension pattern.  A `set_adapter` WebSocket message sets the
+selection; default none leaves the payload unchanged so the base model runs.
+
+Web chat: an adapter dropdown was added next to the Thinking pill, populated
+from `/v1/adapters/available` on ready.  Selecting fires an idempotent `POST
+/v1/adapters` then a `set_adapter` WS message.
+
+CLI: `extensions/mlx-bun-adapter.ts` is a shipped Pi extension exposing
+`/adapter <id> | off | list` and the same hook.  Both UIs control adapters
+through Pi; no separate plumbing.
+
+## 2026-06-17 — First-run onboarding UX
+
+The previous startup story required knowing the right model ID, running a
+separate download command, and manually opening a browser.  The new path:
+`mlx-bun serve` (or bare `mlx-bun`) auto-downloads the starter if absent,
+starts the server, and opens the chat UI.  If a localhost tab is already open it
+focuses that tab rather than opening a duplicate (`--no-open` to skip).
+
+The web chat empty state was updated with mlx-bun-centric starter chips and a
+download-aware greeting.  A live background-download pill shows progress.  The
+system prompt briefs the model on mlx-bun and its own identity, with framing
+appropriate to MiniCPM5 being a fast starter while a larger model optionally
+loads in the background.
+
+`chat-template.ts` also tightened `supportsThinking` to require the `<think>`
+channel we actually split, so e4b / harmony-format models no longer leak raw
+reasoning markers into content for users who haven't configured a thinkingFormat.
+
+The runtime-isolation design doc (`docs/design/runtime-isolation.md`) was also
+written this session: inference running in-process on the Bun event loop is a
+known UI-lag footgun when the GPU is busy.  The doc proposes moving inference to
+a subprocess; the work is explicitly deferred (design, not started) — written
+now so the root-cause analysis is available cold.
+
+## 2026-06-17 — SigLIP vision sidecar for gemma-4 e4b
+
+Ported optiq's `vlm/gemma4` SigLIP encoder as `src/vision/siglip.ts`: a 16-layer
+vision transformer with clippable linears (trained clip bounds), manual-f32
+RMS norms on q/k/v, on-device 2D RoPE, fused SDPA at scale=1.0, and GeGLU →
+3×3 avg-pool → `MultimodalEmbedder` → `/embed_scale`.
+
+The tower selection is driven by `vision_config.model_type`: e2b/e4b get
+SigLIP, the 12B gets the existing encoder-free tower.  Lazy loading means
+text-only sessions never pay for it.  `gemma4.ts`'s `forwardEmbeddings` was
+extended to support per-layer-input models (e2b/e4b) — it previously threw for
+that path — threading zeroed image-token ids into the per-layer forward matching
+optiq's `where(text_mask, ids, 0)`.
+
+Parity: every mlx primitive is bit-exact vs the optiq oracle (verified
+model-free via `op-parity-{dump.py,check.ts}` — rms_norm, gelu, matmul, clip,
+cos, sin, full multidim RoPE, sdpa, pool).  The residual ~1% feature divergence
+is sub-bf16 full-graph composition order amplified by the encoder's scale=1.0
+peaked softmax (the topped-out softmax distribution is highly sensitive to small
+pre-softmax shifts in bf16).  The acceptance bar was exact spliced ids + greedy
+prefix + grounded output, which passes.  Driving the residual to zero would
+require matching optiq's exact op ordering across the full encoder; deferred as
+a TODO.
+
+## 2026-06-17 — Distribution: npm, bunx, Homebrew signed/notarized, direct download
+
+The question was how to ship a standalone binary that runs on a user's Mac
+without requiring Bun, Python, or anything else.  We settled on four parallel
+install paths rather than picking one.
+
+**npm / bunx**: `bunx mlx-bun` for users already in the Bun ecosystem.  A
+`publish-release.sh` script and `bun run publish` alias handle the npm side.
+Published as v0.0.4.
+
+**Homebrew**: `scripts/release-binary.sh` builds a standalone binary, signs it
+with a Developer ID (nested Mach-O first, then the executable with JIT
+entitlements from `packaging/entitlements.plist`), notarizes via `AC_PROFILE`, and
+produces a clean tarball + sha256 for the formula.
+`packaging/homebrew/mlx-bun.rb` is the formula source of truth (self-contained
+`libexec` bundle + `bin` symlink; Bun `realpath`s `execPath` so sibling dylibs
+resolve after relocation).  The tap lives at `joshuarossi/homebrew-tap`.
+`scripts/publish-release.sh` auto-syncs the formula sha256 on each release.
+One fix was required: `--disable-library-validation` in the entitlements so
+brew-relocated dylibs load without `CS_RESTRICT` blocking them.
+
+**Direct download**: the notarized tarball passes Gatekeeper even when
+downloaded via browser + Finder (quarantine worst case), so a stable versionless
+URL (`releases/latest/download/mlx-bun-arm64.tar.gz`) is published alongside the
+versioned asset and documented as a one-liner curl install.
+
+**Embedded library**: existing path, unchanged (`docs/reference/embedding.md`).

package/docs/reference/distribution.md

@@ -54,7 +54,7 @@ and notarized as one artifact.
 This runs [build-binary.sh](../../scripts/build-binary.sh), then:
 - signs every nested Mach-O (dylibs, any `.node`) with the hardened
   runtime, then the executable with the JIT
-  [entitlements](../../scripts/entitlements.plist) (JavaScriptCore JITs —
+  [entitlements](../../packaging/entitlements.plist) (JavaScriptCore JITs —
   without `allow-jit` the binary is killed on launch);
 - notarizes the zipped bundle via `AC_PROFILE` and waits;
 - emits `dist-release/mlx-bun-v<ver>-arm64.tar.gz` (+ `.sha256`) and
@@ -75,19 +75,20 @@ gh release create v0.0.4 dist-release/mlx-bun-v0.0.4-arm64.tar.gz \
 
 ## The tap
 
-Homebrew taps must be a repo named `homebrew-<name>`:
+Homebrew taps must be a repo named `homebrew-<name>`. The canonical tap is
+**`joshuarossi/homebrew-tap`** (Homebrew shorthand: `joshuarossi/tap`).
 
-1. Create a public repo **`joshuarossi/homebrew-mlx-bun`**.
+1. Create a public repo **`joshuarossi/homebrew-tap`** (one-time).
 2. Copy [packaging/homebrew/mlx-bun.rb](../../packaging/homebrew/mlx-bun.rb)
    to `Formula/mlx-bun.rb` in it, updating `version`/`url`/`sha256` from
    the release-script output. Commit and push.
 3. Install:
    ```sh
-   brew install joshuarossi/mlx-bun/mlx-bun
+   brew install joshuarossi/tap/mlx-bun
    mlx-bun --version
    ```
 
-`brew audit --strict --new joshuarossi/mlx-bun/mlx-bun` before pushing
+`brew audit --strict --new joshuarossi/tap/mlx-bun` before pushing
 catches most formula issues; `brew install --build-from-source` +
 `brew test` runs the formula's `test do` locally.
 
@@ -119,3 +120,58 @@ Users then upgrade with:
 ```sh
 brew upgrade joshuarossi/tap/mlx-bun
 ```
+
+## Direct download (stable URL)
+
+`publish-release.sh` uploads two assets per release: the versioned tarball
+(`mlx-bun-v<ver>-arm64.tar.gz`) **and** a versionless stable copy
+(`mlx-bun-arm64.tar.gz`) pinned to `releases/latest`. This lets the
+install script target a URL that never changes:
+
+```
+https://github.com/joshuarossi/mlx-bun/releases/latest/download/mlx-bun-arm64.tar.gz
+```
+
+The user-facing install path is the script at
+[`website/public/install.sh`](../../website/public/install.sh) (served at
+`mlx-bun.dev/install.sh`), which resolves that URL, extracts the bundle, and
+links `mlx-bun` onto `PATH`:
+```sh
+curl -fsSL https://mlx-bun.dev/install.sh | sh
+```
+
+Without the script, the same stable URL streams straight to `tar`:
+```sh
+curl -fsSL https://github.com/joshuarossi/mlx-bun/releases/latest/download/mlx-bun-arm64.tar.gz \
+  | tar -xz -C /usr/local/bin mlx-bun
+```
+
+Note: **browser downloads and `curl`-to-file** (e.g. `curl -O ...`) are
+quarantined by Gatekeeper — macOS sets the `com.apple.quarantine` xattr on
+the saved file. The piped one-liner above streams directly to `tar` and
+never writes a quarantined file, so it runs without any Gatekeeper prompt.
+For the browser/curl-to-file path, the notarization ticket resolves the
+Gatekeeper check online; `xattr -d com.apple.quarantine mlx-bun` is the
+manual escape hatch if offline.
+
+## npm / bunx
+
+mlx-bun is published to npm as `mlx-bun` (current: 0.0.4). Install or
+run without a permanent install:
+
+```sh
+# one-off (no install)
+bunx mlx-bun
+
+# permanent global install
+bun install -g mlx-bun
+mlx-bun --version
+```
+
+The npm package wraps the same signed binary via a postinstall download
+step (it does not bundle the ~228 MB dylibs/metallib in the npm tarball
+itself). Publish to npm as part of a release:
+
+```sh
+bun publish   # from repo root; package.json version must already be bumped
+```

package/docs/reference/library-api.md

@@ -2,8 +2,9 @@
 
 The HTTP server is one consumer of a library-first API. Everything it
 does — generation, sampling, KV schemes, LoRA, vision, caching, memory
-admission — is importable directly into a Bun process. Not yet
-published to npm; import from `src/`.
+admission — is importable directly into a Bun process. Published as
+`mlx-bun` on npm (0.0.4); `bunx mlx-bun` works out of the box, or
+`import … from "mlx-bun"` for programmatic use.
 
 One rule everywhere: **one generation at a time per process** (one GPU,
 batch=1). The server serializes through a queue; do the same.
@@ -11,17 +12,24 @@ batch=1). The server serializes through a queue; do the same.
 ## Quick start
 
 ```ts
-import { loadModelConfig } from "./src/config";
-import { Weights } from "./src/weights";
-import { Gemma4Model } from "./src/model/gemma4";
-import { loadTokenizer } from "./src/tokenizer";
-import { ChatTemplate } from "./src/chat-template";
-import { generate } from "./src/generate";
+import {
+  loadModelConfig,
+  Weights,
+  createModel,       // dispatches to Gemma4Model / MiniCPM5Model / Qwen35Model
+  loadTokenizer,
+  ChatTemplate,
+  generate,
+} from "mlx-bun";   // or "./src/index" in-repo
 
 const dir = "/path/to/hf-snapshot";          // mlx-bun ls prints these
-const model = new Gemma4Model(await Weights.open(dir), await loadModelConfig(dir));
+const config = await loadModelConfig(dir);
+const model = createModel(await Weights.open(dir), config);   // returns RuntimeModel
 const tok = await loadTokenizer(dir);
 const template = await ChatTemplate.load(dir);
+// Gemma4Model and MiniCPM5Model can still be imported directly from "mlx-bun"
+// if you need the concrete type. Qwen35Model is NOT exported from the public
+// package — use createModel/RuntimeModel (preferred dispatch), or import from
+// "./src/model/qwen35" in-repo.
 
 const ids = tok.encode(template.render([{ role: "user", content: "hi" }]));
 const promptIds = ids[0] === ids[1] && ids[0] === tok.bosTokenId ? ids.slice(1) : ids;
@@ -67,7 +75,7 @@ prompt time), the decode clock from there.
 ## Serving pieces
 
 ```ts
-import { loadContext, createServer } from "./src/server";
+import { loadContext, createServer } from "mlx-bun";
 
 const ctx = await loadContext(dir, "my-model", { memoryBudgetBytes: 12e9 });
 const server = createServer(ctx, 8090, {
@@ -84,6 +92,12 @@ completions, Anthropic `/v1/messages`, OpenAI Responses, adapters, and
 
 ## Prompt cache (in-process prefix reuse)
 
+> **Note:** `PromptCache`, `AdapterManager`, `saveKvCache`, and
+> `loadKvCache` are **not** re-exported from the public `"mlx-bun"`
+> entry point (`src/index.ts`). Import them from their internal paths
+> as shown below; this is intentional — they are implementation-level
+> pieces rather than stable public API surface.
+
 ```ts
 import { PromptCache } from "./src/prompt-cache";
 const cache = new PromptCache(2e9);                  // BYTE-capped LRU
@@ -102,9 +116,11 @@ under one adapter must not seed another's prefill).
 ```ts
 import { saveKvCache, loadKvCache } from "./src/kv-store";
 saveKvCache("/tmp/prefix.kv", tokens, caches);       // page-aligned file
-const { tokens, caches, mmap } = loadKvCache("/tmp/prefix.kv", model);
+const { tokens, caches, mmap } = loadKvCache("/tmp/prefix.kv", model as Gemma4Model);
 // reload is a zero-copy MAP_PRIVATE mmap straight to the GPU (~1 ms);
 // keep `mmap` referenced as long as the caches live
+// Note: loadKvCache is typed to accept Gemma4Model, but only uses model.layers.length
+// at runtime — a cast from RuntimeModel is safe for any model that has a layers array.
 ```
 
 Quantized caches are not persistable yet (documented gap).
@@ -112,8 +128,7 @@ Quantized caches are not persistable yet (documented gap).
 ## Registry + fit (model discovery and memory math)
 
 ```ts
-import { Registry } from "./src/registry";
-import { fit, skuMatrix } from "./src/fit";
+import { Registry, fit, skuMatrix } from "mlx-bun";
 
 const reg = new Registry();           // bun:sqlite, scans the HF cache
 await reg.scan();
@@ -150,4 +165,4 @@ any amount of switching.
   handler → process death). Use `fit`/admission before generating, not
   try/catch around it.
 - Never read a typed array that native code wrote in a hot path; use
-  the `read.*` helpers (bun#32054 — see PLAN.md Phase 4 findings).
+  the `read.*` helpers (bun#32054 — see PLAN-archive.md Phase 4 findings).