mlx-bun 0.4.0 → 1.0.0

package/docs/planning/ResearchTopics.md

@@ -1,119 +0,0 @@
-# Research Topics
-
-Speculative, exploratory directions for mlx-bun — theses worth prototyping, not
-committed work. Each entry carries enough detail to pick up cold. PLAN.md is for
-work in flight; this is for ideas waiting their turn.
-
----
-
-## 1. Tonal Grading for Token Distributions ("HDR Sampling")
-
-> Replace the global scalar sampling knobs (temperature, top-k, top-p) with a
-> region-aware **transfer function** applied to logits before softmax — the
-> color-grading move, ported to token distributions.
-
-**Framing — the Wes Anderson point.** A film's symmetrical pastel look isn't a
-colorist failing to neutralize; it's a *target*. Grade Marvel that way and it's
-"wrong" only against Marvel's acceptance function, not against any absolute.
-Different films, different targets — and you have a lot of films. The
-Lucien-synthesis agent and the test-writing agent want different *looks*, and
-"temperature + a system-prompt vibe word" is the crude contrast slider standing
-in for a real grade.
-
-**Core thesis.** Current LLM sampling is "SDR": the available controls
-(temperature, top-k, top-p) are crude *global* knobs — the equivalent of a single
-contrast slider. Color grading abandoned "just turn the contrast knob" long ago
-in favor of transfer functions with region-specific behavior. The same move is
-available for sampling: replace global scalar knobs with a region-aware transfer
-function applied to the logits before softmax.
-
-**The reframe that makes it work.** Signal vs. noise is defined by *intent*, not
-by magnitude — exactly as on a camera sensor, where the noise floor is "noise"
-because it doesn't represent what's wanted, not because it's small. The full
-logit vector is the model's signal, but it is not the signal *I* care about. My
-acceptance criterion defines which part is signal. The low-logit tail is
-therefore legitimately treatable as a noise floor to roll off.
-
-**The three regions** — one curve, three treatments. This is precisely what
-temperature *cannot* do, since it scales everywhere uniformly:
-
-- **Shoulder** (head / confident tokens): soft rolloff instead of hard
-  truncation. Top-k/top-p slam into the ceiling and clip; a shoulder eases the
-  top tokens toward co-equality while *preserving their ordering* — it tames
-  single-token dominance without destroying the relationships among the spared
-  tokens.
-- **Midtones** (the plausible-but-distinct band): steepen for local contrast.
-  Push apart the tokens that matter so detail comes out sharp. This has no name
-  in current sampling because sampling has no concept of distributional
-  midtones — and it's the most novel piece.
-- **Toe** (tail / unwanted tokens): roll into the floor. Defines them as noise
-  relative to intent.
-
-**Curve vocabulary, ported from grading:**
-
-- **Pivot/cut** placed *relative* to the distribution (by rank or percentile),
-  recomputed every step — never an absolute logit value, because logits are only
-  meaningful up to an additive constant (softmax shift-invariance). HLG's
-  relative / scene-referred design is the right philosophical match; LogC3 is
-  more absolute / exposure-locked.
-- **Slope-matched splices** between regions (C1 continuity — no kink in how
-  adjacent-ranked tokens get treated).
-- **Curve family:** signed / `log1p`-style forms to stay finite on negative
-  logits. An HLG-style architecture (gamma toe + log shoulder, slope-matched) is
-  the leading candidate because its two-regime structure happens to match the
-  two-regime structure of a real logit distribution — temperature-style shaping
-  on the uncertain tail, log-style compression on the overconfident head.
-
-**Why it's not overclaiming.** I am *not* claiming buried capability that a curve
-"unclips" — the logits aren't a compressed encoding of something richer. The
-honest claim: there is an acceptable manifold the model can't articulate and the
-default transform doesn't reliably land on; the curve encodes my
-*observer-relative acceptance function* onto a distribution that has no opinion
-of its own. This is why sampling knobs exist at all — "friendly / neutral /
-professional" are crude gestures at exactly this, and a grade is the
-high-resolution version.
-
-**The per-agent-target insight (the payoff).** "Correctly transformed to display"
-is not universally acceptable — it's unacceptable for an Oscar-buzz feature and
-fine for a sports broadcast, because they have different acceptance functions.
-Grading Marvel like Wes Anderson isn't incompetence; it's the wrong *target*. My
-agents are different films:
-
-- A **conversation→article synthesis** agent wants range, texture, and
-  surprising-but-apt word choices — "cinematic."
-- An **acceptance-criteria→tests** agent wants tight, predictable, low-variance
-  determinism — "broadcast neutral."
-
-They want different *grades*, not different *temperatures*. Per-task tonal
-curves, not per-task scalar knobs.
-
-**Open engineering question — build this first.** The colorist works in a tight
-perceptual loop against a calibrated reference monitor; a sampler's curve is
-chosen *upstream* of seeing output, with a slower, noisier judgment loop. So the
-gating question isn't "is the curve good" — it's **"what is the monitor?"** What
-readout tells me the midtones got crisper and the highlights rolled off well, per
-target? Candidates: held-out preference, diversity-at-fixed-quality, task-specific
-pass rates (for the test agent) vs. human preference (for the synthesis agent).
-Define the per-target reference monitor first; the curve design follows from it
-the way a grade follows from a trustworthy display.
-
-**Implementation.** A pure elementwise function on the logits tensor inside
-mlx-bun's per-step logits-processor hook: sort once for the rank-relative pivot,
-apply the slope-matched toe/mid/shoulder curve, renormalize, sample. The hook
-already exists — logits processors run at
-[`generate.ts:236`](../../src/generate.ts:236)
-(`for (const p of processors) logits = disposing(logits, p(history, logits))`),
-built by `makeLogitsProcessors` in [`src/sampler.ts`](../../src/sampler.ts) as pure
-`(history, logits) => logits` functions applied before `toLogprobs` and the
-sampler — so the curve drops in as one more processor in that loop.
-
-Given dispatch dominates decode wall time (~90–96%), the curve's arithmetic cost
-rides inside work already dispatched — wall-time-negligible. Parameterize: pivot
-rank, toe exponent, midtone slope, shoulder compression. A/B against plain
-temperature **at matched effective entropy**, judged on the per-target monitor —
-looking for outputs that land in the target region *unreachably* by
-temperature-plus-top-p at any setting, not merely "different."
-
-*Relation to current code:* the `--temperature` / `--top-p` / `--top-k` server
-defaults (and `generation_config.json`) are the SDR knobs this topic proposes to
-supersede — useful as the A/B baseline, not the destination.

package/docs/planning/release-notes-v0.4.0.md

@@ -1,45 +0,0 @@
-# mlx-bun v0.4.0
-
-This release brings shared batched decode, generated-history cache reuse and
-resumable generation together through separate scheduling, inference, sampling
-and persistence interfaces.
-
-- Shared ordinary and speculative execution supports Qwen MTP, prompt lookup,
-  standalone, assistant, DeepSpec and DSpark/DFlash providers on qualified
-  model/request combinations. Single requests run at B=1 through the scheduler.
-- Full and rotating affine/TurboQuant cache layouts compose with delayed and
-  per-layer precision, speculative rollback and generated target/draft state.
-- Completed decode populates the shared RAM/SSD cache so later turns can reuse
-  generated history. SSD persistence is queued and runs during idle ownership.
-- Ordinary interrupted-generation resume now works through shared execution,
-  including compatible adapters and quantized caches. Restored sampler history,
-  pending tokens and usage accounting preserve the continuation contract.
-- Ordinary paged storage and adapter-qualified prompt lookup use the same state
-  and execution interfaces. GLM native MTP gains shared cache transactions and
-  row state; final real-artifact testing remains deferred.
-- Measured Trellis v13 is selected by default, and eligible Qwen KV4 speculation
-  is enabled. Workload-dependent lookup and other experimental settings remain
-  opt-in. Memory fit estimates are advisory unless users choose explicit limits.
-- Benchmark reporting retains request/source identities, paired comparisons,
-  failures and configuration differences. CI and the release build use Bun 1.4.2.
-
-Serving keeps its existing `--batch 8` default. A lone eligible request starts
-immediately at B=1; the cap does not wait for eight requests. `--batch 1` retains
-the serial executor, and unsupported compositions can still use it. Removing
-serial and completing every feature combination are follow-up work.
-
-Checkpoint capture is eventually persistent: queued snapshots can remain in RAM
-during uninterrupted traffic. A successful explicit flush is the durability
-boundary. Media, fill/grammar-jump and direct paged-attention work remain tracked.
-No DSpark/DFlash checkpoint has been trained for this campaign; execution fixtures
-do not establish trained-drafter performance.
-
-Measured performance and its workload limits are in the
-[benchmark record](https://github.com/joshuarossi/mlx-bun/blob/v0.4.0/docs/reference/benchmarks.md).
-This release does not claim a matched full-task Kanban speedup or completion of
-the entire optimization program. Supported settings are documented in the
-[server configuration reference](https://github.com/joshuarossi/mlx-bun/blob/v0.4.0/docs/reference/server-config.md).
-
-Requires Apple Silicon and macOS 14 or later. npm installs require Bun 1.4.0 or
-later; Bun 1.4.2 is the tested version. The existing native runtime pack 0.4.0
-provides MLX 0.32.2 and its matching C API.

package/docs/planning/video-series.md

@@ -1,421 +0,0 @@
-# mlx-bun video series
-
-## Episode 1: local AI on a Mac without the setup ritual
-
-### Production brief
-
-- **Working title:** Running a 27B Model Locally on a Mac, Without Python
-- **Thumbnail text:** 27B. LOCAL. NO PYTHON.
-- **Target length:** 8 minutes
-- **Audience:** developers who know what local models are but have not used MLX
-- **Machine:** whichever machine is used for the final recorded benchmark; show
-  its chip and memory alongside the results
-- **Demo model:** `mlx-community/Qwen3.8-27B-OptiQ-4bit`, already downloaded
-- **Presentation:** calm, direct, mostly terminal and browser capture with short
-  pieces to camera
-- **Question the episode answers:** Can local AI on a Mac feel like installed
-  software instead of an ML engineering project?
-
-This episode proves the product before explaining the implementation. It does
-not tour every feature. Memory, training, model ports, Bun FFI, and the failed
-experiments belong in later episodes.
-
-### Recording setup
-
-Prepare two terminal profiles:
-
-1. A clean profile for the installation shot. Do not begin the model download
-   during the take unless the download duration is known and useful on camera.
-2. The normal development profile with Qwen3.8-27B already downloaded. Use
-   this for the live server and API demonstrations.
-
-Before recording, confirm the exact model selector accepted on the machine:
-
-```sh
-mlx-bun ls
-mlx-bun fit Qwen3.8-27B --ctx 8192
-```
-
-Start the demo server shortly before the live take if model loading or browser
-placement would interrupt the opening. The visible command should still be the
-real command:
-
-```sh
-mlx-bun serve Qwen3.8-27B
-```
-
-Use a fresh chat and disable notifications. Set the terminal to at least 18 pt.
-Keep the browser and terminal at the same zoom level throughout.
-
-### Shot-by-shot script
-
-#### 0:00 to 0:12, cold open
-
-**Picture**
-
-Close shot of an empty terminal. Type `mlx-bun serve Qwen3.8-27B`. Cut directly
-to the browser as the chat opens. Ask a short question whose answer starts
-quickly.
-
-Suggested prompt:
-
-> Why can a 27-billion-parameter model fit on this 32 GB Mac? Answer in three
-> sentences and show the memory arithmetic.
-
-Do not wait for the full answer. Once several words appear, cut to camera.
-
-**Voiceover**
-
-> That is a 27-billion-parameter Qwen model running locally on a Mac. There is no
-> Python server behind it, no API key, and no request leaving the machine. One
-> command started the model, the chat interface, and an API that existing tools
-> can use. This is mlx-bun.
-
-**On-screen text**
-
-```text
-<machine> · Qwen3.8-27B · 20.4 GB weights
-```
-
-#### 0:12 to 0:35, state the problem
-
-**Picture**
-
-Piece to camera. Brief cuts to a deliberately restrained list of familiar setup
-artifacts: a Python environment, package installation, a server command, and a
-model configuration file. Do not use fake error messages.
-
-**Dialogue**
-
-> Local AI already works on a Mac. The rough part is everything around the
-> model. You choose a model that might fit, assemble a Python environment,
-> start a server, and then configure the application that was supposed to use
-> it.
->
-> I wanted the default path to feel like installing software. Run one command,
-> get a useful local assistant, and keep the controls available when you
-> actually need them.
-
-#### 0:35 to 1:10, show the real first-run path
-
-**Picture**
-
-Show the installation command in the clean terminal. Use a pre-recorded,
-time-compressed capture of the first run beside it. Label the time compression.
-
-```sh
-curl -fsSL https://mlx-bun.dev/install.sh | sh
-mlx-bun
-```
-
-Show the starter model becoming ready. Then cut to the explicit Qwen download
-command used before the main recording:
-
-```sh
-mlx-bun get mlx-community/Qwen3.8-27B-OptiQ-4bit
-```
-
-**Voiceover**
-
-> The installer downloads a signed, notarized executable. On a fresh machine,
-> bare `mlx-bun` downloads a sub-gigabyte MiniCPM5 starter so the first chat can
-> begin quickly.
->
-> This episode is about a much larger model, Qwen3.8-27B. Its quantized weights
-> are 20.4 gigabytes, so I downloaded them before recording. I do not think
-> watching a twenty-gigabyte progress bar would improve the video.
-
-**Verification note**
-
-Capture this section again if first-run behavior changes. Do not narrate the
-device-aware model selection described in the product roadmap until that lookup
-is implemented.
-
-#### 1:10 to 1:55, one process with three ways in
-
-**Picture**
-
-Use a simple screen composition with the running `mlx-bun` process in the
-center. Reveal the chat UI, an HTTP request, and a TypeScript file around it.
-
-```text
-Chat UI
-   \
-    mlx-bun + MLX + Metal
-   /                     \
-OpenAI-compatible API    TypeScript library
-```
-
-**Dialogue**
-
-> The visible chat is only one way into the runtime. The same process exposes an
-> OpenAI-compatible API, so applications that already speak that protocol can
-> point at localhost. A Bun application can also call the generation library
-> directly without running a sidecar server.
->
-> Those are different entry points into the same native runtime. The model
-> weights stay on the Mac, and MLX runs the computation through Metal on the
-> Apple Silicon GPU.
-
-#### 1:55 to 2:35, prove the API claim
-
-**Picture**
-
-Keep the server log visible on the left. Run this request on the right:
-
-```sh
-curl http://localhost:8080/v1/chat/completions \
-  -H 'Content-Type: application/json' \
-  -d '{
-    "messages": [{
-      "role": "user",
-      "content": "Name two reasons unified memory helps local inference."
-    }],
-    "max_tokens": 96,
-    "temperature": 0
-  }'
-```
-
-Format or crop the JSON response so the answer and usage fields remain legible.
-
-**Voiceover**
-
-> This is a normal chat-completions request. Change an OpenAI client base URL to
-> `localhost:8080`, and the request goes to the model loaded by mlx-bun. There
-> is no translation service on the internet and no account involved.
-
-Do not say every OpenAI feature is implemented. The claim is protocol
-compatibility for the documented API, not the entirety of OpenAI's platform.
-
-#### 2:35 to 3:25, explain why Bun matters
-
-**Picture**
-
-Piece to camera, followed by a clean runtime diagram. Avoid scrolling through
-large source files here.
-
-```text
-TypeScript application
-        ↓
-       Bun
-        ↓
-  mlx-bun native bindings
-        ↓
-       MLX
-        ↓
- Apple Silicon GPU
-```
-
-**Dialogue**
-
-> Most MLX applications are written in Python. mlx-bun calls MLX's native C API
-> from Bun instead.
->
-> It loads safetensors weights, builds the model graph, manages the attention
-> cache, samples tokens, and serves requests from TypeScript. It does not start
-> Python in the background.
->
-> That does not make GPU matrix multiplication faster because it passed through
-> JavaScript. The GPU still sets that limit. It does make the application easier
-> to distribute, and it gives the server less work to do around each model step.
-
-#### 3:25 to 4:15, show the finite-matrix idea
-
-**Picture**
-
-Run the fit command. Hold on the result long enough to read memory use, safe
-context, and predicted speed. Use the exact live output rather than rebuilding
-it for the edit.
-
-```sh
-mlx-bun fit Qwen3.8-27B --ctx 8192
-```
-
-**Dialogue**
-
-> The project makes one opinionated bet. Local AI on a Mac is a finite problem.
-> Apple ships a known set of chips and memory sizes. mlx-bun supports a curated
-> set of model families. We can measure those combinations instead of asking
-> every user to become a model-serving specialist.
->
-> The `fit` command estimates whether a model and context length are safe before
-> loading the weights. The default path can be simple because the measurements
-> and the failure checks live underneath it.
-
-Do not claim that automatic device-specific recommendation is complete. Today,
-the fit contract is the concrete proof of this idea.
-
-#### 4:15 to 5:20, correctness before speed
-
-**Picture**
-
-Show a two-column comparison titled `Python oracle` and `mlx-bun`. Animate two
-small arrays of identical hexadecimal or decimal values. Follow with a quick
-shot of a parity test name from the repository.
-
-```text
-same model + same input
-
-mlx-lm logits  =  mlx-bun logits
-               bit-exact
-```
-
-**Dialogue**
-
-> Reimplementing a model runtime creates a nasty testing problem. Wrong models
-> still write convincing sentences.
->
-> So I do not use "the answer looked reasonable" as the correctness test. For
-> the compatibility path, the Python MLX implementation is the oracle. The same
-> weights and inputs go through both runtimes, and the tests compare the logits
-> before a token is selected.
->
-> On the validated paths, those logits match bit for bit. That catches errors in
-> attention masks, normalization, position encoding, cache updates, and model
-> loading that a chat demo can easily hide.
->
-> Compatibility work has a strict order here. Match the reference first. Then
-> measure any optimization against that baseline.
-
-#### 5:20 to 6:20, measured performance
-
-**Picture**
-
-Display one benchmark card. Do not show the entire results table.
-
-```text
-Qwen3.8-27B · <machine> · warm HTTP serving
-Preflight-gated run · 2026-08-22 · Bun 1.4.0
-
-                 short decode   decode at 15.8k   warm TTFT   aggregate ×4
-mlx-bun          18.6 tok/s      17.0 tok/s        116 ms      19.5 tok/s
-mlx-lm           16.5 tok/s      14.8 tok/s        284 ms      18.6 tok/s
-```
-
-Add a small source line: `docs/reference/benchmarks.md`.
-
-**Dialogue**
-
-> Here is a measured comparison on this Mac. Same Qwen model snapshot, same
-> machine state, warm HTTP serving.
->
-> On the short request, mlx-bun generated 18.6 tokens per second. mlx-lm
-> generated 16.5. At about 15,800 tokens of context, the result was 17.0 versus
-> 14.8. Warm time to first token was 116 milliseconds for mlx-bun and 284 for
-> mlx-lm.
->
-> This is the kind of comparison I care about. It uses the server path an
-> application actually hits, the same weights, and a machine that passed the
-> benchmark preflight. The GPU sets the speed limit, but the runtime still has
-> to reach that limit without wasting memory or time around each token.
-
-**Editorial note**
-
-These are controlled same-machine results. Put the run date and Bun version on screen.
-If a fresh cleared-machine benchmark is available before publication, replace
-the full card and narration together. Never mix live numbers with this table.
-
-#### 6:20 to 7:05, show the larger project without losing focus
-
-**Picture**
-
-Use four quick clips, about three seconds each:
-
-1. An image input in chat.
-2. The Markdown memory directory.
-3. A training progress display.
-4. A benchmark or parity result from the lab.
-
-Return to camera before naming more features.
-
-**Dialogue**
-
-> Serving is the base of the project, not its boundary. The same binary also has
-> image and video input on supported Qwen models, a local Markdown memory, LoRA
-> training, and a lab for decoding and quantization experiments.
->
-> I am not going to explain those in a thirty-second feature list. Each one has
-> an implementation story, measurements, and a few failures worth discussing.
-> Those will be separate videos.
-
-#### 7:05 to 7:50, close and set up episode 2
-
-**Picture**
-
-Return to the original chat. Show the local URL, server process, and activity
-monitor GPU history in quick succession. End on the repository and site URLs.
-
-**Dialogue**
-
-> mlx-bun is my attempt to make local AI on Apple Silicon behave like software
-> you can install, use, and build on.
->
-> The short version is one command. Underneath it is a TypeScript model runtime,
-> native bindings, model-family ports, memory accounting, and a lot of parity
-> tests.
->
-> In the next video, I am going to trace one generated token through that whole
-> path, from the TypeScript call to MLX and the GPU, then back to the response.
-> That is where the project gets strange.
-
-**On-screen text**
-
-```text
-mlx-bun.dev
-github.com/joshuarossi/mlx-bun
-```
-
-### Required captures
-
-- Clean installer command.
-- Time-compressed first run with MiniCPM5 becoming ready.
-- The explicit Qwen3.8-27B download command or a short time-compressed capture.
-- `mlx-bun serve Qwen3.8-27B` startup and automatic browser open.
-- A short streamed chat response.
-- A successful `/v1/chat/completions` curl request.
-- Live `mlx-bun fit Qwen3.8-27B --ctx 8192` output.
-- One real parity test or stored oracle comparison.
-- Four brief clips for vision, memory, training, and benchmark evidence.
-- Repository and website closing shot.
-
-### Graphics to prepare
-
-Only three graphics are needed:
-
-1. The three-entry-point runtime diagram.
-2. The bit-exact oracle comparison.
-3. The dated Qwen3.8-27B benchmark card.
-
-Keep them monochrome except for one mlx-bun accent color. Use the same terminal
-font in the diagrams and screen recordings.
-
-### Recording-day checks
-
-- Confirm the current release installation command.
-- Confirm the Qwen3.8-27B selector and exact displayed model name.
-- Confirm port 8080 is unused before the take.
-- Confirm no private paths, tokens, conversations, or model-cache usernames are
-  visible.
-- Confirm first-run behavior still matches the narration.
-- Confirm the API request and response fields against the current reference
-  docs.
-- Confirm the benchmark card still matches `docs/reference/benchmarks.md`.
-- Record room tone and five seconds of clean terminal footage for edits.
-- Capture every important terminal action twice.
-
-### Episode 2 handoff
-
-The promised follow-up is: **What happens when mlx-bun generates one token?**
-
-Trace this path without turning the episode into a source-tree tour:
-
-```text
-request
-  → tokenizer
-  → model forward pass
-  → MLX lazy graph evaluation
-  → Metal execution
-  → GPU sampling
-  → one token returned to Bun
-  → streamed protocol event
-```