mlx-bun 0.4.0 → 0.5.0

package/docs/planning/public-readiness-handoff.md

@@ -0,0 +1,244 @@
+# Public readiness — handoff (2026-09-11)
+
+Task list and verified context for the outreach thread. The scope and
+acceptance criteria for each milestone live in
+[public-readiness.md](public-readiness.md); milestone state lives in
+[PLAN.md](../../PLAN.md#public-readiness). This document says what was
+verified today, what that changes, and what to do next, in order.
+
+## Update after handoff
+
+The GPTQ report is filed as [mlx-lm #1878](https://github.com/ml-explore/mlx-lm/issues/1878).
+The issue contains the complete runnable reproduction and proposed fix; the
+local reproduction files have been removed. Independent-reference regression
+coverage and upstream follow-up remain open. Presentation changes and GitHub
+metadata are also complete on this branch. The sections below preserve the
+review handoff as written; their original state labels predate this update.
+
+## What was verified today
+
+**The listing target is stalled.** The mlx-framework.org "community" section
+is a single link to [discussion #654](https://github.com/ml-explore/mlx/discussions/654);
+there is no separate curated page. The pinned list in #654 was last edited
+2025-10-17. Every comment since March 2026 (at least 20, including two
+projects named `mlx-serve`, `mlx-node`, `atlas`, `squish`, `mlx-chronos`,
+`mlx-tsfm`) has no maintainer reply and none were added. Only `awni` or an
+ml-explore member with write access can edit that post.
+
+**Awni Hannun no longer maintains MLX.** He left Apple in February 2026 (last
+`ml-explore/mlx` commit 2026-02-24) and is at Anthropic. He still comments on
+mlx-lm issues occasionally (last seen 2026-08-24) but is not the person who
+will act on a listing request. Named successors: Angelos Katharopoulos
+(`angeloskath`), Cheng Zhao (`zcbenz`), Jagrit Digani (`jagrit06`),
+Anastasiia Filippova (`nastya236`), Ronan Collobert. Recent mlx-lm committers:
+`michalk8`, `zcbenz`, `angeloskath`.
+
+**The GPTQ bug is live upstream and unreported.** `mlx_lm/quant/gptq.py`
+on `main` (1b3594b, 2026-09-09) still contains both defects at lines 126–127:
+the `k : k + j` window that propagates past the block edge, and
+`err[..., k : k + 1]` indexing a group-local buffer with a global column
+index (a silent no-op for every block after the first). Issue search for
+`gptq` in ml-explore/mlx-lm returns only perf/memory reports (#1094, #1098,
+#1122). The reproduction is included in [mlx-lm #1878](https://github.com/ml-explore/mlx-lm/issues/1878).
+
+**The category is crowded.** Already in the #654 thread: `mlx-serve`
+(ddalcu, Zig, no Python, Gemma 4 + Qwen 3.x, MTP, TurboQuant KV, OpenAI +
+Anthropic APIs), `mlx-serve` (raspoli), `mlx-node` (Node.js inference and
+training APIs), `mlx-omni-server`, `squish`. "No Python OpenAI-compatible
+server" is not a differentiator. "TypeScript library" needs a comparison
+with `mlx-node` before it is claimed as one.
+
+**Other places a project actually gets listed.** `raullenchai/awesome-mlx`
+(466 stars, last push 2026-04-11) accepts PRs. `igurss/mlx-chronos` (last
+push 2026-08-22) is a benchmark suite and leaderboard for MLX inference
+engines with reproducible JSON submissions.
+
+## What this changes
+
+- A #654 comment is the last step and should be expected to sit. The
+  outcomes that are actually reachable, in order of value: a merged or
+  acknowledged upstream fix; a Show-and-tell post that a current maintainer
+  replies to; an `awesome-mlx` PR; an `mlx-chronos` submission; and, if any
+  of those produce a conversation with a maintainer, the listing ask goes in
+  that thread rather than into the #654 pile.
+- Outreach copy should not be written for Awni. It should be written for
+  `angeloskath`/`zcbenz`-style readers: correctness first, measured numbers
+  with machine labels, an upstream contribution in hand, no digs at mlx-lm.
+- The first-impression fixes matter more, not less: a maintainer visit will
+  be rare and there is one chance at it.
+
+## State of the work (working tree, this branch)
+
+Done or in flight in the uncommitted diff on `chore/public-readiness`:
+
+- `docs/planning/public-readiness.md` rewritten as P0–P5; PLAN block updated.
+- `docs/reference/benchmarks.md` restructured: parity table first, latest
+  h2h second; campaign ledger moved to
+  `docs/archive/investigations/benchmark-ledger-through-v0.4.0.md`.
+- README lead paragraph narrowed to library + server; STATUS/PLAN link
+  removed from the README table.
+- `lab/repro/mlx-lm-gptq-block-indexing/repro.py` written (CPU-only, no
+  weights; asserts the installed source matches, runs original vs corrected).
+- `docs/reference/models.md` edited (roster/evidence reconciliation, partial).
+
+Not started: oracle setup script and lock (P2), package ownership map (P3),
+tarball/first-run/library-consumer verification and GitHub metadata (P4),
+every outreach draft (P5).
+
+## Tasks, in order
+
+Each task names its owner: **agent** (can be done in a session), **Josh**
+(needs approval, an account, a server, or a download). Session rules apply:
+no long-running servers or multi-GB downloads from a session.
+
+### T1 — Upstream GPTQ report (P0) — agent drafts, Josh submits
+
+Filed 2026-09-11 as [mlx-lm #1878](https://github.com/ml-explore/mlx-lm/issues/1878)
+(both defects, CPU-only fixture, observed result on 0.31.3 and upstream
+`e5962529`, proposed two-line patch). Recorded in
+[turboquant.md](../design/turboquant.md) and `scripts/turboquant/tq-gptq.py`.
+
+Fix open as [mlx-lm PR #1880](https://github.com/ml-explore/mlx-lm/pull/1880)
+from `joshuarossi/mlx-lm:fix-gptq-block-indexing` (`d4d38d9`): the two-line
+change in `mlx_lm/quant/gptq.py` plus `tests/test_gptq.py`, which compares
+`gptq_quantize` against a column-by-column reference (paper form, no lazy
+block update) over four groups, with a single-group control. On upstream
+`e596252` the multi-group test fails (max abs dequantized diff 0.38) and the
+control passes; on the branch both pass. black 25.1.0 / isort 6.0.0 / ruff
+0.16.6 clean. The issue body was rewritten in Josh's words and the PR
+description discloses AI-drafted code, per mlx-lm's CONTRIBUTING.md.
+
+Remaining: **follow up.** Check the PR for maintainer questions every few
+days; answer with test output, not prose (Josh writes the replies; the
+policy covers comments too). Record the outcome (merged, changed,
+declined) in turboquant.md.
+
+Acceptance: any maintainer question answered; outcome recorded.
+
+### T2 — Finish the internal-voice scrub (P1) — agent
+
+Current counts in public entry points (`rg -c "Josh"`): STATUS.md 4,
+`docs/reference/environment.md` 3. `benchmarks.md` is clean after the
+restructure; check the archive ledger is linked, not rendered by the site.
+Also grep the website source and every `docs/reference/*.md` for
+`authorized`, `diagnostic mode`, `machine policy`, `Josh requested`,
+`per Josh` and rewrite as technical conditions (keep the CPU/swap/memory
+facts, drop the approval framing). Do not touch `PLAN.md`/design docs for
+this; they are internal by design and no longer linked from the README.
+
+Acceptance: the greps return zero in README, STATUS, CONTRIBUTING,
+`docs/reference/*`, `website/src/**`; `bun scripts/check-hygiene.ts` passes.
+
+### T3 — Fix the site claims (P1) — agent
+
+`website/src/content/docs/index.mdx` "Why it's different" and the numbers
+table:
+
+- "Fastest to first token … of any stack tested" → "vs `mlx_lm.server` and
+  optiq serve" and state that it is server overhead at warm cache, not
+  model speed.
+- Remove "no segfaults on exit".
+- "bit-exact to the Python reference" → "bit-exact logits vs mlx-lm on the
+  validated configurations (link to the parity table)".
+- Reduce the eight feature cards to library, server/API, correctness,
+  measured performance; fold memory/curves/training/diffusion into one
+  "also in the box" line.
+- First-run model: the site shows `MiniCPM5-1B` (index.mdx:24,
+  quickstart.md:16, installation.mdx:81); `DEFAULT_REPO_ID` in `src/fit.ts:88`
+  is the 7 GB e4b. Pick one and make code, README and site agree.
+
+Acceptance: `bun run --cwd website build` passes; a narrow-viewport check
+of the landing page; `tests/docs-surface.test.ts` passes.
+
+### T4 — Roster vs parity evidence (P1) — agent
+
+`models.md` labels Qwen3.5-4B, Qwen3.8-27B and eleven Tier-0 archs L1;
+the parity table in `benchmarks.md` lists four Gemma/MiniCPM rows. For each
+L1 row, either add the artifact + test file to the parity table (the Qwen
+oracle cases and universal goldens exist under `tests/parity/` and
+`scripts/oracle/gen-universal-golden.py`) or mark the row "family-supported,
+artifact-unvalidated" in the roster. No row may claim L1 without a named
+test.
+
+Acceptance: every L1 in `models.md` resolves to a row in `benchmarks.md §1`.
+
+### T5 — Reproducible oracle (P2) — agent builds, Josh runs the clean check
+
+1. Add a new `setup.sh` under `scripts/oracle/` (plus a committed lock, pip
+   freeze or `uv` lock) that recreates the pinned venv anywhere and prints
+   the `MLX_BUN_ORACLE_VENV=` line. Until the file exists, do not write its
+   full path into a live doc: hygiene check 7 fails on script paths that do
+   not exist, and `public-readiness.md` P2 currently trips it. Pins come from
+   [environment.md](../reference/environment.md) (mlx 0.32.2, mlx-metal
+   0.32.2, mlx-lm 0.31.3, Python 3.13.5).
+2. Document one command chain that ends in a real pass, not a skip:
+   setup → `mlx-bun get MiniCPM5` (sub-GB) → regen the MiniCPM goldens with
+   `scripts/regen/minicpm5.ts` → `bun test tests/parity/minicpm5-parity.test.ts`.
+   Make the test fail loudly when goldens or venv are missing.
+3. Explain, in that section, machine-layered goldens and the fast-SDPA
+   dispatch boundary (L ≥ 16) so a reader on another Mac knows why their
+   binary goldens differ from the tracked manifest.
+4. Josh: run the chain on the other dev box from a clean checkout and record
+   wall time. Do not print a duration in docs until this run supports it.
+
+Acceptance: the chain passes on a machine that never had Josh's venv path.
+
+### T6 — Package ownership map (P3) — agent
+
+Produce the dependency map the milestone asks for
+(`reports/public-readiness/` already has source-import and dry-run
+reports): which `src/` roots the `mlx-bun`, `mlx-bun/engine`, `mlx-bun/client`
+exports reach, which runtime deps and native assets each needs, and what
+the CLI/server/app layers add. Recommend keep-as-is or extract, with the
+reason. Compare against `mlx-node`'s packaging while doing it, since that is
+the closest JS-side comparison and the pitch depends on the difference.
+
+Acceptance: the map and decision recorded in engine architecture §12.15;
+no code move unless justified in that record.
+
+### T7 — Candidate, tarball, metadata (P4) — agent prepares, Josh verifies
+
+- `bun pm pack --dry-run` and inspect: no `reports/`, `runs/`, goldens
+  blobs, or research output; `bin`, `src`, `docs` present.
+- GitHub repo settings (currently empty): description, homepage
+  `https://mlx-bun.dev`, topics `mlx apple-silicon bun llm metal
+  openai-compatible`. `gh repo edit joshuarossi/mlx-bun --description ...
+  --homepage ... --add-topic ...`.
+- Josh, on a clean user account or fresh machine: `curl … | sh`, first
+  `mlx-bun serve`, browser chat, one streaming request, one `openai` SDK
+  call, one library consumer importing `mlx-bun/engine`. Record what broke.
+
+Acceptance: as written in public-readiness.md P4.
+
+### T8 — Outreach drafts (P5) — agent drafts, Josh posts
+
+Write all four before posting any, so they agree:
+
+1. **Discord intro** (MLX Community, verify channel first): three
+   sentences, the parity command from T5, one ask ("does the TypeScript
+   integration make sense to you; what would you want from the library
+   API"). No numbers without a machine label.
+2. **Show and tell** post on ml-explore/mlx: what it is, how to reproduce
+   the parity claim, the h2h table with losses, the upstream GPTQ PR link,
+   known limits, the model roster. This is the artifact a maintainer might
+   actually read.
+3. **awesome-mlx PR** and an **mlx-chronos** submission (check the
+   submission format in that repo; our `bench-serve.ts` output may map to
+   it).
+4. **#654 comment**: the one-line sentence already drafted in
+   public-readiness.md P5.
+
+Post order: T1's upstream PR → Discord → Show and tell → awesome-mlx /
+chronos → #654. If a maintainer replies anywhere, ask for the #654 add in
+that reply.
+
+## Do not
+
+- Do not write copy aimed at Awni or predict Apple's reaction.
+- Do not compare against mlx-lm without the machine, artifact and settings
+  on the same line as the number.
+- Do not claim family-wide bit-exactness; claim it per validated artifact.
+- Do not start servers or multi-GB downloads from a session for T5/T7.
+- Do not touch `PLAN.md`, `public-readiness.md`, `benchmarks.md`, `README.md`
+  or `models.md` in this tree without checking `git status` first; another
+  session was editing them on 2026-09-11.

package/docs/planning/public-readiness.md

@@ -0,0 +1,149 @@
+# Preparing mlx-bun for the MLX community
+
+Success means that a curious developer can understand the project, reproduce
+its numerical claims, and build something with it. The listing is a discovery
+link; the repository and working examples have to earn further attention.
+
+The public pitch is MLX inference as a TypeScript/Bun library, with a signed
+executable serving OpenAI/Anthropic-compatible APIs. Correctness claims name
+the tested artifact and configuration. Specialized kernels, quantization,
+and reusable cache state support that product; ancillary applications do not
+need equal space in the introduction.
+
+Open milestones live in [PLAN.md](../../PLAN.md#public-readiness).
+Numbers belong in [benchmarks.md](../reference/benchmarks.md), with historical
+experiments in its linked ledger. Package decisions belong in the
+[engine architecture](../design/unified-engine-frontier-plan.md#1215-package-boundaries-before-community-outreach).
+
+## P0. Contribute the GPTQ finding upstream
+
+Check the reported indexing errors against a pinned upstream revision and
+search existing issues and pull requests before filing. Produce a minimal
+CPU-only reproduction that needs no model download. Demonstrate both the
+out-of-block update window and the global index into a block-local error
+buffer; a one-line fix is not enough. Keep the report about the numerical bug,
+with a proposed patch and a useful regression test.
+
+Acceptance: the reproduction runs against the cited revision, a reviewed
+issue or pull request is submitted when authorized, and the discussion is
+followed up. Upstream acceptance is not required to proceed with community
+feedback. Submitted as [mlx-lm #1878](https://github.com/ml-explore/mlx-lm/issues/1878) with the complete CPU-only
+reproduction, observed results and proposed two-line correction. Follow-up
+and an independent-reference regression test remain open.
+
+## P1. Make the first visit clear and credible
+
+Remove session-specific instructions, personal approvals, and handoff prose
+from README, STATUS, model/configuration references, benchmarks, and pages
+rendered by the website. Preserve observed CPU, swap, memory, artifact, and
+sampling conditions as technical facts. Do not turn diagnostic measurements
+into controlled experiments by removing their qualifications.
+
+The benchmark reference should open with its parity summary, then the latest
+standard head-to-head with wins and losses, then reproduction instructions.
+Move historical campaign details into a linked ledger and preserve old
+section links. Map every roster row claiming L1 to a concrete artifact and
+test, or explicitly describe the remaining evidence gap. Family support does
+not establish parity for every checkpoint or modality.
+
+Lead the README and site with the library and server. Link to contribution
+instructions instead of internal work tracking from the main README. Keep
+secondary applications in a short, separate line. Distinguish the first-run
+starter from the preferred model used once suitable weights are available.
+
+Acceptance: the public wording agrees with the implementation; no personal
+approval language remains in these public entry points; local links and the
+website build pass; desktop and narrow-screen presentation are reviewed.
+The claims about numerical agreement remain bounded by recorded evidence.
+
+## P2. Make correctness independently reproducible
+
+Implemented: [pinned setup and MiniCPM comparison](../reference/environment.md#reproduce-the-minicpm5-logit-comparison).
+A clean source export, new oracle environment and empty model cache pass on
+M1 Max with 100 exact vectors and tokens. Missing inputs fail explicitly.
+Complete per-roster artifact/run provenance remains open.
+
+Provide `setup.sh` under `scripts/oracle/` and a committed dependency lock for the
+reference environment. Select it through `MLX_BUN_ORACLE_VENV`; no contributor
+should need the maintainer's directory layout. Pin the native MLX/Metal build,
+Python dependencies, model revision, tokenizer/template inputs, and oracle
+fixture-generation command. Explain the machine-specific fixtures and the
+attention-dispatch boundary that makes another Mac's binary goldens unsuitable.
+
+Publish a small-model recipe ending in the MiniCPM parity test. It must create
+fixtures from the Python oracle and fail clearly on missing inputs, rather
+than reporting skipped tests as success. Measure the setup and test duration;
+do not promise ten minutes until a clean-machine run supports it. Explain
+what is compared before sampling and which execution shapes are tested.
+
+Acceptance: a clean checkout can install the pinned reference, obtain the
+specified artifact, regenerate its own oracle fixtures, and run a documented
+logit comparison. Publish enough provenance for each summary table row to be
+traceable. Existing accepted checks do not need wholesale reruns; this work
+checks whether an outsider can reproduce them.
+
+## P3. Settle package ownership
+
+Map imports, public APIs, dependencies, native assets, builds, tests and
+release packaging. Distinguish the reusable engine from HTTP/CLI assembly
+and applications. Decide whether workspaces improve independent installation
+and development enough to introduce them before outreach.
+
+Keep `mlx-bun`, `mlx-bun/engine`, and `mlx-bun/client` imports compatible.
+Preserve in-process execution, specialized kernels, tensor ownership and the
+self-contained executable. Interfaces do not each need their own package.
+No package extraction should add transport hops or copies in the decode path.
+
+Acceptance: record and implement justified pre-outreach moves, or document
+which internal moves can safely follow later. Test an installed consumer and
+bundled assets when extraction changes them. Postponing the decision without
+a dependency map does not complete this milestone.
+
+## P4. Verify the candidate and its presentation
+
+Inspect the npm tarball and release bundle. Keep runtime assets complete and
+exclude accidental research output. Check first-run chat, streaming, model
+selection, a small library consumer, and actionable errors against the
+candidate artifact. Reuse release and benchmark evidence where code and
+configuration have not changed. Default memory guidance remains advisory.
+
+Prepare a short demonstration and evidence bundle. Set an accurate GitHub
+repository description, homepage and topics: `mlx`, `apple-silicon`, `bun`,
+`llm`, `metal`. Verify that the intended release is easy to find; use the
+available GitHub release/highlight controls without promising a nonexistent
+pinning feature. Include straightforward issue reporting and contributor setup.
+
+Acceptance: the installed candidate and demo work as documented, repository
+metadata matches the pitch, and every public performance claim points to its
+machine and workload. Persistent servers and large downloads follow the
+repository's existing session rules.
+
+## P5. Engage, then request the listing
+
+Order: upstream GPTQ report, Discord introduction, GitHub Show and tell,
+then the community-project comment. Verify current venue instructions before
+posting. The [MLX Community Discord invite](https://discord.gg/wXk8tEqAcG)
+is a starting point to verify, not evidence about audience size or reception.
+
+Ask for feedback on the library integration and reproducible correctness
+comparison. Credit MLX, mlx-c, mlx-lm and mlx-optiq accurately. Treat rotation
+and trellis findings as a separate technical discussion with their quality
+tradeoffs; do not imply that all quantization methods preserve L1 numerics.
+
+The [Show and tell category](https://github.com/ml-explore/mlx/discussions/categories/show-and-tell)
+provides a durable, searchable introduction containing the example,
+reproduction steps, known limits and upstream contribution. Incorporate
+Discord feedback before preparing that post. Then follow the instructions in
+[community-project discussion #654](https://github.com/ml-explore/mlx/discussions/654).
+
+Draft listing sentence, for review:
+
+> [mlx-bun](https://github.com/joshuarossi/mlx-bun) brings MLX inference to TypeScript/Bun applications and ships a signed executable serving OpenAI/Anthropic-compatible APIs on Apple Silicon, with bit-exact logit tests against mlx-lm for validated configurations.
+
+The listing sentence deliberately omits an unqualified family-wide parity
+claim. Add the reproducible example and coverage link to the Show-and-tell
+post, where readers can inspect the precise scope.
+
+Acceptance: review the concrete drafts, post when authorized, triage feedback,
+and submit the listing comment. Website placement and maintainer enthusiasm
+are external outcomes, not completion promises. Hacker News comes later.