mlx-bun 0.0.4

package/docs/design/compat-cli-surface-design.md

@@ -0,0 +1,227 @@
+# Compat CLI surface + parity harness — design
+
+Status: design (2026-06-12). Engineering tracking lives in PLAN.md
+Phase 17. UX/DX rationale lives in docs/planning/PRODUCT_ROADMAP.md ("The three
+modes"). This doc is the surface contract: every verb, the flag
+vocabulary, how optiq is absorbed, the parity harness, and **the
+matrix of what we still lack**.
+
+## Goal
+
+Design mlx-bun's entire CLI/flag surface from scratch (nothing is
+published, nothing is set in stone) so that one coherent set of verbs
+serves three audiences at three depths — automatic / compat /
+stick-shift (see PRODUCT_ROADMAP). mlx-lm is the vocabulary we borrow,
+because it is the Apple-WWDC-referenced default and outweighs the
+alternative ~425× on monthly installs (1.64M vs 3.85k; mlx-lm 5.8k
+stars / 765 forks; mlx-optiq has no public repo — measured
+2026-06-12).
+
+## Scope boundary: surface parity ≠ architecture parity
+
+This is CLI/flag-**surface** parity. It is NOT a promise to run every
+model architecture mlx-lm supports — the design principle "scope is the
+survival strategy: target Gemma, Qwen, one MoE; not parity with mlx-lm's
+dozens" still holds. A compat verb invoked against an unsupported
+architecture returns a clear `architecture not supported by mlx-bun yet
+(supported: …)` error, never a silent failure or a wrong answer. The
+flag is accepted; the model may not be.
+
+## The three laws (from PRODUCT_ROADMAP, restated as surface rules)
+
+1. **One surface, mlx-lm vocabulary.** `mlx-bun.<verb>` is a pure alias
+   of `mlx-bun <verb>`. Adopt mlx-lm's flag name wherever it is the
+   standard; name a flag ourselves only where mlx-lm has no concept for
+   it (free to beat optiq's name there).
+2. **Superset, never intersection.** Every documented mlx-lm flag
+   behaves faithfully so scripts drop in unchanged; we never gate a free
+   capability to resemble an upstream (OpenAI+Anthropic+Responses all
+   stay on).
+3. **Gate only on conflict.** A capability is always on unless leaving
+   it on would degrade the automatic or compat experience (changes
+   output determinism, trades quality, surprises a script) — only then
+   does it move behind a stick-shift flag.
+
+## Verb map (the spine = mlx-lm)
+
+Each verb is reachable as `mlx-bun <verb>` and the dotted alias
+`mlx-bun.<verb>`. Flags follow `mlx_lm.<verb> --help` exactly for the
+shared subset; optiq-only features are additive (marked ⊕).
+
+| Verb | Maps to | mlx-bun engine today | Notes |
+|---|---|---|---|
+| `generate` | mlx_lm.generate | ✅ `generate.ts` (no CLI verb) | wire verb + full flag set |
+| `chat` | mlx_lm.chat | engine ✅, web chat ✅; no terminal REPL (`tui.ts` is help-formatting) | build REPL frontend over `generate` |
+| `server` | mlx_lm.server / optiq serve | ✅ `serve` | rename flags; ⊕ `--kv-config`, `--anthropic` honored (default on) |
+| `convert` | mlx_lm.convert + optiq convert | ❌ | quant pipeline; ⊕ `--target-bpw/--candidate-bits/--reference` |
+| `lora` | mlx_lm.lora + optiq lora | serve/hot-swap ✅, train ❌ | `--train/--test`; ⊕ `--rank-scaling by_bits\|by_kl`; `lora info` |
+| `fuse` | mlx_lm.fuse | ❌ | fuse adapters, dequant, GGUF export, upload |
+| `cache_prompt` | mlx_lm.cache_prompt | runtime ✅, CLI ❌ | precompute + save reusable KV cache file |
+| `benchmark` | mlx_lm.benchmark / optiq benchmark | ✅ `benchmark` | flag-name parity (`-p/-g/-b/-n`); ppl half ❌ |
+| `evaluate` | mlx_lm.evaluate / optiq eval | ❌ (`evals` = our bench viewer) | task harness; ⊕ optiq task names |
+| `perplexity` | mlx_lm.perplexity | ❌ | ppl on HF dataset |
+| `upload` | mlx_lm.upload | ❌ | push MLX dir to HF |
+| `manage` | mlx_lm.manage | ✅ `get`/`scan`/`ls` | add `--delete/--pattern/--scan` parity |
+| `kv-cache` | optiq kv-cache | runtime ✅, profiler ❌ | per-layer sensitivity → `kv_config.json` |
+| `latency` | optiq latency | ✅ `fit` | alias + `--calibrate` |
+| `awq` | mlx_lm.awq | ❌ | activation-aware quant |
+| `dwq` | mlx_lm.dwq | ❌ | distilled quant |
+| `dynamic_quant` | mlx_lm.dynamic_quant | ❌ | sensitivity mixed-bit (overlaps convert/TurboQuant) |
+| `gptq` | mlx_lm.gptq | ❌ | GPTQ |
+| `share` | mlx_lm.share | ❌ | distributed cache share (lowest priority) |
+
+Our own verbs stay (`fit`, `pi`, `harness`, `doctor`, no-args
+appliance) — they are the automatic/stick-shift surface and have no
+mlx-lm equivalent. See the native-surface section below.
+
+## mlx-bun-native surface (the differentiators)
+
+These have NO mlx-lm/optiq equivalent — they are not parity items, they
+are what makes mlx-bun *more* than a faster mlx-lm. The compat layer
+(Phase 17) is the "drop-in for mlx-lm" half; this is the "more than
+mlx-lm" half.
+
+### pi — the built-in agentic coding CLI
+
+(Phase 16; full plan in **docs/pi-builtin-investigation.md**.) pi is an
+MIT, Bun-native (`bun build --compile`) coding agent — the "CLI
+experience inside our package" that neither Python package has. Two
+commands, one for each audience:
+
+1. **`mlx-bun pi`** (the appliance, exists) — start the server (reuse if
+   healthy) and run pi's own `InteractiveMode` **in-process**: one
+   binary, one command, a real agent on a local model with nothing to
+   install (pi is a bundled dep). Custom mlx-bun coding-agent prompt,
+   full coding tools + web tools, pi's native approval prompts.
+   `-p`→`runPrintMode`, `--mode json/rpc`→`runPrintMode`/`runRpcMode`.
+2. **`mlx-bun harness pi`** (for people who already use pi, exists) —
+   point the user's OWN pi at mlx-bun's local model, via a generated
+   extension that dynamically discovers models from our `/v1/models`.
+   No models.json surgery; reversible. This is where the full pi flag
+   surface (`--continue`, `--resume`, `@file`, extensions, themes) lives.
+
+History: an earlier cut shipped `mlx-bun pi` as a subprocess launcher of
+the user's own pi (and briefly a `--external-pi` flag); both were dropped
+(2026-06-14) — a user who already has pi already knows how to run it, so
+it was pure duplication. The embedded agent + `harness pi` is the clean
+split.
+
+### Built-in web UI (OptIQ-Lab parity, pi-powered)
+
+The web UI's agent backbone is pi's RPC / `AgentSession.subscribe()`
+event stream (investigation Option 4) — the SAME events drive the
+embedded TUI and the browser, so tool calls / steering / progress come
+for free. Tiles: chat ✅; quantize / fine-tune / training-data gate on
+the capability matrix.
+
+## Flag-vocabulary reconciliation (where we currently diverge)
+
+- `serve --kv-quant N` → adopt mlx-lm's `--kv-bits` + `--kv-group-size`
+  + `--quantized-kv-start` (keep `--kv-quant` as a deprecated alias).
+- `serve --prompt-cache` → mlx-lm's `--prompt-cache-size` (entries) +
+  `--prompt-cache-bytes` (our byte-cap is already the right model).
+- Our runtime levers (`--compiled-decode`, `--perf-kernel`,
+  `--fused-decode`, `--fused-sdpa`, `--force-wire`) have no mlx-lm
+  equivalent → keep, these are stick-shift flags.
+- Adopt mlx-lm's sampling names verbatim: `--temp`, `--top-p`,
+  `--top-k`, `--min-p`, `--max-tokens/-m`, `--seed`, `--xtc-*`.
+
+## What we still lack — capability gap matrix
+
+Legend: ✅ have · 🔌 wiring-only (engine exists, needs verb/flags) ·
+🟥 missing capability (real build). The compat **layer** is the 🔌
+column; the 🟥 column is the capability matrix that fills in behind it.
+
+| Capability | Verb(s) | Status | Bucket / owning phase |
+|---|---|---|---|
+| One-shot generate | `generate` | 🔌 | Phase 17 (compat) |
+| Chat REPL | `chat` | 🔌 | Phase 17 |
+| OpenAI/Anthropic/Responses serve | `server` | ✅ | done (Phase 4/11) |
+| Server flag parity | `server` | 🔌 | Phase 17 |
+| Benchmark (throughput) | `benchmark` | ✅ flags 🔌 | Phase 17 |
+| Latency prediction | `latency`/`fit` | ✅ | done; `--calibrate` 🔌 |
+| Model cache manage | `manage` | 🔌 | Phase 17 |
+| Precompute prompt cache | `cache_prompt` | 🔌 | Phase 17 |
+| LoRA serve / hot-swap | `lora` (serve) | ✅ | done (Phase 8) |
+| `lora info` reporting | `lora info` | 🔌 | Phase 17 |
+| **HF→MLX convert → new model artifact** | `convert` | 🟥 | Model-quant (new phase) |
+| **Sensitivity / mixed-precision weights** | `convert`, `dynamic_quant` | 🟥 | Model-quant |
+| **AWQ / DWQ / GPTQ (weights)** | `awq`/`dwq`/`gptq` | 🟥 | Model-quant |
+| KV-cache quant: bf16 / uniform / **mixed per-layer** (kv_config.json) | `server` (`off`/`N`/`config`) | ✅ | done (Phase 9/10; config.ts reads, generate.ts applies per-layer) |
+| KV sensitivity *profiler* (authors a NEW kv_config.json) | `kv-cache` | 🟥 | model-prep, low pri (shipped artifacts already include one) |
+| TurboQuant — optional extra quant *method* (rotation VQ) | scheme, not a verb | 🟥 | Phase 13 (KV plumbing already exists) |
+| **LoRA / DoRA / full training** | `lora --train` | 🟥 | Training (new phase) |
+| **Sensitivity-aware rank scaling** | `lora --rank-scaling` | 🟥 | Training |
+| **Fuse adapters (+GGUF/upload)** | `fuse` | 🟥 | Training |
+| **lm-eval-harness tasks** | `evaluate` | 🟥 | Eval (new phase) |
+| **optiq eval suite (kl/gsm8k/ifeval/bfcl/hashhop…)** | `evaluate` | 🟥 | Eval |
+| **Perplexity** | `perplexity` | 🟥 | Eval |
+| **HF upload** | `upload` | 🟥 | Distribution |
+| **Distributed share** | `share` | 🟥 | Distribution (lowest pri) |
+| **Web UI: quantize** | `lab`/UI | 🟥 | gated on Quantize |
+| **Web UI: fine-tune** | `lab`/UI | 🟥 | gated on Training |
+| **Web UI: training-data template + gen** | `lab`/UI | 🟥 | new (no engine dep) |
+| Web UI: chat | `/chat` | ✅ | done (publishing gate 2) |
+| **Native: pi embedded** (`mlx-bun pi`, in-process TUI) | — | ✅ (P3/P4) | Phase 16 (flagship) |
+| **Native: connect user's own pi** (`harness pi`) | — | ✅ (P1) | Phase 16 |
+
+Summary: serving and the read-only/measurement verbs are **🔌
+wiring-only** — the compat layer ships over what the engine already
+does. The real capability work splits along two axes:
+
+- **Model quantization** (offline, makes a new artifact): `convert`,
+  `awq`, `dwq`, `gptq`, `dynamic_quant`. Its own new phase.
+- **Inference-time KV-cache quant** (online, no artifact): bf16 /
+  uniform / **mixed per-layer** from kv_config.json all ✅ today
+  (Phase 9/10 — we read & apply shipped configs). The only KV gaps are
+  optional and not inference plumbing: a *profiler* to author a new
+  kv_config.json (model-prep, low pri) and extra methods like TurboQuant
+  (Phase 13).
+- **Training**: LoRA/DoRA/full + `fuse`.
+- **Eval**: harness + `perplexity`.
+- **Distribution**: `upload`/`share`.
+
+The OptIQ-Lab web-UI tiles gate on these: quantize ← model-quant,
+fine-tune ← training, training-data ← independent, chat ✅.
+
+Plus the **native half** (no parity equivalent — the differentiators):
+the pi agentic coding CLI (Phase 16: `mlx-bun pi` is the embedded
+in-process agent, `harness pi` connects the user's own pi) and the web
+UI, both riding pi's `AgentSession` event stream. Compat makes us a
+drop-in for mlx-lm;
+the native half makes us more than it.
+
+## Parity harness (turns "drop-in" into a passing test)
+
+Same discipline as logit parity. For each compat verb, a test diffs
+mlx-bun against the Python oracle in `/Users/joshrossi/Code/mlx-lm/.venv`:
+
+- **Flag-surface parity**: parse `mlx_lm.<verb> --help`, assert every
+  documented flag is accepted by `mlx-bun <verb>` (or explicitly
+  rejected with a clear message — never silently ignored).
+- **Behavior parity** where deterministic: `generate --seed --temp 0`
+  output, `manage --scan` listing shape, `cache_prompt` round-trip,
+  `benchmark` column schema.
+- **Gap honesty**: a not-yet-built verb (`convert`, `evaluate`, …)
+  exits non-zero with `not implemented in mlx-bun yet` — the harness
+  asserts the message, so gaps are visible, not silent.
+
+Gated by `benchmark.sh`-style preflight only where it measures speed;
+correctness diffs need no clean machine.
+
+## Build sequencing
+
+1. **Phase 17 — this layer**: verb router + dotted aliases + flag
+   vocabulary + reconcile existing commands + parity harness, over
+   existing capabilities (`generate`, `chat`, `server`, `benchmark`,
+   `manage`, `cache_prompt`, `lora info`, `latency`). Honest stubs for
+   🟥 verbs.
+2. Model quantization (new phase) → `convert`/`dynamic_quant`/`awq`/
+   `dwq`/`gptq` (new model artifacts) + web-UI quantize tile.
+3. Inference-time KV quant → already done (bf16 / uniform / mixed
+   per-layer via kv_config.json, Phase 9/10). Optional later: a profiler
+   to author new configs (low pri) + the TurboQuant method (Phase 13).
+4. Training → `lora --train`/`fuse` + web-UI fine-tune tile.
+5. Eval → `evaluate`/`perplexity`.
+6. Distribution → `upload`/`share`.
+7. Web UI training-data tile (independent of engine work).

package/docs/design/docs-reorg-plan.md

@@ -0,0 +1,152 @@
+# Docs & repo-organization cleanup plan
+
+**Status:** executed 2026-06-14 (P0 + P1 + P2 #11). Held: P2 #9/#10
+(src/ and scripts/ moves — they change import paths / script references
+and need a test run, so deferred to their own pass). Authored 2026-06-14.
+
+**Scope & ethos.** Private repo, single author + agents, MLX-by-identity.
+No OSS/community scaffolding (CONTRIBUTING, issue templates, badges) —
+out of scope by design. **Prefer move / consolidate over delete.** The
+one exception is regenerable benchmark *output* (md/json/html report
+artifacts): those are noise, not experiments, and are explicitly
+deletable per Josh's direction. Losing *experiments* and code paths are
+never deleted (see memory: "don't delete optionality on one
+measurement").
+
+---
+
+## The centerpiece: benchmark provenance (corrected)
+
+The original analysis mis-framed this. The correct policy:
+
+- Benchmark **runs** are date- and machine-stamped, regenerable, and
+  machine-specific. They do **not** belong in the durable repo. The
+  existing `.gitignore` rule `benchmarks-h2h-*` (ignore all raw runs) is
+  **correct and stays**.
+- The actual defect today: a handful of raw run files leaked into git
+  history (committed before that ignore rule existed), and the README's
+  headline performance claims cite one of them
+  (`benchmarks/benchmarks-h2h-2026-06-11-Joshs-MBP-2025.md`) as the
+  source of truth. The repo's central perf claim depends on exactly the
+  kind of ephemeral artifact the gitignore says shouldn't be tracked.
+
+**The fix:** one curated, intentionally-maintained record, structured
+around the **three distinct kinds of measurement** below; raw runs stay
+local/ignored; the eval DB remains the structured backing record.
+
+### The three kinds of benchmarks (the record's structure)
+
+These are categorically different — one is pass/fail correctness, one is
+a perf number, one is a quality delta — and `RESULTS.md` must keep them
+in separate sections rather than one undifferentiated table. They map to
+the project's three-tier parity model (L1/L2/L3).
+
+1. **Parity (porting correctness) — pass/fail, not a number.** Are we
+   bit-for-bit with the upstream oracle under like-for-like config?
+   - mlx-lm + standard (bf16) KV cache → bit-exact (L1).
+   - mlx-optiq + mixed-precision KV cache → bit-exact (L2).
+   Record as a matrix of model × mode → ✓/✗ (+ commit, oracle versions).
+   This is the gate the perf/quality numbers are only meaningful *under*.
+
+2. **Performance — like-for-like numbers, two comparison axes.** Under
+   the parity constraints above, measure tok/s, prefill/s, decode/s,
+   TTFT, startup, RSS, peak memory. Two axes:
+   - **vs the oracles** — mlx-bun vs mlx-lm / mlx-optiq, same model, same
+     mode, same machine, same day.
+   - **vs our own compat mode** — mlx-bun's NEW optimizations vs
+     mlx-bun's own compatible (bit-exact) path, to prove an optimization
+     actually beats the baseline it diverges from.
+
+3. **Quality — for our non-bit-exact optimizations only (L3).** When a
+   custom path trades exactness for speed, quantify what the trade costs:
+   - **mean score across the 6 eval tasks** (`src/eval/tasks/`: bfcl,
+     gsm8k, hashhop, humaneval, ifeval, mmlu), vs the compatible upstream.
+   - **KL divergence** vs the compatible distribution.
+   Purpose: enable statements like *"the optimized path gains 23% tok/s
+   while holding ±5% on the 6-test mean"* — the perf win is only worth
+   claiming alongside its measured quality cost.
+
+### B1. Create `benchmarks/RESULTS.md` (the new durable record)
+- Three top-level sections: **Parity** (model × mode → ✓/✗), **Performance**
+  (oracle comparison + our-optimized-vs-our-compat), **Quality** (6-test
+  mean + KL for non-exact paths). Each row carries provenance (machine,
+  date, commit, model, mode, oracle versions).
+- Named so it does **not** match `benchmarks-h2h-*`, so it's tracked
+  deliberately and future raw runs never auto-collide with it.
+- Migrate the headline numbers Josh wants to keep (currently the 06-11
+  matrix, plus the 06-13/06-14 follow-ups) into the right section
+  **before** removing the raw files, so README's claims never lose their
+  source.
+- Point README's "Full table" link at `benchmarks/RESULTS.md`.
+
+### B2. Remove the old committed benchmark artifacts
+`git rm` the leaked raw runs and verification dumps (numbers preserved in
+B1 + the eval DB):
+- `benchmarks/benchmarks-h2h-2026-06-10-Joshs-MBP-2025.md`
+- `benchmarks/benchmarks-h2h-2026-06-10-Joshs-MacBook-Pro.md`
+- `benchmarks/benchmarks-h2h-2026-06-10.md`
+- `benchmarks/benchmarks-h2h-2026-06-11-Joshs-MBP-2025.md`
+- `benchmarks/lab-verification.json`
+- `benchmarks/lab-verification-mixed.json`
+
+Also clear the loose **untracked** root spew (delete or move to an
+ignored `reports/`): `benchmarks-h2h-2026-06-13-*.md`,
+`benchmarks-h2h-2026-06-14-*.{md,html}`, `benchmark-report-*.html`,
+`kernel-fusion-report-*.html`, `project-review.html`.
+
+`scripts/verify-lab.ts` keeps writing `benchmarks/lab-verification.json`
+locally — it'll just be gitignored now (add `benchmarks/lab-verification*.json`
+to `.gitignore`).
+
+### B3. Eval DB — leave where it is (decided 2026-06-14)
+- `~/.cache/mlx-bun/evals.sqlite` ([src/evaldb.ts:8](../../src/evaldb.ts))
+  stays user-local — it is the structured backing record of every run.
+  **Not tracked in-repo** (binary SQLite in git doesn't diff, churns, and
+  grows history). `RESULTS.md` is the committed human-readable record; the
+  DB is its machine-local source. Decision confirmed by Josh — closed.
+
+### Do NOT touch
+`goldens/*.json` and `goldens/**` are the correctness oracle golden
+files, not benchmark output. Leave entirely alone.
+
+---
+
+## P0 — Quick, safe wins (low risk)
+
+| # | Action | Why |
+|---|---|---|
+| 1 | Collapse `AGENTS.md` → `CLAUDE.md` (make AGENTS.md a 1-line pointer or symlink). | They're byte-identical except the title line — guaranteed to drift. |
+| 2 | B1–B2 above: create `RESULTS.md`, migrate numbers, `git rm` old artifacts, point README at it, gitignore `lab-verification*.json`. | Fixes benchmark provenance; declutters; keeps gitignore policy honest. |
+| 3 | Relocate untracked root HTML/md spew into an ignored `reports/` (or delete). | De-clutters the working-tree root. |
+| 4 | Drop the hardcoded "All 72 tests" in CLAUDE.md (say "the suite" / reference the count command). | 57 test files now — a baked-in number that's already wrong. |
+
+## P1 — Structural consolidation (the real navigability win)
+
+| # | Action | Why |
+|---|---|---|
+| 5 | Add `STATUS.md` (or a pinned top section in PLAN.md) holding only live handoff: current state + next action + blockers. Move PLAN.md's `NEXT UP` / `NEXT SESSION PICKUP` / `SESSION SWEEP` blocks into it. | Separates transient "do this next" from 2,475 lines of durable phase history; gives agents one entry point. |
+| 6 | Create `docs/planning/` and move `PRODUCT_ROADMAP.md`, `IDEAS.md`, `ResearchTopics.md`, `journal.md` under it. Leave only README + PLAN.md (+ STATUS.md) at root. Add a ~6-line "doc map" to README. | Root is currently a 6-file planning thicket; extends the taxonomy that already works in `docs/`. |
+| 7 | Pick one home for the optimization plan: fold `docs/design/optimization_plan.md` into PLAN.md Phase A–E, or have PLAN.md link out to it as canonical. | Two homes for the same plan = drift. |
+| 8 | Resolve the archive overlap: `archive/pi-builtin-investigation.html` duplicates [docs/investigations/pi-builtin-investigation.md](../investigations/pi-builtin-investigation.md). Decide whether `archive/` (tracked; 8 generated HTML files incl. a 374 KB pi-session transcript) should be tracked at all, or moved to an ignored location. Keep the `.md`, drop the duplicate. | Stops committing heavyweight generated HTML; one source per investigation. |
+
+## P2 — Polish (optional)
+
+| # | Action | Why |
+|---|---|---|
+| 9  | Group `src/pi/` (`harness-pi.ts`, `pi-provider.ts`, `pi-terminal.ts`, `pi-web.ts`). | `src/` root is 31 flat files; this is an obvious cohesive cluster. |
+| 10 | Sort `scripts/` into `scripts/{bench,regen,build}/` (40 files, clear prefixes). | Navigability only; low urgency. |
+| 11 | Reconcile CLI vocabulary: README quickstart uses the shipped `mlx-bun` verb (with a one-line "from a clone, `bun src/cli.ts` is the same"). | README/roadmap/package.json currently mix `bun src/cli.ts` and `mlx-bun`. |
+
+---
+
+## Execution order (after slots work is pushed)
+
+1. P0 #1, #4 (trivial, isolated).
+2. Benchmark block B1 → B2 → README relink (do B1 fully before B2).
+3. P0 #3 (root spew).
+4. Confirm B3 eval-DB decision with Josh.
+5. P1 #5–#8 (the structural moves; one commit per item so history stays legible).
+6. P2 as appetite allows.
+
+Each step keeps the test suite green and changes no source behavior — this
+is docs + file layout only.