mlx-bun 0.0.4 → 0.0.6

package/docs/design/adapters-end-to-end.md

@@ -1,5 +1,16 @@
 # Adapters end-to-end — notes / TODO (2026-06-16)
 
+**Status legend:** ✅ DONE — shipped and verified in code | ⏳ PENDING — not yet implemented
+
+| Section | Status |
+| --- | --- |
+| §A Pi CLI extension (`extensions/mlx-bun-adapter.ts`) | ✅ DONE |
+| §B Web chat adapter selector (`src/web/app.html` + `before_provider_request` hook in `pi-web.ts`) | ✅ DONE |
+| §C `/v1/adapters/available` discovery (`src/server.ts:1182`, `src/lora.ts:139`) | ✅ DONE |
+| §D Overnight training from the web UI — monitor/stream endpoint + UI | ⏳ PENDING |
+| §E `--force-adapter` CLI flag (absent from `src/cli.ts`) | ⏳ PENDING |
+| Repo-id keying written into adapter configs at save time | ⏳ PENDING |
+
 Working notes for wiring trained LoRA adapters through the whole stack: Pi (CLI +
 web), the web Chat page UI, and the server's discovery surface. Captured from the
 2026-06-16 session (serving verified working; pi-extension mechanism confirmed from
@@ -31,16 +42,16 @@ API should let you move an adapter between these states explicitly.
 So **loaded** and **selected** are done. The gap is **available** (discovery) plus
 the two front-ends (Pi extension, web Chat selector).
 
-## TODO
+## Shipped (§A–§C) ✅
 
-### A. Pi CLI/terminal — adapter via extension  *(default none)*
+### A. Pi CLI/terminal — adapter via extension  *(default none)* ✅ DONE
 
-Mechanism (confirmed from the pi docs in-session): a Pi extension using the
-`before_provider_request` hook to inject the field into the outgoing
-OpenAI-compatible payload. `models.json` can't add arbitrary body fields — the hook
-is the right tool.
+Wired a Pi extension using the `before_provider_request` hook to inject the `adapter`
+field into the outgoing OpenAI-compatible payload. `models.json` can't add arbitrary
+body fields — the hook was the right tool.
 
-Create `~/.pi/agent/extensions/mlx-bun-adapter.ts`:
+Shipped as `~/.pi/agent/extensions/mlx-bun-adapter.ts` (copy in-repo + one-liner
+installer):
 
 ```ts
 import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
@@ -60,53 +71,44 @@ export default function (pi: ExtensionAPI) {
 - Selection via `PI_MLX_ADAPTER=chunk pi --provider mlx-bun --model local -p "..."`
   (no native `-p --adapter` flag; unknown CLI flags get rejected). Optional shell
   helper: `pi-chunk() { PI_MLX_ADAPTER=chunk pi --provider mlx-bun --model local -p "$@"; }`.
-- Put it in the global `~/.pi/agent/extensions/` dir so it loads for non-interactive
-  `-p` too. `/reload` to pick it up.
-- **Ship this** as part of mlx-bun (a copy in-repo + a one-liner installer), so users
-  don't hand-write it. Consider also reading the *selected* adapter from a small
-  state file the web UI / a `mlx-bun adapter use <id>` command writes, so CLI and web
-  share one selection.
+- Lives in the global `~/.pi/agent/extensions/` dir so it loads for non-interactive
+  `-p` too. `/reload` picks it up.
+- Also reads the *selected* adapter from a small state file the web UI / a
+  `mlx-bun adapter use <id>` command writes, so CLI and web share one selection.
 
-### B. Web Chat page — selector + load/unload  *(default none)*
+### B. Web Chat page — selector + load/unload  *(default none)* ✅ DONE
 
 Path: `src/web/app.html` (UI) ↔ `/ws/chat` (`src/pi-web.ts`) ↔ in-process Pi ↔ model.
 
 - **Selector** in `app.html`: a dropdown of *available* adapters with **"none"
   selected by default**. Changing it sets the active selection for subsequent
-  messages. Allow multi-select → `"a+b"` (resolveSpec already supports stacking).
+  messages. Multi-select → `"a+b"` (resolveSpec already supports stacking).
 - **Load / unload controls**: buttons that call `POST /v1/adapters` (load) and
-  `DELETE /v1/adapters/:id` (unload). Show each adapter's state (available / loaded /
-  selected) so the three concepts are visible. An adapter must be **loaded** before
-  it can be **selected** (selecting an unloaded id errors) — UI should either
-  auto-load on select or gate selection to loaded ones.
-- **Inject via the `before_provider_request` hook — the primary mechanism, identical
-  to the CLI (§A).** `pi-web.ts` ALREADY registers an inline extension this way
-  (`pi.on("tool_call", …)` wired into `createAgentSession`, pi-web.ts:469/605), so
-  `pi.on("before_provider_request", (e) => ({ ...e.payload, adapter }))` drops in the
-  same way and reads the current selection from the WS session state. **No custom
+  `DELETE /v1/adapters/:id` (unload). Each adapter's state (available / loaded /
+  selected) is shown so the three concepts are visible. Selecting an unloaded adapter
+  auto-loads it (fires `POST /v1/adapters`) before injecting the id.
+- **Injected via the `before_provider_request` hook — identical mechanism to the CLI
+  (§A).** `pi-web.ts` registers the inline extension in `createAgentSession`
+  (pi-web.ts:469/605), so `pi.on("before_provider_request", (e) => ({ ...e.payload,
+  adapter }))` reads the current selection from WS session state. **No custom
   provider plumbing, no new model-call wiring.** The hook is the spine for BOTH UIs.
-- **Selection → hook**: the dropdown just updates the session's selected-adapter value
-  that the hook reads — a tiny client→server message (or reuse the `set_thinking`-style
-  toggle path), NOT a heavyweight protocol. Default none → hook returns the payload
-  unchanged → base model.
-- **Mounting is the only thing the hook can't do**: the injected id must be mounted
-  server-side or `resolveSpec` rejects it. Auto-load-on-select handles it — selecting
-  fires one `POST /v1/adapters`, then the hook injects the id. Discovery (§C) and a
-  richer protocol are optional polish; the dropdown can start from `GET /v1/adapters`.
-
-### C. Server — adapter **discovery** (the missing "available" list)
-
-- Add a store the server scans for available adapters:
-  - `~/.cache/mlx-bun/adapters/` (the trainer already defaults saves here —
-    `src/server.ts:1653`) and/or a **sidecar next to the model dir**.
-- New endpoint, e.g. `GET /v1/adapters/available` →
+- **Selection → hook**: the dropdown updates the session's selected-adapter value via
+  a small client→server message (reusing the `set_thinking`-style toggle path). Default
+  none → hook returns the payload unchanged → base model.
+
+### C. Server — adapter **discovery** (`GET /v1/adapters/available`) ✅ DONE
+
+- Scans `~/.cache/mlx-bun-finetunes/` and `~/.cache/mlx-bun/adapters/`
+  (`src/server.ts:1192`) and any sidecar next to the model dir.
+- `GET /v1/adapters/available` (`src/server.ts:1182`, `src/lora.ts:139`) →
   `[{ id, path, source_repo, rank, scale, compatible }]`, where each entry is read
   from the adapter's `optiq_lora_config.json` (`source_model`, `rank`, `scale`).
-- **Filter by the served model.** Today `source_model` is a *machine/snapshot-specific
-  path* (`/Users/joshrossi/.cache/huggingface/.../snapshots/<hash>`), which won't
-  match across machines/snapshots. Key compatibility on the **repo id**
-  (`mlx-community/MiniCPM5-1B-OptiQ-4bit`) instead — see "repo-id keying" below. The
-  server already tracks `repoId` per model; `compatible = adapter.source_repo === served.repoId`.
+- **Filters by the served model** on **repo id**
+  (`mlx-community/MiniCPM5-1B-OptiQ-4bit`) rather than the fragile
+  machine/snapshot-specific path. The server already tracks `repoId` per model;
+  `compatible = adapter.source_repo === served.repoId`.
+
+## TODO (§D–§E)
 
 ### D. Overnight training from the web UI
 
@@ -117,8 +119,10 @@ Training must be fully drivable from the web UI — not just a script.
   (rank / scale / num_layers / LR / dropout / iters / seq / segmentSize — the
   configurable-params task, mlx-lm defaults), **start**, and leave it running.
 - **Monitor**: live loss / iter / ETA, and survives the browser closing (the run is
-  server-side; the page reconnects). `POST /v1/fit` already exists
-  (`src/server.ts:1652`) — needs a status/stream endpoint + a UI.
+  server-side; the page reconnects). The training backend already exists:
+  `POST /api/finetune/submit` (`src/server.ts:1781`). What §D still needs is a
+  status/stream endpoint + the overnight-monitoring UI. (Note: `GET /fit` is the
+  unrelated memory-fit solver in `src/fit.ts` — not a training endpoint.)
 - **Completion**: the finished adapter lands in the store and shows up as *available*
   (state A in the three-state model), ready to load/select or auto-serve.
 - This closes the loop with the chunk-task roadmap: iterate data → train overnight →

package/docs/design/compat-cli-surface-design.md

@@ -6,6 +6,8 @@ modes"). This doc is the surface contract: every verb, the flag
 vocabulary, how optiq is absorbed, the parity harness, and **the
 matrix of what we still lack**.
 
+**Update 2026-06-17:** Several items have shipped since the 2026-06-12 snapshot: **LoRA training** is live (`src/train/`, `scripts/chunk-finetune.ts`, segmented-backward Phase A MiniCPM5 + Phase B e4b both merged); the **web UI fine-tune tab** (`src/web/app.html`) is live with SFT/DPO workflow, loss curve, merge, and export; the **chat adapter selector** (web UI) is live; and **adapters end-to-end** (web selector + CLI `/adapter`) are done — the matrix below reflects these as ✅. The **Phase 17 compat parity harness itself** (the verb router, dotted aliases, flag-surface parity tests) remains **unbuilt**.
+
 ## Goal
 
 Design mlx-bun's entire CLI/flag surface from scratch (nothing is
@@ -54,7 +56,7 @@ shared subset; optiq-only features are additive (marked ⊕).
 | `chat` | mlx_lm.chat | engine ✅, web chat ✅; no terminal REPL (`tui.ts` is help-formatting) | build REPL frontend over `generate` |
 | `server` | mlx_lm.server / optiq serve | ✅ `serve` | rename flags; ⊕ `--kv-config`, `--anthropic` honored (default on) |
 | `convert` | mlx_lm.convert + optiq convert | ❌ | quant pipeline; ⊕ `--target-bpw/--candidate-bits/--reference` |
-| `lora` | mlx_lm.lora + optiq lora | serve/hot-swap ✅, train ❌ | `--train/--test`; ⊕ `--rank-scaling by_bits\|by_kl`; `lora info` |
+| `lora` | mlx_lm.lora + optiq lora | serve/hot-swap ✅, train ✅ | `--train/--test`; ⊕ `--rank-scaling by_bits\|by_kl`; `lora info` |
 | `fuse` | mlx_lm.fuse | ❌ | fuse adapters, dequant, GGUF export, upload |
 | `cache_prompt` | mlx_lm.cache_prompt | runtime ✅, CLI ❌ | precompute + save reusable KV cache file |
 | `benchmark` | mlx_lm.benchmark / optiq benchmark | ✅ `benchmark` | flag-name parity (`-p/-g/-b/-n`); ppl half ❌ |
@@ -83,7 +85,7 @@ mlx-lm" half.
 
 ### pi — the built-in agentic coding CLI
 
-(Phase 16; full plan in **docs/pi-builtin-investigation.md**.) pi is an
+(Phase 16; full plan in **docs/investigations/pi-builtin-investigation.md**.) pi is an
 MIT, Bun-native (`bun build --compile`) coding agent — the "CLI
 experience inside our package" that neither Python package has. Two
 commands, one for each audience:
@@ -111,8 +113,8 @@ split.
 The web UI's agent backbone is pi's RPC / `AgentSession.subscribe()`
 event stream (investigation Option 4) — the SAME events drive the
 embedded TUI and the browser, so tool calls / steering / progress come
-for free. Tiles: chat ✅; quantize / fine-tune / training-data gate on
-the capability matrix.
+for free. Tiles: chat ✅; fine-tune ✅ (shipped: SFT/DPO workflow, loss curve, merge,
+export); quantize / training-data gate on the capability matrix.
 
 ## Flag-vocabulary reconciliation (where we currently diverge)
 
@@ -150,7 +152,7 @@ column; the 🟥 column is the capability matrix that fills in behind it.
 | KV-cache quant: bf16 / uniform / **mixed per-layer** (kv_config.json) | `server` (`off`/`N`/`config`) | ✅ | done (Phase 9/10; config.ts reads, generate.ts applies per-layer) |
 | KV sensitivity *profiler* (authors a NEW kv_config.json) | `kv-cache` | 🟥 | model-prep, low pri (shipped artifacts already include one) |
 | TurboQuant — optional extra quant *method* (rotation VQ) | scheme, not a verb | 🟥 | Phase 13 (KV plumbing already exists) |
-| **LoRA / DoRA / full training** | `lora --train` | 🟥 | Training (new phase) |
+| **LoRA / DoRA / full training** | `lora --train` | ✅ | Training (done: Phase A MiniCPM5 + Phase B e4b) |
 | **Sensitivity-aware rank scaling** | `lora --rank-scaling` | 🟥 | Training |
 | **Fuse adapters (+GGUF/upload)** | `fuse` | 🟥 | Training |
 | **lm-eval-harness tasks** | `evaluate` | 🟥 | Eval (new phase) |
@@ -159,7 +161,7 @@ column; the 🟥 column is the capability matrix that fills in behind it.
 | **HF upload** | `upload` | 🟥 | Distribution |
 | **Distributed share** | `share` | 🟥 | Distribution (lowest pri) |
 | **Web UI: quantize** | `lab`/UI | 🟥 | gated on Quantize |
-| **Web UI: fine-tune** | `lab`/UI | 🟥 | gated on Training |
+| **Web UI: fine-tune** | `lab`/UI | ✅ | done (SFT/DPO workflow, loss curve, merge, export) |
 | **Web UI: training-data template + gen** | `lab`/UI | 🟥 | new (no engine dep) |
 | Web UI: chat | `/chat` | ✅ | done (publishing gate 2) |
 | **Native: pi embedded** (`mlx-bun pi`, in-process TUI) | — | ✅ (P3/P4) | Phase 16 (flagship) |
@@ -177,7 +179,7 @@ does. The real capability work splits along two axes:
   optional and not inference plumbing: a *profiler* to author a new
   kv_config.json (model-prep, low pri) and extra methods like TurboQuant
   (Phase 13).
-- **Training**: LoRA/DoRA/full + `fuse`.
+- **Training**: LoRA/DoRA/full training ✅ (shipped); `fuse` verb (adapter merge, GGUF export, upload) 🟥 remaining.
 - **Eval**: harness + `perplexity`.
 - **Distribution**: `upload`/`share`.
 
@@ -221,7 +223,7 @@ correctness diffs need no clean machine.
 3. Inference-time KV quant → already done (bf16 / uniform / mixed
    per-layer via kv_config.json, Phase 9/10). Optional later: a profiler
    to author new configs (low pri) + the TurboQuant method (Phase 13).
-4. Training → `lora --train`/`fuse` + web-UI fine-tune tile.
+4. Training → ✅ `lora --train` + web-UI fine-tune tile (both shipped); remaining: `fuse` verb (adapter merge, GGUF export, upload).
 5. Eval → `evaluate`/`perplexity`.
 6. Distribution → `upload`/`share`.
 7. Web UI training-data tile (independent of engine work).

package/docs/design/docs-reorg-plan.md

@@ -4,6 +4,19 @@
 (src/ and scripts/ moves — they change import paths / script references
 and need a test run, so deferred to their own pass). Authored 2026-06-14.
 
+**Execution state (verified 2026-06-17):**
+- P0 #1 — **DONE**: `AGENTS.md` is a symlink → `CLAUDE.md`; drift risk resolved.
+- P0 #2 — **DONE**: `benchmarks/RESULTS.md` created; old committed benchmark artifacts `git rm`'d. Root untracked spew (`benchmarks-h2h-2026-06-13-*` etc.) is now cleaned — no untracked report files found at root.
+- P0 #3 — **DONE**: root HTML/md spew cleared (no `benchmarks-h2h-*.md`, `benchmark-report-*.html`, `kernel-fusion-report-*.html`, or `project-review.html` at root).
+- P0 #4 — **DONE**: hardcoded test count removed from CLAUDE.md (no "All 72 tests" present).
+- P1 #5 — **DONE**: `STATUS.md` exists at root.
+- P1 #6 — **DONE**: `docs/planning/` exists with `IDEAS.md`, `PRODUCT_ROADMAP.md`, `ResearchTopics.md`, `journal.md`.
+- P1 #7 — open (optimization_plan.md still lives in docs/design/ as a standalone).
+- P1 #8 — status unknown (archive/ not verified in this pass).
+- P2 #9 — **NOT DONE**: `harness-pi.ts`, `pi-provider.ts`, `pi-terminal.ts`, `pi-web.ts` still live flat in `src/` root (no `src/pi/` subdirectory).
+- P2 #10 — **PARTIALLY DONE**: ~57 one-off research/debug scripts moved to `scripts/experiments/`; production + tooling + bench-* + regen-* + parity-* + gen-model + eval/serve remain at `scripts/` root (the full bench/regen/build split was not applied).
+- P2 #11 — **DONE**: CLI vocabulary reconciled.
+
 **Scope & ethos.** Private repo, single author + agents, MLX-by-identity.
 No OSS/community scaffolding (CONTRIBUTING, issue templates, badges) —
 out of scope by design. **Prefer move / consolidate over delete.** The
@@ -127,7 +140,7 @@ files, not benchmark output. Leave entirely alone.
 | 5 | Add `STATUS.md` (or a pinned top section in PLAN.md) holding only live handoff: current state + next action + blockers. Move PLAN.md's `NEXT UP` / `NEXT SESSION PICKUP` / `SESSION SWEEP` blocks into it. | Separates transient "do this next" from 2,475 lines of durable phase history; gives agents one entry point. |
 | 6 | Create `docs/planning/` and move `PRODUCT_ROADMAP.md`, `IDEAS.md`, `ResearchTopics.md`, `journal.md` under it. Leave only README + PLAN.md (+ STATUS.md) at root. Add a ~6-line "doc map" to README. | Root is currently a 6-file planning thicket; extends the taxonomy that already works in `docs/`. |
 | 7 | Pick one home for the optimization plan: fold `docs/design/optimization_plan.md` into PLAN.md Phase A–E, or have PLAN.md link out to it as canonical. | Two homes for the same plan = drift. |
-| 8 | Resolve the archive overlap: `archive/pi-builtin-investigation.html` duplicates [docs/investigations/pi-builtin-investigation.md](../investigations/pi-builtin-investigation.md). Decide whether `archive/` (tracked; 8 generated HTML files incl. a 374 KB pi-session transcript) should be tracked at all, or moved to an ignored location. Keep the `.md`, drop the duplicate. | Stops committing heavyweight generated HTML; one source per investigation. |
+| 8 | Resolve the archive overlap: `archive/pi-builtin-investigation.html` duplicates [docs/investigations/pi-builtin-investigation.md](../investigations/pi-builtin-investigation.md). Decide whether `archive/` (tracked; 2 HTML files: `mlx-bun-lab-report.html` + `wwdc-mlx-bun.html`) should be tracked at all, or moved to an ignored location. Keep the `.md`, drop the duplicate. | Stops committing heavyweight generated HTML; one source per investigation. |
 
 ## P2 — Polish (optional)
 

package/docs/design/hlg-sampling.md

@@ -1,6 +1,6 @@
 # HLG sampling — a piecewise tone curve on the logits
 
-Status: **design + phasing** (no code yet; this doc is the contract)
+Status: **implemented, found neutral, superseded by Curve Designer** — `applyHlg` shipped in `src/sampler.ts`; investigation found HLG a wash vs plain temperature; the Curve Designer (`src/curve-sampler.ts`) is its v2 replacement. See [docs/investigations/hlg-sampling-investigation.md](../investigations/hlg-sampling-investigation.md) for findings. This doc remains as the design rationale.
 Owner: sampling layer (`src/sampler.ts`)
 Default: **off** (`--hlg-sampling off` = today's temperature sampler, untouched)
 
@@ -10,6 +10,12 @@ Default: **off** (`--hlg-sampling off` = today's temperature sampler, untouched)
 > [docs/reference/server-config.md](../reference/server-config.md) and
 > [docs/reference/server-api.md](../reference/server-api.md) once shipped.
 
+## Outcome / Retrospective
+
+HLG was implemented through **Piece 2** (pure curve + neutrality tests + serial wiring, `applyHlg` live in `src/sampler.ts`). Evaluation found it **neutral vs plain temperature** — no measurable gain in diversity or quality at matched output entropy. As a result, HLG is **superseded by the Curve Designer** (`src/curve-sampler.ts`), which generalizes the same region-shaping idea with an editable control-point interface and cleaner UX. Full empirical findings are in [docs/investigations/hlg-sampling-investigation.md](../investigations/hlg-sampling-investigation.md).
+
+The sections below (Goal through Correctness) remain as design rationale. Pieces 3–6 of Phasing and the Open Questions were never executed; see the [Phasing](#phasing-the-piece-n-increments) and [Open Questions](#open-questions--risks) sections for historical context.
+
 ## Goal
 
 Add a sampling transform — gated entirely behind `--hlg-sampling on` and a
@@ -290,6 +296,8 @@ quality + diversity rather than parity — except for the degeneracy gates, whic
 
 ## Phasing (the "piece N" increments)
 
+> **Historical / superseded.** Pieces 1–2 shipped. Pieces 3–6 were never executed — HLG was evaluated after Piece 2 and found neutral; the feature was superseded by the Curve Designer before Piece 3 began.
+
 1. **Pure curve + neutrality tests** — `applyHlg(lp, params)` in `sampler.ts`,
    top-anchored pivot only; degeneracy + monotonicity tests green across all
    four models. (No flag wired yet — pure, fully-tested function.)
@@ -305,6 +313,8 @@ quality + diversity rather than parity — except for the degeneracy gates, whic
 
 ## Open questions / risks
 
+> **Historical / superseded.** These questions were open at the time of design. HLG evaluation concluded with a neutral verdict before they were answered; the Curve Designer supersedes this path. Questions are preserved as design context.
+
 - **Default knob values** are calibration-pending — set empirically by the
   Piece 5 sweep, not guessed. Initial exploration points: `c ≈ 6`, `w ≈ 4`,
   `β_h ≈ 4`, `β_t ≈ 6` nats; confirm or replace with measured values.
@@ -337,4 +347,4 @@ for it. That a transfer function is standard practice in another domain says
 nothing about whether it helps token sampling — that is decided only by
 measurement (the KL characterization, the capability guardrail, and the
 diversity lens in [Correctness](#correctness--evaluation)). The full empirical
-write-up goes in `docs/investigations/hlg-sampling-investigation.md` once run.
+write-up is in [docs/investigations/hlg-sampling-investigation.md](../investigations/hlg-sampling-investigation.md).