mlx-bun 0.0.5 → 0.0.6

package/docs/investigations/trainer-validation-experiment.md

@@ -0,0 +1,173 @@
+# Trainer validation experiment
+
+**Status:** in progress (started 2026-06-20). **Goal:** prove the from-scratch
+SFT/DPO/ORPO trainer actually works before trusting any downstream result — general
+capability (Goal 1) or task-specific (Goal 2).
+
+## Why this exists — the smoking gun
+
+The margin/accuracy is recorded in `metrics.json` (`valTrajectory[].margin`,
+`bestMargin`), not the stdout log. The prior `cpm5-uf` ORPO run (λ=0.1) shows:
+
+| step | 100 | 200 | 300 | 400 | 500 | 600 | 700 | 800 |
+|---|---|---|---|---|---|---|---|---|
+| accuracy | .507 | .514 | .534 | .541 | .534 | .520 | .547 | .547 |
+| margin | .132 | .124 | .110 | .120 | .136 | .128 | .118 | .123 |
+
+**Preference accuracy is ~0.52 (near the 0.50 coin-flip) and the margin is flat.**
+ORPO's defining job is to push accuracy up and widen the margin. This is *not* a clean
+"it works" signal. Caveats: 800 steps ≈ 0.05 epoch (undertrained), provenance
+(from-base vs warm-start) unconfirmed. But it's enough to make validation mandatory:
+**we cannot currently claim our ORPO learns the preference at all.**
+
+## Method-decomposition (the validation key)
+
+`ORPO_loss = NLL(chosen) + λ·L_OR`. NLL is exactly the SFT objective; L_OR is the
+reference-free odds-ratio preference term. So we validate the building blocks, and ORPO
+is covered by its parts plus a small ORPO-specific surface (the OR term + λ).
+
+mlx-lm has **no DPO/ORPO** — so the oracle strategy is:
+- **SFT/NLL** → numeric parity vs **mlx-lm tuner** (oracle env exists).
+- **DPO + OR loss** → **numeric unit test vs the paper's equations** on a fixed tiny
+  input (hand-computed expected loss + grad). No external dep. Optional stronger
+  cross-check: **TRL** `DPOTrainer`/`ORPOTrainer` loss (PyTorch; multi-GB env →
+  approve-first, not default).
+
+Every positive control is designed so "weak data / weak lever" is **not** an available
+excuse — success must be unambiguous.
+
+## PILLAR 1 — Prove the trainer (gate for everything)
+
+- **1a · SFT positive control** *(infra GREEN, behavioral check WEAK)*. Train
+  `fixtures/adapters/data-upper` from scratch on MiniCPM5; assert generations are
+  actually uppercased (oracle: the repo's pre-trained `upper` adapter). Existing
+  `train-e2e.test.ts` only checks loss↓ — add the behavioral assert. *(seconds)*
+- **1b · DPO positive control** *(MISSING)*. Trivial preference (chosen uppercase,
+  rejected lowercase). Assert **margin increases** and **preference accuracy → ~1.0**.
+- **1c · ORPO positive control + the real question.** (i) Same trivial preference →
+  accuracy → ~1.0. (ii) **From-BASE ORPO on UF**, multiple epochs, watch margin widen
+  and accuracy climb well above chance — directly resolves the near-chance finding
+  (undertraining vs warm-start vs bug).
+- **1d · Loss-formula parity.** SFT NLL vs mlx-lm; DPO/ORPO loss + grad vs hand-computed
+  paper equations (unit test). The training-side analogue of the inference logit-parity
+  oracle — currently MISSING.
+- **1e · vs the paper.** Internal proxy first: preference accuracy robustly > chance on
+  UF. Then (approve-first, LLM-judge) AlpacaEval/MT-Bench **direction** vs base (not
+  absolute — paper used 7B). optiq suite (incl. IFEval) is an already-local downstream.
+
+> **Gate:** until 1a–1c are green, Pillars 2 & 3 sit on an unverified trainer.
+
+## PILLAR 2 — Measure general-capability impact (optiq suite)
+
+All 6 datasets are present locally (`~/.cache/mlx-bun/eval-data/`: mmlu, gsm8k, ifeval,
+bfcl, humaneval, hashhop); harness = `scripts/eval.ts capability`.
+- **2a · Eval parity:** run on base CPM5, reproduce optiq's **30.28** within tolerance
+  (validates the *measurement* before trusting deltas; confirm the README's open BFCL
+  "no-calls" / MMLU "at-chance" flags aren't broken on our side).
+- **2b · Delta:** trained checkpoint vs base, on the 6-task mean.
+- **2c · Targeting:** mean is floored by **BFCL 0.0 / GSM8K 2.7 / HashHop 4.0**; UF hits
+  the already-strong IFEval, so expect small mean movement. Highest-leverage Goal-1 lever
+  is **BFCL** (0.0 is format-trainable; 0→30 = +5 on the mean).
+
+## PILLAR 3 — Task-specific (separate; gated on Pillar 1)
+
+Chunk segmenter via `scripts/chunk-eval.ts` (boundary/label accuracy), using the
+hard-negative ORPO data (remove-a-cut / add-a-cut, balanced to kill the length
+shortcut). Measured by chunk accuracy, not val loss / capability. Credibility depends on
+Pillar 1.
+
+## Sequencing & gates
+
+1. Pillar 1 positive controls (1a→1b→1c-trivial) — fast, "does the machinery work."
+2. Pillar 1 real replication (1c from-base UF) + 1d loss-formula parity.
+3. Pillar 2a eval-parity (reproduce 30.28) → 2b deltas.
+4. Pillar 3 — once the trainer is trusted.
+
+## Execution log
+
+### 2026-06-20 — Pillar 1 quick validations: PASS
+
+**Machinery + gradient correctness — GREEN.** `MLX_BUN_TEST_TRAIN=1 bun test
+train-autograd train-e2e train-orpo-e2e` → 8 pass / 0 fail, real training on
+MiniCPM5 (11.3s). `train-autograd` confirms the backward matches autograd.
+
+**ORPO positive control (prefer-uppercase, 12 train / 4 val, 120 iters, lr 1e-3) — PASS.**
+Held-out preference accuracy **1.000 by step 10**, margin **5.5 → 4.0** (noisy at lr=1e-3
+but accuracy solid). Logged: `runs/orpo-pc.{json,csv}`.
+
+**DPO positive control (same data) — PASS, cleaner.** Accuracy **1.000 from step 10**,
+margin **monotonic 6.46 → 9.67**. Logged: `runs/dpo-pc.{json,csv}`.
+
+**Conclusion: the preference machinery is validated.** On a learnable preference both
+DPO and ORPO drive accuracy to 1.0 and the margin wide open (vs UF's flat ~0.12). A
+broken trainer could not. ⇒ **The UF near-chance result is the DATA, not a bug** — UF
+pairs are genuinely subtle for a 1B (both responses plausible); "prefer uppercase" is
+trivially separable. The smoking gun is explained: implementation sound, lever weak.
+
+**Gotcha found (recorded so we don't repeat it):** `AdapterManager.mount` (the *serving*
+hot-swap path) does **not** reflect into `branchLogpMean` (the *training* forward) — an
+external base-vs-adapter `orpoMetrics` measurement via mount returns bit-identical
+numbers (measured: 0.500/0.1463 both). Measure training-side metrics via the
+`trainLora` emit / loraState path (`scripts/experiments/pref-control.ts`), not mount.
+
+**Logging for plotting:** `runs/<stem>.{json,csv}` — per-step train `{loss, accuracy,
+margin}` + per-eval val `{loss, accuracy, margin}`. CSV columns: `phase,step,loss,
+accuracy,margin`. Tools: `scripts/experiments/pref-control.ts` (train+log),
+`scripts/experiments/measure-pref.ts` (standalone adapter eval — superseded by the
+emit path for training metrics; see gotcha).
+
+### 2026-06-20 — Pillar 2a: capability-harness parity. Two bugs found + fixed.
+
+**The first-measure step caught two harness bugs** (exactly its purpose — never trust a
+delta until the harness reproduces the published baseline):
+
+1. **Thinking mode.** The eval rendered the chat template's *default* (thinking-on for
+   MiniCPM5); optiq's numbers are **non-thinking** (HF card: IFEval 64.7 is non-thinking).
+   Symptom: IFEval **36.7%** vs 64.7. **Fix:** `src/eval/runner.ts` `generateText` now
+   renders `enableThinking: false` by default (env `MLX_BUN_EVAL_THINK=1` to re-enable).
+   → IFEval **73.3%** (n=30), optiq ballpark.
+2. **HumanEval oracle-python path.** `src/eval/tasks/humaneval.ts` hardcoded
+   `/Users/joshrossi/Code/mlx-lm/.venv/bin/python` — wrong on this machine (it's
+   `mlx-lm-example`), so every code-exec spawned a missing interpreter → **0%**. **Fix:**
+   env-overridable `MLX_BUN_ORACLE_PYTHON` (committed default = reference box; never
+   re-commit the path). → HumanEval **53.3%** (n=15), optiq 57.9.
+
+**Post-fix parity (base CPM5, non-thinking, n=20 — all ballpark):**
+
+| | MMLU | GSM8K | IFEval | BFCL | HumanEval | HashHop |
+|---|---|---|---|---|---|---|
+| ours | 55.0 | 10.0 | 75.0 | 0.0 | ~53 | 0.0 |
+| optiq | 52.4 | 2.7 | 64.7 | 0.0 | 57.9 | 4.0 |
+
+⇒ **harness reproduces optiq.** Small-n inflation on GSM8K/IFEval will settle toward
+optiq on the full set. **Exp-1 baseline harness VALIDATED.**
+
+### Ready-to-fire sequence (everything staged; only GPU time remains)
+
+All measurement tools validated. `scripts/eval.ts` now takes `--adapter <dir>` (mounts the
+trained LoRA → x2). `ORC=/Users/joshrossi/Code/mlx-lm-example/.venv/bin/python` below.
+
+1. **x1** (running): `MLX_BUN_ORACLE_PYTHON=$ORC bun scripts/eval.ts capability --candidate MiniCPM5`
+2. **Train** (user): from-base ORPO/UF → `./adapters/cpm5-uf-frombase` (command in chat).
+3. **x2 (Exp 1 after):** `MLX_BUN_ORACLE_PYTHON=$ORC bun scripts/eval.ts capability --candidate MiniCPM5 --adapter ./adapters/cpm5-uf-frombase` → compare to x1.
+4. **Exp 2 gen (GPU):** `bun scripts/experiments/prefbench.ts gen mtbench ./adapters/cpm5-uf-frombase mtbench-uf` (and `gen alpaca … alpaca-uf 200` for a subset).
+5. **Exp 2 judge (codex, anytime):** `bun scripts/experiments/prefbench.ts judge mtbench-uf` → TRAINED win-rate vs BASE.
+
+### Next
+- **Full baseline:** see step 1 (running). **(long → user-launched)**
+- **Train (the "do"):** from-base ORPO on UF (`pref-control.ts`, command in chat). **(long → user-launched)**
+- **Re-measure → x2**, compare to x1. Exp 1 done.
+- **Exp 2 judge — VALIDATED (keyless).** No OpenAI key needed: the judge shells out to
+  `codex exec -m gpt-5.4-mini` (codex's own auth), forcing structured `{"winner":...}` via
+  `--output-schema` and reading the clean verdict from `-o`. `scripts/experiments/llm-judge.ts`
+  — `judgeOnce` + `judgePair` (judges BOTH orders to cancel small-judge position bias).
+  Positive control: **3/3 obvious good-vs-bad pairs, good won both orders, zero flips.**
+  1bx vs 1bx1; consistency (same judge both sides), not absolute comparability to the paper's GPT-4.
+- **Exp 2 harness — STAGED.** Prompts downloaded: `~/.cache/mlx-bun/eval-data/{mt_bench.jsonl
+  (80), alpaca_eval.json (805)}`. Two-phase runner `scripts/experiments/prefbench.ts`:
+  `gen <mtbench|alpaca> <adapterDir> <stem> [limit]` (GPU — base then trained, non-thinking)
+  → `runs/<stem>.responses.jsonl`; `judge <stem> [limit]` (codex) → TRAINED win-rate vs BASE,
+  judging trained-vs-base **directly** (head-to-head, not vs a fixed reference). Only the `gen`
+  phase needs GPU; everything else is validated. Fire after the from-base training produces
+  `./adapters/cpm5-uf-frombase`.
+- **1c real / 1d parity** — still open (lower priority now the positive controls + harness parity are green).

package/docs/reference/memory.md

@@ -0,0 +1,77 @@
+# Memory
+
+mlx-bun memory is a local Markdown wiki that the built-in pi agents can read as durable user context.
+
+Core rule:
+
+> Chat agents read memory articles. `mlx-bun memory synthesize` is the only writer.
+
+The current release includes the vault, read path, Obsidian open flow, and scheduler. Synthesis is scaffolded but still a no-op M1 stub. The planned synthesis engine uses the same local models you chat with, switched into an editorial pipeline with stage-specific LoRA adapters.
+
+## Storage
+
+- Vault: `~/.mlx-bun/wiki/`
+- User articles: `~/.mlx-bun/wiki/articles/*.md`
+- Read-only reference docs: `~/.mlx-bun/wiki/Reference/*.md`
+- Meta pages: `~/.mlx-bun/wiki/Meta/*.md`
+- Talk pages: `~/.mlx-bun/wiki/Talk/`
+- Override: `MLX_BUN_WIKI=/path/to/wiki`
+- Derived synthesis DB: `~/.cache/mlx-bun/memory.sqlite`
+
+The vault is plain Markdown, git-tracked, and opens directly in Obsidian. `Reference/` contains read-only symlinks to mlx-bun's own docs so memory has useful built-in residents on day one; synthesis must never write them.
+
+## CLI
+
+```bash
+mlx-bun memory init              # create the wiki; optional import + schedule
+mlx-bun memory status            # path, article count, git, schedule, recent articles
+mlx-bun memory open              # open wiki in Obsidian/Finder
+mlx-bun memory open <article>    # open a specific article
+mlx-bun memory list              # list article stems + Reference/* docs
+mlx-bun memory search <query>    # search articles
+mlx-bun memory toc <article>     # headings + anchors
+mlx-bun memory section <article> <anchor>
+mlx-bun memory links <article>   # outbound/inbound wikilinks
+mlx-bun memory read <article>    # print article body
+mlx-bun memory synthesize --dry-run
+mlx-bun memory schedule --at 03:00
+mlx-bun memory unschedule
+```
+
+`memory synthesize` is currently safe and side-effect-free: it reports that M1 stage bodies are not implemented and writes nothing. In the full flow, `memory init` will be able to offer the HF-hosted memory adapter pack download so synthesis can run locally.
+
+## Agent tools
+
+Both `mlx-bun pi` and the web chat expose these read-only tools:
+
+- `memory_search`
+- `memory_read`
+- `memory_toc`
+- `memory_section`
+- `memory_links`
+- `memory_list` — user articles plus read-only `Reference/*` docs
+- `memory_status` — vault path, setup state, article count, git state, last-synthesis placeholder, schedule state, recent changed articles
+
+The tools are always registered. If no vault exists, they return a setup message instead of disappearing. Search/read results prefixed with `Reference/` are mlx-bun docs, not user memory articles.
+
+## Obsidian
+
+```bash
+mlx-bun memory open
+mlx-bun memory open Archie_Project
+```
+
+The first command opens the vault. The second opens one article. Obsidian is preferred; Finder/default Markdown app is the fallback.
+
+## Synthesis roadmap
+
+M1/M2 will turn saved pi sessions into articles using the local model plus four specialized LoRA adapters trained by Josh from existing Lucien-scale pipeline data and distributed from Hugging Face:
+
+1. deterministic ingest of saved web/terminal pi session JSONL;
+2. **memory-chunk LoRA** — conversation → topic chunks (already trained as a WIP artifact);
+3. **memory-bucket LoRA** — chunk → existing bucket or new article proposal;
+4. **memory-synthesis LoRA** — chunks + existing article → conservative article update candidate;
+5. **memory-editor LoRA** — candidate article → polished wiki Markdown;
+6. deterministic footnote/wikilink normalization, quality gates, and git commit.
+
+The adapters propose; the pipeline validates and writes. The pipeline writes only `articles/`; `Reference/` is immutable context. Until synthesis lands, memory works with imported/hand-written user articles plus the built-in read-only mlx-bun reference docs.

package/docs/reference/orpo-quickstart.md

@@ -6,16 +6,55 @@ One launcher runs the complete new system — **flash-CCE Metal head** (steel fw
 **falls back and logs** if a precondition isn't met (e.g. a row whose chosen/rejected
 prompts aren't identical → two-forward for that row).
 
-## Command
+## Command — `mlx-bun train` (recommended)
 
 ```bash
-MODEL=/path/to/snapshot DATA=/path/to/datadir bun scripts/train-orpo.ts
+mlx-bun train <model> --data <dir>
 ```
-- `DATA` is a directory with `train.jsonl` (+ optional `valid.jsonl`); rows are
+- `<model>` is a query or snapshot path (e.g. `e4b`, `MiniCPM5`); omit it to auto-pick.
+- `--data <dir>` is a directory with `train.jsonl` (+ optional `valid.jsonl`); rows are
   `{"prompt": "...", "chosen": "...", "rejected": "..."}`.
-- The script **auto-detects e4b/Gemma** and sets its required training env flags
+- The command **auto-detects e4b/Gemma** and sets its required training env flags
   (`MLX_BUN_PERF_KERNEL=0`, `MLX_BUN_FUSED_GELU=0`) before loading the model — you don't
-  set them yourself.
+  set them yourself. The full stack (flash head + prefix-share + segmented) is on by default.
+- It prints a plan box (dataset counts, format, the resolved knobs), streams per-step loss
+  + peak memory, and saves a **mountable** adapter (`mlx-bun serve <model> --adapter <dir>`).
+- `--dry-run` inspects the dataset and prints the plan without training. `mlx-bun help train`
+  lists every flag.
+
+```bash
+# e4b ORPO on your preference data, checkpointing every 200 steps; run it detached
+nohup mlx-bun train e4b --data ./prefs --iters 1000 --save-every 200 > train.log 2>&1 &
+# resume from a checkpoint (warm-start the LoRA weights)
+mlx-bun train e4b --data ./prefs --resume ~/.cache/mlx-bun/mlx-bun-finetunes/orpo-e4b/checkpoints/step-00200
+```
+
+| flag | default | meaning |
+|---|---|---|
+| `--method` | `orpo` | `sft` · `dpo` · `orpo` |
+| `--iters` | 100 | iterations |
+| `--lr` | orpo 1e-5 · dpo 5e-5 · sft 2e-4 | learning rate (cosine + short warmup) |
+| `--rank` / `--scale` | orpo 16 / 2.0 | LoRA rank (by_bits scaled) / scale |
+| `--seq` | e4b 8192 · cpm 4096 | max sequence length |
+| `--lambda` | 0.1 | ORPO odds-ratio weight |
+| `--seg` | 2 | layers/segment (segmented backward); `--no-segment` to disable |
+| `--save-every` | off | mountable checkpoint every N steps |
+| `--resume <dir>` | — | warm-start LoRA weights from a checkpoint/adapter |
+| `--no-flash` / `--no-prefix` | (on) | drop the flash Metal head / prefix-sharing |
+| `--adapter <dir>` | `~/.cache/mlx-bun/mlx-bun-finetunes/<method>-<model>` | output dir |
+
+> **Long runs:** launch detached from your own shell (`nohup … &`). Foreground is fine for
+> short runs; the command blocks and streams until done.
+
+## The launcher (env-var alternative)
+
+`scripts/train-orpo.ts` is the same stack driven by env vars — handy for scripting:
+
+```bash
+MODEL=/path/to/snapshot DATA=/path/to/datadir bun scripts/train-orpo.ts
+```
+- `DATA` is a directory with `train.jsonl` (+ optional `valid.jsonl`).
+- Auto-detects e4b/Gemma and sets the training env flags, same as the CLI.
 
 ### Preconfigured defaults (override via env)
 | env | default | meaning |

package/docs/reference/training.md

@@ -17,18 +17,22 @@ for the long-context memory mechanism.
 
 ## Entry points
 
-There is **no `mlx-bun train` CLI verb.** Training runs as a subprocess job,
-reachable four ways:
+Training is reachable several ways — all of them drive the **same** runner
+(`finetuneRunner` in `src/train/job.ts`):
 
 | Path | How | Use when |
 |---|---|---|
-| **Web UI** | `mlx-bun serve`, open `/finetune` — pick model → dataset → hyperparameters → train; watch live train/val loss; merge/export the adapter | Interactive, the default |
+| **CLI** | `mlx-bun train <model> --data <dir> [options]` — foreground, full ORPO stack on by default; auto-detects e4b/Gemma + sets its env; `--save-every`, `--resume`, `--dry-run`; streams loss to the terminal. `mlx-bun help train` for all flags. Run long jobs detached: `nohup mlx-bun train … &` | **The default for CLI users** |
+| **Web UI** | `mlx-bun serve`, open `/finetune` — pick model → dataset → hyperparameters → train; watch live train/val loss; merge/export the adapter | Interactive, in the browser |
 | **HTTP API** | `POST /api/finetune/submit` (job id + SSE events); `POST /api/finetune/inspect-dataset` to probe a file; `POST /api/finetune/merge` to fold an adapter into base weights | Scripted / remote |
-| **Script** | [`scripts/chunk-finetune.ts`](../../scripts/chunk-finetune.ts) — an env-driven wrapper that calls the runner directly | Repeatable CLI runs |
+| **ORPO launcher** | [`scripts/train-orpo.ts`](../../scripts/train-orpo.ts) — the same stack driven by env vars (`MODEL=`, `DATA=`, `RANK=`, `RESUME=`, …). See [orpo-quickstart](./orpo-quickstart.md) | Env-var scripting |
 | **Shell recipe** | [`scripts/ft-e4b-v2.sh`](../../scripts/ft-e4b-v2.sh) `probe`\|`train` — the actual e4b run we use; sets the required env (see [What we actually run](#what-we-actually-run-the-e4b-recipe)) | Reproducing our e4b fine-tune |
-| **ORPO launcher** | [`scripts/train-orpo.ts`](../../scripts/train-orpo.ts) — preconfigured ORPO with the **full stack on by default** (flash-CCE head + segmented backward + prefix-sharing); auto-detects e4b and sets its env; `RESUME=<dir>` warm-starts. See [orpo-quickstart](./orpo-quickstart.md) | Running an ORPO fine-tune |
 | **Library** | `import { finetuneRunner } from "./src/train/job"` and call it with a config + emitter | Embedding training in your own TS |
 
+The CLI and the launcher run the runner **in-process / foreground** (stream to the
+terminal, `nohup`-able); the Web UI and HTTP API run it as a **subprocess job** (crash
+isolation + a single GPU lease across concurrent server requests).
+
 ### Quick start (HTTP API)
 
 ```bash

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mlx-bun",
-  "version": "0.0.5",
+  "version": "0.0.6",
   "description": "Native MLX inference for Bun on Apple Silicon — local LLM server (OpenAI/Anthropic-compatible) and TypeScript library. No Python.",
   "type": "module",
   "license": "MIT",