mlx-bun 0.2.0 → 0.3.0

package/docs/design/w4a16-compute-precision-spike.md

@@ -1,232 +0,0 @@
-# Research spike — W4A16 head GEMM (compute precision, not storage)
-
-**Status:** proposed (not started) · **Opened:** 2026-06-19 · **Owner:** Josh
-**Surface:** `src/train/flash-cce.ts` (steel + SG kernels) · **Gate:** the existing
-1e-5 CCE parity gate + greedy-prefix logit parity.
-
-> One-line thesis (Josh): *we've maxed **storage** quant (4-bit weights), but we
-> dequant into fp32 and run the matmul in fp32 — so the head GEMM is **W4A32**.
-> The untouched lever is **compute** precision: dequant 4-bit → half, matmul in
-> `half` operands with an fp32 accumulator (**W4A16**), gate on parity. Same 4-bit
-> storage, untouched. The only change is the float width we dequant into and
-> multiply in.*
-
----
-
-## TL;DR
-
-- The flash-CCE head (forward logit GEMM `E@Wᵀ` and backward phase-2 `coeffᵀ@W`)
-  currently runs **W4A32**: 4-bit weights are dequantized into **fp32** threadgroup
-  tiles and multiplied with **fp32** simdgroup matrices. Verified in source (below).
-- The 4-bit packing bought the cheap HBM read; the fp32 matmul throws away the
-  compute advantage. Apple GPUs run **fp16 FMA at ~2× fp32** on the regular SIMD
-  ALUs (no tensor cores). Both head GEMMs are **compute-bound** (see roofline note),
-  so half operands are the lever.
-- **Industry-standard for the forward** (MLX `quantized_matmul`, llama.cpp, every
-  on-device 4-bit stack do exactly W4A16). Low risk; do it first.
-- **Backward needs the mixed-precision caveat** — fp32 accumulate is non-negotiable
-  (already planned), and the **operands** (`coeffᵀ`, `h`) need a dynamic-range probe
-  before trusting fp16, specifically the near-zero coeff tails after the skips fire.
-- **M1 Max constraint:** no `bf16` simdgroup_matrix (M2+ only), so **fp16 is the
-  only half option** — which makes the range-probe matter more here than on M3.
-- Gate exactly like the existing skip epsilons: **fp32 stays the default-exact
-  parity mode**; half is the training default; the 1e-5 gate test runs against fp32.
-
----
-
-## Current state — verified W4A32
-
-The default production path for **both** forward and backward is the **steel**
-kernel (MLX's verbatim `BlockMMA` GEMM, `steel-qmm-header.ts`), selected when
-`H,V,blockV0 % 32 == 0` (`flash-cce.ts:1034`, `:1130`). The `simdgroup_float8x8`
-SG kernels and the scalar/lane kernels are fallbacks for non-32-divisible shapes or
-explicit `MLX_BUN_CCE_*` overrides.
-
-Every one of these instantiations is fp32-operand:
-
-| path | site | instantiation | operand width |
-|---|---|---|---|
-| steel fwd (default) | `flash-cce.ts:950` | `BlockMMA<float,float,…>` | fp32 |
-| steel bwd phase-1 (default) | `flash-cce.ts:799` | `BlockMMA<float,float,…>` (lmma_t) | fp32 |
-| steel bwd phase-2 (default) | `flash-cce.ts:800` | `BlockMMA<float,float,…>` (dmma_t) | fp32 |
-| steel staging tiles | `:596–598`, loaders `:801` | `threadgroup float` / `BlockLoader<float,…>` | fp32 |
-| SG fwd (fallback) | `:251–254` | `simdgroup_float8x8`, `Csh`/`Esh` = `threadgroup float` | fp32 |
-| SG bwd (fallback) | `:617`, `:684–704` | `simdgroup_float8x8`, `Wsh` = `threadgroup float` | fp32 |
-
-The dequant itself (`Csh[…] = sc * q + bi`, `flash-cce.ts:244`) computes into fp32
-and stores fp32. The dh accumulator is already fp32 (`MMATile<float,…>`,
-`flash-cce.ts:828`) — i.e. **the accumulator is already where it should be**; only
-the matmul *operands* are wider than they need to be.
-
----
-
-## The lever — what actually changes
-
-W4A16 = **same 4-bit storage, half-width operands, fp32 accumulate.** Concretely:
-
-1. **Steel operand type** — the `BlockMMA<float,float,…>` first template param is the
-   GEMM input/operand type; flip operand to `half` while keeping the
-   accumulator/fragment in `float`. (**Spike task 0: confirm the exact MLX steel
-   template param that selects operand vs accumulator width** in
-   `steel-qmm-header.ts` before editing — don't assume the param order.)
-2. **Staging tiles** — `Esh`, `Cstg`, `Wsh`, `Csh` go `threadgroup half`; dequant
-   arithmetic can stay fp32 then narrow on store, or compute in half.
-3. **Loaders** — `BlockLoader<float,…>` → `BlockLoader<half,…>` to stage half tiles.
-4. **SG fallback** — `simdgroup_float8x8` → `simdgroup_half8x8` for the operands,
-   keep an `simdgroup_float8x8` accumulator (Apple supports the mixed `half×half→float`
-   `simdgroup_multiply_accumulate`).
-
-Everything reduction-shaped (the online-softmax `m`/`d` recurrence, the dh accum,
-logsumexp) **stays fp32**.
-
----
-
-## First thoughts (mine) — what to add to Josh's analysis
-
-The analysis is right and the change is well-targeted. Five additions, each grounded
-in the current source:
-
-**1. Aim at the steel kernel, not the SG snippet.** Josh's note led with the SG path
-(`simdgroup_float8x8`, `Csh`). That path is a *fallback*. The lever that moves the
-measured 180 ms fwd / 754 ms bwd (e4b) numbers is the **steel `BlockMMA<float,float>`**
-instantiations (`:950`, `:799`, `:800`) and their `BlockLoader<float>`/staging tiles.
-Do the SG path too for completeness, but the steel path is where the production time is.
-
-**2. The 2× is an upper bound gated by the FMA fraction — get the roofline split
-first.** The whole flash-CCE design is `[M,V]`-free and amortizes dequant across
-`BLOCK_B` tokens specifically because the **dequant ALU** was "the dominant cost in
-v1" (`flash-cce.ts:59`). The runtime splits into (a) nibble-unpack + `sc·q+bi`
-dequant and (b) the FMA matmul. **W4A16 only speeds up (b).** The kernel is
-unambiguously compute-bound, not BW-bound — **confirmed live** (see the measured box
-below), and Josh flagged it from `mactop` before this doc existed. *But* the win
-ceiling is the FMA share of that ALU time. The dequant-amortization work was done
-precisely to shrink (a)'s share, which *should* leave (b) dominant and favorable — but
-**measure the (a)/(b) split before predicting 2×.** The live numbers prove the
-*whole kernel* is ALU-bound; they do **not** resolve the intra-kernel (a)/(b) split,
-which is what actually bounds the reclaim. If the kernel is still dequant-ALU-bound,
-half operands move it less.
-
-> **Live roofline — measured (M1 Max 32 GB, MiniCPM5-1B ORPO, `--no-segment`, seq 4096):**
-> GPU **100% @ 1296 MHz, 81 °C, Nominal** (sustained, not throttling) · memory
-> **BW 129.2 GB/s ≈ ⅓ of the M1 Max's ~400 GB/s ceiling** (bus two-thirds idle) ·
-> mem **17.37 / 32 GB (54%)**, swap 0 · CPU ~5%, ANE 0%. The training `bun` proc is
-> 92.6% GPU. **GPU pegged + memory bus mostly idle = ALU-bound.** This is the regime
-> where W4A16 pays; if it were BW-bound (BW near 400, GPU spiky) the spike would be
-> moot. The 4-bit storage already did its job — *that's why BW is low* — and the GPU
-> is now spending its cycles on the fp32 FMAs.
-
-**3. Threadgroup memory is a free secondary win — possibly a compounding one.** The
-staging tiles (`Esh`/`Cstg`/`Wsh`/`Csh`) are the kernel's threadgroup-memory pressure,
-and tile sizing is *already* occupancy-constrained — see the note at `flash-cce.ts:71`
-("the 8KB `Csh` tile measured SLOWER"). Half-width staging **halves that footprint**,
-which can either raise occupancy or admit larger tiles. So W4A16 may buy more than the
-raw 2× FMA: it could relieve the exact constraint that capped the current tiling.
-Worth re-sweeping `BLOCK_B`/tile dims *after* the precision change, not just measuring
-the drop-in.
-
-**4. SDPA in the segmented forward is probably already half — don't chase it.** Josh's
-aside ("if your SDPA tiles are also fp32, same logic"). The segmented forward uses
-`ops.sdpa` — MLX's own fused flash kernel ([[opssdpa-dk-vjp-bug]]) — which runs in the
-model's dtype (bf16), **not** our fp32 tiles. So that lever is likely already pulled by
-delegating to MLX. One-line confirm, then drop it; the head GEMM is the real target.
-
-**5. Free instrumentation hooks already exist for the backward range-probe.** The
-backward already computes `cmax = simd_max(coeff)` for the FILTER_EPS skip
-(`flash-cce.ts:682`). That same reduction is the natural hook to log the coeff dynamic
-range (min nonzero |coeff|) across a real run *before* committing the operand to fp16 —
-the underflow-to-zero case Josh flagged. Pair it with a max-abs probe on `h`: at e4b's
-`scale=1.0` peaked attention the residual stream can carry outliers
-([[e4b-forward-length-sensitivity]], [[siglip-vision-parity-cross-build]]), and with no
-bf16 on M1 Max we can't lean on a wider exponent to eat them.
-
----
-
-## Risk analysis (preserving the forward/backward split)
-
-**Forward — low risk, boring industry default.** Logits are well-scaled; the
-online-softmax max-subtraction bounds the `exp` args. W4A16 here is what the dense
-path already assumes. Validate against the 1e-5 gate (fp16 operands won't hold 1e-5 —
-that's expected; the **fp32 path stays the exact-parity gate**, half is graded by
-greedy-prefix logit agreement, same as the vision tier-a bar).
-
-**Backward — fp16 matmul is mainstream, but as *mixed precision*, with eyes on
-operand range.** What makes the forward safe (well-scaled, max-subtracted) is **not**
-automatically true here:
-- `coeff_v = g·(onehot − softmax)·sech²` — after the skips fire, the survivors are the
-  **near-zero tails**. Small magnitudes are exactly where fp16's ~3-digit mantissa and
-  narrow range (subnormals < ~6e-5) bite. **`coeffᵀ` as an fp16 operand is the highest
-  risk** — tail coeffs underflowing to zero silently drop gradient contribution. Probe
-  the range first (task 5 above).
-- `h` (activations) as fp16 operand — residual outliers; the classic bf16-beats-fp16
-  case, but bf16 isn't available on M1 Max. Range-check before trusting.
-- **fp32 accumulate is non-negotiable** (already planned; `MMATile<float>` already is).
-- **Validate grads against autograd, not a hand-derived backward** ([[opssdpa-dk-vjp-bug]]).
-
----
-
-## Spike plan
-
-Sequenced so each step is independently measurable and revertible (Josh: *"Don't flip
-both at once — you'll learn more, and if parity drifts you'll know which GEMM did it."*).
-
-- [ ] **0. Confirm the steel template semantics.** In `steel-qmm-header.ts`, identify
-  which `BlockMMA`/`BlockLoader` template params are operand vs accumulator width.
-  Exit: a one-line mapping, no code change.
-- [x] **1a. Whole-kernel roofline — DONE (live).** GPU 100% / BW 129 GB/s (~⅓ of 400)
-  on M1 Max confirms the kernel is ALU-bound, not BW-bound (see the measured box above).
-  The spike is worth chasing.
-- [ ] **1b. Intra-kernel split (still open).** Profile the steel fwd (180 ms) and bwd
-  (754 ms, e4b) into dequant-ALU vs FMA fractions — 1a says the GPU is the wall, this
-  says *which part* of the GPU work. Exit: a number that bounds the expected win. *If
-  FMA < ~40% of either kernel, reset expectations before coding.*
-- [ ] **2. Add the precision gate.** New codegen `#define` (e.g. `CCE_A_HALF`) +
-  env flag (e.g. `MLX_BUN_CCE_A16`), mirroring `CCE_BWD_FILTER_EPS`/`CCE_BWD_BLOCK_EPS`
-  (`flash-cce.ts:732`,`:739`). Default: fp32 (exact) for the parity suite; half is the
-  training default. Per [[three-level-fidelity-tree-model]] the exact path must stay
-  selectable.
-- [ ] **3. Forward → half operands.** Steel fwd `BlockMMA`/loader/staging to half.
-  Validate: 1e-5 gate (fp32 path) + greedy-prefix logit parity (half path) + measure
-  the fwd GEMM delta. Land as its own change.
-- [ ] **4. Backward range probe.** Instrument coeff (via the `simd_max` hook) and `h`
-  dynamic range across a real MiniCPM5/e4b ORPO step. Exit: min-nonzero-coeff and
-  max-|h| numbers + a go/no-go on fp16 operands.
-- [ ] **5. Backward → half operands** (only if 4 says go). `coeffᵀ`/`h`/`W` operands
-  half, dh accumulator stays fp32. Validate grads **vs autograd** + measure bwd delta.
-- [ ] **6. Re-sweep tiling.** With half-width staging, re-check `BLOCK_B`/tile dims and
-  occupancy (first-thought #3) — the precision win may unlock a tiling win.
-
-**Exit criteria for the spike:** a measured fwd/bwd speedup on e4b (V=262144) and
-MiniCPM5 (V=130560) on the bench machine (M4 Pro 24 GB; numbers verified against
-`~/.cache/mlx-bun/evals.sqlite`, not docs — [[verify-benchmarks-against-evaldb-not-docs]]),
-with the fp32 exact path still passing the 1e-5 gate and the half training path inside
-the greedy-prefix parity bar. Either it lands as the training default behind the gate,
-or the roofline/range data explains why it didn't pay and the spike closes with that.
-
----
-
-## Open questions
-
-- Steel `BlockMMA` mixed operand/accum: does MLX's steel already support
-  `half`-operand / `float`-accum, or does it need a local template tweak? (task 0)
-- Is the dequant cheaper to do **in half** (fewer ALU bits) or **in fp32 then narrow**
-  (preserves the scale/bias precision)? A/B inside task 3.
-- Does the half staging change the `+4-float` padding trick (`flash-cce.ts:197`, the
-  BK_padded bank-conflict avoidance)? Half-width may re-introduce or shift bank
-  conflicts — re-derive the pad.
-
----
-
-## References
-
-**Code:** `src/train/flash-cce.ts` — steel fwd `:937–1028`, steel bwd `:763–935`, SG
-fwd `:174–270`, SG bwd `:561–726`, gates `:732`/`:739`/`:1034`/`:1130`; staging tiles
-`:596–598`; dequant `:244`; coeff `simd_max` hook `:682`; dh fp32 accum `:828`.
-`src/train/steel-qmm-header.ts` (the verbatim MLX BlockMMA). `src/mlx/metal-kernel.ts`
-(kernel infra).
-
-**Cross-links:** [[flash-cce-metal-kernel-prototype]] · [[three-level-fidelity-tree-model]]
-· [[opssdpa-dk-vjp-bug]] · [[e4b-forward-length-sensitivity]] ·
-[[verify-benchmarks-against-evaldb-not-docs]] · `docs/archive/investigations/steel-flash-cce-handoff.md`.
-
-**Sibling deferred lever** (queued together, post-run): `docs/design/orpo-future-enhancements.md`
-→ "Deferred levers" (expose `--grad-accum`).

package/docs/design/web-chat-beat-matrix.md

@@ -1,315 +0,0 @@
-# The mlx-bun beat matrix
-
-Purpose: an axis-by-axis competitive requirement table for making mlx-bun's
-web chat UI THE SINGLE BEST macOS local-AI chat surface — not best-in-a-niche,
-best, period. Every row is a capability judged against the best current
-implementation anywhere in the surveyed field (mlx-lm, mlx-optiq, Ollama, Open
-WebUI, LM Studio, Odysseus, Jan, Msty, GPT4All, AnythingLLM, Enchanted,
-macMLX/mlx-serve, ChatGPT, Claude.ai). Sourced from the 10-agent market-research
-corpus plus a direct line-level audit of `src/web/app.html`, `src/pi-web.ts`,
-`src/server.ts` and the merged plan at
-`docs/design/web-chat-redesign.md`. Supplemented with a fresh read of
-mlx-optiq's live Lab docs (mlx-optiq.com/docs/lab/{chat,arena,hub}) — optiq is
-the closest-cousin MLX product, so its rows are held to the strictest
-standard: the operating rule for this matrix is **zero "I miss how
-<competitor> had X" moments**, so every capability below is MATCH, BEAT, or an
-explicit SKIP with a stated engine-substrate reason.
-
-*(Caveat: the Odysseus rows rest on the research agent's public-web
-identification of "Odysseus" — Cookbook recommender / Skills /
-Observer-Reflector memory / SSRF CVEs. If that's not the Odysseus meant,
-those rows describe the wrong product and need a re-run; no verdict in this
-matrix flips on them.)*
-
-Verdict legend:
-- **MATCH** — users expect it; parity is sufficient (e.g. code-block copy).
-- **BEAT** — an axis where mlx-bun can be clearly, structurally better because
-  of something we alone have (in-process server, `/fit`, whole-stack control,
-  MLX-native, memory vault, adapters). The "what beating requires" column says
-  *how*, not "do it better."
-- **SKIP** — the engine genuinely lacks the substrate (image gen, voice). One
-  line each so silence is never mistaken for an oversight.
-
----
-
-## Axis 1 — Chat fundamentals
-
-| Capability | Best today (who + what makes theirs best) | Verdict | What beating them concretely requires | Backend status |
-|---|---|---|---|---|
-| Streaming responses with visible Stop | Universal table stakes (ChatGPT/Claude/all local apps) — Stop is prominent, never buried in a menu | MATCH | Already shipped (`app.html` composer) | exists |
-| Stop actually kills in-flight tool work | optiq Lab: the stop button SIGKILLs sandboxed tool subprocesses, not just the token stream | MATCH | Ensure our stop aborts running tool calls (bash/web_fetch) through the pi bridge, not only the generation — a half-stop that leaves a shell command running is a visible miss vs. the closest-cousin MLX product | thin addition (verify + wire abort through the pi tool layer) |
-| Chat persistence to disk, no account | optiq persists to `~/.optiq/lab/chats/`; universal among local apps | MATCH | Already shipped — pi `SessionManager` disk-persisted sessions with richer lifecycle (fork) than optiq's | exists |
-| Regenerate response | ChatGPT/Claude cycle response variants (1/2, 2/2…); Open WebUI same | MATCH | Add a regenerate button per assistant turn, reusing the existing send pipeline | thin addition |
-| Edit-and-resend (non-destructive branch) | ChatGPT/Claude: edit a prior user message silently forks; `< 1/2 >` toggle above the edited message | MATCH | Reuse `SessionManager.forkFrom` (already exists server-side) at message granularity instead of whole-session only | thin addition (server primitive exists, needs per-message wiring) |
-| Full conversation tree view | Neither ChatGPT nor Claude ships this natively — only a linear toggle; third-party extensions (BranchGPT) exist because users want it | BEAT | Build a genuine node-graph view of every edit-branch, reusing `forkFrom` — a documented gap neither cloud incumbent has closed. Phase 4 stretch, not launch-blocking. | new build (UI only; fork primitive exists) |
-| Live steering (type while streaming to redirect) | mlx-bun already has this wired server-side (`queue_update`, distinct `steering`/`followUp` arrays) — no competitor in the survey does this | BEAT | Fix the existing truthy-vs-length-tested array bug (`web-ui-pass-plan.md` #2) so the feature that already exists actually renders | exists (buggy — Phase 0 fix) |
-| Follow-up queue while a turn streams | Same as above — server already distinguishes queued follow-ups from live steering | BEAT | Same fix as above; this is a genuine differentiator once rendering is correct | exists (buggy) |
-| Stop/regenerate reachable by keyboard | ChatGPT ships a full shortcut sheet (`Cmd+/`) | MATCH | `Cmd/Ctrl+Shift+O` new chat, `Shift+Esc` focus composer, etc. — adopt ChatGPT's vocabulary verbatim, zero cost to originality | thin addition |
-| Message-level copy (full assistant text) | Universal — every competitor has full-message copy, not just code-fence copy | MATCH | Add copy button to the message action row | thin addition |
-| Delete a single message | Common in Open WebUI, LM Studio | MATCH | Add to message action row | thin addition |
-| Multi-turn context management / auto-compaction | mlx-lm's byte-capped LRU prompt cache is the correctness bar; pi already does session compaction | MATCH | Already exists server-side; only gap is a UI seam marking *when* compaction happened | thin addition (mark the seam) |
-
----
-
-## Axis 2 — Rendering & content
-
-| Capability | Best today | Verdict | What beating them concretely requires | Backend status |
-|---|---|---|---|---|
-| GFM markdown (tables, task lists, blockquotes, autolinks) | mlx-bun's own hand-rolled `mdToHtml` is already competitive with library output | MATCH | Nothing — already good, keep it | exists |
-| Syntax-highlighted code blocks | Every mainstream competitor (ChatGPT, Claude, Open WebUI, LM Studio) has this; mlx-bun renders plain monochrome `<pre><code>` | MATCH | Vendor Shiki (or a lighter WASM-free alternative) self-hosted, no CDN, applied only to completed blocks | new build (thin — rendering only) |
-| LaTeX/math rendering | Universal via KaTeX in modern chat UIs; mlx-bun has none | MATCH | Vendor KaTeX self-hosted, same completed-block-only pass | new build (thin) |
-| Diagram rendering (Mermaid) | Present in several competitors' RAG-heavy stacks | MATCH | Optional — low priority; add if code-heavy audience demands it | new build |
-| O(n²) streaming re-render fix | Streamdown/t3.chat's block-memoized architecture (split completed vs. live-tail block, skip re-parse on stable blocks) | BEAT | mlx-bun is the only local competitor whose whole pitch is serving *speed* — a frontend that stutters on a fast, spec-decode-accelerated response undercuts the product's own thesis. Port the block-memoization *idea* into the existing vanilla renderer (no framework). This unblocks every other visual change touching the render path. | new build (architecture fix, no new backend) |
-| Unterminated-markdown repair mid-stream | Streamdown's context-aware auto-close of open fences/emphasis | MATCH | Port the repair-pass idea alongside the block-memoization fix | new build (thin, bundled with above) |
-| Sandboxed HTML/canvas preview for fenced HTML | optiq Lab's Canvas: fenced HTML blocks render live in a sandboxed allow-scripts iframe with a view-source toggle — shipped today in the closest-cousin MLX product | MATCH | Add the same allow-scripts iframe render + view-source toggle for HTML fences, opt-in-by-heuristic like ChatGPT's Canvas walk-back (not mandatory). Explicit "no I-miss-optiq-Canvas" requirement. | new build |
-| Artifacts/Canvas side-panel | Claude Artifacts / ChatGPT Canvas: persistent versioned side panel, highlight-to-scope-edit, version scrubber | MATCH (opt-in, not launch-required) | OpenAI itself partially walked Canvas back in May 2026 for the common case — ship opt-in-by-heuristic (auto-open above ~10 lines / high fence density), not a mandatory pane. Real effort; explicitly Phase 4. | new build |
-| logprobs-derived confidence visualization | Nobody in the local-AI survey ships this in-chat at all | BEAT | Server already validates and returns `logprobs`/`top_logprobs` (mlx-lm-matched shape) with zero UI. An opt-in per-token confidence overlay is pure whitespace — no competitor's server even supports it as cleanly. | exists (server), new build (UI) |
-| Structured-output / JSON-schema picker | optiq Lab ships JSON mode via lm-format-enforcer (schema-constrained, deliberately pure-Python at a stated ~1ms/token cost); mlx-lm has zero `response_format` support; most local UIs don't expose it even when the server supports it | BEAT | Server already supports `response_format` json_schema/json_object + `guided_grammar`/`guided_regex`/`guided_choice` via **xgrammar token bitmasks** — structurally faster than optiq's ~1ms/token pure-Python enforcer, and broader (grammar/regex/choice, not just JSON schema). Ship a schema/grammar picker with live validation; we beat the closest cousin on both overhead and surface. | exists (server), thin addition (UI) |
-
----
-
-## Axis 3 — Model discovery & management
-
-| Capability | Best today | Verdict | What beating them concretely requires | Backend status |
-|---|---|---|---|---|
-| Hardware-fit traffic light on model selection | LM Studio: green/yellow/red against detected RAM/VRAM before download commit | BEAT | We already have per-SKU fit prediction measured against *this specific Mac's bandwidth* (`GET /fit`, `src/fit.ts`) — not a generic heuristic like LM Studio's. Wire it into both the model list and the in-chat model switcher. | exists (server), thin addition (UI) |
-| One-click model library browse + load + chat (Hub) | optiq Lab's Hub: three sources in one browser — published OptiQ quants auto-discovered from mlx-community (family/size/bit-profile/download counts), HF search of any MLX-compatible model (OptiQ-only filter), and local converted dirs (vision badge when a sidecar is present) — with one-click load-to-server AND new-chat per row | BEAT | Match the three-source browse→load→chat loop, then beat it where optiq is documented weakest: their Hub has **no hardware-fit indicator beyond raw size** — put a `/fit`-computed green/yellow/red badge (predicted decode tok/s + safe context for *this* Mac) on every row, at the exact point of download/load commitment | exists (`/fit`), new build (library UI) |
-| Capability badges (vision/tool-use/reasoning) | LM Studio's `model.yaml` badges; Ollama's library tag chips | MATCH | Surface known properties (SigLIP vision, quant tier) as UI chrome — we already know these, just not rendered | thin addition |
-| In-chat model switching (no restart) | optiq Lab Hub: hot-swap the serving model in place in ~5-30s without restarting the Lab; Jan's router mode (single process, on-demand load/unload) | MATCH | `#nav-model` is currently a dead `<span>` with no click handler — wire a clickable picker that triggers the same in-place hot-swap; match optiq's 5-30s bar | thin addition |
-| Model swap preserves session history + re-primes cleanly | Every competitor surveyed is documented as bad at this (LM Studio has an open bug clearing the selector on swap) | BEAT | Since we control the whole stack (session + KV + adapter layer), a swap that keeps history and re-primes cleanly is a believable, demonstrable win nobody else in the survey has actually shipped correctly | thin addition (UI + verify session continuity) |
-| Resident multi-model pinning / LRU pool | macMLX's per-model pin/LRU-evict pool | MATCH | Extend existing adapter-namespaced KV logic to full models for users bouncing between two models | new build |
-| Running-models resource table | Ollama's `ollama ps` (PROCESSOR split, UNTIL auto-unload countdown) | BEAT | Build the equivalent Status-tab table PLUS columns no competitor can show: LoRA adapter attached, KV tier/quant, spec-decode draft pairing | new build |
-| Live per-layer quantization sensitivity visualization | optiq computes this internally (KL cost per layer per bit-width) but never renders it — a documented gap even in the best-in-class quant tool | BEAT | We already compute or could compute per-layer bit allocation; a heatmap/bar-strip nobody in the entire surveyed field has shipped, turning an opaque flag into a screenshot-able trust signal | thin addition (Quantize tab exists; visualization is new) |
-| One-command coding-tool integration (`ollama launch <tool>`) | Ollama: `ollama launch claude --model X` wires Claude Code/Codex/OpenCode at itself with zero config | BEAT | We already speak both OpenAI- and Anthropic-compatible protocols — `mlx-bun launch <tool>` is nearly free to build and makes us infrastructure other daily tools sit on | thin addition (CLI, not chat UI, but a real distribution wedge) |
-
----
-
-## Axis 4 — Customization: prompts / presets / sampling
-
-| Capability | Best today | Verdict | What beating them concretely requires | Backend status |
-|---|---|---|---|---|
-| Full sampler parameter set (min_p, XTC, penalty families + windows, logit_bias) | mlx-lm's own `/v1/chat/completions` (the oracle) accepts all of these; mlx-bun's composer only exposes 3 of ~11 | MATCH | Server already implements the full set (`src/server.ts`) — wire `min_p`/XTC/penalties/logit_bias/seed into the sampling popover behind an Advanced disclosure. Table-stakes gap against our own drop-in-replacement claim. | exists (server), thin addition (UI, WS `set_sampling` currently hardcodes 3 fields) |
-| Per-model correct sampling defaults | LM Studio auto-binds presets per model | MATCH | Fix the hardcoded MiniCPM5-shaped `SAMP_REC` — resolve defaults from `generation_config.json` per served model, which the server already supports | exists (server), thin addition (UI) |
-| Per-MESSAGE sampling granularity (temp / max_tokens / thinking per turn, in the composer) | optiq Lab: temperature, max tokens, and enable_thinking overridable per message directly in the chat composer — per-turn granularity, not a per-session settings modal | MATCH | Our per-turn plumbing already exists (`injectSampling` per-turn overrides via `set_sampling`, per-model thinking toggle) — the gap is presentation: keep the popover per-turn-scoped and visible at the composer, not buried, so optiq's per-message granularity has a direct equal | exists (per-turn `injectSampling` + thinking toggle), thin addition (composer placement) |
-| Saved/shareable prompt presets (system prompt + params bundle) | LM Studio Presets: exportable, importable, auto-bound per model | MATCH | Build a preset save/load layer over the existing sampling+system-prompt state | new build |
-| Slash-command prompt library with variable-filled forms | Open WebUI's `/name` + `{{var}}` popup | MATCH | Well-tested pattern, no meaningful downside; build as a Phase-2/3 item | new build |
-| Model-string variant shorthand (`:precise`/`:no-think`) | optiq's `:no-think`/`:precise`/`:creative` suffixes for rigid clients | MATCH | Cheap compatibility hack for agent clients that only set `model` — trivial to add given presets already exist conceptually | new build (thin) |
-| Persona/bundle system (prompt + adapter + tools + memory scope, named, switchable) | Msty's Persona Studio + Crew Conversations; AnythingLLM workspace pinning | BEAT | We're the only one who can bind a *named* bundle to an actual LoRA weight swap (not just a system-prompt swap) — Msty/AnythingLLM personas are prompt bundles only. Phase 4 item; data model should be shaped now so it's a thin layer later. | new build |
-| Warm/preload a context explicitly | mlx-lm's `cache_prompt` as a first-class CLI verb | MATCH | Expose an explicit "warm this session" action turning a backend perf trick into a felt speed win | new build |
-
----
-
-## Axis 5 — Knowledge: documents & RAG
-
-| Capability | Best today | Verdict | What beating them concretely requires | Backend status |
-|---|---|---|---|---|
-| Zero-config document attach (point-and-chat) | GPT4All's LocalDocs: point at a folder, fully offline, no pipeline setup | MATCH | Table-stakes risk: our Chat tab currently has no document/RAG story at all. Ship the simplest possible attach-and-answer path first. | new build |
-| Transparent inline-vs-RAG mode selection | LM Studio: auto-decide inline (fits context) vs. chunk+embed (doesn't), zero user configuration | MATCH | Copy the zero-config decision engine; then extend the *same* engine to also route into the memory vault (see Axis 6) so "attach a file" and "recall memory" feel like one retrieval system | new build |
-| Chat-with-files RAG + citation badges + Sources panel | optiq Lab ships this today in the closest-cousin MLX product: attached non-image files are indexed with a **dependency-free BM25 retriever**, chunks retrieved with inline `[n]` citation markers plus a Sources panel; Open WebUI's SourcePanel adds page-level PDF deep links | MATCH (v1 = optiq's bar) | Match optiq's exact v1 shape — BM25 index over attachments, `[n]` markers, Sources panel (filename+snippet, not full PDF-page rendering) — then extend the same retrieval component to vault articles (Axis 6), which optiq has no concept of | new build (optiq's dependency-free BM25 proves the v1 needs no vector infra) |
-| Unified `#`/`@` retrieval mention (files + docs) | Open WebUI's `#` command pulls a KB doc or raw URL as one interaction | MATCH | Extend to both attached files and vault articles as *one* retrieval gesture — this is where the RAG axis and the memory axis (Axis 6) genuinely merge for us | new build |
-| Retrieval-mode toggle (chunked search vs. full-context) | Open WebUI's Focused/Full-Context switch | MATCH | Cheap, well-proven; add per-document toggle | new build |
-| Hybrid RAG (BM25 + vector + reranking) | Open WebUI's serious RAG maturity (configurable hybrid retrieval, reranking, full-context bypass) | SKIP (full hybrid, for now) | The *hybrid/reranking* tier is what's skipped, not RAG itself — the BM25 v1 above is required (optiq proves it's cheap), and embeddings exist (`src/embed.ts`) for a later vector upgrade; full Open WebUI-grade reranking infrastructure is not warranted at launch | new build (large, deferred tier) |
-| Local semantic search over the user's OWN chat history | Nobody in the survey does this — competitors' RAG is document-centric, not "my own past conversations" | BEAT | The memory vault + session history IS a corpus no competitor's RAG concept reaches; making "search my own past" a first-class retrieval target ties directly into the local-to-you thesis | new build |
-
----
-
-## Axis 6 — Memory & personalization
-
-| Capability | Best today | Verdict | What beating them concretely requires | Backend status |
-|---|---|---|---|---|
-| Auto-extracted, editable memory list (facts/preferences) | AnythingLLM's Observer/Reflector; ChatGPT's Manage Memories (dated rows, per-item delete, search/sort) | BEAT | mlx-bun's memory is architecturally deeper: a git-tracked Markdown vault, read-only at chat time, written only by nightly synthesis — not a flat fact list. The UI's job is making that *visible*: a real Memory panel (vault browser + search), not inventing a new prompt-stuffing mechanism. | exists (`src/memory/tools.ts`, `vaultStatus()`), thin server addition (REST wrappers) + new UI |
-| Per-turn provenance ("the assistant looked this up") | ChatGPT/Claude show *what* is remembered but never *how*; nobody shows live per-turn citations for memory reads | BEAT | Memory tool calls already fire silently in web chat with zero distinct rendering. Give them a citation-chip card style (not the generic wrench icon) — the single highest-leverage rendering change in the whole plan. | exists (tools fire), thin addition (card styling) |
-| Git-diff / history view of what memory synthesis changed | No competitor's architecture can do this — ChatGPT/Claude synthesis runs on someone else's infrastructure, unauditable even in principle | BEAT | Our synthesis runs locally and is git-tracked. A per-article git-log/diff toggle lets a user *watch* an article evolve after a correction — the sharpest wedge against both cloud incumbents' otherwise-superior memory UX, because ours is mechanistically inspectable and theirs structurally cannot be. | exists (git history is real), thin server addition (log/diff read route) + new UI |
-| Wikilink knowledge-graph view | Nobody in the survey has a git-tracked personal wiki at all, so nobody has this | BEAT | `memory_links` already resolves inbound/outbound wikilinks per article — cheap to expose as a graph view; a "knowledge graph of what the model knows about you" nobody else's architecture supports | exists (`memory_links` tool), new build (graph UI) |
-| Scope control (this-session / this-surface / global) | AnythingLLM's workspace-vs-global memory scoping | MATCH | Vault is currently single-scope; scope selection is a reasonable v2 addition once the panel exists | new build |
-| Pause vs. reset as distinct actions | ChatGPT/Claude: pause (stop reading/writing, keep data) vs. reset (irreversible wipe) as two different stakes levels | MATCH | Cheap UX pattern to copy once settings/memory panel exists | thin addition |
-| Memory that graduates into trained weights (not just context) | Nobody — not AnythingLLM, not ChatGPT, not Claude — synthesizes memory into actual model weights; all of them are context-window-bounded fact lists | BEAT | This is mlx-bun's single deepest structural moat: the nightly "dreaming" pipeline can eventually bake a fact into a memory LoRA, not just a growing prompt. A visible marker distinguishing "raw context" vs. "baked into your memory LoRA" is a distinction no competitor's architecture can even express. | partial (synthesis-to-context exists; synthesis-to-LoRA is a further phase per project memory) |
-| First-run memory consent flow | AnythingLLM/Odysseus bury consent in settings; nobody makes it a first-60-seconds moment | BEAT | `mlx-bun memory init` today is CLI-only, invisible to anyone who never opens a terminal. A one-line skippable consent card in the hero screen puts personalization consent in the same 60 seconds as "install to chat under a minute" — no competitor frames it this tightly. | exists (CLI), thin addition (hero-screen card) |
-| Nightly synthesis schedule status visible | Odysseus runs nightly audits but doesn't surface schedule state in mainstream UI docs | MATCH | `vaultStatus()` already returns schedule state, last-run time — just render it | exists, thin addition |
-| Self-evolving procedural memory ("Skills") | Odysseus: auto-distills reusable markdown playbooks from multi-tool-call sessions, teacher-escalation on failure | SKIP (v1) | Genuinely novel mechanic but a large, separate system (distillation trigger, confidence scoring, LLM-as-judge harness) — not core to the memory-vault thesis; defer as a later differentiator, not a launch requirement | new build (large, deferred) |
-
----
-
-## Axis 7 — Tools & agents
-
-| Capability | Best today | Verdict | What beating them concretely requires | Backend status |
-|---|---|---|---|---|
-| Tool-call visibility as a distinct card (not silent) | Universal table stakes now (Open WebUI, LM Studio, Jan, AnythingLLM all show tool cards) | MATCH | Already shipped (`app.html` running/ok/fail cards) | exists |
-| Inline, non-blocking approval cards | Jan v0.8.0: moved away from blocking modal to inline cards embedded in the tool-call bubble, plus global "Allow All" | MATCH | We already have the approval round-trip (`installApprovalGate`) — the presentation layer needs to match Jan's proven inline pattern | exists (backend), thin addition (styling matches Jan's pattern already) |
-| Editable arguments before approval | LM Studio: edit the proposed call's arguments, not just accept/deny | MATCH | Extend the existing approval card to allow inline argument editing before execute | thin addition |
-| Durable, tool-identity-keyed "always allow" | Universally fragile across the field — Claude Desktop's persistence resets on app update; Cowork doesn't persist across sessions at all | BEAT | This is an easy, concrete bar: store approvals in a real versioned config file keyed by stable tool identity, with a migration test across simulated version bumps — fixing a failure mode every competitor visibly has | new build (small — config file + test) |
-| Gated tools (bash/edit/write) actually reachable in web chat | optiq Lab ships a terminal tool (bash one-liners with dangerous-command filtering) live in chat today; mlx-bun's own approval-card UI is fully built but currently dead code — the browser allowlist excludes every `GATED_TOOL` | MATCH (fix our own regression) | Wire `codingTools: true` behind an explicit off-by-default "let it touch files" toggle so the already-shipped UI stops being inert; our per-call approval gate is a *stronger* trust model than optiq's static dangerous-command filter once it's actually reachable | exists (backend + UI both exist, just disconnected) |
-| Sandboxed Python execution tool | optiq Lab: built-in Python tool running in a sandbox with AST-level security checks, killed cleanly on Stop | MATCH | Add a Python-interpreter tool to the pi tool surface (or route through the existing gated bash path with an interpreter preset); single-user local means "runs on your machine" is the honest framing, with the approval gate as the control — full AST-sandbox parity is optional, the *capability* is the requirement | new build (moderate; approval plumbing exists) |
-| Self-healing tool calls (auto-correct malformed formats) | optiq Lab: auto-corrects six distinct malformed tool-call formats instead of failing the turn — directly addresses the reliability gap mlx-lm's own server logs warn about (truncated/failed parses, no recovery) | MATCH | Add a repair layer in the tool-call parse path (`src/tool-call.ts`) covering the common malformation classes; small local models flub tool-call syntax constantly, so without this every optiq user switching over feels a reliability regression | new build (parser hardening; tool-call parsing exists) |
-| Tool-loop hygiene (dedup, retry budget, turn cap, force-finish) | optiq Lab: skips identical consecutive successful calls (dedup), gives 3 failed attempts then forces course-correction (retry budget), caps the loop at 25 turns, and force-generates a text answer if the cap is hit (never ends a turn tool-looping with no answer) | MATCH | Implement the same four guards in the pi agent-loop bridge; the force-finish guarantee ("you always get an answer, never a stuck loop") is the piece users notice most | new build (loop-policy layer over existing agent loop) |
-| Argument/diff preview before approval | Neither ChatGPT nor Claude previews the concrete blast radius (file diff, shell command, bytes written) before approval — both show name + yes/no only | BEAT | mlx-bun's approval hook already has the data; showing an actual diff/args preview clears a bar neither cloud incumbent has cleared | thin addition |
-| Per-session disk-touch tally (trust log) | Nobody in the survey ships a running "what did the agent touch on disk this session" ledger distinct from tool-call cards | BEAT | Single-user local execution means this is honest, cheap, and uniquely credible — "runs on your machine, with your files" instead of Open WebUI/Odysseus's sandbox theater for multi-tenant deployments we don't need | new build |
-| Curated pre-wired tool pack on first run | Cherry Studio: filesystem, web search, GitHub, fetch, memory pre-registered, zero JSON config | MATCH | Ship a default pack including our own wiki-read/vault tool — turns first-run into a memory-flywheel demo instead of a config chore | new build (small) |
-| MCP server support | Open WebUI native MCP + mcpo proxy; LM Studio Cursor-compatible `mcp.json`; Jan MCP marketplace | MATCH | Add MCP client support behind the existing tool-call plumbing — real engineering, but table stakes for 2026 agentic chat | new build (moderate-large) |
-| Web search + URL fetch, no API key | optiq Lab: built-in DuckDuckGo search + URL fetch requiring zero API keys, on by default in chat; Odysseus/Open WebUI need SearXNG/config | MATCH | `web_search`/`web_fetch` tools already exist, keyless and read-only-safe — but `web_fetch` is currently excluded from the hardcoded `WELCOME_TOOLS` allowlist; widen the toolset per served model and expose an explicit per-turn toggle | exists (tools), thin addition (allowlist + UI toggle) |
-| Approval-fatigue-aware defaults | Field-wide complaint (Jan, LM Studio, Claude Desktop users all report fatigue) | MATCH | Ship read-only/local tools (file read, our wiki-read) as always-allow-by-default, reserve prompts for write/network/execute-class tools | thin addition |
-| tool_choice forcing a specific function | API-only today (`src/server.ts`); no chat-UI concept | SKIP (chat UI) | Real capability but a power-user/API concern, not a chat-composer affordance — low priority for the redesign | exists (API), not a UI priority |
-| No-code visual agent-flow builder | AnythingLLM's Agent Flows canvas; Odysseus admin-gated automation | SKIP | Building a general visual flow builder from scratch is a multi-quarter distraction with no differentiator payoff; if we need automation, scope it narrowly (a "schedule a synthesis-style job" UI) rather than a general canvas | not planned |
-| Deep-research-style multi-step trace panel | Odysseus's Deep Research; Open WebUI's Interleaved Thinking | MATCH | Table stakes for agentic 2026 chat UIs; wrap a visible plan/trace UI around existing tool-call infra as a research-flavored preset | new build |
-
----
-
-## Axis 8 — Multimodal
-
-| Capability | Best today | Verdict | What beating them concretely requires | Backend status |
-|---|---|---|---|---|
-| Vision input (image attach + inference) | Universal now (Ollama desktop, LM Studio, optiq's bf16 sidecar) | MATCH | Already wired end-to-end correctly (capability negotiation via `ready` frame's `vision` flag); only fix needed is stale "switch to 12B" copy now that e2b/e4b/26B/31B are all vision-capable | exists |
-| Vision + prompt-cache unification (no perf tax when text-only) | LM Studio's Unified MLX Engine: single code path conditionally attaches a VisionAddOn, ~25x faster follow-up TTFT via shared caching | MATCH | Verify our own vision path already avoids a text-only tax (SigLIP sidecar pattern is architecturally similar); if not already true, this is a real perf investment worth matching | exists (verify) |
-| Image generation | Not in scope for any MLX-native serving engine surveyed except bolted-on extras (mlx-serve) | SKIP | No image-generation model/pipeline exists in the engine at all — genuine substrate gap, not a UI decision | not built |
-| Speech-to-speech voice mode | ChatGPT Advanced Voice Mode: single model hears tone/emotion, replies in 2-3s | SKIP | No speech/TTS/STT engine exists. This is a hard, expensive model-latency problem, not a UI problem — a naive STT→LLM→TTS bolt-on would read as a visible regression against user expectations set by the cloud bar. Correctly scoped as out of v1. | not built |
-| Native OS voice I/O (dictation, TTS readback) | Enchanted's native SwiftUI voice I/O across Apple devices | SKIP | Same substrate gap as above — no TTS/STT pipeline; Enchanted rides macOS system APIs we haven't wired at all | not built |
-| Document (PDF/DOCX) attach with inline-vs-RAG | LM Studio's transparent dual mode | MATCH | Covered under Axis 5 (Knowledge) — same capability, cross-referenced here for multimodal completeness | new build |
-| Screenshot capture in a floating companion window | ChatGPT macOS's Option+Space + instant screenshot; mlx-serve's Ctrl-Space quick launcher | BEAT | The Tauri shell already wraps the same web UI with zero bespoke chrome; adding a global hotkey + floating window wired to the existing vision pipeline is a concrete way to earn native-app status a browser SPA structurally can't claim | exists (vision pipeline), new build (Tauri-specific, Phase 4) |
-
----
-
-## Axis 9 — Performance & transparency
-
-| Capability | Best today | Verdict | What beating them concretely requires | Backend status |
-|---|---|---|---|---|
-| Live tok/s + TTFT readout | LM Studio 0.4's ambient context-fullness/tok-s chrome; macMLX's Benchmark tab | MATCH | Already computed (`app.html` post-turn readout) — just under-designed as small dim text | exists |
-| Context-window meter | LM Studio's persistent fullness indicator | MATCH | Already exists (`app.html:1696-1703`) | exists |
-| Serving-lane badge (serial / spec-decode / batched) | Nobody in the entire surveyed field integrates lane-awareness into the *live chat surface* — macMLX/mlx-serve have a separate Benchmark tab, not live-turn lane visibility | BEAT | This is the one thing no competitor's architecture allows, because none of them control both spec-decode and batching as internally routed lanes AND expose that routing per-turn in the chat itself. Needs a small per-turn `lane` field added server-side (usage payload + pi-web frame) — the single highest-differentiation, lowest-backend-cost item in the whole matrix. | thin server addition (field doesn't exist yet) + UI |
-| Prompt-cache-hit telemetry per turn | mlx-lm's LRUPromptCache tracks reuse internally but never surfaces it to a user; nobody in the survey shows "served from cache" per turn | BEAT | We already compute cache-hit counts server-side; surfacing "this turn reused N cached tokens, saved Xms" turns an invisible backend optimization into a felt trust/speed signal | exists (server), thin addition (UI) |
-| Built-in Benchmark tab (prefill/decode tok/s, TTFT, peak RSS) | macMLX's Benchmark tab; mlx-serve's "+39% geomean" claim | MATCH | We already discipline every perf claim with `benchmark.sh` (per CLAUDE.md ground rules) — exposing a lightweight in-app version turns internal QA into user-facing proof | exists (harness), new build (in-app UI) |
-| Compare/split view of two configs on one prompt | optiq's Model Arena is the closest-cousin implementation: side-by-side 2-model comparison on identical prompts, model B spun up in a second server on port+1 on demand, both panes streaming live with tok/s + token counts, both models staying loaded while Arena is open, thinking force-disabled for reasoning models so comparisons show real answers; Msty's Split Chat adds synced typing across N panes | BEAT | Match the Arena mechanics (two live-streaming panes, tok/s per pane, thinking force-disabled — copy that design decision verbatim), then beat it on two documented Arena gaps: (1) compare things optiq *can't* — base vs. LoRA-adapted variant, bit-exact parity mode vs. L3 fast path — since Arena only compares whole models; (2) add a quality signal (Arena shows only speed + token counts, no voting/ranking — its own docs' weakness), e.g. a memory-grounded check or preference capture wired into the ORPO data pipeline | new build |
-| Concurrency / batch-slot visibility | optiq's documented `--max-concurrent-equivalent` guidance | MATCH | Document + add a live concurrency/slot indicator tied to our shipped `--batch N` concurrent serving | thin addition |
-| Hardware-fit prediction (`/fit`) surfaced in chat | LM Studio's traffic light is generic-heuristic; we measure actual bandwidth per SKU | BEAT | `GET /fit` already predicts decode tok/s and safe context for *this specific Mac* — surface it in the model switcher and adapter picker, not buried in the Status tab only | exists (server), thin addition (surface beyond Status tab) |
-| Auto-sized context window to available memory | Ollama has open feature requests for exactly this (currently a conservative fixed default) | BEAT | Our M4-Pro-tuned memory-headroom knowledge (Metal/unified memory) lets us size this more precisely than Ollama's generic GPU detection; the context-window meter already exists as the display surface | thin addition (logic; UI meter exists) |
-| Provable "zero cloud calls" per session | No cloud competitor can make this claim structurally; local competitors don't bother proving it | BEAT | A per-session network-activity indicator (there were zero egress calls) is trivially true for us and impossible for ChatGPT/Claude's "temporary chat" (still retained 30 days per OpenAI's own disclosure) | new build (small) |
-
----
-
-## Axis 10 — Polish: onboarding / theming / keyboard / a11y / mobile
-
-| Capability | Best today | Verdict | What beating them concretely requires | Backend status |
-|---|---|---|---|---|
-| Dark + light theme, `prefers-color-scheme` | Universal table stakes; mlx-bun is dark-only today (`--bg:#000` hardcoded) | MATCH | Add `prefers-color-scheme` handling + a light theme; audit design tokens | thin addition |
-| `prefers-reduced-motion` respect | WCAG baseline; ambient blooms currently unconditional | MATCH | Guard shimmer/bloom animations behind the media query | thin addition |
-| Full keyboard shortcut sheet (`Cmd/Ctrl+/`) | ChatGPT's documented vocabulary is the de facto standard | MATCH | Adopt ChatGPT's exact bindings where sensible — zero cost to originality, reuses existing copy-button code for "copy last response" | thin addition |
-| Command palette (`Cmd/Ctrl+K`) | ChatGPT/Claude: universal "do anything" entry point | MATCH | Build a palette covering new chat/session search/model-adapter switch/thinking toggle | new build |
-| Mobile responsive layout down to 375px | Universal table stakes; mlx-bun's sidebar vanishes at 760px with zero replacement | MATCH | Replace the vanishing sidebar with a slide-over drawer + hamburger — table stakes, not differentiation | new build (small) |
-| PWA installability (manifest + service worker) | Open WebUI ships "Install app" as a first-class menu item | MATCH | Add a manifest + minimal cache-first shell service worker; scope explicitly as shell-instant-load + installability, not offline chat (app is useless without the local server running) | new build (small) |
-| A11y: `role="log" aria-live="polite"` on the thread | WCAG ARIA23 documented technique; mlx-bun has 11 total `aria-*` attributes across 2737 lines | MATCH | Throttle to paragraph/sentence boundaries, never per-token; verify against actual VoiceOver testing, not just docs | new build (small) |
-| Full keyboard tab order + focus rings + Escape-to-close | Baseline a11y expectation; currently no documented strategy | MATCH | Trap focus in popovers, Escape returns focus to trigger | thin addition |
-| Progressive disclosure (Simple / Developer mode) | LM Studio's User/Power-User/Developer toggle — one codebase, one switch, not a fork | MATCH | Collapse Quantize/Fine-tune/Build-Dataset/Status/Curves/Routes behind one Developer toggle so Chat is the whole app for a first-time user — directly serves "install to chat under a minute" | new build (IA move, no new backend) |
-| Personalized, server-driven onboarding chips | Nobody ties onboarding chips to actual server state — LM Studio/Ollama's onboarding is generic | BEAT | Replace 4 static marketing chips with server-driven ones: one drawn from real vault content, one offering to try an adapter, a first-run memory-consent card — because our onboarding can reflect real personalization state nobody else's architecture has to reflect | thin addition (`vaultStatus()`, `/v1/adapters/available` already exist) |
-| Chat export (Markdown/PDF/plain text) | LM Studio exports to PDF/MD/plain text; Open WebUI exports JSON | MATCH | Add export action to session actions; local-only, no server cost concern | new build (small) |
-| Chat sharing (shareable link) | Open WebUI: shareable chat links | SKIP | Requires a hosting/sharing surface that conflicts with the single-user, local-only, no-contributors identity — a shareable *link* implies a server reachable by someone else, which is out of scope by design, not a capability gap | not planned (scope decision) |
-| Full-text search across message bodies | Claude's own search is title-only — a widely-cited annoyance; local storage removes any indexing-cost excuse | BEAT | No server-cost constraint exists locally the way it does for Claude/ChatGPT's cloud indexing — ship real full-text body search, not just title search, and be visibly better than the cloud incumbents on this specific point | new build (small) |
-| Session/conversation folders and pins | Open WebUI folders/tags; ChatGPT pin/star | MATCH | Add pin + lightweight tag/folder grouping to the sidebar | new build |
-| Native menu-bar/tray behavior (clean quit, no zombie process) | Ollama gets public complaints for exactly the failure mode to avoid (can't hide dock icon, can't fully quit) | MATCH | Cheap insurance in the Tauri shell — verify clean quit/hide-dock-icon before shipping, avoid inheriting a known competitor failure mode | thin addition (Tauri config) |
-
----
-
-## Axis 11 — Sessions & organization
-
-| Capability | Best today | Verdict | What beating them concretely requires | Backend status |
-|---|---|---|---|---|
-| New / list / open / fork / delete sessions | mlx-bun already round-trips all of this through pi's `SessionManager` | MATCH | Already shipped, real depth | exists |
-| Session search box | Table stakes; mlx-bun's sidebar has none today | MATCH | Add a filter/search input above the session list | thin addition (small) |
-| Per-message branch-from-here (not just whole-session fork) | ChatGPT/Claude's edit-creates-branch convention | MATCH | `SessionManager.forkFrom` already exists server-side — expose at message granularity, not just the sidebar | thin addition |
-| Projects (scoped knowledge + memory per workspace) | Claude Projects: persistent system instructions + files + isolated memory summary per project | BEAT | A Project is effectively "routed context + adapter scope" in UX terms even though Anthropic implements it as RAG-over-files; we can implement the same UX shape with an actual weight-level adapter swap underneath, something Claude's Projects structurally cannot do since they don't ship adapter weights per customer | new build (persona/bundle concept, Phase 4) |
-| Temporary/incognito chat | ChatGPT/Claude: one-click pill in composer header, per-conversation scope | BEAT | Cloud "temporary chat" is still retained 30 days per OpenAI's own disclosure "for safety" — ours can be **provably** never written to disk at all, because we control the whole stack. Pair with a per-session "zero cloud calls" indicator. | new build (small) |
-| Multi-model / multi-adapter compare | Msty's Split Chat; optiq's Model Arena; LM Studio's Split View | BEAT | Covered in Axis 9 — cross-referenced here as an organizational pattern (parallel panes as a session-management primitive, not just a perf demo) | new build |
-| Conversation export/import (portable JSON) | Open WebUI: JSON export/import | MATCH | Local JSON already backs every session on disk — expose an explicit export/import action | thin addition |
-
----
-
-## Axis 12 — App-aware assistant (added 2026-07-06, from the PortfolioManager evaluation)
-
-The chat agent can see and act on the app it lives in — a structured DOM
-snapshot (no screenshots, no vision round-trip), agent-driven navigation,
-and element spotlighting. The product theory: guided tours fail on timing
-(help before intent → skipped), chatbot help fails on grounding (moment of
-intent, but blind — a search bar with a face); this is the only shape with
-both — help at the moment of intent, grounded in what's actually on screen.
-Acceptance test: it feels like someone is there with you. Verified against
-every other axis: **no product in the surveyed field has any of this** — the nearest neighbors are generic
-browser-automation agents (wrong tool, heavyweight) and static product
-tours (not conversational). Reference implementation:
-`~/Code/PortfolioManager` (`server/src/agent/{ui-catalog,portfolio-tools}.ts`,
-`client/src/lib/{ui-snapshot,spotlight,resolve-spotlight}.ts`,
-`ui-catalog.json`) — same pi SDK as our `src/pi-web.ts`, so the port is
-mechanical.
-
-| Capability | Best today | Verdict | What beating them concretely requires | Backend status |
-|---|---|---|---|---|
-| Agent sees the live UI (structured snapshot, not pixels) | Nobody in the field; PortfolioManager (private) proves the mechanism: on route change the browser pushes a `uiSnapshot` of visible interactive elements + labeled regions ({ref, label, role, selector}, ~120-element cap, agent chrome excluded) as a WS `context` frame; a `get_current_app_context` tool returns it | BEAT | Add the snapshot builder to the web app + a `context` ClientMessage + the read tool in `pi-web.ts`. Snapshot is DOM-derived (a11y-tree-like) — kills the screenshot→upload→describe round-trip entirely, locally, with no vision model in the loop | new build (small; pattern proven in-house) |
-| Agent navigates the app for you ("take me to Quantize" → it clicks the tab) | Nobody; PortfolioManager's `navigate_app` validates against a curated `ui-catalog.json` route map → `ui_navigate` WS frame → browser routes | BEAT | Catalog our routes (incl. Developer-toggle views) + the tool + frame. Navigation is reversible and safe — no approval needed. Synergy: resolves the Developer-toggle discoverability tension (§8) — hidden tabs become conversationally discoverable | new build (small) |
-| Agent spotlights a control with a hint ("the source-model field is here") | Nobody; PortfolioManager's `spotlight_ui` locates by ref/label/selector/catalog-id, optionally navigates first, highlights via overlay + popover, auto-dismisses | BEAT | Curated `data-spotlight` anchors on stable controls + a hand-rolled overlay (backdrop cutout + popover, ~80 lines, matches our tokens — no need to vendor driver.js). The agent answers "how do I X here?" by pointing at the actual screen | new build (small) |
-| Approval-gated agent UI actions (fill this form, start this job) | Nobody — PortfolioManager deliberately stops at navigate+point; generic computer-use agents act but with no in-app trust surface | BEAT (stretch) | Route `ui_act` tools (set field, submit job) through the EXISTING approval-card gate (`installApprovalGate`) so guided actions render as visible allow/deny cards with argument preview — the trust model no browser-automation layer has. Phase 5; requires the approval surface from Phase 2 to be live first | new build (moderate; approval gate exists) |
-
----
-
-## The 10 beats
-
-Ranked by (visible wow) × (structural defensibility) — the ten rows where
-mlx-bun can be clearly, demonstrably better than the current best, not just
-equal.
-
-1. **Serving-lane badge in live chat (Axis 9).** No competitor surveyed —
-   not macMLX, not mlx-serve, not optiq — integrates spec-decode/batch/serial
-   lane-awareness into the actual chat surface; they all bury it in a separate
-   Benchmark tab. Demo: watch the badge flip from "serial" to "spec-decode
-   active" mid-conversation on the `dspark` branch and see tok/s visibly jump
-   in the same breath.
-2. **Memory git-diff / self-healing view (Axis 6).** ChatGPT and Claude show
-   *what* is remembered but never *how* — their synthesis is a black box on
-   someone else's infrastructure. Demo: correct a fact mid-chat, run nightly
-   synthesis, then open the article's git diff and watch the exact line
-   change.
-3. **Per-turn memory provenance chip (Axis 6).** Memory tools already fire
-   silently today with zero UI trace — this is a near-free wiring fix with
-   outsized visible payoff. Demo: ask a personal question, see a small
-   "read: Photography" citation chip appear and expand to the source
-   article.
-4. **LoRA adapter routing made legible + stackable (Axis 4/6).** LM Studio's
-   own bug tracker confirms adapter hot-swap is unsupported industry-wide;
-   optiq proved the mechanism but shipped it Python-API-only. Demo: swap from
-   "general-purpose" to a named adapter mid-session with a visible RAM-cost
-   note, then stack a second adapter (`"a+b"`) live.
-5. **Adapter/parity compare view (Axis 9/11).** optiq's Arena (the
-   closest-cousin implementation: two live panes, port+1 second server,
-   tok/s per pane, thinking force-disabled) can only compare whole *models*
-   and shows no quality signal at all — its own documented gap. Nobody
-   compares a base model against its own LoRA-adapted variant, or bit-exact
-   parity mode against the fast path, on the identical prompt. Demo: fire
-   one prompt at base vs. adapted, side by side, and see the adapter
-   visibly change the answer's grounding — a comparison Arena literally
-   cannot express.
-6. **Model swap that actually preserves the session (Axis 3).** Every
-   competitor surveyed is documented as bad at this (LM Studio's own open
-   bug). Demo: swap the served model mid-thread and keep scrolling the same
-   conversation with no reset, no lost history.
-7. **Prompt-cache-hit telemetry per turn (Axis 9).** mlx-lm's LRUPromptCache
-   tracks this internally and never shows it; nobody in the field surfaces
-   "served from cache" as a felt signal. Demo: a repeated near-identical
-   prompt visibly shows "reused 1,400 cached tokens" and a corresponding
-   TTFT drop.
-8. **Provably-local temporary chat + zero-cloud-calls indicator (Axis 11).**
-   Cloud "temporary chat" is retained 30 days per OpenAI's own disclosure —
-   ours is structurally different because we control the whole stack. Demo:
-   toggle temporary chat, chat normally, then show a network-activity panel
-   proving zero egress the whole time.
-9. **Tool-call diff/argument preview before approval (Axis 7).** Neither
-   ChatGPT nor Claude previews the actual blast radius (file diff, shell
-   command) before approval — both show name + yes/no. Demo: approve a
-   `write` call and see the literal diff that will land on disk before
-   clicking allow.
-10. **Full-text search across message bodies (Axis 10/11).** Claude's own
-    search is title-only, a widely-cited annoyance, purely because
-    server-side indexing has a cost model we don't have locally. Demo:
-    search a phrase buried in the middle of a months-old conversation and
-    jump straight to it.
-11. **The app-aware assistant (Axis 12 — added 2026-07-06).** Nobody's
-    local chat agent can see the app it lives in; on onboarding wow alone
-    this ranks top-3. Demo: ask "how do I make a smaller model?" — the
-    assistant answers from the live screen, reveals the Developer tabs,
-    navigates to Quantize, and spotlights the source-model field, with
-    zero screenshots taken.