mlx-bun 0.3.0 → 0.5.0

package/docs/design/unified-engine-frontier-plan.md

@@ -215,6 +215,17 @@ fusedSdpaRuntimeOk(q, mask)` in `src/model/generated/gemma4-*.ts`) that are
 per-generation-decidable and should be hoisted out of the token path; compiled
 decode already achieves the debranched line dynamically (trace once, replay).
 
+The shared greedy operation uses native logsumexp followed by tiled normalized
+score/index reductions in `src/mlx/normalized-argmax.ts`. It preserves rounding
+and lowest-ID ties without materializing the normalized vocabulary array.
+The step sampler applies processors and grammar first, and selects this operation
+only for built-in greedy requests that do not require metadata. Independent
+verification calls the same operation. Fixed-shape compilation caches each
+geometry; enclosing shapeless graphs retain native operations. No scheduler,
+model-family branch or separate sampling setting selects this optimization.
+Actual-model and operation comparisons, plus both-machine serving measurements,
+are recorded in benchmarks.md under fused normalized greedy selection.
+
 ### Layer 0 — the SSD spill substrate
 
 It is almost always faster to cache to SSD and mmap it back than to
@@ -414,12 +425,12 @@ satisfy, with the state of `GenerationGateway.#supportsContinuous` as read on
 | layer | today (src) | end state |
 |---|---|---|
 | dedicated per-model graph | both mechanisms (profile-routed) | unchanged |
-| mixed-precision KV (`affine-config`) | continuous when every configured layer is a plain or rotating KVCache the scheme can convert (`KvScheme.batchable` + the gateway's cache probe); uniform `kvBits`, SSM-layer configs, and TurboQuant route serial | one active scheme per server, any B |
-| LoRA adapters | serial only (`shape.hasAdapters`) | per-slot adapter state (vLLM Punica/SGMV pattern: every row runs the adapter math, non-adapter rows at scale 0) |
-| speculative decoding (two-model · assistant · dspark · deepspec · native MTP · ngram) | serial only; a mounted draft routes every request serial (`shape.hasDraft`, mlx_lm.server's `is_batchable = draft is None`); GLM-5.2 MTP is on by default and pins the serial verify lane (`--mtp off` restores batching) | per-slot drafting behind the `DraftSource` seam |
+| mixed-precision KV (`affine-config`) | supported per-layer and uniform affine KV use continuous groups; incompatible layouts remain open | one active scheme per server, any B |
+| LoRA adapters | compatible ordered adapter sets share a group; context and cache identity bind under the execution lease | per-slot adapter state (vLLM Punica/SGMV pattern: every row runs the adapter math, non-adapter rows at scale 0) |
+| speculative decoding (two-model · assistant · dspark · deepspec · native MTP · ngram) | actual speculation uses the serial verifier; supported ordinary requests can batch with a mounted draft | per-slot drafting behind the `DraftSource` seam |
 | structured output (grammar) | both (per-row matchers; `MLX_BUN_GRAMMAR_BATCH=0` forces serial) | unchanged; keep the conformance gate |
-| sampling (temp/top-p/top-k/min-p/XTC/penalties/HLG/seed) | per-row `StepSampler` batches, including repetition penalty and logits extras; a user-fixed `seed` forces serial (mlx-lm's `_is_batchable`) | fully per-row, seeds included |
-| logprobs capture | serial only (`shape.wantsLogprobs`) | per-row readback in the scheduler |
+| sampling (temp/top-p/top-k/min-p/XTC/penalties/HLG/seed) | per-row sampling sessions, including explicit seeds, repetition penalty and logits extras | fully per-row, seeds included |
+| logprobs capture | shared row sampling and readback in continuous groups | preserve method composition |
 | vision / audio media prompts | serial only (`shape.hasVision`; embeddings-prefill bypasses the prompt cache) | batched media prefill |
 | prompt cache / SSD tier | both: scheduler joiners `take()` the longest usable prefix at admission, never-merged rows `put()` back on finish; the cold tier restores inside `take()` | block-granular sharing across rows |
 | paged KV (`--paged-kv`) | serial only, Gemma4 bf16, refuses `--batch N>1`, `--kv-quant`, `--draft-model`; bypasses the prompt cache | block-level CoW prefix sharing (vLLM oracle) |
@@ -472,132 +483,13 @@ entries so a caller cannot mutate conversion or accounting after resolution.
    `BatchScheduler.#prefillChunk` in `src/serve/batch-scheduler.ts` (the same
    conventions re-implemented over `PrefillState` for interleaved admission).
    Both are gated bit-exact, but a convention change must land in two places.
-2. **Spec eligibility is decided after placement.** `planRequest` only knows
-   `hasDraft = !!ctx.draft` (server-level), which routes every request serial
-   while a draft is mounted. The real per-request eligibility — text-only, no
-   adapters, no logprobs capture, bf16 KV (no `kvBits`/`kvConfig`/`turboQuant`),
-   not paged — is evaluated inside `runGeneration` after `place()`; ineligible
-   requests silently fall through to plain serial decode, and the lane label is
-   corrected from `serial+spec` to `serial` only when `stats.spec` is absent
-   (`finalLane` in the executor). Resolving this means lifting the eligibility
-   predicate into `RequestShape` so placement and the lane label are decided
-   once.
-
-Other seams worth knowing: `runGeneration` also decides the prompt-cache
-bypass (media, paged) and the adapter cache namespace; `--isolate` wraps the
-ENTIRE server as an engine child behind a proxy (`src/serve/isolate.ts`,
-`ModelPool` for `--model-pool` LRU residency and model switching by the
-request's `model` field) — the inter-process API is the /v1 surface itself, no
-second protocol.
-
-## 7. The flag surface (src truth, grouped by layer)
-
-Read from `SERVER_FLAGS` and `applyDecodeRoute` in `src/cli.ts`. Every
-`MLX_BUN_*` value is read through the immutable runtime snapshot
-(`src/runtime-config.ts` `runtimeValue/runtimeFlag`, `src/flags.ts` `flagOn`);
-feature code never reads `process.env`, and the CLI installs tier/fork
-overrides with `configureRuntime()` before model modules load.
-
-**Parity tier (aliases):** `--l1` (default) · `--l2`. `--l3` hard-errors with a
-pointer to this doc.
-
-**L2 — quantization:** `--kv-quant config|off|4|8|turbo[:k<bits>v<bits>]`
-(default off). An older draft of this doc listed only `config|4|8|off`; the
-`turbo` axis is real and mutually exclusive with the affine modes.
-
-**L3 — scheduler / decode policy:** `--batch <n>` (default 8;
-`--decode-concurrency` alias) · `--kv-budget <GB>` · `--draft-model <query>` ·
-`--draft-kind two-model|assistant|dspark|deepspec|mtp|ngram` ·
-`--num-draft-tokens` · `--ngram-max` / `--ngram-min` · `--mtp on|off`
-(GLM-5.2 native MTP, on by default) · `--context-length` (GLM-5.2 reservation)
-· `--paged-kv` / `--paged-kv-block-size` (env `MLX_BUN_PAGED_KV=1`).
-
-**L4 — serving surface:** `--model` · `--host` · `--port` · `--memory-budget`
-· `--prompt-cache <GB>` · `--ssd-cache <dir>` · `--ssd-cache-max` ·
-`--ssd-demote-idle` · `--ssd-cache-verify` · `--isolate` · `--model-pool` ·
-`--unix` (internal) · `--no-open` · `--allow-private-media` · `--adapter`
-(`--adapter-path` alias) · `--thinking` · `--temperature`/`--temp` · `--top-p`
-· `--top-k` · `--max-tokens` · the HLG sampler family (`--hlg-sampling`,
-`--hlg-width`, `--hlg-shoulder`, `--hlg-toe`, `--hlg-pivot-offset`).
-
-**Kill switches (bit-exact A/B levers, each selects a slower same-parity
-path):** `--compiled-decode on|off` (`MLX_BUN_COMPILED_DECODE`) ·
-`--compiled-activations on|off` (`MLX_BUN_COMPILED_GEGLU` +
-`MLX_BUN_COMPILED_SWIGLU`; only gemma geglu and MiniCPM5 swiglu have an
-uncompiled form — qwen3/qwen3.5/universal compile unconditionally) ·
-`--fused-sdpa on|off` (`MLX_BUN_NO_FUSED_SDPA`, inverted; default follows
-`--kv-quant`) · `--force-wire` (`MLX_BUN_FORCE_WIRE`) · `--expert-offload`
-(`MLX_BUN_EXPERT_OFFLOAD`). An older draft said these were env-only; the three
-decode kill switches are CLI flags as well.
-
-**Lab / diagnostic env flags (no CLI flag; live with a bench and an expiry):**
-`MLX_BUN_GRAMMAR_JUMP=1` (jump-forward decoding, serial non-spec loop),
-`MLX_BUN_GRAMMAR_BATCH=0` (grammar back to serial), `MLX_BUN_BATCH_SSM=0`
-(SSM caches back to serial), `MLX_BUN_BATCH_NO_PIPELINE`,
-`MLX_BUN_BATCH_STEP_TRACE`, `MLX_BUN_BATCH_VEC_SAMPLE`, `MLX_BUN_BATCH_EXTEND`,
-`MLX_BUN_PREFILL_TAIL_SPLIT`, `MLX_BUN_FLASH_MIN_M`, the `MLX_BUN_CCE_*`
-training-kernel switches, and the memory/trace loggers. Anything
-output-changing is Lab by definition.
-
-**Deleted (2026-07-05, §11):** `--fused-decode`, `--fused-gelu`,
-`--perf-kernel`, `--l3`, and their env twins. None exist in `src/`.
-
-## 8. Design invariants proven en route (durable, not changelog)
-
-- **Readbacks in a pipelined loop must not create ops.** `toFloat32()` on the
-  int token array enqueued a cast BEHIND the next dispatched step, stalling the
-  "overlapped" read a full GPU step per token. Use `MlxArray.toIntTokens()`.
-- **A lone row keeps bare serial-class caches.** `KVCache.makeMask(1)` is the
-  empty mask and scalar rope; the batched step then dispatches the same graph
-  the serial loop builds. The per-layer `BatchedDecodeMaskCache` wrapper is
-  only for padded batches.
-- **Rope-array step-stability contract.** A batched cache's `ropeOffsetArr`
-  must be stable within a decode step and refresh only at `releaseRopeArr()`;
-  re-reading it mid-update ropes K and Q one position apart.
-- **Generated-file guards accept batched subclasses.** The generated forwards'
-  per-layer `instanceof` guards pass any batched cache that subclasses a
-  serial class, so an all-quant gemma batch decodes through the generated
-  fast path — a feature (B=1 proved bit-exact) but a contract: batched caches
-  must behave serially under re-reads. Test with FULL configs; one bf16 layer
-  anywhere fails the guard and drops to the monolith.
-- **Quantization packs along HEAD_DIM**, so token-axis batch surgery over
-  (packed, scales, biases) triples is byte-safe; solo rows convert at serial
-  chunk boundaries (bit-exact by construction).
-- **A scheme-less path must refuse, never silently drop quantization.** The
-  gateway only threads a `KvScheme` to the scheduler when it is batchable; the
-  scheduler refuses an unsupported scheme at construction.
-- **Prefix sharing is non-consuming.** `PromptCache.take()` serves zero-copy
-  clones (ref-counted retain so a demoted donor never unmaps pages a clone
-  reads); the donor stays put; `put()` supersedes same-namespace prefix
-  ancestors when the new entry is trimmable. Kills the cannibalization flaw
-  (agent B consuming agent A's entry). v1 shares COMPUTE and durability;
-  concurrent rows still hold separate physical KV — one shared physical prefix
-  across rows is the block-KV frontier item, and whole-entry duplication in
-  the disk tier is the block-granularity revisit trigger.
-- **Prompt-boundary snapshot.** The prompt+generation entry is untrimmable
-  past a wrapped ring or under quantized KV, so every substantial request also
-  snapshots a strict prompt prefix (cap `len-1`), the mlx-lm
-  `insert_segments` invariant; a re-rendered next turn always matches.
-- **Never a JS callback as an mlx buffer destructor** — last-ref `Data` dtors
-  run on the Metal completion thread (native `dlsym(free)` dtor, process-pinned
-  mmaps).
-- **Isolation proxies the whole server**, not the gateway: grammar WASM,
-  vision arrays, and sampler closures don't serialize; the /v1 surface is the
-  IPC.
-
-## 9. The frontier program (order matters)
-
-Ranked by expected frontier shift (memory ↓ / intelligence ↑ first, then
-speed). Each item is a Lab program with its own doc; `decode-speed-program.md`
-ranks the speed levers and is the only doc that does.
-
-1. **Mixed-precision weights** — knapsack `--target-bpw` (ours; OptiQ-style
-   sensitivity port in `src/quantize/sensitivity.ts`) and rotation-folded
-   quantization (`docs/design/turboquant.md`, `--rotate-weights`). Gate:
-   perplexity + frozen 6-task eval at equal bpw.
-2. **TurboQuant KV** (`docs/design/turboquant.md`, landed v1) — orthogonal to
-   allocation: mixed precision is the ALLOCATION axis, TurboQuant the
-   QUANTIZER axis; they compose. Solo-only in v1.
+2. **Method selection now precedes placement.** `resolveExecution` selects
+   speculative or ordinary decoding from the actual request and backend
+   capabilities. Configuring a draft no longer forces supported ordinary work
+   into the serial executor. The final design has one shared executor; changing
+   method must not select a retired execution path. Concurrent speculation is
+   still an implementation gap, tracked in batching.md.
+
 3. **Speculative decoding depth** — DFlash/DSpark, native MTP, behind the
    `DraftSource` seam (`docs/design/speculative-decoding.md`).
 4. **Per-model graph work from the baseline** — unroll a model's flat DAG,
@@ -767,14 +659,12 @@ implementations throughout migration. New numerical work has separate paired
 validation, so structural changes cannot conceal a quality or performance
 tradeoff. §12.11 defines the shared frontier scorecard.
 
-Future design question (2026-09-05): consider a monorepo with the inference
-engine kept together as one package, and application modules such as memory
-and the web UI in separate packages. CLI/TUI and training boundaries can be
-considered in that discussion. This is an unplanned idea, not part of the
-current optimization campaign. Before choosing boundaries, map dependencies,
-native assets, builds, tests, and releases; preserve in-process execution and
-the single-binary distribution. Package boundaries should follow demonstrated
-ownership and dependency needs without adding abstractions to the decode path.
+Package organization is now a planned decision in the
+[public-readiness program](../../PLAN.md#public-readiness), separate from
+the optimization campaign. §12.15 records the initial boundary audit and
+remaining evidence needed before choosing a workspace layout. Preserve
+in-process execution and the single-binary distribution; package boundaries
+must follow ownership and dependency needs without adding decode overhead.
 
 ### 12.2 Dependency structure
 
@@ -833,8 +723,11 @@ Small stateless ports may be implemented by functions or object literals.
 | Interface | Owns | Must not own |
 |---|---|---|
 | `CompletionClient` | Semantic completion request, event stream, terminal result, cancellation | Tensor/state handles, model class selection |
-| `PromptPreparer` | Template/tokenizer policy, token IDs, validated media descriptors, stable-prefix boundary | GPU scheduling or request-global model mutation |
-| `ExecutionPlanner` | Effective immutable policy, implementation selection, capability negotiation, fallback reasons, plan identity | Starting generation or inventing new feature support |
+| `PromptPreparer` | Coordinate renderer, tokenizer and media preparation; produce token IDs and a stable-prefix boundary | Template syntax, tokenizer algorithms, GPU scheduling or request-global model mutation |
+| `RequestPolicyResolver` | Resolve defaults and explicit overrides once into immutable policy | Executing model work or resampling defaults inside a token loop |
+| `ChatTemplateRenderer` | Render messages, tools and reasoning markers; identify stable rendered boundaries | Tokenizer algorithms, scheduling or cache retention |
+| `Tokenizer` | Encode/decode, token metadata and incremental encoding | Chat-template policy, sampling or scheduling |
+| `ExecutionPlanner` | Effective immutable policy, implementation selection, capability negotiation, declared implementation choices, plan identity | Starting generation or inventing new feature support |
 | `InferenceEngine` | Session creation, runtime coordination, shutdown | Per-family forward math |
 | `GenerationSession` | Lifecycle, committed output, result, cancellation and close | Token-at-a-time assumption, KV offsets, concrete method branches |
 | `InferenceMethod` | AR/speculative/diffusion algorithm and private per-run state | HTTP framing or application sessions |
@@ -1010,14 +903,20 @@ inside forward/decode after binding. Experiment commands construct separate
 bindings with explicit snapshots. Diagnostics may observe a running request
 but cannot change its numeric policy halfway through execution.
 
-Admission is two-stage: validate metadata and reserve a conservative preparation
-budget before compiling grammar or running media towers; then refine the
-reservation using prepared sizes before execution. Count shared resident
-weights once, per-run state, draft resources, media/preparation transients,
-compiled state, pending output, and retained snapshot/write-behind buffers.
-Reservation ownership transfers across stages and releases on every failure.
-Keep prompt/context clamping semantics and aggregate KV limits consistent with
-current policy. A rejected prompt must not first allocate its full GPU payload.
+Configuration follows ownership: scheduling owns concurrency and queue policy;
+methods own sampling, speculation and stopping; backends own kernels and state
+layouts; storage owns cache budgets, tiers and persistence. The application
+composition root binds these interfaces. A single-request caller supplies the
+same method configuration as a batched caller. Consolidate duplicate controls
+as implementations migrate, retaining documented CLI aliases where needed.
+
+Default memory estimates are advisory by Josh's instruction. Attempt the
+requested work without predicted-memory refusals or automatic token clamping.
+Explicit user limits and actual layout requirements still apply. Account for
+shared weights once, live state, draft resources, preparation transients and
+retained/write-behind buffers. Release actual resources on failure; a
+conservative estimate does not establish that execution is impossible. This
+supersedes the earlier mandatory reservation-before-preparation design.
 
 ### 12.6 State, ownership, and persistence
 
@@ -1074,11 +973,35 @@ and define read-old/write-new compatibility or explicit invalidation per change.
 ### 12.7 Scheduling and inference methods
 
 Keep policy independent from mechanics. Scheduler chooses eligible work,
-fairness, queue bounds, active-row limits, and resource reservations. A method's
+fairness, queue bounds and active-row limits. A method's
 scheduling adapter creates execution groups and implements join/leave/advance
 over its own state. Native batch merge and filter stay inside the MLX adapter.
 Scheduler must not import KV classes or perform a per-layer tensor loop.
 
+The consolidation target is single-request execution as a batch of one through
+these same contracts. Methods publish reusable checkpoints through the shared
+cache interface at completed boundaries. The cache owns prefix selection,
+retention and RAM/SSD placement; scheduling does not choose a storage tier, and
+decode does not inspect the request queue. Model/method codecs describe the
+state needed to restore a prefix, including draft state when applicable.
+Prefill and generated tokens populate the same cache. R17 in the decode-speed
+program owns the current implementation and acceptance work.
+
+Full batch feature parity is required, including speculation. The current
+serial exclusions are migration work, not the target architecture. Keep an
+explicit feature matrix in the batching design and retire duplicate request
+lifecycles only after the batched implementation covers every serial feature
+and equals or beats serial performance in matched measurements. B=1 latency
+and feature-specific costs must pass independently of aggregate throughput.
+The optimized kernels already built remain behind the backend interface;
+scheduling policy does not choose or reimplement their arithmetic. Continue
+building faster specializations, fusing operations, removing work and avoiding
+materialization. Interfaces must permit complete fused work units rather than
+force one call or intermediate tensor per logical operation. Measure kernel
+changes separately from composition refactors so their effects are attributable. Interface
+extraction must not force tensor copies, materialization, synchronization or
+additional dispatches; compare B=1 latency and B>1 throughput at fixed settings.
+
 An advancement is a bounded safe unit, not necessarily a token: one prefill
 chunk, a pipelined decode step, one verify round, or one denoising iteration.
 Budgets describe admitted resources and work hints separately. A synchronous
@@ -1876,3 +1799,189 @@ existing placement and numerical defaults. Their independent experiment programs
 still own promotion/deletion decisions. Quiet performance, exact target quants,
 unavailable oracle/drafter fixtures and the second machine remain acceptance
 work, not missing interface code. No frontier advance is claimed by this refactor.
+
+### 12.14 Request flow and interface ownership
+
+This is the target responsibility map for Phase 6/18. A request carries one
+resolved policy: measured defaults plus explicit user overrides. Each component
+consumes its own part of that policy. The table below distinguishes existing
+contracts from implementation and migration work still in progress.
+
+```mermaid
+flowchart TB
+    client["Client: chat, Pi, API or library"]
+    server["Server adapter<br/>receive requests and send responses"]
+    policy["Request policy interface<br/>resolve defaults and user overrides once"]
+    template["Chat template interface<br/>messages and tools → rendered prompt"]
+    tokenizer["Tokenizer interface<br/>rendered prompt → token IDs"]
+    session["InferenceEngine / GenerationSession<br/>own request lifecycle and cancellation"]
+    scheduler["Scheduler / ExecutionGroup<br/>choose when ready work runs"]
+    method["InferenceMethod<br/>ordinary decode or MTP proposal / verification"]
+    graph["Model graph interface<br/>prefill, forward passes and specialized kernels"]
+    sampler["SamplingSession<br/>apply settings and grammar; select tokens"]
+    output["Response interface<br/>decode committed tokens; emit text and tool events"]
+    state["Active state interface<br/>target KV, recurrent state, draft state and positions"]
+    cache["PrefixCache interface<br/>restore and retain aligned token / state checkpoints"]
+    ram["RAM cache<br/>retain reusable checkpoints"]
+    persistence["Persistence interface<br/>queue background writes and restore from SSD"]
+
+    client --> server --> policy --> template --> tokenizer --> session
+    session -->|ready work| scheduler
+    scheduler -->|admit and advance a batch of 1 or more| method
+    method -->|inputs and state| graph
+    graph -->|device logits| sampler
+    sampler -->|selected tokens or proposals| method
+    method -->|committed tokens only| output
+    output -->|response events| server
+    method <-->|borrow, update, snapshot, rollback| state
+    session <-->|restore / publish prefill checkpoint| cache
+    method -->|publish generated checkpoint| cache
+    cache <-->|fast reuse| ram
+    ram <-->|spill and restore when needed| persistence
+```
+
+The scheduler chooses **when** work runs and which requests are active. The
+inference method chooses **what numerical work** advances them. A batch of one
+uses the same method and state contracts as a larger batch. The final design
+has one shared executor and no separate serial executor to fall back to.
+
+The model graph returns logits to the sampler. The sampler applies temperature,
+penalties, grammar, sampling filters and RNG policy, and returns selected tokens
+with requested logprobs. MTP keeps proposals private until verification commits
+them. The response formatter sees committed tokens only.
+
+Live KV is working state for the current computation. A reusable checkpoint
+contains token IDs plus the corresponding target, recurrent and optional draft
+state. Both prefill and generated output must feed the same cache interface.
+Only an exact, rendered-stable token boundary can be published for state that
+cannot be trimmed. RAM retention, eviction, asynchronous SSD writes and restart
+restore belong behind that interface. Scheduling does not wait for an ordinary
+SSD spill; an explicit durability flush can wait for persistence.
+
+| Piece | Current code or contract | Remaining work |
+|---|---|---|
+| Server / response adapters | `CompletionEngine`, `CompletionSink`, `GenerationOutput` | Preserve the same semantics through the final shared-executor cutover. |
+| Request policy | `RequestPlan`, request preparation and execution planning | Consolidate configuration into one resolved policy; choose measured defaults for typical multi-turn use and retain user overrides. |
+| Chat template | `ChatTemplate.render` | Expose the renderer as its own narrow port instead of requiring a concrete serving context. Record rendered-stable boundaries for generated checkpoint alignment. |
+| Tokenizer | `LoadedTokenizer` | Reuse the same encode/decode contract across preparation, caching and output. Keep incremental encoding inside this component. |
+| Session / scheduling | `InferenceEngine`, `GenerationSession`, `ExecutionGroup`, `SchedulingClock` | Close all feature and performance gaps before deleting the remaining serial executor. |
+| Inference method | `InferenceMethod`; shared ordinary and speculative methods | Complete every shared-method composition and acceptance gate. Draft depth belongs here, independently of queue policy. |
+| Numerical graph / kernels | `AutoregressiveGraph`, `GraphFactory`, backend bindings | Continue hardware-specific fusion and kernel optimization behind these contracts. |
+| Sampler | `SamplingSession` | Retain one set of sampling/history/logprob semantics for ordinary and speculative execution. |
+| Active state / codec | `Cache`, `KvAttentionState` / `KvAttentionView`, row layouts, `SpeculativeTransaction`, checkpoint attachments | Appending and attending are separate operations. One captured view can serve multiple model layers while retaining pre-write positions and tensor ownership. Delayed affine/TurboQuant and rotating layouts are integrated; qualify additional model/layout combinations separately. |
+| Reusable cache | `PrefixCache.take/put`, shared `PromptCache` | Prefill and processed generated output populate the same immutable RAM/SSD cache. Session lookup and rendered-token alignment are integrated. |
+| Persistence | `ColdTier`, `SpillSink`, `SsdCacheStore`, `SsdDurabilityCoordinator` | Queued persistence, native/restart checks and full Kanban durability acceptance are complete. RAM/SSD retention remains behind cache operations. |
+
+An interface boundary does not require copying tensors, downloading logits,
+or adding an RPC or synchronization per token. Device handles can stay lazy;
+the backend may fuse compatible graph and sampling operations while preserving
+policy, ownership and observable results. Each component can improve its own
+implementation without adding model, cache or sampling decisions to scheduling.
+
+Contract sources: [generation](../../src/contracts/generation.ts),
+[scheduling](../../src/contracts/scheduling.ts),
+[sampling](../../src/contracts/sampling.ts),
+[prefix cache](../../src/contracts/prefix-cache.ts),
+[graph](../../src/inference/graph.ts),
+[tokenizer](../../src/tokenizer.ts),
+[cache tiers](../../src/prompt-cache.ts).
+Performance evidence and default selection live in
+[benchmarks.md](../reference/benchmarks.md); open acceptance remains in Phase 6/18.
+
+#### Kernel configuration ownership
+
+Shared and serial bindings execute within one immutable `RuntimeConfig`.
+Numerical helpers read that scope; they do not read process environment in
+forward loops. The serial speculative trace switch is captured once per run.
+Trellis mode resolves through the same configuration port at weight creation.
+
+TurboQuant caches capture their codec's fused-decode choice at construction.
+RAM clones, full-prefill row restoration, batch layout factories, extraction
+and delayed conversion preserve that choice. KV maintenance binds its runtime
+before conversion, including conversion performed after a host setting changes.
+SSD formats contain numerical state, not a frozen process configuration; their
+receiving cache factory selects the destination binding's policy.
+
+The configuration tests change host settings around scoped construction,
+copy/extraction and delayed conversion. Existing codec/donor tests now select
+both fused settings through `configureRuntime`, so they exercise the same
+configuration contract as serving. Paged request policies also bind the numerical
+namespace and construction policy before deferred lookup or allocation, including
+when a gateway outlives later host configuration changes. The serving reference
+maps flags to their owning interfaces and records supported combinations. The
+legacy internal `flagOn` wrapper is removed; CLI controls remain compatible.
+The new paged-policy tests first reproduce both deferred-setting failures, then
+pass with the captured policy. Paged operation/layout checks pass on both Macs;
+the full local model-free suite and all typechecks pass.
+
+### 12.15 Package boundaries before community outreach
+
+Initial audit, 2026-09-11. The public-readiness program owns the packaging
+work; the existing interface and optimization milestones retain their scope.
+The three published entry points need different treatment:
+
+| Entry or layer | Verified dependency boundary | Packaging consequence |
+|---|---|---|
+| `mlx-bun/client`, `src/client.ts` | Imports only completion/host contracts; uses standard Request, Response, fetch and cancellation APIs | A candidate for independent client installation. The current npm package still has Darwin/arm64 metadata and application dependencies; a portable source file alone does not change that install contract. |
+| `src/contracts/`, `src/engine/`, `src/inference/` | Separate TypeScript project excludes Bun and DOM types; the AST boundary test limits imports by layer | Keep these interfaces and policies together inside the engine. They do not each need a published package. |
+| `mlx-bun/engine`, `src/library.ts` | CPU-safe entry exports engine/client APIs, but `initializeMlx()` dynamically imports the broad root API; `openIsolatedHost()` imports serving isolation and locates the CLI | Preserve this compatibility entry. Separate native engine initialization from application/worker assembly before treating it as a standalone engine package. |
+| `mlx-bun`, `src/index.ts` | Re-exports concrete models, generation, downloads, registry, fit and server construction | Already a published compatibility API. A workspace move needs forwarding exports, not a silent reduction of available functionality. |
+| Browser application | Own TypeScript project; boundary test allows web code and contracts | A candidate private application workspace. Its assets still participate in the CLI/native bundle. |
+| Website | Already has its own package manifest and build; generates reference pages from canonical docs | Keep one source for reference facts. A root workspace declaration alone would not improve this ownership. |
+
+The source audit followed static imports, re-exports, type imports and literal
+`import()`/`require()` references using TypeScript module resolution. The
+client closure contained four source files and no external module imports.
+The high-level engine entry reached the broad root API, including Pi and web
+assets. This is a dependency inventory, not a claim that importing the
+CPU-safe entry eagerly loads those modules. Non-code assets and executable
+paths also need a packaging check; import analysis cannot establish that they
+ship correctly.
+
+Recommended direction for the remaining audit: one engine package containing
+models, native bindings, cache codecs, inference methods and their interfaces;
+application assembly outside it; a small client package if independent
+installation is useful. Keep specialized kernels and tensor ownership within
+that engine. Memory, Pi and training require their own dependency and asset
+review before deciding whether they become private workspaces. A general MLX
+array binding is a separate API commitment and is not implied by this split.
+
+Before choosing and implementing the layout, map the engine-to-application
+edges through server construction, native bootstrap, worker commands and
+bundled assets. Verify the package tarball and a consumer installed outside
+the repository. Public import compatibility and the single-binary release are
+acceptance requirements. The initial audit does not yet close the package
+organization milestone.
+
+The initial `bun pm pack --dry-run --ignore-scripts` also confirms that the
+package includes the whole `docs/` tree, including archived investigations
+and active plans. Review that allowlist during packaging work; these are not
+runtime dependencies. This dry run inventories the candidate but does not
+replace installation and asset checks against an actual tarball.
+
+
+### 12.15 Prefill workspace policy
+
+`PrefillPolicy` selects a request's chunk size independently of scheduling,
+cache placement and sampling. The MLX backend captures model geometry and
+explicit configuration when it binds the policy. Ordinary generation,
+shared rows and speculative preparation consume the selected size through
+their existing prefill interfaces. A caller's explicit request value wins.
+The automatic recurrent-attention policy bounds the temporary materialized
+SDPA score workspace; it does not inspect live free memory or refuse work.
+Default values and precedence live in [server configuration](../reference/server-config.md).
+
+### 12.16 Committed append cache formats
+
+`MlxTokenAppend` declares supported affine precisions and TurboQuant K/V
+formats. Qualified Qwen27B bindings expose KV4/KV8 and K8V3.
+Request preparation supplies template-derived candidates; placement selects
+an eligible method without inspecting numerical cache formats. The ordinary
+method consumes the binding's declaration once and uses its committed-append
+operation. Qwen shares projection work over known tokens while retaining each
+query's one-token affine attention calculation and causal KV extent.
+`KvMaintenance.maxAppendTokens` bounds a committed chunk at a pending
+precision transition; the method runs maintenance before continuing.
+Verification remains a separate method capability: enabling committed
+quantized appends does not enable quantized echo verification. No default fill setting or
+shared-group support is inferred from that declaration.

package/docs/planning/PRODUCT_ROADMAP.md

@@ -6,6 +6,22 @@ the evolving UI/UX/DX north star.
 
 ## Product north star
 
+Josh's guiding principle for the entire project is "The absolute BEST local
+AI experience on a mac". This governs installation, application development,
+everyday use and inference performance.
+
+Offer multiple simple installation methods without asking users to manage
+a Python virtual environment. Let developers import local AI into their
+applications without managing a separate inference sidecar. Make JavaScript
+and TypeScript first-class ways to build with local AI, without requiring
+Python. Make more capable models practical on the same Mac and reduce the
+time users wait for successful results.
+
+Every workstream must serve that experience. Evaluate engineering choices
+by the setup, integration and operating burden they remove, as well as the
+quality, reliability, speed and hardware efficiency they deliver. Research
+and publication can support the project; user benefit determines priority.
+
 Make the answer to "what is the best way to use local AI on my Mac?"
 boringly simple:
 
@@ -49,12 +65,11 @@ disclosure: appliance defaults at the front, sharp instruments underneath.
 
 ## The three modes (the organizing design law)
 
-mlx-bun has not published anything: the entire CLI, command, and flag
-surface is still ours to invent, and we are not married to any flag
-name. Where a de-facto standard name exists (mlx-lm's), we adopt it
-rather than reinvent. The surface is designed once, to serve three
-audiences at three depths — progressive disclosure over ONE set of
-verbs and one flag vocabulary, not three separate CLIs:
+mlx-bun is published. Existing commands, flags, and package exports are
+compatibility commitments; changes need documented migration behavior.
+Where an mlx-lm flag has matching semantics, prefer its established name.
+The CLI serves three audiences through one set of verbs, with more explicit
+configuration available as users need it:
 
 1. **Automatic mode** — first-time local-AI users. `mlx-bun` (and bare
    verbs) with zero flags. Smart, sane defaults from the device-profile