mlx-bun 0.4.0 → 0.5.0

package/docs/design/speculative-decoding.md

@@ -1108,16 +1108,13 @@ The bar for this tier is NOT token identity (sampling never guaranteed that):
 it is a paired A/B on task success and wall clock over mocked-replay agent
 sessions. Default off until that lands.
 
-**Composition.** Serial lane only, and it never FORCES a request serial — a
-batch-placed request simply does not fill, because `generate()` is the only
-site that reads `options.fill`. Refused for: a compiled grammar (forced tokens
-are its job), `logprobs`/`top_logprobs` (injected tokens have no distribution
-row — the same rule as `shouldUseGrammarJump`), a user-fixed `seed`
-(reproducibility: the step index would skip injected positions), media
-prompts, a mounted draft model, quantized/TurboQuant KV (post-conversion
-multi-token append is L-generic but unvalidated), and sliding-window models
-(RotatingKVCache multi-token append is O(window) via `#updateConcat` — one
-warning, then no fill).
+**Composition.** Strict fill has serial and shared methods. Shared fill keeps
+request-local sampler history and absolute seeded positions while using the
+same preparation and encoded-row interfaces as other methods. Echo verification
+remains an explicit serial operation. Grammar, probability metadata, media,
+mounted drafting, paging and interrupted-generation checkpoints remain separate
+compositions. Quantized committed append is supported by the qualified model
+bindings described below; quantized echo verification remains inactive.
 
 **Mismatch policy.** If `parseGeneratedToolCalls` rejects the emitted markup
 on a request whose rows fired, `usage.fill.parseFallback` increments and
@@ -1415,22 +1412,30 @@ decode. The machine still fails the quiet preflight, so these are diagnostic
 results. Both external-drive failures remain recorded separately. Evidence:
 `fill-append-http-internal-repeat.json` and `fill-append-http-internal-review.json`.
 
-Quantized-KV composition remains disabled. A direct R6 append screen with
-TurboQuant k8v3 preserves hidden rows, logits, live cache state and four
-continuations at prefixes 128 and 1021, but exposes retained temporary state
-views between chunks. `appendFillHidden` now uses the existing
-`leaseCacheStates` ownership interface, as prefill does. Explicit release
-removes the extra 21,528,576 and 131,153,920 active bytes in those two checks;
-both ordinary and fused TurboQuant decoding then have identical post-arm
-allocation and numerical results. This fixes the research composition's
-view lifetime; the ordinary served path already excludes this combination.
-Seven targeted ownership tests and all three typechecks pass. The affine
-KV4 comparison fails both contexts, first changing recurrent state at layer
-4 after the first attention layer. Keep its failure and the composition
-guard. Other quant/cache modes, longer contexts and actual serving
-performance remain unqualified. Evidence:
-`fill-append-quant-cache-review.json` and
-`fill-append-state-ownership-native-test.log`.
+The original quantized-KV append screen passed TurboQuant K8V3 after fixing
+retained temporary state views through `leaseCacheStates`, but affine KV4
+changed recurrent state after the first full-attention layer. That failure
+is now resolved: multi-token affine attention selected different arithmetic
+from one-token decode, both in tiled and ordinary unfused forms. The model's
+committed-append path now computes each query against its own causal KV
+prefix using the one-token attention operation, while retaining shared
+projection work. No sampled decode or ordinary prefill arithmetic changes.
+
+On M4 Pro, packed and RTN4 Qwen27B both preserve complete hidden/logit/cache
+bytes and four subsequent continuations for KV4/KV8 at prefixes 128 and
+1021, including the attention boundary. The real KV4 generation loop also
+preserves emitted IDs, final state, subsequent logits and active allocation,
+with nonzero fill and zero verification. The append binding declares affine
+format support; HTTP preparation and placement leave that decision to the
+method. K8V3 TurboQuant now uses the same declaration and delayed-conversion
+boundary interface. Both Qwen artifacts pass real-generation state/continuation
+and six-pair HTTP comparisons; quantized echo verification remains separate.
+Full numerical and HTTP measurements live in benchmarks.md. Tests:
+`tests/unit/quantized-append-attention.test.ts`,
+`tests/parity/qwen-affine-append.test.ts`, and
+`MLX_BUN_TEST_FILL_KV_BITS=4` with the existing strict-fill parity test.
+Earlier failure/lease evidence remains `fill-append-quant-cache-review.json`
+and `fill-append-state-ownership-native-test.log`.
 
 The literal/reasoning checks deliberately decline ambiguous text, including some
 otherwise valid prose with unmatched quote characters. This is bounded
@@ -1600,6 +1605,140 @@ grammar/KV/batch/logprobs/seed exclusions individually after their own gates.
 The full experimental matrix and completion rule are in
 [decode-speed-program.md](decode-speed-program.md#7-qwen38-27b-research-program).
 
+#### Shared strict-fill method
+
+`backends/mlx/fill-group.ts` composes `MlxPrefillCohort`, `MlxStateRows`, the
+model graph, the sampler and the common prompt-cache interface. The scheduler
+continues to select work through `MlxGroupedMethod`. It does not compile tool
+schemas or choose filled tokens. A request owns its `FillSession`, partial
+proposal and sampler history.
+
+Ordinary samples retain build-next-before-read overlap. The method commits a
+token to sampler history only when that token becomes model input. An in-flight
+sample replaced by a strict continuation never enters history. Known positions
+skip sampling; if every next position is known, the method also omits the
+vocabulary projection. The model binding supplies the maximum committed chunk
+size for the current row geometry. When every row has a known continuation and
+the model permits wider chunks, the method uses the same append operation as
+explicit serial execution. Qualified Qwen B1 uses its specialized multi-position
+operation; wider rows retain one-position arithmetic and pipeline overlap.
+Performance parity remains a measured acceptance gate.
+
+Before a row joins, the method flushes pending outputs. Retirement and
+cancellation settle exactly the emitted prefix of a proposal once. Cache
+publication uses the tokens actually consumed by the model, which can lead
+visible output by one pipelined position or the remaining committed span.
+The cache owns RAM retention and
+queued SSD persistence. No storage work is awaited by the method.
+
+The row-aware `appendHiddenRows` interface separately accepts borrowed `[B,L]`
+inputs and asks the model for its maximum chunk length at the actual B. The
+qualified Qwen binding retains its specialized B1 append and declares one
+position for wider rows. Full hidden outputs, live cache bytes and subsequent
+logits match ordinary same-B forwards after retirement on M1 MiniCPM/Qwen0.8B
+and M4 packed Qwen27B with bf16, KV4 and k8v3.
+
+`tests/unit/fill-group.test.ts` exercises actual shared execution, including
+mixed sampled/filled rows, head omission, stream stops, length limits,
+consumer failures, cancellation and late joins. `tests/parity/shared-fill.test.ts`
+compares ordinary, empty-fill and saved-continuation arms at matched B1/B2/B4
+with seeded sampling, presence/repetition penalties and bf16/KV4/k8v3. All
+nine cases pass on M1 MiniCPM, Qwen0.8B and Gemma4-e4b and M4 packed Qwen27B.
+Gemma's sliding layers remain bf16 in the TurboQuant arm.
+
+The M4 delayed-conversion gate caught two composition defects. Shared row
+layouts did not expose their pending precision boundary to committed append;
+the affine attention view also bypassed the model's single-position reduction
+policy. The cache now publishes `maxAppendTokens()` from logical row offsets,
+including padding and retirement, and the attention view carries the qualified
+committed-position policy to its codec. Both delayed KV4 and k8v3 seeded cases
+pass after splitting the five-token span into two and three positions at the
+boundary. Separate B1 full-hidden/cache-byte/continuation comparisons pass for
+bf16, KV4, k8v3 and both delayed formats. Earlier failed logs are retained.
+
+The first unequal-prompt comparison did not fix the admission geometry and
+produced different later tokens; preserve it as an unmatched experiment.
+A later fixture reused a short suffix which recurred during stochastic
+sampling and wrongly supplied the same continuation twice. The numerical
+gate now supplies a saved span at one exact position. Parser/source validity
+has its own tests; numerical replay does not prove arbitrary suffix asserts.
+The failed logs and corrected runs remain in `reports/prefill-observation/`.
+Serving binding tests also cover seeded placement and `usage.fill` propagation.
+The six-arm M4 HTTP comparison preserves all 108 responses and all paired token/cache counts, with six durable SSD restarts. Sequential shared strict-fill performance matches serial within 0.13%; the fill-off comparison and concurrent latency tradeoffs are recorded in benchmarks.md. Model-declared codec eligibility is resolved once at binding: unsupported multi-position formats still use shared one-position fill. A separate k4v3 seeded gate proves that execution with no specialized append.
+
+#### Shared echo verification
+
+The fill method accepts both asserted and verified proposals. A pending
+sample checks the first echo token without replacing the sampler's choice.
+When a live row has a verified echo continuation, the method supplies
+its proposal to `advanceSpeculativeOutputs`, the same grouped target
+verification and output-aligned rollback operation used by speculative
+providers. Ordinary rows join with empty proposals and retain only their
+consumed inputs. Asserted spans keep their append policy. Proposals discovered
+while publishing pending output stay within the scheduler's declared token
+work; any remainder belongs to the request's next iteration.
+No scheduler or storage branch chooses echo tokens.
+
+Each request retains its own seed and processor history. The sampler adapter
+commits a token when it becomes model input; rejected proposals never enter
+that history. Eligible plain-greedy rows use the sampler's existing independent
+verification kernel. Output delivery determines how much state survives a consumer
+stop, then the common cache receives the corresponding tokens and row state.
+The cache retains RAM donors and owns SSD persistence. Checkpoint telemetry
+includes the shared cohort's begin and resolve time for each participating
+request. Fill remains opt-in. Shared paged execution is unsupported. Qwen MTP can
+consume verified echo continuations through the provider operation below.
+
+`shared-echo.test.ts` audits the real sampler against an independent sampler
+on the exact same score tensor and selected-token history, across B1/B2/B4,
+bf16, KV4, k8v3 and delayed quantization. It covers accepted, first-rejected,
+partially rejected, mixed and staggered proposals plus a consumer stopping inside
+the span. Published cache keys must match emitted history and every layer offset.
+B2 retained state also survives SSD store/reopen with identical state and two
+subsequent logit vectors against its RAM clone. These checks pass on M1
+Qwen0.8B/Gemma-e4b and M4 packed Qwen27B. HTTP tests cover actual copied tool arguments,
+seeded concurrent requests, follow-up reuse and durable SSD restart.
+
+An early experiment compared multi-position verification against one-position
+ordinary decoding and found changed sampled tokens. That comparison changes
+numerical geometry and is preserved as an unmatched experiment, not an exact
+oracle. The original strict-fill exact replay gate remains unchanged. Echo
+acceptance preserves the sampler's decision for the verified logits; it does
+not promise identical text to a different forward shape. Request timing and
+paired-response results are recorded in benchmarks.md.
+
+#### Echo with a mounted MTP provider
+
+`DraftRowGroup.consume` consumes externally verified target tokens and hidden
+context, retaining the specified prefix length for each row. Provider
+capabilities advertise this operation before execution. The Qwen MTP provider
+uses its existing true-hidden prefill graph and per-row rollback; it neither
+samples draft tokens nor projects its vocabulary during this operation.
+
+The speculative method first compares ordinary MTP output with a pending copy
+proposal. A first-token mismatch uses no additional target forward. Once output
+establishes a matching prefix, the method verifies the remaining copied span.
+Other rows supply empty proposals. The existing verifier and
+sampler select output, then output delivery determines how many inputs survive.
+The provider consumes that same prefix before ordinary MTP resumes. Target and
+companion state are published together through the unchanged cache interface.
+No scheduling or storage implementation selects between echo and MTP.
+
+`FillSession.observe` records sampled output independently of proposal timing.
+The method asks for verified proposals at committed output boundaries. Under
+MTP, asserted template spans remain ordinary sampled output; this combination
+adds echo without changing the strict append algorithm. Grammar, probability
+metadata, media, paging and providers without external-token consumption remain
+outside this combination's serving coverage.
+
+`usage.fill` counts verified echo proposals. Learned-provider drafted/accepted
+counts, rounds and acceptance histograms exclude echo. Target calls and tokens
+per forward cover both kinds of round. A request can legitimately use no
+learned drafts if echo supplies every continuation. Both-machine companion
+oracle checks cover external prefixes and subsequent full decoder output;
+shared tests cover sampling, mixed rows, rejection, stop and paired SSD state.
+
+
 ### 7.5 Logical jump-ahead prefill
 
 Proposal by Josh Rossi, recorded 2026-09-08. Research proposal, not an

package/docs/design/turboquant.md

@@ -357,7 +357,7 @@ Gemma-4-e4b's D512 deferred consumer also passes: 20 forward/state cases, four
 generations and 1,220 matching cache reads, one specialization and 1,057,864
 final active bytes per arm. Evidence: tq-kv-gemma-e4b-full{,-review}.json.
 
-The joint decoder is now integrated as an experimental opt-in. The environment
+The joint decoder was initially integrated as an experimental opt-in. The environment
 setting and supported inputs are documented in server-config.md. A shared
 `src/mlx/turboquant-kv-decode.ts` owns only the packed Metal operation; the existing
 codec owns inverse rotation and the cache captures operation selection once.
@@ -517,8 +517,8 @@ longer prompt. Sampled RSS establishes no reduction. The server-exit observer
 still sees resident model arrays, so its active capacity is not a disposal-to-
 zero test. Native operation and full-model ownership gates remain separate.
 These are M4 Pro 24 GB diagnostics with k8v3, greedy seed 42 and 128 generated
-tokens. TurboQuant KV requires serial placement in the existing execution
-planner. Evidence: mlx-upgrade-0.32.2/deferred-kv-http{,-review}.json. This closes
+tokens. Those measurements used serial placement; shared integration is covered below.
+Evidence: mlx-upgrade-0.32.2/deferred-kv-http{,-review}.json. This closes
 the deferred-model serving comparison; broader pressure and quiet gates remain.
 
 Direct packed-key attention is closed as an additional speed candidate. A
@@ -612,9 +612,9 @@ snapshots with no missing, pending, dropped or failed writes. The constrained
 arms cause six eviction spills; the async observer confirms 3,264 submissions.
 These sequential shared-server pressure checks cause substantial swapping;
 B4 correctness is a separate native gate, and no speed or memory-reduction
-claim follows. M4 pressure and strict paired performance acceptance remain open.
-The shared integration is now adopted in the unreleased working tree; the
-released v0.3.0 binary retains its earlier placement limits.
+claim follows. The subsequent M4 combined pressure gate also passes with both asynchronous and blocking submissions exercised; full usage, responses and final persistence match. Current measurements and provenance live in benchmarks.md. This closes the named pressure gate; performance and default selection remain separate.
+Shared TurboQuant integration ships in v0.4.0; v0.3.0 retained the earlier
+placement limits.
 Evidence: `reports/qwen38-closeout/composition-baseline/turboquant-composition-manifest.json`.
 Integration evidence: `reports/qwen38-closeout/composition-baseline/turboquant-serving/`.
 Broader family checks pass MiniCPM seeded/logprob serving but reproduce a
@@ -626,6 +626,19 @@ Gemma seeded sampling, affine KV4 controls and Qwen MTP3 B4 serving.
 MiniCPM prefix checks allow the existing policy to supersede trimmable donors
 while proving byte identity and continuation through retirement.
 
+### Fused codec default selection
+
+The unreleased decoder selects packed K/V fusion by default when a caller
+chooses TurboQuant. `turboQuantFusedDecode` resolves the runtime policy once;
+ordinary storage, shared rows and delayed conversion capture the same value.
+Copies and restored rows retain the codec selection through their existing
+interfaces. `MLX_BUN_TURBOQUANT_FUSED_DECODE=0` selects the original operations.
+The numerical kernels and eligibility rules are unchanged from the accepted
+opt-in. TurboQuant quantization itself remains an explicit cache-format choice.
+The current shared-serving comparison and combined pressure measurements are
+recorded in benchmarks.md; the earlier six-pair timing and native/state gates
+above remain the evidence for the fused implementation.
+
 ## KV-leg limits and non-goals (recorded so they don't creep)
 
 - Full-attention layers only; rotating/sliding-window layers stay bf16 (warn
@@ -950,7 +963,7 @@ since e·Hinv[k,k] = w−q) and `err[..., k:k+1]` indexes a group-local buffer
 with the GLOBAL k (mlx out-of-range slice assignment silently no-ops →
 cross-group propagation lost after block 0). Fixing only err
 DOUBLE-compensates [j, k+j) and REGRESSES below RTN (measured: 7.22/7.50);
-fixing both = paper GPTQ. Worth an upstream report. Fork:
+fixing both = paper GPTQ. [mlx-lm #1878](https://github.com/ml-explore/mlx-lm/issues/1878) contains the CPU-only reproduction, observed results against mlx-lm 0.31.3 and upstream `e5962529`, and proposed two-line correction; [mlx-lm PR #1880](https://github.com/ml-explore/mlx-lm/pull/1880) carries the fix plus `tests/test_gptq.py` (column-by-column reference, four groups fail on `main`, single-group control passes on both). Fork:
 `scripts/turboquant/tq-gptq.py` (also restricts GPTQ+fallback to language
 modules — vision H is a zero scalar and stays bf16).
 

package/docs/design/unified-engine-frontier-plan.md

@@ -215,6 +215,17 @@ fusedSdpaRuntimeOk(q, mask)` in `src/model/generated/gemma4-*.ts`) that are
 per-generation-decidable and should be hoisted out of the token path; compiled
 decode already achieves the debranched line dynamically (trace once, replay).
 
+The shared greedy operation uses native logsumexp followed by tiled normalized
+score/index reductions in `src/mlx/normalized-argmax.ts`. It preserves rounding
+and lowest-ID ties without materializing the normalized vocabulary array.
+The step sampler applies processors and grammar first, and selects this operation
+only for built-in greedy requests that do not require metadata. Independent
+verification calls the same operation. Fixed-shape compilation caches each
+geometry; enclosing shapeless graphs retain native operations. No scheduler,
+model-family branch or separate sampling setting selects this optimization.
+Actual-model and operation comparisons, plus both-machine serving measurements,
+are recorded in benchmarks.md under fused normalized greedy selection.
+
 ### Layer 0 — the SSD spill substrate
 
 It is almost always faster to cache to SSD and mmap it back than to
@@ -648,14 +659,12 @@ implementations throughout migration. New numerical work has separate paired
 validation, so structural changes cannot conceal a quality or performance
 tradeoff. §12.11 defines the shared frontier scorecard.
 
-Future design question (2026-09-05): consider a monorepo with the inference
-engine kept together as one package, and application modules such as memory
-and the web UI in separate packages. CLI/TUI and training boundaries can be
-considered in that discussion. This is an unplanned idea, not part of the
-current optimization campaign. Before choosing boundaries, map dependencies,
-native assets, builds, tests, and releases; preserve in-process execution and
-the single-binary distribution. Package boundaries should follow demonstrated
-ownership and dependency needs without adding abstractions to the decode path.
+Package organization is now a planned decision in the
+[public-readiness program](../../PLAN.md#public-readiness), separate from
+the optimization campaign. §12.15 records the initial boundary audit and
+remaining evidence needed before choosing a workspace layout. Preserve
+in-process execution and the single-binary distribution; package boundaries
+must follow ownership and dependency needs without adding decode overhead.
 
 ### 12.2 Dependency structure
 
@@ -1826,7 +1835,7 @@ flowchart TB
     output -->|response events| server
     method <-->|borrow, update, snapshot, rollback| state
     session <-->|restore / publish prefill checkpoint| cache
-    method -.->|publish generated checkpoint: still open| cache
+    method -->|publish generated checkpoint| cache
     cache <-->|fast reuse| ram
     ram <-->|spill and restore when needed| persistence
 ```
@@ -1856,12 +1865,12 @@ SSD spill; an explicit durability flush can wait for persistence.
 | Chat template | `ChatTemplate.render` | Expose the renderer as its own narrow port instead of requiring a concrete serving context. Record rendered-stable boundaries for generated checkpoint alignment. |
 | Tokenizer | `LoadedTokenizer` | Reuse the same encode/decode contract across preparation, caching and output. Keep incremental encoding inside this component. |
 | Session / scheduling | `InferenceEngine`, `GenerationSession`, `ExecutionGroup`, `SchedulingClock` | Close all feature and performance gaps before deleting the remaining serial executor. |
-| Inference method | `InferenceMethod`; shared MTP method in the isolated candidate | Complete every shared-method composition and acceptance gate. Draft depth belongs here, independently of queue policy. |
+| Inference method | `InferenceMethod`; shared ordinary and speculative methods | Complete every shared-method composition and acceptance gate. Draft depth belongs here, independently of queue policy. |
 | Numerical graph / kernels | `AutoregressiveGraph`, `GraphFactory`, backend bindings | Continue hardware-specific fusion and kernel optimization behind these contracts. |
 | Sampler | `SamplingSession` | Retain one set of sampling/history/logprob semantics for ordinary and speculative execution. |
-| Active state / codec | `Cache`, `KvAttentionState` / `KvAttentionView`, row layouts, `SpeculativeTransaction`, checkpoint attachments | Appending and attending are separate operations. One captured view can serve multiple model layers while retaining pre-write positions and tensor ownership. Delayed rotating affine and broader model/layout composition remain open. |
-| Reusable cache | `PrefixCache.take/put`, shared `PromptCache` | Generated-output checkpoints and next-request token alignment remain open. Prefill reuse is already implemented. |
-| Persistence | `ColdTier`, `SpillSink`, `SsdCacheStore`, `SsdDurabilityCoordinator` | Complete combined M4 pressure/restart acceptance. RAM and SSD remain interchangeable retention tiers behind cache operations. |
+| Active state / codec | `Cache`, `KvAttentionState` / `KvAttentionView`, row layouts, `SpeculativeTransaction`, checkpoint attachments | Appending and attending are separate operations. One captured view can serve multiple model layers while retaining pre-write positions and tensor ownership. Delayed affine/TurboQuant and rotating layouts are integrated; qualify additional model/layout combinations separately. |
+| Reusable cache | `PrefixCache.take/put`, shared `PromptCache` | Prefill and processed generated output populate the same immutable RAM/SSD cache. Session lookup and rendered-token alignment are integrated. |
+| Persistence | `ColdTier`, `SpillSink`, `SsdCacheStore`, `SsdDurabilityCoordinator` | Queued persistence, native/restart checks and full Kanban durability acceptance are complete. RAM/SSD retention remains behind cache operations. |
 
 An interface boundary does not require copying tensors, downloading logits,
 or adding an RPC or synchronization per token. Device handles can stay lazy;
@@ -1878,3 +1887,101 @@ Contract sources: [generation](../../src/contracts/generation.ts),
 [cache tiers](../../src/prompt-cache.ts).
 Performance evidence and default selection live in
 [benchmarks.md](../reference/benchmarks.md); open acceptance remains in Phase 6/18.
+
+#### Kernel configuration ownership
+
+Shared and serial bindings execute within one immutable `RuntimeConfig`.
+Numerical helpers read that scope; they do not read process environment in
+forward loops. The serial speculative trace switch is captured once per run.
+Trellis mode resolves through the same configuration port at weight creation.
+
+TurboQuant caches capture their codec's fused-decode choice at construction.
+RAM clones, full-prefill row restoration, batch layout factories, extraction
+and delayed conversion preserve that choice. KV maintenance binds its runtime
+before conversion, including conversion performed after a host setting changes.
+SSD formats contain numerical state, not a frozen process configuration; their
+receiving cache factory selects the destination binding's policy.
+
+The configuration tests change host settings around scoped construction,
+copy/extraction and delayed conversion. Existing codec/donor tests now select
+both fused settings through `configureRuntime`, so they exercise the same
+configuration contract as serving. Paged request policies also bind the numerical
+namespace and construction policy before deferred lookup or allocation, including
+when a gateway outlives later host configuration changes. The serving reference
+maps flags to their owning interfaces and records supported combinations. The
+legacy internal `flagOn` wrapper is removed; CLI controls remain compatible.
+The new paged-policy tests first reproduce both deferred-setting failures, then
+pass with the captured policy. Paged operation/layout checks pass on both Macs;
+the full local model-free suite and all typechecks pass.
+
+### 12.15 Package boundaries before community outreach
+
+Initial audit, 2026-09-11. The public-readiness program owns the packaging
+work; the existing interface and optimization milestones retain their scope.
+The three published entry points need different treatment:
+
+| Entry or layer | Verified dependency boundary | Packaging consequence |
+|---|---|---|
+| `mlx-bun/client`, `src/client.ts` | Imports only completion/host contracts; uses standard Request, Response, fetch and cancellation APIs | A candidate for independent client installation. The current npm package still has Darwin/arm64 metadata and application dependencies; a portable source file alone does not change that install contract. |
+| `src/contracts/`, `src/engine/`, `src/inference/` | Separate TypeScript project excludes Bun and DOM types; the AST boundary test limits imports by layer | Keep these interfaces and policies together inside the engine. They do not each need a published package. |
+| `mlx-bun/engine`, `src/library.ts` | CPU-safe entry exports engine/client APIs, but `initializeMlx()` dynamically imports the broad root API; `openIsolatedHost()` imports serving isolation and locates the CLI | Preserve this compatibility entry. Separate native engine initialization from application/worker assembly before treating it as a standalone engine package. |
+| `mlx-bun`, `src/index.ts` | Re-exports concrete models, generation, downloads, registry, fit and server construction | Already a published compatibility API. A workspace move needs forwarding exports, not a silent reduction of available functionality. |
+| Browser application | Own TypeScript project; boundary test allows web code and contracts | A candidate private application workspace. Its assets still participate in the CLI/native bundle. |
+| Website | Already has its own package manifest and build; generates reference pages from canonical docs | Keep one source for reference facts. A root workspace declaration alone would not improve this ownership. |
+
+The source audit followed static imports, re-exports, type imports and literal
+`import()`/`require()` references using TypeScript module resolution. The
+client closure contained four source files and no external module imports.
+The high-level engine entry reached the broad root API, including Pi and web
+assets. This is a dependency inventory, not a claim that importing the
+CPU-safe entry eagerly loads those modules. Non-code assets and executable
+paths also need a packaging check; import analysis cannot establish that they
+ship correctly.
+
+Recommended direction for the remaining audit: one engine package containing
+models, native bindings, cache codecs, inference methods and their interfaces;
+application assembly outside it; a small client package if independent
+installation is useful. Keep specialized kernels and tensor ownership within
+that engine. Memory, Pi and training require their own dependency and asset
+review before deciding whether they become private workspaces. A general MLX
+array binding is a separate API commitment and is not implied by this split.
+
+Before choosing and implementing the layout, map the engine-to-application
+edges through server construction, native bootstrap, worker commands and
+bundled assets. Verify the package tarball and a consumer installed outside
+the repository. Public import compatibility and the single-binary release are
+acceptance requirements. The initial audit does not yet close the package
+organization milestone.
+
+The initial `bun pm pack --dry-run --ignore-scripts` also confirms that the
+package includes the whole `docs/` tree, including archived investigations
+and active plans. Review that allowlist during packaging work; these are not
+runtime dependencies. This dry run inventories the candidate but does not
+replace installation and asset checks against an actual tarball.
+
+
+### 12.15 Prefill workspace policy
+
+`PrefillPolicy` selects a request's chunk size independently of scheduling,
+cache placement and sampling. The MLX backend captures model geometry and
+explicit configuration when it binds the policy. Ordinary generation,
+shared rows and speculative preparation consume the selected size through
+their existing prefill interfaces. A caller's explicit request value wins.
+The automatic recurrent-attention policy bounds the temporary materialized
+SDPA score workspace; it does not inspect live free memory or refuse work.
+Default values and precedence live in [server configuration](../reference/server-config.md).
+
+### 12.16 Committed append cache formats
+
+`MlxTokenAppend` declares supported affine precisions and TurboQuant K/V
+formats. Qualified Qwen27B bindings expose KV4/KV8 and K8V3.
+Request preparation supplies template-derived candidates; placement selects
+an eligible method without inspecting numerical cache formats. The ordinary
+method consumes the binding's declaration once and uses its committed-append
+operation. Qwen shares projection work over known tokens while retaining each
+query's one-token affine attention calculation and causal KV extent.
+`KvMaintenance.maxAppendTokens` bounds a committed chunk at a pending
+precision transition; the method runs maintenance before continuing.
+Verification remains a separate method capability: enabling committed
+quantized appends does not enable quantized echo verification. No default fill setting or
+shared-group support is inferred from that declaration.

package/docs/planning/PRODUCT_ROADMAP.md

@@ -65,12 +65,11 @@ disclosure: appliance defaults at the front, sharp instruments underneath.
 
 ## The three modes (the organizing design law)
 
-mlx-bun has not published anything: the entire CLI, command, and flag
-surface is still ours to invent, and we are not married to any flag
-name. Where a de-facto standard name exists (mlx-lm's), we adopt it
-rather than reinvent. The surface is designed once, to serve three
-audiences at three depths — progressive disclosure over ONE set of
-verbs and one flag vocabulary, not three separate CLIs:
+mlx-bun is published. Existing commands, flags, and package exports are
+compatibility commitments; changes need documented migration behavior.
+Where an mlx-lm flag has matching semantics, prefer its established name.
+The CLI serves three audiences through one set of verbs, with more explicit
+configuration available as users need it:
 
 1. **Automatic mode** — first-time local-AI users. `mlx-bun` (and bare
    verbs) with zero flags. Smart, sane defaults from the device-profile