mtplx 2.0.2__tar.gz → 2.2.0__tar.gz

{mtplx-2.0.2 → mtplx-2.2.0}/CHANGELOG.md

@@ -4,6 +4,53 @@ All notable user-facing changes to MTPLX. The format is based on
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versions follow
 [Semantic Versioning](https://semver.org/).
 
+## [2.2.0] - 2026-07-19
+
+The copy-drafting and small-Mac release. Decoding: context-copy
+(prompt-lookup) drafting lands on by default (PR #151 by lBroth) with an
+exact temperature path — copied blocks are accepted with the target's own
+shaped probability, so the output distribution is unchanged at any
+temperature; measured +53% on edit-heavy agent turns at temp 0.6, parity
+on novel text (disable with MTPLX_CONTEXT_COPY=0). Models: the 4B
+zero-acceptance defect (#176) is root-caused and fixed — the engine heals
+raw delta-encoded MTP sidecars at load so existing downloads recover
+without re-downloading, the 4B Speed artifact is rebuilt (227.8 tok/s D3
+on M5 Max, 1.71x AR), and a new 4B Quality artifact ships (191.7 tok/s
+D3, 2.19x — the largest MTP multiplier in the fleet); sub-16GB Macs get
+first-class catalog recommendations. Tune (#177): a 0.0-acceptance depth
+can never win, be saved, or be replayed, and poisoned records are
+quarantined at load. Forge: FP16 precision option with M1/M2 auto-select
+(#166); the fp16 cast can no longer corrupt a sidecar in place;
+degenerate sidecars are quarantined on re-forge. Server: SSD session
+writer crash fixed — encode at enqueue (#169); live requests preempt idle
+cache maintenance. MoE: mtp_depth_max is a ceiling, not the default; the
+35B-A3B launches at its measured D2 (#174 part 1 by davidtai). Also: the
+Qwen 3.6 27B AR decode-trace crash fix (#167 by davidtai), truthful
+per-model profile display, hybrid-model boundary retention across append
+churn, and an experimental --draft-core device (opt-in). Full details in
+docs/releases/v2.2.0.md.
+
+## [2.1.0] - 2026-07-17
+
+The community-fixes release. Memory: the v2.x reports are root-caused and
+closed (MLX allocator cache bounded by default, per-session admission
+re-clamped on sub-96GB machines, paged pool bounded by the context
+window, pressure responder redesigned, q4 kv-quant crash fixed, new
+`--memory-budget` knob). Agent sessions: warm prefix reuse survives every
+tool turn (#121), hybrid-model near-prefix restores no longer collapse to
+the oldest boundary (measured 0.4s instead of 33.8s on a 22k follow-up),
+restart-warm sessions keep their boundary records across SSD generations
+(#159, #144), and cache hits are reported in standard `usage` fields.
+Sampling: presence and frequency penalties fixed in the batched AR lane
+(#156). App: startup and update hang fixed plus a full subprocess
+watchdog sweep (#158), the Hermes tile launches Hermes Desktop, raw
+tool-call XML no longer leaks into no-tools chats (#160). CLI: `start
+opencode` serves the same lane the app serves. Backends: qwen3_5_mtp and
+hy_v3 land (#142, #147). Performance: the model-owner thread is
+QoS-pinned for 8 to 10% faster decode under real multitasking load.
+Operators: `MTPLX_COMPILED_VERIFY_MAX_CONTEXT` is env-overridable. Full
+details in docs/releases/v2.1.0.md.
+
 ## [2.0.2] - 2026-07-09
 
 The agent-reliability release: multi-turn agent sessions now render

{mtplx-2.0.2 → mtplx-2.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mtplx
-Version: 2.0.2
+Version: 2.2.0
 Summary: Native MTP speculative decoding for Qwen3-Next on Apple Silicon.
 Author-email: Youssof Altoukhi <business@youssofal.com>
 License-Expression: Apache-2.0
@@ -25,9 +25,9 @@ License-File: LICENSE
 License-File: NOTICE
 Requires-Dist: fastapi>=0.136
 Requires-Dist: huggingface-hub>=0.36
-Requires-Dist: mlx<0.32,>=0.31; sys_platform == "darwin" and platform_machine == "arm64"
+Requires-Dist: mlx<0.33,>=0.31; sys_platform == "darwin" and platform_machine == "arm64"
 Requires-Dist: mlx-lm<0.32,>=0.31; sys_platform == "darwin" and platform_machine == "arm64"
-Requires-Dist: transformers<5.13; sys_platform == "darwin" and platform_machine == "arm64"
+Requires-Dist: transformers!=5.13.0,<5.14; sys_platform == "darwin" and platform_machine == "arm64"
 Requires-Dist: nanobind>=2; sys_platform == "darwin" and platform_machine == "arm64"
 Requires-Dist: numpy>=2
 Requires-Dist: pydantic>=2

mtplx-2.2.0/mtplx/backends/hy_v3_mtp.py

@@ -0,0 +1,69 @@
+"""Tencent Hy3 (hy_v3) native MTP backend facade.
+
+Hy3 ships one appended MTP layer (num_nextn_predict_layers=1): a full MoE
+decoder layer fed concat[RMSNorm(next-token embedding), RMSNorm(trunk
+pre-final-norm hidden state)] through an eh_proj down-projection, sharing the
+trunk's embeddings and lm_head. The MLX reference implementation exposes it as
+``Model.predict_next_tokens(hidden, token_ids, cache)`` with
+``return_hidden_states=True`` on the trunk forward (see
+mlx-lm ``models/hy_v3.py``, MTP revision).
+
+Like the GLM/DeepSeek facades, drafting and verification are wired through the
+shared speculative sampler in ``generation.py``; this facade gates execution
+behind the verified runtime contract.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+from . import DraftTokens, ModelState, MTPBackend, VerifyOutput
+from mtplx.profiles import DEFAULT_PROFILE_NAME
+
+
+class HyV3MTPBackend(MTPBackend):
+    arch_id = "hy-v3-mtp"
+
+    def load(self, model_path: Path) -> ModelState:
+        from mtplx.mtp_patch import MTPContract
+        from mtplx.runtime import load
+
+        runtime = load(model_path, mtp=True, contract=MTPContract())
+        return ModelState(
+            model_path=Path(model_path),
+            runtime=runtime,
+            metadata={"arch_id": self.arch_id, "contract_gated": True},
+        )
+
+    def verify(self, state: ModelState, draft_tokens: DraftTokens, hidden: Any) -> VerifyOutput:
+        raise NotImplementedError("HyV3MTPBackend.verify is wired through generation.py")
+
+    def propose(self, state: ModelState, hidden: Any) -> DraftTokens:
+        raise NotImplementedError("HyV3MTPBackend.propose is wired through generation.py")
+
+    def recommended_profile(self) -> str:
+        return DEFAULT_PROFILE_NAME
+
+    def health(self) -> dict[str, Any]:
+        return {
+            "arch_id": self.arch_id,
+            "runtime_path": "mtplx.runtime + mtplx.hy_v3_mtp_patch + mtplx.generation",
+            "support_level": "experimental-native-contract-gated",
+            "contract_required": True,
+            "supported_model_types": ["hy_v3"],
+            "mtp_depth_max": 1,
+            "notes": (
+                "Single appended NextN layer with its own 192-expert MoE MLP, "
+                "sigmoid top-8 routing with expert bias, eh_proj over "
+                "concat[enorm(embedding), hnorm(hidden)], shared embeddings "
+                "and head. Draft layer consumes the trunk pre-final-norm "
+                "hidden state. Verification is exact rejection sampling in "
+                "generation.py; the MLX reference (mlx-lm hy_v3 MTP revision) "
+                "verifies greedily and is temp-0 exact."
+            ),
+            "references": [
+                "REFERENCES:TOOLS/vllm-official-main/vllm/model_executor/models/hy_v3_mtp.py",
+                "REFERENCES:TOOLS/mlx-lm/mlx_lm/models/hy_v3.py",
+            ],
+        }

{mtplx-2.0.2 → mtplx-2.2.0}/mtplx/backends/registry.py

@@ -21,6 +21,7 @@ SUPPORTED_ARCH_IDS = {
     "nemotron-h-mtp",
     "gemma4-assistant-mtp",
     "step3p5-mtp",
+    "hy-v3-mtp",
 }
 
 TIER_VERIFIED = "verified"
@@ -411,10 +412,22 @@ ARCHITECTURE_CATALOG: dict[str, ArchitectureSupport] = {
         display_name="HY V3 MTP",
         family="hy",
         backend="hy_v3_mtp",
-        support_level="recognized-backend-pending",
-        runtime_compatibility="recognized-backend-pending",
+        support_level="experimental-native-contract-gated",
+        runtime_compatibility="native-contract-gated",
+        can_run_verified=True,
         aliases=("hy_v3_mtp", "hy_v3"),
-        references=("REFERENCES:TOOLS/vllm-official-main/vllm/model_executor/models/hy_v3_mtp.py",),
+        family_gate="appended-layer-mtp-markers",
+        references=(
+            "REFERENCES:TOOLS/vllm-official-main/vllm/model_executor/models/hy_v3_mtp.py",
+            "REFERENCES:TOOLS/mlx-lm/mlx_lm/models/hy_v3.py",
+        ),
+        notes=(
+            "Hy3 ships one appended NextN layer with its own 192-expert MoE, "
+            "eh_proj over concat[enorm(embedding), hnorm(hidden)], and shared "
+            "embeddings/head. The mlx-lm hy_v3 MTP revision exposes the head "
+            "natively (predict_next_tokens), so injection binds the existing "
+            "surface rather than grafting weights."
+        ),
     ),
     "generic-mtp": ArchitectureSupport(
         arch_id="generic-mtp",
@@ -794,6 +807,35 @@ def _has_all_suffixes_under_prefixes(
     return True
 
 
+_HY_V3_MTP_MARKER_SUFFIXES = (
+    "enorm.weight",
+    "hnorm.weight",
+    "eh_proj.weight",
+    "final_layernorm.weight",
+)
+
+
+def _passes_hy_v3_gate(inspection: Any) -> bool:
+    """Hy3's appended MTP block lives directly under an ``mtp.`` prefix
+    (``mtp.enorm.weight``, ``mtp.hnorm.weight``, ``mtp.eh_proj.weight``,
+    ``mtp.final_layernorm.weight``, ``mtp.layer.*``) rather than the
+    ``mtp.layers.{idx}.`` nesting DeepSeek/GLM/Step use, so it needs its own
+    gate instead of `_passes_appended_layer_gate` (verified against the
+    shipped `hy3-demolition-mlx-*-mtp` checkpoints' safetensors index)."""
+    keys = _weight_keys(inspection)
+    if not keys:
+        return False
+    count = int(getattr(inspection, "mtp_num_hidden_layers", 0) or 0)
+    if count <= 0:
+        return False
+    return _has_marker_under_prefixes(
+        keys,
+        ("mtp.",),
+        _HY_V3_MTP_MARKER_SUFFIXES,
+        ("mtp.layer.",),
+    )
+
+
 def _passes_appended_layer_gate(inspection: Any) -> bool:
     keys = _weight_keys(inspection)
     if not keys:
@@ -910,6 +952,8 @@ def _passes_family_runtime_gate(arch_id: str, inspection: Any, tensor_gate: bool
             tensor_gate
             and int(getattr(inspection, "mtp_num_hidden_layers", 0) or 0) > 0
         )
+    if arch_id == "hy-v3-mtp":
+        return _passes_hy_v3_gate(inspection)
     if arch_id in {
         "deepseek-v3-mtp",
         "glm-moe-dsa-mtp",

{mtplx-2.0.2 → mtplx-2.2.0}/mtplx/cache_bank/cold_tier.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import hashlib
+from collections import deque
 import json
 import logging
 import os
@@ -70,14 +71,15 @@ _COMMITTED_CACHE_POLICIES = frozenset({"committed", "last_window"})
 
 
 def _deferred_encode_enabled() -> bool:
-    """Writer-thread payload encode (kvcache-v2). Off-switch only.
+    """RETIRED (#169, 2026-07-17) — writer-side encode is gone; always False.
 
-    The foreground evaluates payload arrays (ms-scale GPU slice kernels) so
-    the writer thread never evaluates foreign lazy graphs — it only reads
-    settled buffers into bytes (the GB-scale memcpy that used to run on the
-    request thread)."""
-    raw = str(os.environ.get("MTPLX_SSD_DEFERRED_ENCODE", "1")).strip().lower()
-    return raw not in {"0", "false", "off", "no"}
+    The kvcache-v2 writer-thread encode block-sliced tensors at write time
+    (TreeCodec builds lazy slice arrays and mx.eval()s them), which crashed
+    on restore-derived arrays ("There is no Stream(gpu, 1) in current
+    thread") and could serialize donation-mutated KV pages from the writer
+    backlog. put_entry now always encodes at enqueue on the owner thread.
+    MTPLX_SSD_DEFERRED_ENCODE is parsed nowhere and ignored."""
+    return False
 
 
 @dataclass(frozen=True)
@@ -100,6 +102,9 @@ class PendingWrite:
     tensors: dict[str, bytes]
     deferred: DeferredPayload | None = None
     created_at_s: float = field(default_factory=time.time)
+    # Estimated bytes this write pins in memory until the writer drains it
+    # (deferred payloads hold live KV arrays; encoded ones hold the buffers).
+    pinned_nbytes: int = 0
 
 
 @dataclass(frozen=True)
@@ -169,6 +174,10 @@ def default_cold_tier_max_bytes() -> int:
     return DEFAULT_COLD_TIER_MAX_BYTES
 
 
+def _env_size_bytes(name: str, default: int) -> int:
+    return parse_size_bytes(os.environ.get(name), default)
+
+
 def parse_size_bytes(value: str | int | None, default: int) -> int:
     if value is None:
         return int(default)
@@ -270,6 +279,24 @@ class SessionBankColdTier:
         self._queue: queue.Queue[PendingWrite | None] = queue.Queue(
             maxsize=max(1, int(writer_queue_depth))
         )
+        # Backlog byte cap (issue #145): every queued write pins its payload
+        # (deferred ones pin LIVE KV arrays) until the writer drains it. A
+        # count-bounded queue of 32 multi-GB snapshots can pin ~50 GB under
+        # distinct-prefix churn — measured live 2026-07-09 (active memory
+        # climbed 35 -> 66 GB while the bank ledger stayed flat). Cap the
+        # pinned bytes, drop new writes beyond it.
+        self._pending_bytes = 0
+        self._backlog_budget_bytes = _env_size_bytes(
+            "MTPLX_SSD_WRITER_BACKLOG_BYTES", 4 * 1024**3
+        )
+        # Hourly write budget (issue #144: 7 TB written / SSD wear): the
+        # different-repos pattern writes GBs per task and never restores
+        # them (measured 58 GB in 45 min with restore_hits=0). Rolling
+        # one-hour byte budget; beyond it new writes are skipped.
+        self._write_budget_per_hour_bytes = _env_size_bytes(
+            "MTPLX_SSD_WRITE_BUDGET_PER_HOUR", 64 * 1024**3
+        )
+        self._written_window: deque[tuple[float, int]] = deque()
         self._stop = threading.Event()
         self._base_lock = threading.RLock()
         self._disk_usage_lock = threading.Lock()
@@ -336,82 +363,55 @@ class SessionBankColdTier:
         if len(token_ids) < self.min_prefix_tokens:
             self._inc("skipped_too_short")
             return False
+        estimated_nbytes = int(getattr(entry, "nbytes", 0) or 0)
+        if not self._admit_write(estimated_nbytes):
+            return False
         boundaries = tuple(
             (int(r[0]), r[1], r[2] if len(r) > 2 else None)
             for r in (getattr(entry, "gdn_boundaries", None) or [])
         )
-        if _deferred_encode_enabled():
-            try:
-                deferred = DeferredPayload(
-                    cache_snapshot=getattr(entry, "cache_snapshot"),
-                    logits=getattr(entry, "logits"),
-                    hidden=getattr(entry, "hidden"),
-                    mtp_history_snapshot=getattr(entry, "mtp_history_snapshot", None),
-                    gdn_boundaries=boundaries,
-                    has_recurrent=bool(getattr(entry, "has_recurrent", False)),
-                    block_size=self.block_size,
-                )
-                # Settle every payload array on the request thread so the
-                # writer only reads buffers (MLX thread discipline: never
-                # evaluate another thread's lazy graph).
-                _eval_payload_trees(
-                    deferred.cache_snapshot,
-                    deferred.logits,
-                    deferred.hidden,
-                    deferred.mtp_history_snapshot,
-                    deferred.gdn_boundaries,
-                )
-            except Exception as exc:
-                self._inc("skipped_serialize_error")
-                logger.warning(
-                    "SessionBank SSD payload prep skipped: %s: %s",
-                    type(exc).__name__,
-                    exc,
-                )
-                return False
-            metadata = self._metadata_for_entry(
-                entry,
-                capabilities=capabilities or (),
-                payload_nbytes=0,
-            )
-            pending = PendingWrite(
-                entry_id=str(metadata["entry_id"]),
-                token_ids=token_ids,
-                metadata=metadata,
-                payload_spec=None,
-                tensors={},
-                deferred=deferred,
-            )
-        else:
-            try:
-                encoded = encode_payload(
-                    cache_snapshot=getattr(entry, "cache_snapshot"),
-                    logits=getattr(entry, "logits"),
-                    hidden=getattr(entry, "hidden"),
-                    mtp_history_snapshot=getattr(entry, "mtp_history_snapshot", None),
-                    gdn_boundaries=boundaries,
-                    has_recurrent=bool(getattr(entry, "has_recurrent", False)),
-                    block_size=self.block_size,
-                )
-            except Exception as exc:
-                self._inc("skipped_serialize_error")
-                logger.warning("SessionBank SSD serialize skipped: %s: %s", type(exc).__name__, exc)
-                return False
-            metadata = self._metadata_for_entry(
-                entry,
-                capabilities=capabilities or (),
-                payload_nbytes=encoded.nbytes,
-            )
-            pending = PendingWrite(
-                entry_id=str(metadata["entry_id"]),
-                token_ids=token_ids,
-                metadata=metadata,
-                payload_spec=encoded.spec,
-                tensors=encoded.tensors,
+        # Encode ALWAYS happens here, on the enqueueing (owner) thread, never
+        # on the writer thread (#169, 2026-07-17). The retired writer-side
+        # "deferred encode" (kvcache-v2) block-sliced tensors at write time:
+        # TreeCodec._encode_tensor_blocks builds new lazy slice arrays and
+        # mx.eval()s them, which (a) crashed the process on restore-derived
+        # arrays whose graphs referenced the restore stream ("There is no
+        # Stream(gpu, 1) in current thread"), and (b) held live KV references
+        # for seconds in the writer backlog, so under buffer donation the
+        # eventual serialization could capture mutated pages — silently
+        # corrupt persisted sessions that degrade on every restore. Bytes are
+        # captured at snapshot time; the writer thread is pure file IO.
+        try:
+            encoded = encode_payload(
+                cache_snapshot=getattr(entry, "cache_snapshot"),
+                logits=getattr(entry, "logits"),
+                hidden=getattr(entry, "hidden"),
+                mtp_history_snapshot=getattr(entry, "mtp_history_snapshot", None),
+                gdn_boundaries=boundaries,
+                has_recurrent=bool(getattr(entry, "has_recurrent", False)),
+                block_size=self.block_size,
             )
+        except Exception as exc:
+            self._inc("skipped_serialize_error")
+            logger.warning("SessionBank SSD serialize skipped: %s: %s", type(exc).__name__, exc)
+            return False
+        metadata = self._metadata_for_entry(
+            entry,
+            capabilities=capabilities or (),
+            payload_nbytes=encoded.nbytes,
+        )
+        pending = PendingWrite(
+            entry_id=str(metadata["entry_id"]),
+            token_ids=token_ids,
+            metadata=metadata,
+            payload_spec=encoded.spec,
+            tensors=encoded.tensors,
+            pinned_nbytes=max(estimated_nbytes, int(encoded.nbytes)),
+        )
         try:
             self._queue.put_nowait(pending)
         except queue.Full:
+            self._release_pending(pending.pinned_nbytes)
             self._inc("skipped_queue_full")
             logger.warning(
                 "SessionBank SSD writer queue full; skipping prefix_len=%d token_hash=%s",
@@ -573,11 +573,18 @@ class SessionBankColdTier:
     def stats(self) -> dict[str, Any]:
         with self._stats_lock:
             stats = dict(self._stats)
+        with self._stats_lock:
+            pending_bytes = int(self._pending_bytes)
+            written_last_hour = sum(nbytes for _, nbytes in self._written_window)
         stats.update(
             {
                 "enabled": self.enabled,
                 "restorable": self.restorable,
                 "writer_queue_depth": int(self._queue.qsize()),
+                "writer_backlog_bytes": pending_bytes,
+                "writer_backlog_budget_bytes": int(self._backlog_budget_bytes),
+                "written_bytes_last_hour": int(written_last_hour),
+                "write_budget_per_hour_bytes": int(self._write_budget_per_hour_bytes),
                 "dir": str(self.base_dir),
                 "manifest_path": str(self._manifest_path),
             }
@@ -786,6 +793,9 @@ class SessionBankColdTier:
                     self._inc("writes_completed")
                     with self._stats_lock:
                         self._stats["last_write_s"] = time.time()
+                        self._written_window.append(
+                            (time.time(), int(pending.pinned_nbytes))
+                        )
                     logger.info(
                         "SessionBank SSD wrote entry_id=%s prefix_len=%d nbytes=%d",
                         pending.entry_id,
@@ -801,37 +811,54 @@ class SessionBankColdTier:
                     exc,
                 )
             finally:
+                self._release_pending(pending.pinned_nbytes)
                 self._queue.task_done()
 
+    def _admit_write(self, estimated_nbytes: int) -> bool:
+        """Backlog + hourly-budget admission for a new SSD write."""
+
+        now = time.time()
+        with self._stats_lock:
+            if (
+                self._pending_bytes + max(0, estimated_nbytes)
+                > self._backlog_budget_bytes
+            ):
+                self._stats["skipped_backlog_bytes"] = (
+                    int(self._stats.get("skipped_backlog_bytes", 0) or 0) + 1
+                )
+                return False
+            while self._written_window and self._written_window[0][0] < now - 3600:
+                self._written_window.popleft()
+            written_last_hour = sum(nbytes for _, nbytes in self._written_window)
+            if (
+                written_last_hour + max(0, estimated_nbytes)
+                > self._write_budget_per_hour_bytes
+            ):
+                self._stats["skipped_write_budget"] = (
+                    int(self._stats.get("skipped_write_budget", 0) or 0) + 1
+                )
+                return False
+            self._pending_bytes += max(0, estimated_nbytes)
+        return True
+
+    def _release_pending(self, nbytes: int) -> None:
+        with self._stats_lock:
+            self._pending_bytes = max(0, self._pending_bytes - max(0, int(nbytes)))
+
     def _write_pending(self, pending: PendingWrite) -> bool:
         if pending.deferred is not None:
-            # Writer-side encode (kvcache-v2): arrays were settled by the
-            # request thread; this is pure buffer->bytes work off the
-            # foreground. Failures count as write failures, not serialize
-            # skips, so the stats distinguish the two eras.
-            encoded = encode_payload(
-                cache_snapshot=pending.deferred.cache_snapshot,
-                logits=pending.deferred.logits,
-                hidden=pending.deferred.hidden,
-                mtp_history_snapshot=pending.deferred.mtp_history_snapshot,
-                gdn_boundaries=pending.deferred.gdn_boundaries,
-                has_recurrent=pending.deferred.has_recurrent,
-                block_size=pending.deferred.block_size,
-            )
-            metadata = dict(pending.metadata)
-            metadata["nbytes"] = int(
-                max(int(metadata.get("nbytes", 0) or 0), int(encoded.nbytes))
-            )
-            metadata["logical_nbytes"] = int(encoded.nbytes)
-            metadata["physical_nbytes"] = int(encoded.nbytes)
-            pending = PendingWrite(
-                entry_id=pending.entry_id,
-                token_ids=pending.token_ids,
-                metadata=metadata,
-                payload_spec=encoded.spec,
-                tensors=encoded.tensors,
-                created_at_s=pending.created_at_s,
+            # Retired path (#169, 2026-07-17): writer-side encode ran MLX
+            # slice/eval graph work on the writer thread (crash on
+            # restore-stream arrays, donation-corruption window). put_entry
+            # now always encodes at enqueue; a deferred payload reaching the
+            # writer is a programming error, never silently encoded here.
+            self._inc("skipped_deferred_retired")
+            logger.error(
+                "SessionBank SSD writer received a deferred payload "
+                "entry_id=%s; writer-side encode is retired (#169), skipping",
+                pending.entry_id,
             )
+            return False
         with self._base_lock:
             self._ensure_store()
             entry_hash_prefix = pending.entry_id[:2]

{mtplx-2.0.2 → mtplx-2.2.0}/mtplx/cache_state.py

@@ -889,6 +889,17 @@ class VllmMetalPagedKVCache:
             int((self.num_blocks * 3 + 1) // 2),
             int(self.num_blocks) + 1,
         )
+        window_tokens = _env_int("MTPLX_CONTEXT_WINDOW_TOKENS", 0)
+        if window_tokens > 0:
+            # Geometric growth must not overshoot the serving context window
+            # (#150: the 1.5x step at 100k+ ctx allocates GiBs of blocks no
+            # request can ever address). A genuinely larger requirement still
+            # wins — correctness over the clamp.
+            window_blocks = (int(window_tokens) + self.block_size - 1) // self.block_size
+            if window_blocks >= required_blocks:
+                grown_blocks = min(
+                    grown_blocks, max(window_blocks, int(self.num_blocks))
+                )
         if grown_blocks <= self.num_blocks:
             return True
         if self.key_cache is None or self.value_cache is None:
@@ -1437,6 +1448,14 @@ class VllmMetalPagedKVCache:
         outputs: list[Any] = []
         very_negative = mx.array(-1.0e30, dtype=mx.float32)
         eps = mx.array(1.0e-20, dtype=mx.float32)
+        # kv-quant stores values packed; _paged_range dequantizes them back to
+        # the logical head dim recorded in _shape, so the accumulator must be
+        # sized to the dequantized width, not the packed storage width (#150,
+        # q4 crash on the paged split-SDPA path).
+        if self.kv_quant and self._shape is not None:
+            value_dim = int(self._shape[2])
+        else:
+            value_dim = int(self.value_cache.shape[3])
 
         for q_start in range(0, q_len, q_chunk_size):
             q_end = min(q_len, q_start + q_chunk_size)
@@ -1454,7 +1473,7 @@ class VllmMetalPagedKVCache:
                     int(q.shape[0]),
                     int(q.shape[1]),
                     int(q.shape[2]),
-                    int(self.value_cache.shape[3]),
+                    value_dim,
                 ),
                 dtype=mx.float32,
             )

{mtplx-2.0.2 → mtplx-2.2.0}/mtplx/cli.py

@@ -585,7 +585,7 @@ def _add_bridge_prompt_args(parser: argparse.ArgumentParser) -> None:
     )
     parser.add_argument(
         "--chat-template-profile",
-        choices=["local_qwen36", "froggeric_v19", "tokenizer"],
+        choices=["local_qwen36", "froggeric_v19", "froggeric_v21_3", "tokenizer"],
         default="local_qwen36",
         help="Chat template profile for server/OpenCode paths.",
     )
@@ -1924,7 +1924,7 @@ def build_parser() -> argparse.ArgumentParser:
         "--profile",
         type=_profile_arg, metavar=_PROFILE_METAVAR,
         default=DEFAULT_PROFILE_NAME,
-        help="Runtime profile; start defaults to Sustained. Use --profile turbo for the verify-kernel fast path (4/8-bit affine), or --profile performance-cold --max for Burst.",
+        help="Runtime profile. Default resolves per model: Turbo for the quantized 27B and 9B flagships (the app's launch rule), Sustained otherwise. An explicit value always wins. Use --profile performance-cold --max for Burst.",
     )
     start_flow_p.add_argument("--download", action="store_true", help="Download the selected/default model if it is missing")
     start_flow_p.add_argument("--yes", action="store_true", help="Use defaults without interactive model prompts")
@@ -2106,7 +2106,7 @@ def build_parser() -> argparse.ArgumentParser:
         "--profile",
         type=_profile_arg, metavar=_PROFILE_METAVAR,
         default=DEFAULT_PROFILE_NAME,
-        help="Runtime profile. Direct server quickstart defaults to Sustained; use --profile performance-cold --max for Burst.",
+        help="Runtime profile. Default resolves per model (Turbo for the quantized 27B and 9B flagships, Sustained otherwise); use --profile performance-cold --max for Burst.",
     )
     quickstart_server_p.add_argument("--unsafe-force-unverified", action="store_true")
     quickstart_server_p.add_argument("--yes", action="store_true", help="Confirm unsafe non-interactive actions")
@@ -2265,6 +2265,7 @@ def build_parser() -> argparse.ArgumentParser:
     tune_p.add_argument("--mtp-cache-policy", choices=["persistent", "fresh"], default="persistent", help=argparse.SUPPRESS)
     tune_p.add_argument("--mtp-history-policy", choices=["auto", "committed", "full", "last-window", "last_window", "cycle", "none"], default="committed", help=argparse.SUPPRESS)
     tune_p.add_argument("--draft-temperature", type=float, help=argparse.SUPPRESS)
+    tune_p.add_argument("--draft-core", choices=["stock", "device-d2", "device"], default="stock", help=argparse.SUPPRESS)
     tune_p.add_argument("--draft-top-p", type=float, help=argparse.SUPPRESS)
     tune_p.add_argument("--draft-top-k", type=int, help=argparse.SUPPRESS)
     tune_p.add_argument("--prompt-suite", help=argparse.SUPPRESS)
@@ -2330,6 +2331,13 @@ def build_parser() -> argparse.ArgumentParser:
     forge_build_p.add_argument("--max", action="store_true", help="Opt into max-fan verification")
     forge_build_p.add_argument("--max-tokens", type=int, default=2048, help="Verification response budget")
     forge_build_p.add_argument("--suite", help="Verification prompt suite")
+    forge_build_p.add_argument(
+        "--dtype",
+        choices=["auto", "bf16", "fp16"],
+        help="Dtype for non-quantized parameters (overrides recipe body_dtype). "
+        "fp16 prompt-processes faster on M1/M2 Macs, which have no native "
+        "BF16; auto picks fp16 on those chips",
+    )
     forge_build_p.add_argument(
         "--allow-degraded-mtp",
         action="store_true",
@@ -2503,8 +2511,9 @@ def build_parser() -> argparse.ArgumentParser:
         type=_profile_arg, metavar=_PROFILE_METAVAR,
         default=DEFAULT_PROFILE_NAME,
         help=(
-            "Runtime profile. Server defaults to Sustained so long-context "
-            "prefill uses the v0.1.7 fast path; use --profile performance-cold "
+            "Runtime profile. Default resolves per model: Turbo for the "
+            "quantized 27B and 9B flagships (the app's launch rule), "
+            "Sustained otherwise; use --profile performance-cold "
             "--max for Burst."
         ),
     )
@@ -3461,7 +3470,7 @@ def build_parser() -> argparse.ArgumentParser:
     )
     depth_p.add_argument(
         "--draft-core",
-        choices=["stock", "device-d2"],
+        choices=["stock", "device-d2", "device"],
         default="stock",
         help=(
             "Experimental DraftCore backend. device-d2 compiles the greedy D2 "