interp-engine 1.7.2__tar.gz → 1.8.0__tar.gz

{interp_engine-1.7.2 → interp_engine-1.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: interp-engine
-Version: 1.7.2
+Version: 1.8.0
 Summary: A fast, standardized, and easy to use interpretability engine.
 Project-URL: Homepage, https://github.com/decoderesearch/interp-engine
 Project-URL: Repository, https://github.com/decoderesearch/interp-engine

{interp_engine-1.7.2 → interp_engine-1.8.0}/docs/USAGE.md

@@ -94,6 +94,16 @@ eager = load_model("google/gemma-2-2b-it", backend="eager", device="cuda", dtype
 served = load_model("meta-llama/Llama-3.1-8B", backend="vllm", gpu_memory_utilization=0.85)
 ```
 
+Two knobs narrow a bf16 checkpoint on load, with no calibration and no other repo:
+`quantization=` applies `"fp8"` (vLLM backends), `"bnb-4bit"` (either backend) or `"bnb-8bit"`
+(eager) to every linear layer, and `kv_cache_dtype="fp8"` halves vLLM's paged cache. Both take the
+same name on both backends and are refused, naming the alternative, where a backend cannot apply
+them. The [GPU sizer](../gpu-sizer/INPUTS.md) prices both and prints the argument it priced.
+
+```python
+narrow = load_model("meta-llama/Llama-3.3-70B-Instruct", backend="vllm", quantization="fp8", kv_cache_dtype="fp8")
+```
+
 **Construction is cheap and lazy on both backends; `warmup()` is where the cost lands.** On vLLM
 almost the entire load happens there, so call it before you time anything or a first request pays
 for the engine.
@@ -215,7 +225,9 @@ one entry in `chat_formatters.CODE_CHAT_FORMATS`.
 
 To attribute *tokens* to messages there are two methods, and the difference matters. `message_spans`
 gives per-token role, channel and section (`header` / `content` / `footer`), leaving the trailing
-generation scaffold owned by no message — use it to read or display structure. `message_partition`
+generation scaffold owned by no message — use it to read or display structure. A system turn the
+template injects on its own (Llama's knowledge-cutoff preamble, Qwen2.5's default persona) is
+tagged `role="system"` with `message_index=None`, so it can be shown as its own turn. `message_partition`
 gives one contiguous `[start, end)` span per message that together cover every token, which is what
 mean-pooling activations per turn needs:
 

{interp_engine-1.7.2 → interp_engine-1.8.0}/interp_engine/load.py

@@ -50,12 +50,62 @@ def _declares_nothing(value: Any) -> bool:
     return not list(value)
 
 
+def _apply_load_precision(backend: str, quantization: str, kv_cache_dtype: str, backend_kwargs: dict[str, Any]) -> None:
+    """Turn ``quantization`` and ``kv_cache_dtype`` into what ``backend``'s constructor takes.
+
+    Both are one name on ``load_model`` and two different things underneath: vLLM quantizes through
+    an engine argument, transformers through a ``BitsAndBytesConfig``. The table in
+    :data:`interp_engine.memory.QUANTIZATIONS` says which backend applies which scheme, and a scheme
+    the backend cannot apply is refused here with that table's reason -- never passed on to become an
+    opaque ``TypeError`` from a constructor, and never dropped to load the checkpoint as stored.
+    """
+    from interp_engine.memory import QUANTIZATIONS, quantization_refusal
+
+    use_vllm = backend in VLLM_BACKENDS
+    if kv_cache_dtype not in ("auto", "", None):
+        if not use_vllm:
+            raise ValueError(
+                f"kv_cache_dtype={kv_cache_dtype!r} names the dtype of vLLM's paged KV cache, and "
+                f"backend={backend!r} has no such cache. Drop it, or use a vLLM backend."
+            )
+        extra = dict(backend_kwargs.get("extra_vllm_kwargs") or {})
+        extra.setdefault("kv_cache_dtype", kv_cache_dtype)
+        backend_kwargs["extra_vllm_kwargs"] = extra
+
+    if not quantization:
+        return
+    refused = quantization_refusal(quantization, backend)
+    if refused:
+        raise ValueError(f"quantization={quantization!r} on backend={backend!r}: {refused}")
+    scheme = QUANTIZATIONS[quantization]
+    if use_vllm:
+        extra = dict(backend_kwargs.get("extra_vllm_kwargs") or {})
+        if extra.get("quantization") not in (None, scheme.vllm_name):
+            raise ValueError(
+                f"quantization={quantization!r} asks vLLM for {scheme.vllm_name!r}, but extra_vllm_kwargs "
+                f"already names {extra['quantization']!r}. Pass one or the other."
+            )
+        extra["quantization"] = scheme.vllm_name
+        backend_kwargs["extra_vllm_kwargs"] = extra
+        return
+    if backend_kwargs.get("quantization_config") is not None:
+        raise ValueError(
+            f"quantization={quantization!r} builds a BitsAndBytesConfig, and quantization_config= was "
+            f"passed as well. Pass one or the other."
+        )
+    from transformers import BitsAndBytesConfig
+
+    backend_kwargs["quantization_config"] = BitsAndBytesConfig(**scheme.eager_config)
+
+
 def load_model(
     hf_model_id: str,
     *,
     backend: str = "auto",
     device: str | None = None,
     dtype: str = "auto",
+    quantization: str = "",
+    kv_cache_dtype: str = "auto",
     num_gpus: int = 1,
     trust_remote_code: bool | None = None,
     static_points: Any = None,
@@ -86,7 +136,21 @@ def load_model(
         device: Explicit device for the eager backend. None means let the ladder choose.
             Ignored by vLLM, which always initializes on CUDA.
         dtype: ``"auto"`` (the checkpoint's native precision) or an explicit
-            ``"float32"``/``"float16"``/``"bfloat16"``.
+            ``"float32"``/``"float16"``/``"bfloat16"``. This is the width the **activations** run
+            at, and the width an unquantized checkpoint's weights are held at. It is not how to ask
+            for a narrower checkpoint: vLLM rejects ``dtype="fp8"``, and transformers would store
+            fp8 weights with no kernel behind them. That is ``quantization``.
+        quantization: An on-load scheme from :data:`interp_engine.memory.QUANTIZATIONS`, applied to
+            a wider checkpoint as it loads, with no calibration step: ``"fp8"`` (vLLM only),
+            ``"bnb-4bit"`` (either backend) or ``"bnb-8bit"`` (eager only). Empty, the default,
+            loads the checkpoint as stored -- which for a repo that already ships quantized is the
+            right answer, since a quantizer cannot narrow what is already narrower. A scheme the
+            chosen backend cannot apply is refused, naming the one to use instead. Other vLLM
+            schemes still reach the engine through ``extra_vllm_kwargs={"quantization": ...}``.
+        kv_cache_dtype: vLLM's KV cache dtype -- ``"auto"`` (the model dtype, or the scheme the
+            checkpoint declares for its cache) or ``"fp8"``, which halves the cache and so roughly
+            doubles the context or concurrency a card holds. Refused on the eager backend, which
+            has no paged cache to set the dtype of.
         num_gpus: Shard across this many GPUs on one node -- vLLM ``tensor_parallel_size``,
             eager accelerate ``device_map="auto"``. Note that vLLM with ``num_gpus > 1``
             cannot serve per-head ``z`` or DFA, because attention heads are sharded across
@@ -118,7 +182,8 @@ def load_model(
         ValueError: ``backend`` is not one of :data:`BACKENDS`; or ``static_points`` /
             ``static_writes`` was passed on a backend other than ``"vllm-static"``; or
             ``backend="vllm-static"`` declared no taps at all; or ``enforce_eager=True`` was
-            passed alongside a graph-replaying backend.
+            passed alongside a graph-replaying backend; or ``quantization`` / ``kv_cache_dtype``
+            asks the chosen backend for something it cannot apply.
         RuntimeError: a vLLM backend was requested but vLLM is not installed.
         GradientsUnsupported: ``requires_grad=True`` on a vLLM backend, which cannot
             provide gradients through its forward on any configuration.
@@ -182,6 +247,8 @@ def load_model(
         # installs nothing in Worker.load_model, and leaves hooks_available False.
         static_points, static_writes = [], None
 
+    _apply_load_precision(resolved, quantization, kv_cache_dtype, backend_kwargs)
+
     if use_vllm:
         require_vllm(f"backend={resolved!r} requested for {hf_model_id}")
         # `requires_grad` is an eager-only constructor kwarg, so on vLLM it would otherwise land as

{interp_engine-1.7.2 → interp_engine-1.8.0}/interp_engine/memory.py

@@ -44,6 +44,8 @@ from collections.abc import Iterator, Sequence
 from dataclasses import dataclass, field
 from typing import Any
 
+from interp_engine.facts import FP16_EAGER_OVERFLOW_ARCHS
+
 GIB = 1024**3
 
 #: The backends :func:`estimate` understands. Mirrors ``load.BACKENDS`` deliberately rather than
@@ -145,6 +147,22 @@ CALIBRATION: dict[str, Calibration] = {
             "smallest of anything here."
         ),
     ),
+    "quant_on_load_gib": Calibration(
+        value=0.3,
+        unit="GiB",
+        why=(
+            "What vLLM charges against its budget for quantizing at load, past the narrowed tensors "
+            "themselves. It appears in vLLM's own 'weights' figure, so it comes out of the KV cache's "
+            "share the way the CUDA context does, and it does not scale with the model."
+        ),
+        source=(
+            "Qwen3-8B and Qwen3-4B, vllm, bf16 checkpoint with quantization='fp8', RTX 5090, "
+            "utilization 0.9. The fp8 tensors on the card summed to 8.80 and 4.12 GiB; vLLM reported "
+            "'Model loading took' 9.09 and 4.41. The same two models at bf16 reported their tensors "
+            "exactly, and the KV cache came out 0.98x of predicted on fp8 against 1.00x on bf16 -- the "
+            "0.30 GiB is the whole of that gap."
+        ),
+    ),
     "frag_fraction": Calibration(
         value=0.04,
         unit="fraction of card",
@@ -660,7 +678,7 @@ _SCHEME_WIDTH: tuple[tuple[tuple[str, ...], float], ...] = (
 )
 
 
-def _scheme_width(quant_method: str) -> float | None:
+def scheme_width(quant_method: str) -> float | None:
     """Bytes per logical parameter for a quantization scheme, or None when it is not recognized.
 
     ``awq`` and ``gptq`` are listed at 4-bit because that is what they are in practice; an 8-bit GPTQ
@@ -676,6 +694,143 @@ def _scheme_width(quant_method: str) -> float | None:
     return None
 
 
+# ------------------------------------------------------------ on-load quantization
+
+
+@dataclass(frozen=True)
+class OnLoadQuantization:
+    """A scheme a backend can apply to a wider checkpoint **at load**, with no calibration step.
+
+    This is what separates the rows below from AWQ, GPTQ, NVFP4 and MXFP4: those need a calibration
+    run and arrive as a repo, which :class:`WeightBytes` already prices from its headers. A row here is
+    one argument to ``load_model``, so a sizer can offer it on any bf16 checkpoint and the snippet it
+    prints still runs.
+    """
+
+    name: str
+    #: Bytes per **linear** weight once quantized, scales included. Embeddings and the unembed are not
+    #: linear layers to either quantizer and stay at the model dtype -- see :func:`on_load_weight_bytes`.
+    width: float
+    #: The backends that apply it. A backend missing here is refused, with the reason in :attr:`refusals`.
+    backends: tuple[str, ...]
+    #: What vLLM calls it in ``quantization=``; empty when no vLLM backend applies it.
+    vllm_name: str
+    #: The ``BitsAndBytesConfig`` arguments transformers needs; empty when eager does not apply it.
+    eager_config: dict[str, Any]
+    #: Why each absent backend cannot, and what to do instead. ``"*"`` covers every backend not named.
+    refusals: dict[str, str]
+    why: str
+
+    def applies_to(self, backend: str) -> bool:
+        return backend in self.backends
+
+    def refusal(self, backend: str) -> str:
+        """Why this backend cannot apply the scheme, or an empty string when it can."""
+        if self.applies_to(backend):
+            return ""
+        return self.refusals.get(backend) or self.refusals.get("*", f"{self.name} is not available on {backend}")
+
+
+#: Every on-load scheme ``load_model(quantization=...)`` accepts and the sizer prices, by that name.
+#:
+#: The widths carry their scales. bitsandbytes' NF4 stores one fp32 absmax per 64-element block, which
+#: is 0.0625 bytes a weight on top of the half byte; vLLM's fp8 and LLM.int8 keep one scale per output
+#: channel, which rounds to nothing. FP8 on the eager backend is refused rather than priced:
+#: transformers' ``FineGrainedFP8Config`` needs compute 8.9 and reaches for a DeepGEMM Hub kernel, so a
+#: fit there would be a fit for two failure modes at once.
+QUANTIZATIONS: dict[str, OnLoadQuantization] = {
+    "fp8": OnLoadQuantization(
+        name="fp8",
+        width=1.0,
+        backends=VLLM_BACKENDS,
+        vllm_name="fp8",
+        eager_config={},
+        refusals={
+            "eager": (
+                "transformers' on-load FP8 (FineGrainedFP8Config) needs compute 8.9 and a DeepGEMM Hub "
+                "kernel; load a checkpoint that ships in FP8 with dtype='auto' instead, or use a vLLM backend"
+            ),
+        },
+        why="dynamic per-channel FP8 on every linear layer: half of bf16, no calibration",
+    ),
+    "bnb-8bit": OnLoadQuantization(
+        name="bnb-8bit",
+        width=1.0,
+        backends=("eager",),
+        vllm_name="",
+        eager_config={"load_in_8bit": True},
+        refusals={
+            "*": (
+                "vLLM's in-flight bitsandbytes is 4-bit only; use quantization='bnb-4bit' or quantization='fp8' there"
+            ),
+        },
+        why="LLM.int8 through bitsandbytes: the one 8-bit scheme with a backward pass",
+    ),
+    "bnb-4bit": OnLoadQuantization(
+        name="bnb-4bit",
+        width=0.5625,
+        backends=(*VLLM_BACKENDS, "eager"),
+        vllm_name="bitsandbytes",
+        eager_config={"load_in_4bit": True, "bnb_4bit_quant_type": "nf4"},
+        refusals={},
+        why="NF4 through bitsandbytes, on both backends, and with a backward pass on eager",
+    ),
+}
+
+
+def quantization_refusal(quantization: str, backend: str) -> str:
+    """Why ``quantization`` cannot be applied on ``backend``, or an empty string when it can.
+
+    An unknown name is refused too, naming the ones that exist, so a typo does not price as "as stored".
+    """
+    if not quantization:
+        return ""
+    scheme = QUANTIZATIONS.get(quantization)
+    if scheme is None:
+        return f"unknown quantization {quantization!r}; the sizer prices {', '.join(QUANTIZATIONS)}"
+    return scheme.refusal(backend)
+
+
+#: vLLM's spelling of each on-load scheme back to this module's, for the engine's own sizing in
+#: ``vllm_capture/static.py``: ``quantization="bitsandbytes"`` there is in-flight NF4, ``bnb-4bit`` here.
+VLLM_QUANTIZATION_NAMES: dict[str, str] = {
+    scheme.vllm_name: name for name, scheme in QUANTIZATIONS.items() if scheme.vllm_name
+}
+
+
+def narrow_linear_weights(base: int, *, narrow: float, stored: float, embedding_params: int) -> int:
+    """``base`` bytes of weights once every linear layer is narrowed from ``stored`` to ``narrow``.
+
+    **Neither vLLM nor bitsandbytes quantizes the embeddings or the unembed**, so those stay at the
+    stored width: on Llama-3.3-70B that is 2 x 1.05B parameters, about 3.9 GiB that "halve the
+    weights" would have counted away, and 6% of the answer. ``RedHatAI``'s FP8 export of the same
+    model shows the same shape from disk -- 67.68 GiB against 65.7 for 70.56B at one byte.
+
+    A scheme no narrower than the checkpoint changes nothing, which is the direction that OOMs.
+    """
+    if not base or narrow >= stored:
+        return int(base)
+    embeddings = min(max(int(embedding_params * stored), 0), int(base))
+    return int((int(base) - embeddings) * (narrow / stored) + embeddings)
+
+
+def on_load_weight_bytes(facts: ModelMemoryFacts, dtype: str, quantization: str, *, dequantizes: bool) -> int:
+    """Weight bytes on the device once ``quantization`` has narrowed the checkpoint at load.
+
+    Starts from :meth:`WeightBytes.bytes_for_load`, which is the stored answer, and hands it to
+    :func:`narrow_linear_weights`. A checkpoint that already ships quantized is returned as stored:
+    a quantizer does not narrow what is already narrower, and :func:`estimate` says so in a warning.
+    """
+    weights = facts.weights
+    base = weights.bytes_for_load(dtype, dequantizes=dequantizes)
+    scheme = QUANTIZATIONS.get(quantization) if quantization else None
+    if scheme is None or weights.is_quantized:
+        return base
+    stored = dtype_bytes(dtype if dtype not in ("auto", "", None) else weights.stored_dtype)
+    embedding_params = facts.vocab_size * facts.d_model * (1 if facts.tied_embeddings else 2)
+    return narrow_linear_weights(base, narrow=scheme.width, stored=stored, embedding_params=embedding_params)
+
+
 def logical_param_count(elements_by_dtype: dict[str, int], quant_method: str = "", expert_dtype: str = "") -> int:
     """Logical parameters, unpacking whatever the containers hold.
 
@@ -700,7 +855,7 @@ def logical_param_count(elements_by_dtype: dict[str, int], quant_method: str = "
     An unquantized checkpoint is unaffected: its buckets are all float, so containers and parameters
     are the same thing.
     """
-    widths = [w for w in (_scheme_width(quant_method), _scheme_width(expert_dtype)) if w is not None]
+    widths = [w for w in (scheme_width(quant_method), scheme_width(expert_dtype)) if w is not None]
     native = min(widths) if widths else None
     total = 0
     for tag, count in elements_by_dtype.items():
@@ -805,7 +960,7 @@ class WeightBytes:
         if self.is_quantized:
             if not dequantizes:
                 return self.on_disk_bytes
-            native = _scheme_width(self.quant_method)
+            native = scheme_width(self.quant_method)
             if native is not None and wanted <= native:
                 # Asking for the width it is already stored at, or narrower than transformers will
                 # give you: the checkpoint is served natively and the file size stands.
@@ -1015,7 +1170,7 @@ def _scheme_from_headers(elements_by_dtype: dict[str, int]) -> str:
     Requires the container buckets to hold a **majority** of the elements, so a small integer
     side-table cannot make a dense model look packed. Distinguishing 4-bit from 8-bit is then the
     presence of fp8 scales beside a byte payload, which is what every 4-bit export on the Hub looks
-    like; both answers are labels for :func:`_scheme_width`, not claims about a specific vendor
+    like; both answers are labels for :func:`scheme_width`, not claims about a specific vendor
     format, so ``"nvfp4"`` here means "packed two-to-a-byte" rather than NVIDIA's exact encoding.
     """
     total = sum(elements_by_dtype.values())
@@ -1267,6 +1422,10 @@ class ModelMemoryFacts:
     #: NVIDIA's FP4 exports. Not the caller's ``kv_cache_dtype``: this is a property of the weights on
     #: disk, and vLLM honours it whether or not anyone asked. See :func:`hub_kv_quant_algo`.
     kv_quant_algo: str = ""
+    #: Whether the unembed shares the embedding matrix. Read by :func:`on_load_weight_bytes`, since
+    #: both stay at the model dtype under every on-load scheme and a tied pair is one matrix, not two.
+    #: False when unknown, which charges the second matrix -- the direction that does not OOM.
+    tied_embeddings: bool = False
 
     @property
     def kv_width(self) -> int:
@@ -1623,6 +1782,10 @@ class WorkloadSpec:
     #: default is ``"float32"``, not ``"auto"``, which doubles a bf16 checkpoint -- a sizer should
     #: say so rather than reproduce it silently.
     dtype: str = "auto"
+    #: An on-load scheme from :data:`QUANTIZATIONS` (``"fp8"``, ``"bnb-4bit"``, ...), or empty for
+    #: as stored. Not a dtype: vLLM rejects ``dtype="fp8"`` and transformers would store fp8 with no
+    #: kernel behind it, which is why this is its own field and its own ``load_model`` argument.
+    quantization: str = ""
     #: KV cache dtype. ``"auto"`` follows the model dtype.
     kv_cache_dtype: str = "auto"
     max_model_len: int = 0
@@ -2054,13 +2217,42 @@ def estimate(
         )
 
     # Only the eager backend expands a quantized checkpoint to the requested dtype; vLLM reads the same
-    # argument as an activation dtype and serves the packed weights. See `bytes_for_load`.
-    weights_total = facts.weights.bytes_for_load(spec.dtype, dequantizes=spec.backend == "eager")
+    # argument as an activation dtype and serves the packed weights. See `bytes_for_load`. An on-load
+    # scheme then narrows the linear layers and nothing else -- see `on_load_weight_bytes`.
+    refused = quantization_refusal(spec.quantization, spec.backend)
+    weights_total = on_load_weight_bytes(
+        facts, spec.dtype, "" if refused else spec.quantization, dequantizes=spec.backend == "eager"
+    )
+    # True when the scheme took effect: the checkpoint was wider than it, and the backend runs it.
+    narrowed = weights_total < facts.weights.bytes_for_load(spec.dtype, dequantizes=spec.backend == "eager")
     if not weights_total:
         warnings.append(
             "weight bytes are unknown, so every figure below is only the non-weight terms; "
             "pass hf_model_id or a config to model_memory_facts()"
         )
+    if refused:
+        warnings.append(f"quantization={spec.quantization!r} is refused on backend={spec.backend!r}: {refused}")
+    elif spec.quantization and facts.weights.is_quantized:
+        warnings.append(
+            f"quantization={spec.quantization!r} changes nothing: this checkpoint already ships as "
+            f"{facts.weights.quant_method}, and a quantizer does not narrow what is already narrower. "
+            f"Priced as stored"
+        )
+    elif spec.quantization == "fp8" and not gpu.supports_fp8:
+        warnings.append(
+            f"{gpu.name} has no FP8 tensor cores, so quantization='fp8' runs weight-only through the "
+            f"Marlin kernel: the same memory, but slower than on Ada/Hopper or newer"
+        )
+    if (
+        spec.backend == "eager"
+        and spec.dtype in ("float16", "fp16", "half")
+        and (spec.attn_implementation or "eager") == "eager"
+        and facts.architecture in FP16_EAGER_OVERFLOW_ARCHS
+    ):
+        warnings.append(
+            f"{facts.architecture} overflows to NaN in float16 under attn_implementation='eager'; "
+            f"use dtype='bfloat16' or attn_implementation='sdpa'"
+        )
     if facts.weights.is_quantized:
         dequantized = facts.weights.dequantized_bytes() or 0
         if not gpu.supports_mxfp4_kernels and "fp4" in facts.weights.quant_method:
@@ -2103,7 +2295,7 @@ def estimate(
                 "weights",
                 per_card_weights,
                 "eager",
-                f"{facts.weights.param_count / 1e9:.1f}B params at {spec.dtype}"
+                _weights_note(spec, facts, refused)
                 + (f", spread over {tp} GPUs" if tp > 1 else "")
                 + f" [{facts.weights.source}]",
             )
@@ -2126,8 +2318,9 @@ def estimate(
 
         total = sum(term.bytes for term in terms)
         headroom = gpu.total_bytes - total
-        fits = headroom >= 0 and facts.trunk_dims_known
-        if not fits and facts.trunk_dims_known:
+        # A refused quantization is a configuration that cannot start, whatever the arithmetic says.
+        fits = headroom >= 0 and facts.trunk_dims_known and not refused
+        if not fits and facts.trunk_dims_known and not refused:
             advice.extend(_eager_advice(activation, spec, facts))
         return MemoryEstimate(
             spec=spec,
@@ -2197,6 +2390,16 @@ def estimate(
     # Inside the pool. The CUDA context is here rather than outside because vLLM's budget is measured
     # against what the process is ALREADY using -- see CALIBRATION["cuda_context_gib"].
     terms.append(MemoryTerm("cuda_context", context, "pool", "process CUDA context, charged against vLLM's budget"))
+    quant_charge = int(_cal("quant_on_load_gib") * GIB) if narrowed else 0
+    if quant_charge:
+        terms.append(
+            MemoryTerm(
+                "quant_on_load",
+                quant_charge,
+                "pool",
+                f"what vLLM holds for quantizing to {spec.quantization} at load, past the tensors",
+            )
+        )
     if reserved_inside:
         terms.append(
             MemoryTerm(
@@ -2213,7 +2416,7 @@ def estimate(
             "weights",
             per_card_weights,
             "pool",
-            f"{facts.weights.param_count / 1e9:.1f}B params at {spec.dtype}"
+            _weights_note(spec, facts, refused)
             + (f", sharded over TP={tp}" if tp > 1 else "")
             + f" [{facts.weights.source}]",
         )
@@ -2263,7 +2466,7 @@ def estimate(
 
     pool_available = int(spec.gpu_memory_utilization * gpu.total_bytes)
     outside_needed = overshoot + frag + reserved_outside
-    pool_needed = context + reserved_inside + per_card_weights + buffers + graphs + kv_floor
+    pool_needed = context + quant_charge + reserved_inside + per_card_weights + buffers + graphs + kv_floor
 
     # Both constraints have to hold, and they fail differently -- see the module docstring.
     pool_headroom = pool_available - pool_needed
@@ -2275,15 +2478,21 @@ def estimate(
     # here; the warning above says which it is. An all-recurrent trunk reaches the same zero by a
     # different road -- there really is no cache -- and is refused for the same reason: what it holds
     # instead is a state pool nothing here prices.
-    fits = pool_headroom >= 0 and outside_headroom >= 0 and facts.trunk_dims_known and bool(facts.kv_caching_layers)
+    fits = (
+        pool_headroom >= 0
+        and outside_headroom >= 0
+        and facts.trunk_dims_known
+        and bool(facts.kv_caching_layers)
+        and not refused
+    )
 
-    kv_room = max(pool_available - context - reserved_inside - per_card_weights - buffers - graphs, 0)
+    kv_room = max(pool_available - context - quant_charge - reserved_inside - per_card_weights - buffers - graphs, 0)
     per_token = (
         kv_bytes_for_context(facts, spec.max_model_len, kv_dtype=spec.kv_cache_dtype, model_dtype=spec.dtype) / shards
     )
     kv_capacity = int(kv_room / (per_token / spec.max_model_len)) if per_token > 0 else 0
 
-    if not fits:
+    if not fits and not refused:
         advice.extend(
             _vllm_advice(
                 spec=spec,
@@ -2315,6 +2524,15 @@ def estimate(
     )
 
 
+def _weights_note(spec: WorkloadSpec, facts: ModelMemoryFacts, refused: str) -> str:
+    """The first clause of the weights term: what is loaded, at what width."""
+    note = f"{facts.weights.param_count / 1e9:.1f}B params at {spec.dtype}"
+    if spec.quantization and not refused and not facts.weights.is_quantized:
+        stored = spec.dtype if spec.dtype != "auto" else (facts.weights.stored_dtype or "bfloat16")
+        note += f", {spec.quantization} on load (embeddings stay {stored})"
+    return note
+
+
 def _eager_note(name: str, spec: WorkloadSpec, facts: ModelMemoryFacts) -> str:
     if name == "logits":
         return f"{spec.batch_size}x{spec.seq_len} x vocab {facts.vocab_size:,}, plus the fp32 upcast"
@@ -2394,9 +2612,14 @@ def _vllm_advice(
             out.append("dtype='bfloat16' halves the weights, which are the largest term in the pool")
         if weights > gpu.total_bytes * 0.7:
             need = weights / (gpu.total_bytes * 0.6)
+            narrower = (
+                "quantization='fp8' halves the linear layers on load"
+                if not facts.weights.is_quantized and not spec.quantization
+                else "use a quantized checkpoint"
+            )
             out.append(
                 f"the weights alone are {weights / GIB:.1f} GiB of a {gpu.total_gib:.1f} GiB card; "
-                f"num_gpus={max(2, int(need) + 1)} shards them, or use a quantized checkpoint"
+                f"num_gpus={max(2, int(need) + 1)} shards them, or {narrower}"
             )
         if spec.graphs_on:
             out.append(
@@ -2477,6 +2700,7 @@ def model_memory_facts(
         max_position_embeddings=int(getattr(cfg, "max_position_embeddings", 0) or 0),
         architecture=f.architecture,
         kv_quant_algo=hub_kv_quant_algo(hf_model_id, token),
+        tied_embeddings=f.tied_embeddings,
     )
 
 
@@ -2507,6 +2731,7 @@ def fit(
     reservations: Reservations | None = None,
     max_model_len: int = 0,
     dtype: str = "auto",
+    quantization: str = "",
     kv_cache_dtype: str = "auto",
     num_gpus: int = 1,
     static_sites: int = 0,
@@ -2521,6 +2746,9 @@ def fit(
 ) -> tuple[WorkloadSpec, MemoryEstimate] | None:
     """The largest configuration of this shape that fits, or None when none does.
 
+    A ``quantization`` the backend cannot apply returns None before any ladder is walked: the estimate
+    would price the refusal as a warning and refuse to fit, so every rung would come back False.
+
     Solves in the order the constraints actually bind:
 
     1. **Utilization.** Everything outside the pool is fixed by the card and the reservations, so the
@@ -2537,7 +2765,7 @@ def fit(
     """
     res = reservations or Reservations()
 
-    if not facts.trunk_dims_known:
+    if not facts.trunk_dims_known or quantization_refusal(quantization, backend):
         # Short-circuit what would happen anyway. `estimate` refuses to report `fits` on a model whose
         # dims are unknown, so every rung of every ladder below would come back False; returning here
         # just saves walking them. Callers wanting to tell "cannot size" from "does not fit" should
@@ -2566,6 +2794,7 @@ def fit(
             spec = WorkloadSpec(
                 backend="eager",
                 dtype=dtype,
+                quantization=quantization,
                 num_gpus=num_gpus,
                 batch_size=batch_size,
                 seq_len=prompt,
@@ -2611,6 +2840,7 @@ def fit(
             spec = WorkloadSpec(
                 backend=backend,
                 dtype=dtype,
+                quantization=quantization,
                 kv_cache_dtype=kv_cache_dtype,
                 max_model_len=context,
                 # A prefill batch wider than the context is waste: nothing can fill it.