tempest-fastapi-sdk 0.303.0__tar.gz → 0.303.1__tar.gz

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.1}/CHANGELOG.md

@@ -7,6 +7,48 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.303.1] — 2026-10-04
+
+O planejador de hardware passa a dimensionar CPU na precisão que o gerador
+de fato carrega (`float32`), e o offload quantizado que ele sugere fica na
+CPU mesmo com GPU visível.
+
+### Changed
+
+- **`can_run(dtype=...)` agora tem default `None`**, que dimensiona na
+  precisão que o `TextGenerator(dtype="auto")` carrega no device escolhido:
+  `bfloat16` em CUDA/MPS (igual a antes), **`float32` em CPU** (antes
+  `bfloat16`). `dtype=` explícito continua valendo. `recommend()` tenta a
+  precisão nativa → `int8` → `int4`; em GPU a ordem é a mesma de antes. O
+  `CapacityReport` não mudou de forma; muda o `dtype`/`estimated_bytes`
+  que ele traz para CPU.
+- A `suggestion` de quantizar em CPU acrescenta que o bitsandbytes em CPU
+  decodifica mais devagar que `float32` e aponta o `OllamaGenerator`
+  (GGUF) como o caminho de CPU. Medido no Qwen2.5-3B: `generate()` de até
+  16 tokens em ~9 s com int8/int4 contra ~3 s em `float32`.
+
+### Fixed
+
+- **O planejador subdimensionava CPU pela metade.** Sem `dtype=`,
+  `can_run`/`recommend` estimavam CPU em `bfloat16` enquanto o gerador
+  carrega `float32`: para o Qwen2.5-0.5B a estimativa era 1,24 GB e o RSS
+  medido após o `load()` foi 2,13 GB (transformers 4.57.6) / 2,12 GB
+  (5.18.0); para o 3B, 7,71 GB contra 12,51 / 12,48 GB. Agora a estimativa
+  é 2,47 GB / 15,43 GB.
+- A `suggestion` em CPU não propõe mais "Quantize to bfloat16" (não é
+  quantização, e o `TextGenerator` recusa `quantization="bfloat16"`).
+- **`TextGenerator(device="cpu", quantization="int8"|"int4")` carregava na
+  GPU** quando havia CUDA: o caminho quantizado passava sempre
+  `device_map="auto"`, e o `accelerate` punha os pesos em `cuda:0`
+  enquanto `gen.device` dizia `"cpu"` — o offload que o `can_run` sugere
+  era irrealizável. Agora usa `device_map="cpu"` quando o device é CPU
+  (medido com CUDA visível, Qwen2.5-0.5B, transformers 4.57.6 +
+  bitsandbytes 0.50.2: todos os parâmetros em `cpu` em 6/6 loads, N=3 por
+  int8/int4; o código anterior pôs o int4 em `cuda:0`).
+- O exemplo de `recommend()` em `genai-examples` lia `rec.dtype.value`,
+  que levanta `AttributeError`: o `CapacityReport` guarda o valor do enum
+  (`use_enum_values`), então `rec.dtype` já é a string.
+
 ## [0.303.0] — 2026-10-03
 
 Auditoria da documentação contra o código entregue: 150 afirmações da doc

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tempest-fastapi-sdk
-Version: 0.303.0
+Version: 0.303.1
 Summary: Shared FastAPI building blocks: base schemas, ORM model, async repository, exceptions, pagination and settings — the conventions used across Tempest projects.
 Project-URL: Homepage, https://github.com/mauriciobenjamin700/tempest-fastapi-sdk
 Project-URL: Repository, https://github.com/mauriciobenjamin700/tempest-fastapi-sdk

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.1}/SHIPPED.md

@@ -613,7 +613,9 @@ The SDK currently covers (Sep 2025+, post-v0.31.x):
   in slices. **Shipped (v0.96):** hardware capacity check — `probe_hardware`
   → `HardwareInfo` (CPU/RAM/CUDA-VRAM/MPS/disk, degrades without
   psutil/torch), `can_run`/`recommend` → `CapacityReport` (fits? device,
-  estimate vs available, suggestion to quantize/offload), `estimate_model_bytes`/
+  estimate vs available, suggestion to quantize/offload; since v0.303.1 an unquantized load is sized at the precision
+  `TextGenerator(dtype="auto")` loads — `float32` on CPU, not `bfloat16`,
+  measured RSS in `docs/recipes/genai.md#em-cpu`), `estimate_model_bytes`/
   `bytes_per_param`/`fetch_num_params` (Hub metadata, no weight download),
   `ModelDtype`. Capacity fns import WITHOUT the extra. **Shipped (v0.98) —
   `TextGenerator`**: local causal LM (`generate`/`chat`/`stream` async via

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.1}/docs/genai-examples.en.md

@@ -16,7 +16,7 @@ import asyncio
 from tempest_fastapi_sdk.genai import TextGenerator, recommend
 
 MODEL = "Qwen/Qwen2.5-7B-Instruct"
-rec = recommend(model_id=MODEL)          # tries bf16 -> int8 -> int4
+rec = recommend(model_id=MODEL)          # native (bf16 GPU, fp32 CPU) -> int8 -> int4
 
 if not rec.fits:
     raise RuntimeError(rec.reason + " " + (rec.suggestion or ""))
@@ -24,7 +24,7 @@ if not rec.fits:
 gen = TextGenerator(
     MODEL,
     device=rec.device,
-    quantization=rec.dtype.value if rec.dtype.value in ("int8", "int4") else None,
+    quantization=rec.dtype if rec.dtype in ("int8", "int4") else None,
     idle_unload_seconds=300,             # threshold for unload_if_idle()
 )
 

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.1}/docs/genai-examples.md

@@ -16,7 +16,7 @@ import asyncio
 from tempest_fastapi_sdk.genai import TextGenerator, recommend
 
 MODEL = "Qwen/Qwen2.5-7B-Instruct"
-rec = recommend(model_id=MODEL)          # tenta bf16 -> int8 -> int4
+rec = recommend(model_id=MODEL)          # nativa (bf16 GPU, fp32 CPU) -> int8 -> int4
 
 if not rec.fits:
     raise RuntimeError(rec.reason + " " + (rec.suggestion or ""))
@@ -24,7 +24,7 @@ if not rec.fits:
 gen = TextGenerator(
     MODEL,
     device=rec.device,
-    quantization=rec.dtype.value if rec.dtype.value in ("int8", "int4") else None,
+    quantization=rec.dtype if rec.dtype in ("int8", "int4") else None,
     idle_unload_seconds=300,             # limiar do unload_if_idle()
 )
 

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.1}/docs/recipes/genai.en.md

@@ -37,9 +37,9 @@ Loading a too-large model ends in an OOM minutes into the download.
 `can_run` answers first:
 
 ```python
-from tempest_fastapi_sdk.genai import can_run, ModelDtype
+from tempest_fastapi_sdk.genai import can_run
 
-report = can_run(model_id="Qwen/Qwen2.5-7B-Instruct", dtype=ModelDtype.BFLOAT16)
+report = can_run(model_id="Qwen/Qwen2.5-7B-Instruct")
 
 if report.fits:
     print(f"OK on {report.device} — {report.headroom_pct:.0f}% headroom")
@@ -53,9 +53,15 @@ else:
 concrete `suggestion` when it doesn't fit (quantize, offload to CPU, or
 pick a smaller model).
 
+Without `dtype=`, `can_run` sizes the precision `TextGenerator(dtype="auto")`
+loads on the chosen device: `bfloat16` on CUDA/MPS, `float32` on CPU. Pass
+`dtype=` to plan for something else — an explicit `dtype=` on the
+generator, or `ModelDtype.INT8`/`INT4` for `quantization=`.
+
 !!! tip "Let the SDK pick the precision"
-    `recommend(...)` tries `bfloat16` → `int8` → `int4` on the best
-    available device and returns the **first** config that fits:
+    `recommend(...)` tries the native precision of the best available
+    device (`bfloat16` on a GPU, `float32` on CPU) → `int8` → `int4` and
+    returns the **first** config that fits:
 
     ```python
     from tempest_fastapi_sdk.genai import recommend
@@ -64,6 +70,56 @@ pick a smaller model).
     print(best.device, best.dtype, best.fits)   # e.g. cuda int8 True
     ```
 
+### On CPU
+
+Up to 0.303.0 the planner sized every unquantized load at `bfloat16`, but
+on CPU `TextGenerator` loads `float32` — twice the bytes. `recommend()`
+said "fits" for a model that did not. CPU is now sized at `float32`, and
+the rung below it is `int8` (never `bfloat16`, which is not a
+quantization).
+
+Measured with the GPU hidden (`CUDA_VISIBLE_DEVICES=`), i9-13900F, 62 GB
+RAM, WSL2, torch 2.14, bitsandbytes 0.50.2, N=3 per row. "Measured" is the
+process RSS growth from before to after `load()`, median; transformers
+4.57.6 (the floor) / 5.18.0 (the newest on the day):
+
+| Model | Precision | Estimate | Measured (4.57.6 / 5.18.0) |
+| --- | --- | --- | --- |
+| Qwen2.5-0.5B-Instruct | `float32` (was: `bfloat16`) | 2.47 GB (was: 1.24 GB) | 2.13 GB / 2.12 GB |
+| Qwen2.5-0.5B-Instruct | `int8` | 0.62 GB | 0.89 GB / 1.29 GB |
+| Qwen2.5-0.5B-Instruct | `int4` | 0.37 GB | 1.02 GB / 1.62 GB |
+| Qwen2.5-3B-Instruct | `float32` (was: `bfloat16`) | 15.43 GB (was: 7.71 GB) | 12.51 GB / 12.48 GB |
+| Qwen2.5-3B-Instruct | `int8` | 3.86 GB | 3.67 GB / 8.54 GB |
+| Qwen2.5-3B-Instruct | `int4` | 2.31 GB | 3.25 GB / 8.16 GB |
+
+What the table says:
+
+- **`float32` is what CPU loads**, and the new estimate covers the RSS
+  after the load. The peak **during** the load goes past it: 3.12 GB for
+  the 0.5B and 14.71 GB / 18.66 GB for the 3B.
+- **bitsandbytes int8/int4 load and generate on CPU** with bitsandbytes
+  0.50.2 — no error in the table's 24 quantized runs.
+- **The quantized estimate can be optimistic.** For the 0.5B the measured
+  int4 RSS was 2.7× to 4.4× above it (embeddings are not quantized, and
+  weigh more in a small model). For the 3B on transformers 4.57.6, int8
+  came in below the estimate (3.67 against 3.86 GB) and int4 above it
+  (3.25 against 2.31 GB). On transformers 5.18.0 the quantized 3B
+  load leaves the process with 5.65 GB of **file-backed** RSS (`RssFile`,
+  against 0.36 GB on 4.57.6); the whole process's anonymous memory was
+  3.33 GB (int8) and 2.95 GB (int4).
+- **bitsandbytes on CPU is slower than `float32`.** On the 3B, a
+  `generate()` capped at 16 tokens took ~9 s at int8/int4 against ~3 s at
+  `float32`. That is why a CPU quantization `suggestion` points at the CPU
+  path the SDK already ships:
+
+!!! tip "On CPU, GGUF through Ollama"
+    llama.cpp (build 687e778, `-ngl 0`, context 4096) serving the Q4_K_M
+    GGUF that Ollama 0.30.11 pulls for `qwen2.5:3b` sat at 3.60 GB RSS
+    (1.61 GB anonymous + 1.99 GB file-backed) and generated 24 to 27
+    tokens/s; `qwen2.5:0.5b` sat at 0.66 GB and 107 to 134 tokens/s (N=3
+    each). Use [`OllamaGenerator`](#ollama-backend) — memory is in the same
+    range as bitsandbytes int4; the measured advantage is speed.
+
 ## Probing the hardware
 
 ```python

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.1}/docs/recipes/genai.md

@@ -37,9 +37,9 @@ Carregar um modelo grande demais termina num OOM minutos depois do
 download começar. `can_run` responde antes:
 
 ```python
-from tempest_fastapi_sdk.genai import can_run, ModelDtype
+from tempest_fastapi_sdk.genai import can_run
 
-report = can_run(model_id="Qwen/Qwen2.5-7B-Instruct", dtype=ModelDtype.BFLOAT16)
+report = can_run(model_id="Qwen/Qwen2.5-7B-Instruct")
 
 if report.fits:
     print(f"OK em {report.device} — {report.headroom_pct:.0f}% de folga")
@@ -53,9 +53,16 @@ O `CapacityReport` traz: `fits`, `device` (`cuda`/`mps`/`cpu`),
 `suggestion` concreta quando não cabe (quantizar, offload pra CPU, ou
 trocar de modelo).
 
+Sem `dtype=`, o `can_run` dimensiona na precisão que o
+`TextGenerator(dtype="auto")` carrega no device escolhido: `bfloat16` em
+CUDA/MPS, `float32` em CPU. Passe `dtype=` para planejar outra coisa — um
+`dtype=` explícito no gerador, ou `ModelDtype.INT8`/`INT4` para
+`quantization=`.
+
 !!! tip "Deixe o SDK escolher a precisão"
-    `recommend(...)` tenta `bfloat16` → `int8` → `int4` no melhor device
-    disponível e devolve a **primeira** config que cabe:
+    `recommend(...)` tenta a precisão nativa do melhor device disponível
+    (`bfloat16` em GPU, `float32` em CPU) → `int8` → `int4` e devolve a
+    **primeira** config que cabe:
 
     ```python
     from tempest_fastapi_sdk.genai import recommend
@@ -64,6 +71,58 @@ trocar de modelo).
     print(best.device, best.dtype, best.fits)   # ex.: cuda int8 True
     ```
 
+### Em CPU
+
+Até a 0.303.0 o planejador dimensionava todo load sem quantização em
+`bfloat16`, mas em CPU o `TextGenerator` carrega em `float32` — o dobro
+de bytes. O `recommend()` dizia "cabe" para um modelo que não cabia.
+Agora a CPU é dimensionada em `float32`, e o degrau abaixo é `int8` (nunca
+`bfloat16`, que não é quantização).
+
+Medido com o GPU escondido (`CUDA_VISIBLE_DEVICES=`), i9-13900F, 62 GB de
+RAM, WSL2, torch 2.14, bitsandbytes 0.50.2, N=3 por linha. "Medido" é o
+crescimento do RSS do processo entre antes e depois do `load()`, na
+mediana; transformers 4.57.6 (o piso) / 5.18.0 (a mais nova no dia):
+
+| Modelo | Precisão | Estimativa | Medido (4.57.6 / 5.18.0) |
+| --- | --- | --- | --- |
+| Qwen2.5-0.5B-Instruct | `float32` (antes: `bfloat16`) | 2,47 GB (antes: 1,24 GB) | 2,13 GB / 2,12 GB |
+| Qwen2.5-0.5B-Instruct | `int8` | 0,62 GB | 0,89 GB / 1,29 GB |
+| Qwen2.5-0.5B-Instruct | `int4` | 0,37 GB | 1,02 GB / 1,62 GB |
+| Qwen2.5-3B-Instruct | `float32` (antes: `bfloat16`) | 15,43 GB (antes: 7,71 GB) | 12,51 GB / 12,48 GB |
+| Qwen2.5-3B-Instruct | `int8` | 3,86 GB | 3,67 GB / 8,54 GB |
+| Qwen2.5-3B-Instruct | `int4` | 2,31 GB | 3,25 GB / 8,16 GB |
+
+O que a tabela diz:
+
+- **`float32` é o que a CPU carrega**, e a estimativa nova cobre o RSS
+  depois do load. O pico **durante** o load passa dela: 3,12 GB no 0,5B
+  e 14,71 GB / 18,66 GB no 3B.
+- **int8/int4 via bitsandbytes carregam e geram em CPU** com o
+  bitsandbytes 0.50.2 — nenhum erro nas 24 execuções quantizadas da
+  tabela.
+- **A estimativa quantizada pode ser otimista.** No 0,5B o RSS medido do int4
+  ficou 2,7× a 4,4× acima dela (os embeddings não são quantizados, e
+  pesam mais num modelo pequeno). No 3B, no transformers 4.57.6, o int8
+  ficou abaixo da estimativa (3,67 contra 3,86 GB) e o int4 acima (3,25
+  contra 2,31 GB). No transformers 5.18.0 o load
+  quantizado do 3B deixa o processo com 5,65 GB de RSS **mapeado de
+  arquivo** (`RssFile`, contra 0,36 GB no 4.57.6); a memória anônima do
+  processo inteiro ficou em 3,33 GB (int8) e 2,95 GB (int4).
+- **bitsandbytes em CPU é mais lento que `float32`.** No 3B, um
+  `generate()` limitado a 16 tokens levou ~9 s em int8/int4 contra ~3 s
+  em `float32`. Por isso a `suggestion` de quantizar em CPU aponta o
+  caminho de CPU que já está no SDK:
+
+!!! tip "Em CPU, GGUF pelo Ollama"
+    O llama.cpp (build 687e778, `-ngl 0`, contexto 4096) servindo o GGUF
+    Q4_K_M que o Ollama 0.30.11 baixa para `qwen2.5:3b` ficou em 3,60 GB
+    de RSS (1,61 GB anônima + 1,99 GB de arquivo) e gerou 24 a 27
+    tokens/s; o `qwen2.5:0.5b` ficou em 0,66 GB e 107 a 134 tokens/s
+    (N=3 cada). Use o [`OllamaGenerator`](#backend-ollama) — a memória é
+    da mesma ordem do int4 do bitsandbytes; a vantagem medida é a
+    velocidade.
+
 ## Sondando o hardware
 
 ```python

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.1}/docs/recipes/models.en.md

@@ -74,7 +74,8 @@ One enum per task:
 
 !!! tip "Do not memorize VRAM — ask"
     `recommend()` measures the host and picks the precision that fits,
-    from bf16 down to int4:
+    from the device's native one (bf16 on a GPU, fp32 on CPU) down to
+    int4:
 
     ```python
     from tempest_fastapi_sdk.genai import TextModel, recommend

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.1}/docs/recipes/models.md

@@ -73,8 +73,8 @@ Um enum por tarefa:
     **[Pesos de modelos »](model-weights.md#onde-os-pesos-ficam-e-por-que-a-2a-execucao-e-instantanea)**.
 
 !!! tip "Não decore VRAM — pergunte"
-    `recommend()` mede a máquina e escolhe a precisão que cabe, de bf16
-    até int4:
+    `recommend()` mede a máquina e escolhe a precisão que cabe, da nativa
+    do device (bf16 em GPU, fp32 em CPU) até int4:
 
     ```python
     from tempest_fastapi_sdk.genai import TextModel, recommend

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "tempest-fastapi-sdk"
-version = "0.303.0"
+version = "0.303.1"
 description = "Shared FastAPI building blocks: base schemas, ORM model, async repository, exceptions, pagination and settings — the conventions used across Tempest projects."
 readme = "README.md"
 requires-python = ">=3.11"

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.1}/tempest_fastapi_sdk/__init__.py

@@ -1842,7 +1842,7 @@ from tempest_fastapi_sdk.websockets import (
     make_websocket_router as make_websocket_router,
 )
 
-__version__: str = "0.303.0"
+__version__: str = "0.303.1"
 
 from tempest_fastapi_sdk.api import (
     OAuthAudienceVerifier as OAuthAudienceVerifier,

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.1}/tempest_fastapi_sdk/genai/hardware.py

@@ -37,6 +37,19 @@ _BYTES_PER_PARAM: dict[ModelDtype, float] = {
     ModelDtype.INT4: 0.6,
 }
 
+_CPU_QUANTIZATION_NOTE: str = (
+    " bitsandbytes on CPU decodes slower than float32; a GGUF build served"
+    " through OllamaGenerator is the faster CPU path."
+)
+"""Appended to a CPU quantization suggestion.
+
+Measured on Qwen2.5-3B-Instruct with the GPU hidden (torch 2.14,
+bitsandbytes 0.50.2, transformers 4.57.6 and 5.18.0, N=3 each): a
+``generate()`` capped at 16 new tokens took ~9 s with bitsandbytes int8 or
+int4 against ~3 s at float32. llama.cpp on the Q4_K_M GGUF that Ollama
+serves for the same model decoded ~24-27 tokens/s.
+"""
+
 # Inference needs more than the weights (activations, KV cache, CUDA
 # context). Scale the raw weight size by this to plan with headroom.
 _INFERENCE_OVERHEAD: float = 1.25
@@ -266,11 +279,31 @@ def _pick_device(hardware: HardwareInfo) -> str:
     return "cpu"
 
 
+def _native_dtype(device: str) -> ModelDtype:
+    """Return the precision an unquantized model is loaded in on ``device``.
+
+    This is the single source of truth shared with
+    :func:`~tempest_fastapi_sdk.genai.auto_dtype_name`, which is what
+    ``TextGenerator(dtype="auto")`` loads with, so the planner sizes the
+    weights the generator will actually hold. CPU has no fast
+    half-precision path, so it loads ``float32`` — twice the bytes of the
+    ``bfloat16`` used on CUDA and MPS.
+
+    Args:
+        device (str): The concrete device (``"cuda"``, ``"mps"`` or
+            ``"cpu"``).
+
+    Returns:
+        ModelDtype: ``FLOAT32`` on CPU, ``BFLOAT16`` elsewhere.
+    """
+    return ModelDtype.FLOAT32 if device == "cpu" else ModelDtype.BFLOAT16
+
+
 def can_run(
     *,
     num_params: int | None = None,
     model_id: str | None = None,
-    dtype: ModelDtype = ModelDtype.BFLOAT16,
+    dtype: ModelDtype | None = None,
     device: str = "auto",
     hardware: HardwareInfo | None = None,
     token: str | None = None,
@@ -286,7 +319,11 @@ def can_run(
             precedence over ``model_id``.
         model_id (str | None): Hub id to look the parameter count up from
             when ``num_params`` is not given.
-        dtype (ModelDtype): The precision to plan for.
+        dtype (ModelDtype | None): The precision to plan for. ``None`` (the
+            default) plans for the precision ``TextGenerator(dtype="auto")``
+            loads on the chosen device: ``bfloat16`` on CUDA/MPS,
+            ``float32`` on CPU. Pass a value to plan for an explicit
+            ``TextGenerator(dtype=...)`` or ``quantization=...``.
         device (str): ``"auto"``, ``"cuda"``, ``"mps"`` or ``"cpu"``.
         hardware (HardwareInfo | None): Inject a snapshot (tests, or to
             reuse one probe); defaults to a fresh :func:`probe_hardware`.
@@ -311,28 +348,29 @@ def can_run(
         )
 
     chosen = _pick_device(hw) if device == "auto" else device
-    estimated = estimate_model_bytes(params, dtype)
+    planned = dtype if dtype is not None else _native_dtype(chosen)
+    estimated = estimate_model_bytes(params, planned)
     available = _device_capacity(hw, chosen)
     fits = estimated <= available
     headroom = ((available - estimated) / available * 100) if available else -100.0
 
     if fits:
         reason = (
-            f"~{estimated / 1e9:.1f} GB needed at {dtype.value} fits the "
+            f"~{estimated / 1e9:.1f} GB needed at {planned.value} fits the "
             f"~{available / 1e9:.1f} GB free on {chosen}."
         )
         suggestion = None
     else:
         reason = (
-            f"~{estimated / 1e9:.1f} GB needed at {dtype.value} exceeds the "
+            f"~{estimated / 1e9:.1f} GB needed at {planned.value} exceeds the "
             f"~{available / 1e9:.1f} GB free on {chosen}."
         )
-        suggestion = _suggest(hw, params, dtype, chosen)
+        suggestion = _suggest(hw, params, planned, chosen)
 
     return CapacityReport(
         fits=fits,
         device=chosen,
-        dtype=dtype,
+        dtype=planned,
         estimated_bytes=estimated,
         available_bytes=available,
         headroom_pct=round(headroom, 1),
@@ -347,18 +385,37 @@ def _suggest(
     dtype: ModelDtype,
     device: str,
 ) -> str:
-    """Return the best next step when a model doesn't fit as asked."""
-    order = [ModelDtype.BFLOAT16, ModelDtype.INT8, ModelDtype.INT4]
+    """Return the best next step when a model doesn't fit as asked.
+
+    The ladder starts at the device's native precision (see
+    :func:`_native_dtype`) and only goes down, so on CPU — which loads
+    ``float32`` — the first rung below is ``int8``, never ``bfloat16``:
+    ``bfloat16`` is not a quantization, and ``TextGenerator`` rejects it
+    as ``quantization=``.
+
+    Args:
+        hardware (HardwareInfo): The probed hardware.
+        num_params (int): The model's parameter count.
+        dtype (ModelDtype): The precision that did not fit.
+        device (str): The device it did not fit on.
+
+    Returns:
+        str: The suggested next step.
+    """
+    order = [_native_dtype(device), ModelDtype.INT8, ModelDtype.INT4]
     available = _device_capacity(hardware, device)
     for candidate in order:
         if bytes_per_param(candidate) >= bytes_per_param(dtype):
             continue
         if estimate_model_bytes(num_params, candidate) <= available:
-            return (
+            suggestion = (
                 f"Quantize to {candidate.value} (needs "
                 f"~{estimate_model_bytes(num_params, candidate) / 1e9:.1f} GB) "
                 f"to fit {device}."
             )
+            if device == "cpu" and candidate in (ModelDtype.INT8, ModelDtype.INT4):
+                suggestion += _CPU_QUANTIZATION_NOTE
+            return suggestion
     if (
         device == "cuda"
         and estimate_model_bytes(num_params, ModelDtype.INT4)
@@ -378,11 +435,15 @@ def recommend(
     hardware: HardwareInfo | None = None,
     token: str | None = None,
 ) -> CapacityReport:
-    """Pick the best precision that fits, from bf16 down to int4.
+    """Pick the best precision that fits, from the native one down to int4.
 
-    Tries ``bfloat16`` → ``int8`` → ``int4`` on the auto-selected device
-    and returns the first :class:`CapacityReport` that fits (or the int4
-    report when nothing fits, so the caller sees the closest option).
+    Tries the precision ``TextGenerator(dtype="auto")`` loads on the
+    auto-selected device (``bfloat16`` on CUDA/MPS, ``float32`` on CPU),
+    then ``int8``, then ``int4``, and returns the first
+    :class:`CapacityReport` that fits (or the int4 report when nothing
+    fits, so the caller sees the closest option). A report whose ``dtype``
+    is ``int8``/``int4`` maps to ``TextGenerator(quantization=...)``; any
+    other ``dtype`` is what ``dtype="auto"`` already loads.
 
     Args:
         num_params (int | None): The model's parameter count.
@@ -394,21 +455,23 @@ def recommend(
     Returns:
         CapacityReport: The recommended configuration.
 
+    Raises:
+        ValueError: When neither ``num_params`` nor a resolvable
+            ``model_id`` is available.
+
     Notes:
         Two fallbacks are tried in order: a smaller precision on the same
         device first, since that keeps the model on the accelerator, and
         only then CPU RAM when the current device is a GPU.
     """
     hw = hardware or probe_hardware()
+    params = num_params
+    if params is None and model_id is not None:
+        params = fetch_num_params(model_id, token=token)
+    native = _native_dtype(_pick_device(hw))
     report = None
-    for dtype in (ModelDtype.BFLOAT16, ModelDtype.INT8, ModelDtype.INT4):
-        report = can_run(
-            num_params=num_params,
-            model_id=model_id,
-            dtype=dtype,
-            hardware=hw,
-            token=token,
-        )
+    for dtype in (native, ModelDtype.INT8, ModelDtype.INT4):
+        report = can_run(num_params=params, dtype=dtype, hardware=hw)
         if report.fits:
             return report
     assert report is not None

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.1}/tempest_fastapi_sdk/genai/text.py

@@ -31,7 +31,7 @@ from tempest_fastapi_sdk.genai.generation_cache import (
     GenerationCache,
     cached_generate,
 )
-from tempest_fastapi_sdk.genai.hardware import probe_hardware
+from tempest_fastapi_sdk.genai.hardware import _native_dtype, probe_hardware
 from tempest_fastapi_sdk.genai.hub import ModelRef
 from tempest_fastapi_sdk.genai.metrics import GenAIMetrics
 from tempest_fastapi_sdk.genai.schemas import (
@@ -469,9 +469,11 @@ def auto_dtype_name(device: str) -> str:
 
     Returns:
         str: ``"bfloat16"`` on CUDA/MPS, ``"float32"`` on CPU (which has
-        no fast half-precision path).
+        no fast half-precision path). The capacity planner
+        (:func:`~tempest_fastapi_sdk.genai.can_run`) sizes an unquantized
+        load from the same rule, so the two cannot drift.
     """
-    return "float32" if device == "cpu" else "bfloat16"
+    return _native_dtype(device).value
 
 
 def _require_transformers() -> tuple[Any, Any]:
@@ -985,6 +987,13 @@ class TextGenerator:
     def _build(self) -> None:
         """Load the tokenizer and weights; called once, under the load lock.
 
+        A quantized load pins ``device_map="cpu"`` when the resolved device
+        is CPU. ``device_map="auto"`` lets ``accelerate`` place the weights
+        on any GPU it sees, so on a host with CUDA a generator built with
+        ``device="cpu", quantization="int4"`` — the offload the capacity
+        planner suggests — would load onto ``cuda:0`` while ``self.device``
+        still said ``"cpu"``.
+
         Raises:
             ImportError: When the ``[genai]`` (or ``[genai-quant]``) extra
                 is missing.
@@ -996,7 +1005,7 @@ class TextGenerator:
             kwargs["quantization_config"] = transformers.BitsAndBytesConfig(
                 **{f"load_in_{bits}bit": True},
             )
-            kwargs["device_map"] = "auto"
+            kwargs["device_map"] = "cpu" if self.device == "cpu" else "auto"
         else:
             kwargs.update(precision_kwarg(getattr(torch, self.dtype.value)))
             kwargs["device_map"] = self.device if self.device != "cpu" else None