tempest-fastapi-sdk 0.303.0__tar.gz → 0.303.2__tar.gz

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.2}/CHANGELOG.md

@@ -7,6 +7,133 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.303.2] — 2026-10-04
+
+Três defeitos achados ao medir agente e planejador em CPU: resposta vazia
+do modelo deixa de contar como sucesso, a geração do Ollama não refaz mais
+`ReadTimeout` (que custava três timeouts), e o planejador de hardware para
+de tratar "RAM não medida" como "zero bytes livres" — no Linux, mede pelo
+`/proc/meminfo` mesmo sem `psutil`.
+
+### Added
+
+- **`RetryPolicy.retry_on_read_timeout`** (default `True`): com `False`, o
+  `HTTPClient` deixa de refazer `ReadTimeout` em `request()` e na abertura
+  de `stream()`, e continua refazendo `ConnectError` e os `retry_statuses`.
+  Quem usa `HTTPClient` direto não vê mudança.
+
+- **Flags de "medido" no planejador de hardware.** Campos novos, todos
+  `bool` com default `True` (sem quebra de tipo — os campos de bytes
+  continuam `int`): `HardwareInfo.ram_measured`,
+  `HardwareInfo.disk_measured` (o `disk_free_bytes` também virava `0` quando
+  o `shutil.disk_usage` levantava `OSError`) e
+  `CapacityReport.memory_measured`. Em GPU o veredito segue medido pela
+  VRAM; só a sugestão de offload para CPU, que depende da RAM, vira "a RAM
+  não foi medida" em vez de "o host é pequeno demais".
+
+### Fixed
+
+- **Geração mais longa que o timeout custava três timeouts.** O
+  `HTTPClient` que `OllamaGenerator`, `OllamaEmbedder` e
+  `OpenAICompatGenerator` criam refazia `ReadTimeout` até 3 vezes, mas o
+  Ollama aborta a geração quando o cliente desiste (cada tentativa aparece
+  no log do 0.30.11 como um `500` de exatamente o timeout), então nenhuma
+  tentativa terminava. Medido com `qwen2.5:3b` em CPU
+  (`options={"num_gpu": 0}`) e `timeout=10`: 31,54 s até a exceção, com 3
+  requisições no daemon (mediana, N=3); agora 10,01 s, com 1 (mediana,
+  N=5). Com o `timeout=120` padrão, a conta passa de ~6 min para ~2 min.
+  Esses clientes passam a usar `RetryPolicy(retry_on_read_timeout=False)`
+  quando `retry_policy=` não é dado; uma política explícita é usada como
+  veio, e um `http_client=` injetado mantém a dele.
+
+- **Resposta vazia do modelo não termina mais como sucesso.** Uma mensagem
+  sem texto (ou só com espaço em branco) e sem `tool_calls` virava
+  `StopReason.COMPLETED`, e a execução saía com `succeeded=True` e
+  `output == ""`. Agora termina no novo `StopReason.EMPTY_RESPONSE`
+  (`"empty_response"`), com `succeeded=False` — vale para `run`, `stream`,
+  a delegação (`agent_tool` marca `[stopped: empty_response]`), o
+  `run_until(..., until=succeeded)` (que passa a repetir a rodada) e o
+  `run_structured` (`parse_error="the run produced no answer"`, sem a
+  chamada de extração). O agente não repete a pergunta sozinho. Medido com
+  `OllamaGenerator("qwen2.5:0.5b", options={"num_gpu": 0})` no Ollama
+  0.30.11, meta que pede uma ferramenta e a resposta: antes, 58 de 200
+  execuções saíam `completed` com `output` vazio; depois, 73 de 200
+  terminaram `empty_response` e 0 de 200 saíram `succeeded` vazias (a taxa
+  oscilou entre 25% e 40% de um lote para outro). Com
+  `run_until(..., until=succeeded, max_rounds=3)`, 58 de 60 laços terminaram
+  aceitos.
+
+- **RAM desconhecida virava "sem memória livre".** Sem `psutil` (que vem
+  no `[metrics]`, não no `[genai]`), o `probe_hardware()` reportava
+  `ram_available_bytes=0` e o planejador lia isso como zero byte livre.
+  Medido na 0.303.1 numa venv limpa só com o pacote base, máquina de 62 GB,
+  `CUDA_VISIBLE_DEVICES=`: `recommend(num_params=500_000_000)` devolvia
+  `fits=False` em `int4`, `headroom_pct=-100.0` e *"use a smaller model or
+  add memory"*.
+- **No Linux a RAM é medida sem `psutil`**, lendo `MemTotal`/`MemAvailable`
+  do `/proc/meminfo` (o arquivo que o próprio `psutil` lê). Na mesma venv
+  limpa, o `recommend(num_params=500_000_000)` agora dá `fits=True` em
+  `float32`; o total lido bateu com o `free -b` (67 430 916 096 bytes) e,
+  no mesmo processo, os dois bytes bateram exatamente com o `psutil` 7.2.2.
+  Kernel sem `MemAvailable` (ou com `0`) conta como não medido, em vez de
+  aproximar por `MemFree + Buffers + Cached`. Limitação medida e
+  documentada: em container com `--memory=512m`, o `/proc/meminfo` (e o
+  `psutil`) mostram a RAM do host, e o planejador diz `fits=True` para um
+  load de 2,5 GB.
+- **Quando nenhuma fonte responde** (Windows/macOS sem `psutil`, ou Linux
+  com o `/proc/meminfo` ilegível), o relatório diz "não medido" em vez de
+  "não cabe": `memory_measured=False`, `dtype` na precisão nativa (sem
+  descer a escada de quantização), `headroom_pct=0.0`, e uma sugestão que
+  diz por plataforma como medir (*"on darwin measuring RAM requires psutil
+  (pip install 'tempest-fastapi-sdk[metrics]')"*). `fits` continua `False`
+  quando nada foi verificado, para um `if report.fits:` seguir
+  conservador.
+- `tempest model hardware` imprime a RAM lida do `/proc/meminfo` no Linux
+  sem `psutil`, e `ram total  : unknown (<motivo da plataforma>)` em vez de
+  `0.0 GB` quando nenhuma fonte responde.
+
+## [0.303.1] — 2026-10-04
+
+O planejador de hardware passa a dimensionar CPU na precisão que o gerador
+de fato carrega (`float32`), e o offload quantizado que ele sugere fica na
+CPU mesmo com GPU visível.
+
+### Changed
+
+- **`can_run(dtype=...)` agora tem default `None`**, que dimensiona na
+  precisão que o `TextGenerator(dtype="auto")` carrega no device escolhido:
+  `bfloat16` em CUDA/MPS (igual a antes), **`float32` em CPU** (antes
+  `bfloat16`). `dtype=` explícito continua valendo. `recommend()` tenta a
+  precisão nativa → `int8` → `int4`; em GPU a ordem é a mesma de antes. O
+  `CapacityReport` não mudou de forma; muda o `dtype`/`estimated_bytes`
+  que ele traz para CPU.
+- A `suggestion` de quantizar em CPU acrescenta que o bitsandbytes em CPU
+  decodifica mais devagar que `float32` e aponta o `OllamaGenerator`
+  (GGUF) como o caminho de CPU. Medido no Qwen2.5-3B: `generate()` de até
+  16 tokens em ~9 s com int8/int4 contra ~3 s em `float32`.
+
+### Fixed
+
+- **O planejador subdimensionava CPU pela metade.** Sem `dtype=`,
+  `can_run`/`recommend` estimavam CPU em `bfloat16` enquanto o gerador
+  carrega `float32`: para o Qwen2.5-0.5B a estimativa era 1,24 GB e o RSS
+  medido após o `load()` foi 2,13 GB (transformers 4.57.6) / 2,12 GB
+  (5.18.0); para o 3B, 7,71 GB contra 12,51 / 12,48 GB. Agora a estimativa
+  é 2,47 GB / 15,43 GB.
+- A `suggestion` em CPU não propõe mais "Quantize to bfloat16" (não é
+  quantização, e o `TextGenerator` recusa `quantization="bfloat16"`).
+- **`TextGenerator(device="cpu", quantization="int8"|"int4")` carregava na
+  GPU** quando havia CUDA: o caminho quantizado passava sempre
+  `device_map="auto"`, e o `accelerate` punha os pesos em `cuda:0`
+  enquanto `gen.device` dizia `"cpu"` — o offload que o `can_run` sugere
+  era irrealizável. Agora usa `device_map="cpu"` quando o device é CPU
+  (medido com CUDA visível, Qwen2.5-0.5B, transformers 4.57.6 +
+  bitsandbytes 0.50.2: todos os parâmetros em `cpu` em 6/6 loads, N=3 por
+  int8/int4; o código anterior pôs o int4 em `cuda:0`).
+- O exemplo de `recommend()` em `genai-examples` lia `rec.dtype.value`,
+  que levanta `AttributeError`: o `CapacityReport` guarda o valor do enum
+  (`use_enum_values`), então `rec.dtype` já é a string.
+
 ## [0.303.0] — 2026-10-03
 
 Auditoria da documentação contra o código entregue: 150 afirmações da doc

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tempest-fastapi-sdk
-Version: 0.303.0
+Version: 0.303.2
 Summary: Shared FastAPI building blocks: base schemas, ORM model, async repository, exceptions, pagination and settings — the conventions used across Tempest projects.
 Project-URL: Homepage, https://github.com/mauriciobenjamin700/tempest-fastapi-sdk
 Project-URL: Repository, https://github.com/mauriciobenjamin700/tempest-fastapi-sdk

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.2}/SHIPPED.md

@@ -286,7 +286,11 @@ The SDK currently covers (Sep 2025+, post-v0.31.x):
   (httpx wrapper with retry/backoff/circuit-breaker /
   `X-Request-ID` propagation), `async_retry` (v0.280.0 — applies a
   `RetryPolicy` to any coroutine, not just an HTTP call; `RetryPolicy`
-  moved to `utils/retry.py`, old import path kept; **`logger=` is typed
+  moved to `utils/retry.py`, old import path kept;
+  **`retry_on_read_timeout` since v0.303.2** — `False` stops retrying a
+  `ReadTimeout` while still retrying connect errors and `retry_statuses`;
+  the Ollama/OpenAI-compat generators and `OllamaEmbedder` default to it,
+  because a generation the server aborted is never finished by a retry; **`logger=` is typed
   `RetryLogger` since v0.281.0** — a `Protocol` with `warning`/`error`, so
   the `LogUtils` this SDK hands the service is accepted where
   `logging.Logger | None` rejected it, and `logger=logger.logger` is no
@@ -612,8 +616,15 @@ The SDK currently covers (Sep 2025+, post-v0.31.x):
   `[genai-quant]` = bitsandbytes) — `tempest_fastapi_sdk.genai`, delivered
   in slices. **Shipped (v0.96):** hardware capacity check — `probe_hardware`
   → `HardwareInfo` (CPU/RAM/CUDA-VRAM/MPS/disk, degrades without
-  psutil/torch), `can_run`/`recommend` → `CapacityReport` (fits? device,
-  estimate vs available, suggestion to quantize/offload), `estimate_model_bytes`/
+  psutil/torch; since v0.303.2 RAM falls back to
+  `/proc/meminfo` on Linux without psutil (host memory, not the cgroup
+  limit — same as psutil), and an unread RAM/disk is flagged
+  `ram_measured`/`disk_measured=False` instead of passing as `0` free, so
+  `CapacityReport.memory_measured=False` answers "not measured", not
+  "add memory"), `can_run`/`recommend` → `CapacityReport` (fits? device,
+  estimate vs available, suggestion to quantize/offload; since v0.303.1 an unquantized load is sized at the precision
+  `TextGenerator(dtype="auto")` loads — `float32` on CPU, not `bfloat16`,
+  measured RSS in `docs/recipes/genai.md#em-cpu`), `estimate_model_bytes`/
   `bytes_per_param`/`fetch_num_params` (Hub metadata, no weight download),
   `ModelDtype`. Capacity fns import WITHOUT the extra. **Shipped (v0.98) —
   `TextGenerator`**: local causal LM (`generate`/`chat`/`stream` async via
@@ -917,7 +928,8 @@ The SDK currently covers (Sep 2025+, post-v0.31.x):
   answers a chat *turn*). `Agent.run/stream` → `AgentRun` (output + `steps` +
   `artifacts` + `stop_reason`); `AgentBudget` bounds steps/wall-clock/tool
   calls and `StopReason` names which fired (`succeeded` is `COMPLETED` only —
-  a truncated run still carries text). **Three deliberate properties:** a
+  a truncated run still carries text; a model reply with no text and no tool
+  call is `EMPTY_RESPONSE`, not `COMPLETED`, since v0.303.2). **Three deliberate properties:** a
   raising tool becomes an observation fed back to the model, never a crashed
   run; every ceiling is enforced *and reported* (`max_seconds` defaults to 120
   because steps alone do not bound a hung call); binary results never enter

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.2}/docs/genai-examples.en.md

@@ -16,7 +16,7 @@ import asyncio
 from tempest_fastapi_sdk.genai import TextGenerator, recommend
 
 MODEL = "Qwen/Qwen2.5-7B-Instruct"
-rec = recommend(model_id=MODEL)          # tries bf16 -> int8 -> int4
+rec = recommend(model_id=MODEL)          # native (bf16 GPU, fp32 CPU) -> int8 -> int4
 
 if not rec.fits:
     raise RuntimeError(rec.reason + " " + (rec.suggestion or ""))
@@ -24,7 +24,7 @@ if not rec.fits:
 gen = TextGenerator(
     MODEL,
     device=rec.device,
-    quantization=rec.dtype.value if rec.dtype.value in ("int8", "int4") else None,
+    quantization=rec.dtype if rec.dtype in ("int8", "int4") else None,
     idle_unload_seconds=300,             # threshold for unload_if_idle()
 )
 

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.2}/docs/genai-examples.md

@@ -16,7 +16,7 @@ import asyncio
 from tempest_fastapi_sdk.genai import TextGenerator, recommend
 
 MODEL = "Qwen/Qwen2.5-7B-Instruct"
-rec = recommend(model_id=MODEL)          # tenta bf16 -> int8 -> int4
+rec = recommend(model_id=MODEL)          # nativa (bf16 GPU, fp32 CPU) -> int8 -> int4
 
 if not rec.fits:
     raise RuntimeError(rec.reason + " " + (rec.suggestion or ""))
@@ -24,7 +24,7 @@ if not rec.fits:
 gen = TextGenerator(
     MODEL,
     device=rec.device,
-    quantization=rec.dtype.value if rec.dtype.value in ("int8", "int4") else None,
+    quantization=rec.dtype if rec.dtype in ("int8", "int4") else None,
     idle_unload_seconds=300,             # limiar do unload_if_idle()
 )
 

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.2}/docs/recipes/agents-advanced.en.md

@@ -132,7 +132,9 @@ or cut by a budget, it returns `data=None` and `parse_error` says why.
     without calling `final_answer`, and neither the text parse nor the
     extraction found the shape (or `allow_text_fallback=False` turned that
     net off). A spent budget is a different case — that run is no longer
-    `succeeded`. Either way, `run.parse_error` says why. And
+    `succeeded` — and so is an empty model reply: it ends in
+    `empty_response`, with `parse_error="the run produced no answer"` and
+    no extra extraction call. Either way, `run.parse_error` says why. And
     small models sometimes leave a field empty rather than omitting it:
     validate the values, not just the presence of the object.
 

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.2}/docs/recipes/agents-advanced.md

@@ -132,7 +132,9 @@ O `output` da execução passa a ser a resposta em JSON, e é esse texto que o
     modelo terminou sem chamar `final_answer` e nem o parse do texto nem a
     extração acharam a forma (ou `allow_text_fallback=False` desligou essa
     rede). Orçamento esgotado é outro caso — aí a execução já não é
-    `succeeded`. Nos dois, `run.parse_error` diz o motivo. E modelos pequenos às vezes deixam um campo vazio em vez
+    `succeeded` —, e resposta vazia do modelo também: termina em
+    `empty_response`, com `parse_error="the run produced no answer"` e sem
+    a chamada extra de extração. Nos dois, `run.parse_error` diz o motivo. E modelos pequenos às vezes deixam um campo vazio em vez
     de omitir: valide os valores, não só a presença do objeto.
 
 Para insistir até a forma chegar, repita a execução e julgue com

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.2}/docs/recipes/agents-concepts.en.md

@@ -200,6 +200,7 @@ what keeps the request from staying open indefinitely.
 | `max_tool_calls` | The agent | Almost always a repeat-the-same-tool loop |
 | `error` | The backend | The model failed; `output` carries the error |
 | `blocked` | Moderation | Goal or answer refused |
+| `empty_response` | The model | Replied with no text and no tool; retry with `run_until` |
 
 ## Agent, chat, pipeline or loop?
 
@@ -320,6 +321,12 @@ None of that is a flaw in the module — it is what changes when the model fits
 on your machine. Design for it: few well-described tools, short outputs, a
 criterion checkable outside the model.
 
+On CPU, size turns into time as well: a 3B read ~132 prompt tokens per
+second in the measurements of
+[Agents on CPU](agents.md#agents-on-cpu-backend-size-and-budget), so every
+thousand tokens a tool returns cost ~7.6 s per step. Short output is the
+same rule, now with a number.
+
 ## Failure modes you will meet
 
 | Symptom | Likely cause | Where to look |

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.2}/docs/recipes/agents-concepts.md

@@ -201,6 +201,7 @@ que impede a requisição de ficar aberta indefinidamente.
 | `max_tool_calls` | O agente | Quase sempre é laço de repetição da mesma ferramenta |
 | `error` | O backend | O modelo caiu; `output` traz a mensagem do erro |
 | `blocked` | A moderação | Objetivo ou resposta recusados |
+| `empty_response` | O modelo | Respondeu sem texto e sem ferramenta; repetir com `run_until` |
 
 ## Agente, chat, pipeline ou laço?
 
@@ -323,6 +324,12 @@ Nada disso é defeito do módulo — é o que muda quando o modelo cabe na sua
 máquina. Projete para isso: ferramentas poucas e bem descritas, saídas curtas,
 critério verificável fora do modelo.
 
+Em CPU, o tamanho também vira tempo: um 3B leu ~132 tokens de prompt por
+segundo nas medições de
+[Agente em CPU](agents.md#agente-em-cpu-backend-tamanho-e-orcamento), então
+cada mil tokens que uma ferramenta devolve custam ~7,6 s por passo. Saída
+curta é a mesma regra, agora com número.
+
 ## Modos de falha que você vai encontrar
 
 | Sintoma | Causa provável | Onde olhar |

{tempest_fastapi_sdk-0.303.0 → tempest_fastapi_sdk-0.303.2}/docs/recipes/agents.en.md

@@ -157,18 +157,30 @@ reasons are the agent cutting the run short:
 
 | `stop_reason` | What happened |
 | --- | --- |
-| `completed` | The model answered without asking for another tool. |
+| `completed` | The model answered with text, without asking for another tool. |
 | `max_steps` | The step budget ran out first. |
 | `timeout` | The wall-clock budget ran out first. |
 | `max_tool_calls` | The tool-call budget ran out first. |
 | `error` | The model backend failed. |
 | `blocked` | Moderation rejected the goal or the answer. |
+| `empty_response` | The model ended its turn with no text and no tool call. |
 
 !!! warning "A truncated run still carries text"
     The `output` of a cut-short run is the last thing the model said —
     partial work, not a final answer. A caller that ignores `stop_reason`
     presents half-finished work as done.
 
+!!! info "An empty reply is not `completed`"
+    A model message with no text (or whitespace only) and no `tool_calls`
+    ends in `empty_response`, with `succeeded=False`. Up to 0.303.1 it
+    became `completed` with `output == ""`: a success with no answer.
+    `qwen2.5:0.5b` on Ollama 0.30.11, in CPU, with this page's goal, ended
+    58 of 200 runs that way. The agent does not re-ask on its own — that
+    would spend a step of the budget and hide how often the model fails.
+    To retry, use
+    [`run_until(agent, goal, until=succeeded)`](agents-advanced.md#loop-keep-going-until-it-passes-a-check): on the
+    same model, 58 of 60 loops with `max_rounds=3` ended accepted.
+
 ## Budget
 
 ```python title="budget.py" hl_lines="13"
@@ -710,6 +722,214 @@ which also means an `<img src>` works directly.
     and why it is a thread pool and not a semaphore are in
     [Self-hosted generative AI](genai.md#several-requests-at-once-on-cpu-max_concurrent).
 
+## Agents on CPU: backend, size and budget
+
+With no GPU, every turn of the loop is a full generation on the CPU, and
+three choices move a run's time by a factor of five or more: which backend
+runs the model, how big the model is, and how much text the tools return.
+The budget and the timeout have to fit those numbers, not the other way
+around.
+
+```python title="cpu_agent.py" hl_lines="11-16 20"
+import asyncio
+
+from agent_setup import weather_tool
+from tempest_fastapi_sdk.agents import Agent, AgentBudget
+from tempest_fastapi_sdk.genai import GenerationConfig, OllamaGenerator
+
+
+def build_cpu_agent() -> Agent:
+    """Build an agent sized for a CPU-only host."""
+    model = OllamaGenerator(
+        "qwen2.5:3b",
+        num_ctx=8192,
+        options={"num_gpu": 0},
+        timeout=120.0,
+        config=GenerationConfig(max_new_tokens=256),
+    )
+    return Agent(
+        model,
+        tools=[weather_tool],
+        budget=AgentBudget(max_steps=12, max_seconds=120),
+    )
+
+
+async def main() -> None:
+    """Run once and print how long each step took."""
+    agent = build_cpu_agent()
+    run = await agent.run("Qual o tempo no Recife? Use a ferramenta.")
+
+    print(run.stop_reason, f"{run.seconds:.1f}s")
+    for step in run.steps:
+        print(step.kind, step.name, f"{step.seconds:.2f}s")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+```text
+completed 1.9s
+model chat 0.86s
+tool get_weather 0.00s
+model chat 1.05s
+```
+
+(One run with the model already loaded in the daemon; the numbers behind it
+are just below.)
+
+Piece by piece:
+
+- **`OllamaGenerator("qwen2.5:3b")`** — a 4-bit GGUF served by Ollama, not
+  the `TextGenerator` in `float32`. Why is in the numbers below; memory
+  sizing (and why bitsandbytes on CPU is not the way out) is in
+  [Self-hosted generative AI › On CPU](genai.md#on-cpu).
+- **`num_ctx=8192`** — the agent's prompt grows every step, and Ollama cuts
+  whatever passes the window with no error. See
+  [The model runs on the generator's defaults](#the-model-runs-on-the-generators-defaults).
+- **`options={"num_gpu": 0}`** — forces the CPU on a host that has a GPU
+  (that is how the measurements below were taken). On a CPU-only host, drop
+  it.
+- **`config=GenerationConfig(max_new_tokens=256)`** — a token ceiling per
+  turn. On CPU generation is the expensive part, and the loop passes no
+  ceiling of its own.
+- **`timeout=120.0` and `max_seconds=120`** — the defaults, written out
+  because they move together; [The Ollama timeout](#the-ollama-timeout-and-the-budget)
+  explains why.
+
+### What each step costs
+
+Measured on an i9-13900F under WSL2 (12 visible logical CPUs = 6 cores × 2,
+62 GB of RAM, GPU hidden), torch 2.14 + transformers 4.57.6 for the
+`TextGenerator` in `float32`, Ollama 0.30.11 with `num_gpu: 0` for the
+Q4_K_M GGUFs (`size_vram` = 0 in `/api/ps`). The agent is the one in this
+section, with each backend's default generation settings, and the goal
+takes **one** tool call plus the answer (three steps). "Cold" is the first
+run of a fresh process, with the weight load inside the first step
+(weights already in the OS disk cache); N=3. "Warm" is the next five runs
+of each process; N=15. Medians; part of the rounds had another process
+busy on one logical CPU.
+
+With the tool returning one line (a ~230-token prompt):
+
+| Backend and model | Cold run | Warm run | Warm model step | Generation |
+| --- | --- | --- | --- | --- |
+| `TextGenerator` Qwen2.5-0.5B `float32` | 4.39 s | 2.12 s | 1.02 s | 25.6 tokens/s |
+| `TextGenerator` Qwen2.5-3B `float32` | 13.87 s | 11.59 s | 5.50 s | 5.2 tokens/s |
+| `OllamaGenerator` `qwen2.5:0.5b` | 1.52 s | 0.64 s | 0.29 s | 130 tokens/s |
+| `OllamaGenerator` `qwen2.5:3b` | 4.83 s | 1.94 s | 0.84 s | 30.6 tokens/s |
+
+With the tool returning ~4,100 tokens that differ on every run (last step's
+prompt: ~4,400 tokens). The `TextGenerator` rereads the whole prompt on
+every step; on Ollama the observation changes from run to run so the
+prefix cache does not hide the cost:
+
+| Backend and model | Warm run | Last step | Last step's prefill |
+| --- | --- | --- | --- |
+| `TextGenerator` Qwen2.5-0.5B `float32` | 7.60 s | 6.56 s | 819 tokens/s |
+| `TextGenerator` Qwen2.5-3B `float32` | 43.08 s | 37.51 s | 134 tokens/s |
+| `OllamaGenerator` `qwen2.5:0.5b` | 9.65 s | 9.34 s | 509 tokens/s |
+| `OllamaGenerator` `qwen2.5:3b` | 35.91 s | 35.06 s | 132 tokens/s |
+
+All 180 runs finished inside the default `AgentBudget` (`max_steps=12`,
+`max_seconds=120`): the slowest took 51.3 s, and the slowest step 43.0 s
+(3B `float32`, cold, long observation).
+
+What the numbers say:
+
+- **GGUF wins at generating, not at reading the prompt.** On the 3B,
+  Ollama generated 30.6 tokens/s against 5.2 for `float32` — 6× — and the
+  short run dropped from 11.59 s to 1.94 s. But prefill came out the same
+  (132 against 134 tokens/s): with a 4k-token observation the step is
+  almost all prefill, and the gap shrinks to 35.91 s against 43.08 s.
+- **Ollama reuses the prefix; the `TextGenerator` rereads everything.**
+  With the same long observation repeated verbatim from one run to the
+  next, the last step's prefill dropped to 0.05 s on `qwen2.5:3b` and
+  stayed at 32.66 s on the 3B `float32`. The system prompt and tool list
+  are the same prefix in every run, which is why the first warm step's
+  prefill costs 0.04 s on `qwen2.5:3b` and 1.65 s on the 3B `float32`.
+- **Observations are what costs.** On a 3B on this CPU, every thousand
+  tokens a tool returns cost ~7.6 s of prefill (1,000 / 132) on either
+  backend. A tool that returns the summary rather than the dump is a time
+  lever, not only a context one.
+- **0.5B on Ollama returned empty replies.** In 12 of 54 runs
+  `qwen2.5:0.5b` did not call the tool and Ollama returned the message with
+  no text and no `tool_calls`. Up to 0.303.1 that finished `completed` with
+  an empty `output` — a success with no answer; it now ends
+  [`empty_response`](#always-check-stop_reason), with `succeeded=False`. The same
+  model through the `TextGenerator` called the tool in all 36 runs, and
+  `qwen2.5:3b` in all 54. On CPU, the 3B through Ollama was the smallest
+  that completed every run.
+- **The first step pays for the load.** Cold against warm, the first step
+  cost +2.2 s on the 0.5B and +2.6 s on the 3B in `float32`, and Ollama
+  reported 1.4 s to 2.2 s of `load_duration` on the 3B — with the weights
+  already in the disk cache. Call the `TextGenerator`'s `load()` at service
+  startup (through `asyncio.to_thread`) so the first user does not pay for
+  it.
+
+### Sizing the budget
+
+A run's time is the sum, step by step, of **new prompt tokens ÷ prefill
+rate** plus **generated tokens ÷ generation rate**. With the rates measured
+above, the arithmetic matches the measurement: on the 3B `float32` with the
+long observation, 229/140 + 21/5.4 + 4,381/134 + 23/4.6 ≈ 43.2 s, against
+43.08 s measured.
+
+Use that sum, with **your** machine's rates, to pick the ceiling:
+
+- **`max_new_tokens` weighs more than it looks.** The `TextGenerator`
+  default is 256 tokens per turn; at 5.2 tokens/s, a turn that uses the
+  whole ceiling is ~49 s of generation on the 3B `float32`, and two of them
+  already pass 120 s. At 30.6 tokens/s, on the GGUF, the same turn is ~8 s.
+- **`max_seconds` is an HTTP request's ceiling**, so it comes from what
+  your client will wait for; what you tune to fit inside it is the model,
+  the observation size and `max_new_tokens`. At 132 tokens/s, the default
+  120 s holds ~15k tokens of observation summed over the whole run, before
+  counting generation.
+- **The queue counts on the clock.** Concurrent runs on a `TextGenerator`
+  with `max_concurrent=1` wait for one another, and the wait comes out of
+  the same `max_seconds` — see
+  [Several runs at once on a CPU model](#serving-it-over-http). Concurrency
+  inside Ollama was not measured here.
+
+### The Ollama timeout and the budget
+
+The `OllamaGenerator`'s `timeout=` (default 120 s) is **per request**, and
+the generator's HTTP client does **not** retry a request that times out:
+when the client gives up, the daemon aborts the generation, so a new
+attempt would start from zero and time out again. The `ReadTimeout` arrives
+after one timeout. Measured with `qwen2.5:3b` on CPU and `timeout=10.0`:
+the call raised `ReadTimeout` after 10.01 s (median, N=5), with one 10.0 s
+`POST /api/chat` in the Ollama log. A connection error and a `429`/`5xx`
+are still retried.
+
+??? note "Up to v0.303.1, the timeout was retried three times"
+    The client made three attempts, waiting 0.5 s and 1 s between them. On
+    the same scenario, with a ~4,400-token prompt (~33 s of prefill), the
+    call raised `ReadTimeout` after 31.53 s (a single run), the Ollama log showed three
+    10.0 s `POST /api/chat`, and inside an agent the run ended in `error`
+    after 33.73 s. On those versions, pass
+    `retry_policy=RetryPolicy(max_attempts=1)` (with
+    `from tempest_fastapi_sdk import RetryPolicy`): with it, the agent run
+    ended in `error` after 10.84 s — one attempt, which is the default
+    behaviour from v0.303.2 on.
+
+Two setups that behave:
+
+- **`timeout` greater than or equal to `max_seconds`** (the defaults): the
+  budget cuts before the timeout. With `max_seconds=20`, the run ended in
+  `timeout` after 20.02 s.
+- **A timeout shorter than `max_seconds`**: the turn that outlives the
+  timeout fails after one timeout, and the run ends in `error`. For the
+  agent to treat it as the end of the budget, prefer the previous setup.
+
+!!! tip "A starting point on CPU"
+    `OllamaGenerator` with a 3B GGUF model, an explicit `num_ctx`, an
+    explicit `max_new_tokens`, tools that return little text, and the
+    `AgentBudget` defaults. Then measure one `run.steps` on your machine and
+    redo the sum: the rates on this page come from an i9-13900F, and your
+    CPU changes every one of them.
+
 ## Recap
 
 - **`Agent.run(goal)`** returns an `AgentRun`: answer, trace, artifacts and
@@ -725,6 +945,9 @@ which also means an `<img src>` works directly.
   and only `AgentToolError` text reaches the trace in full.
 - **`make_agent_router`** publishes `/run`, `/run/stream` and artifact
   download by `run_id`; `owner=` keeps each caller's runs apart.
+- **On CPU**, the GGUF through `OllamaGenerator` generates 6× faster than
+  `float32` on the 3B, but prefill is the same: observation size and
+  `max_new_tokens` are what make a run fit in `max_seconds`.
 
 Next: [AI agents (advanced)](agents-advanced.md) — typed structured output,
 the three memory layers, skills loaded on demand, delegation between agents,