hipengine 0.2.0__tar.gz → 0.2.2__tar.gz

{hipengine-0.2.0 → hipengine-0.2.2}/CHANGELOG.md

@@ -6,6 +6,82 @@ This changelog is for package/API releases. Performance rollup history remains i
 [`benchmarks/CHANGELOG.md`](benchmarks/CHANGELOG.md), with detailed benchmark
 evidence under [`benchmarks/results/`](benchmarks/results/).
 
+## Unreleased
+
+No changes yet.
+
+## v0.2.2 - 2026-05-26
+
+Patch release improving server startup context preallocation, KV memory
+admission, and request defaults.
+
+### Added
+
+- Server-wide resident context/KV preallocation controls:
+  `--max-context-tokens`, `--kv-storage`, `--kv-scale-dtype`, and
+  `--kv-scale-granularity`. Eager startup prepares the resident PARO session for
+  the configured context, and requests beyond that context or with a different
+  KV policy are rejected instead of resizing/reloading the model.
+- Automatic server context sizing when `--max-context-tokens` is omitted: after
+  resident weights load, the runtime estimates the selected KV dtype plus
+  persistent context metadata and preallocates
+  `min(model_max_context_tokens, allocatable_context_tokens)`.
+- Fast PARO retained-KV capacity estimate during resident session build. The
+  runtime uses current `hipMemGetInfo` after model weights load to report the
+  estimated max context for the selected KV dtype and for INT8 KV, warning when
+  INT8 still falls below the model's advertised max context.
+
+### Changed
+
+- Chat requests that omit `max_tokens` now use `max_tokens=auto`, meaning the
+  remaining admitted context (`max_context_tokens - prompt_tokens - 1`).
+
+### Fixed
+
+- Clean up partially-built PARO resident sessions if capacity preflight or
+  allocation fails, avoiding leaked resident buffers on startup/admission OOM.
+
+## v0.2.1 - 2026-05-25
+
+Patch release improving server session management, streaming, and
+OpenAI-compatible reasoning output.
+
+### Added
+
+- Eager model warmup on server startup: the configured model and a short
+  warmup generation run before uvicorn reports ready, so the first real
+  request does not pay load/compile cost. Controlled by `--eager-load` /
+  `--no-eager-load` (default: on), `--eager-load-prompt`, and
+  `--eager-load-max-tokens`, with `HIPENGINE_EAGER_LOAD`,
+  `HIPENGINE_EAGER_LOAD_PROMPT`, and `HIPENGINE_EAGER_LOAD_MAX_TOKENS`
+  environment variable equivalents.
+- `LLM.stream()` method for single-prompt token-by-token generation when
+  the underlying text generator supports it.
+- Reasoning-content splitting for chat completions: `<think>…</think>`
+  spans (Qwen/DeepSeek-style) are now separated into
+  `message.reasoning_content` (non-streaming) or `delta.reasoning_content`
+  chunks (streaming), matching the OpenAI reasoning-content convention.
+
+### Changed
+
+- PARO text generators and their resident sessions are now cached on the
+  `LLM` instance and reused across requests. Session capacity is bucketed
+  (floor 4 Ki tokens, configurable via `HIPENGINE_SESSION_MIN_TOKENS` and
+  `HIPENGINE_SESSION_BUCKET_TOKENS`) so normal chat-history growth does not
+  force reallocation every turn.
+- Chat `stream=true` now yields token-level SSE chunks from the resident
+  decode loop instead of buffering the full response and wrapping it in a
+  single SSE frame.
+- Chat completions default `max_tokens` raised from 16 to 8192 so clients
+  that omit the field get usable reply lengths, including verbose
+  chain-of-thought reasoning.
+
+### Fixed
+
+- Fixed `LLM.generate()` re-resolving the generation factory on every call,
+  which discarded generator-local caches and caused the PARO resident
+  session (layer weights, KV buffers) to be allocated and freed per request.
+
 ## v0.2.0 - 2026-05-25
 
 Minor release for the GGUF runtime path and W7900 benchmark refresh. GGUF is a

{hipengine-0.2.0 → hipengine-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hipengine
-Version: 0.2.0
+Version: 0.2.2
 Summary: ROCm-native local LLM inference engine with a torch-free runtime hot path
 Project-URL: Homepage, https://github.com/shisa-ai/hipEngine
 Project-URL: Repository, https://github.com/shisa-ai/hipEngine
@@ -75,7 +75,7 @@ supported GPUs and models.
 
 ## Status
 
-**v0.2.0 alpha.** The runtime hot path is torch-free by construction, and the
+**v0.2.1 alpha.** The runtime hot path is torch-free by construction, and the
 first two 35B-class model-loading surfaces are now available on gfx1100:
 [shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed](https://huggingface.co/shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed)
 (19.07 GiB, 4.68 bpw) in packed
@@ -336,13 +336,20 @@ Install the optional server extra and run the FastAPI layer:
 ```bash
 pip install -e ".[server]"
 python -m hipengine.server \
-  --model /path/to/model \
+  --model shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed \
   --quant w4_paro \
   --served-model-name qwen-paro
 ```
 
-Supported v0.1 endpoints: `GET /v1/models`, `POST /v1/completions`, and
-`POST /v1/chat/completions` (including one-chunk SSE for `stream=true`). See
+`--model` accepts either a local filesystem path or a Hugging Face model ID
+already present in the local HF cache; hipEngine resolves IDs locally and does
+not download weights during startup.
+
+Supported endpoints: `GET /v1/models`, `POST /v1/completions`, and
+`POST /v1/chat/completions` with token-level SSE streaming. Chat responses
+separate `<think>` reasoning into `reasoning_content` (matching the OpenAI
+reasoning-content convention). The server eagerly warms the model on startup
+by default so the first request does not pay load/compile cost. See
 [`docs/API.md`](docs/API.md) for request examples, bearer-token auth, and
 current limitations.
 

{hipengine-0.2.0 → hipengine-0.2.2}/README.md

@@ -37,7 +37,7 @@ supported GPUs and models.
 
 ## Status
 
-**v0.2.0 alpha.** The runtime hot path is torch-free by construction, and the
+**v0.2.1 alpha.** The runtime hot path is torch-free by construction, and the
 first two 35B-class model-loading surfaces are now available on gfx1100:
 [shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed](https://huggingface.co/shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed)
 (19.07 GiB, 4.68 bpw) in packed
@@ -298,13 +298,20 @@ Install the optional server extra and run the FastAPI layer:
 ```bash
 pip install -e ".[server]"
 python -m hipengine.server \
-  --model /path/to/model \
+  --model shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed \
   --quant w4_paro \
   --served-model-name qwen-paro
 ```
 
-Supported v0.1 endpoints: `GET /v1/models`, `POST /v1/completions`, and
-`POST /v1/chat/completions` (including one-chunk SSE for `stream=true`). See
+`--model` accepts either a local filesystem path or a Hugging Face model ID
+already present in the local HF cache; hipEngine resolves IDs locally and does
+not download weights during startup.
+
+Supported endpoints: `GET /v1/models`, `POST /v1/completions`, and
+`POST /v1/chat/completions` with token-level SSE streaming. Chat responses
+separate `<think>` reasoning into `reasoning_content` (matching the OpenAI
+reasoning-content convention). The server eagerly warms the model on startup
+by default so the first request does not pay load/compile cost. See
 [`docs/API.md`](docs/API.md) for request examples, bearer-token auth, and
 current limitations.
 

{hipengine-0.2.0 → hipengine-0.2.2}/WORKLOG.md

@@ -26555,3 +26555,169 @@ uv run --extra dev python -m pytest -q
 ```
 
 Next: commit, push `main`, move annotated tag `v0.2.0` to the fix commit, force-push the tag per user approval, and watch the publish workflow complete via trusted publishing.
+
+### v0.2.0 publish workflow result
+
+Committed and pushed the HIP guard fix (`36e3516`), force-moved annotated tag `v0.2.0` to that commit per user approval, and re-pushed the tag. The `Publish to PyPI` workflow reran as `26370452261` and completed successfully: build validation passed, artifact attestation was generated, and `Publish to PyPI (trusted publishing)` succeeded. Verified published install path:
+
+```bash
+uvx --refresh --from "hipengine[server]==0.2.0" hipengine-server --help
+# downloaded hipengine 0.2.0 from PyPI and printed CLI help
+```
+
+Final release pointers: GitHub release `https://github.com/shisa-ai/hipEngine/releases/tag/v0.2.0`, publish workflow `https://github.com/shisa-ai/hipEngine/actions/runs/26370452261`, release tag commit `36e351607955158e821b51f443bac042b140624f`.
+
+## 2026-05-25 — HF cache model id resolution for server/LLM
+
+Added local Hugging Face cache resolution for public model references so `hipengine-server --model` and `LLM(model=...)` can take either a filesystem path or a cached HF model id such as `shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed`. Resolution remains local-only: existing paths win, then `huggingface_hub.snapshot_download(..., local_files_only=True)` is used if available, then the standard HF cache layout is inspected directly; no weights are downloaded during startup. GGUF discovery now resolves the model reference first, so cached GGUF repos/directories work as well as direct `.gguf` file paths.
+
+Also updated README/API server examples to show the HF model id form and note that cache resolution is local-only. User's `uv run --extra server ...` command is the right shape; the `--no-build` error indicates a uv no-build mode/config/environment for the local editable package, not a hipEngine server argument issue. Workaround is to unset no-build for local editable runs or use the published wheel via `uvx --from "hipengine[server]==0.2.0"`.
+
+Validation:
+
+```bash
+python -m py_compile hipengine/loading/hf_cache.py hipengine/loading/safetensors.py hipengine/loading/gguf.py hipengine/llm.py
+python -m pytest -q tests/test_hf_cache.py tests/test_llm_generate.py tests/test_loading_safetensors.py tests/test_server_api.py
+# 19 passed
+uv run --extra dev python -m pytest -q
+# passed
+```
+## 2026-05-23 — llama.cpp wrapper metadata fix for RX 7900 XTX 5-run sweep
+
+Before rerunning `benchmarks/7900XTX.md` with `--repetitions 5`, fixed `scripts/llamacpp_bench_with_peak.py` artifact metadata so it no longer hardcodes W7900 or a single-shot `--repetitions 1` note. The artifact now records the selected amdgpu card name/PCI/VRAM in `hardware` and emits a dynamic repetitions note. Validation: `python3 -m py_compile scripts/llamacpp_bench_with_peak.py`.
+
+## 2026-05-24 — server resident-session reuse smoke
+
+Debugged the local OpenAI-compatible PARO server after LAN chat showed short `<think>` replies and apparent reloads. Root causes:
+
+- `LLM.generate()` re-resolved the generation factory and constructed a new text generator on every call, discarding generator-local caches.
+- `Qwen35ParoOneTokenGenerator` then constructed `Qwen35ParoResidentSession` inside each prompt call and closed it immediately, so resident layer weights/KV buffers were materialized and freed per request.
+- The OpenAI chat server default `max_tokens` was 16; clients that omit `max_tokens` saw very short replies. The Qwen/PARO model config advertises `max_position_embeddings=262144`, but the runtime session capacity was allocated as `len(prompt_ids) + max_tokens + 1` per request.
+- `stream=true` only returned an SSE wrapper around the completed response; it did not yield tokens while generation was running.
+
+Fixes made in-tree:
+
+- Cache the resolved text generator on `hipengine.LLM` so server `app.state.hipengine_llm` keeps its backend/model/quant generator across requests.
+- Cache/reuse `Qwen35ParoResidentSession` inside the PARO generator when the existing session capacity and KV policy cover the next request; reset it between prompts instead of closing it. Session capacity now floors/buckets at 4096 tokens by default (`HIPENGINE_SESSION_MIN_TOKENS`, `HIPENGINE_SESSION_BUCKET_TOKENS`) so normal chat history growth does not force reallocation every turn.
+- Add a single-prompt streaming path using resident `step()` per token, and route chat `stream=true` through it. Chat default `max_tokens` is now 256.
+
+Validation:
+
+```bash
+uv run --extra dev python -m pytest -q tests/test_llm_generate.py tests/test_generation_qwen35_paro.py tests/test_server_api.py
+# 17 passed
+```
+
+Restarted the running LAN server from the local checkout with TheRock ROCm library paths:
+
+```bash
+uv run --extra server hipengine-server --model shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed --quant w4_paro --served-model-name qwen-paro --host 0.0.0.0 --port 8000
+```
+
+Smoke evidence on the running server after restart from the local checkout: first streamed `max_tokens=4` request sent the role chunk immediately, then took 30.25s to warm/materialize and yielded token chunks (`<think>`, newline, `Here`, `'s`); VRAM reached 19.54GB. A second chat-history request reused the resident 4096-token session, yielded the first content chunk at 0.46s, completed at 0.49s, and VRAM stayed resident at 19.55GB. Server PID 9305, wrapper PID recorded in `/tmp/hipengine-server-8000.pid`, log `/tmp/hipengine-server-8000.log`.
+
+## 2026-05-24 — eager server warmup and reasoning-channel split
+
+Implemented eager server warmup and OpenAI-compatible reasoning segregation for the local PARO server.
+
+Changes:
+
+- Added `ServerConfig.eager_load` plus CLI `--eager-load/--no-eager-load` (default on), `--eager-load-prompt`, and `--eager-load-max-tokens`. Startup now constructs `LLM` and runs a one-token warmup before uvicorn reports startup complete, so `/v1/models` is only reachable after the model/session is resident.
+- Default warmup prompt is `one two three four`; the initial `hello` default failed native prefill because Qwen35/PARO requires at least `linear_conv_kernel_dim` (4) prompt tokens.
+- Split Qwen/DeepSeek-style `<think>...</think>` spans in chat responses. Non-stream responses put visible answer text in `message.content` and hidden reasoning text in `message.reasoning_content`. Streaming responses emit `delta.reasoning_content` for reasoning chunks and `delta.content` for final answer chunks.
+
+Validation:
+
+```bash
+uv run --extra dev python -m pytest -q tests/test_server_api.py tests/test_llm_generate.py tests/test_generation_qwen35_paro.py
+# 19 passed
+```
+
+Restarted the LAN server from the local checkout with TheRock ROCm library paths. Eager startup completed after 31s, `/v1/models` returned `qwen-paro`, and VRAM was already resident at 19.54GB before serving requests. A subsequent streamed chat request returned the role chunk at 0.01s and reasoning chunks as `delta.reasoning_content` from 0.49s onward, with VRAM staying resident at 19.54GB. Server PID 12107, wrapper PID 12082, log `/tmp/hipengine-server-8000.log`.
+
+## 2026-05-25 — PARO KV capacity preflight for server sizing
+
+Revisited the first-request reload diagnosis. The chat default should remain `max_tokens=8192`; the real issue is that session sizing is request-driven, so a warmed 4096-token resident session can be discarded when a later request needs a larger KV arena. Before changing admission/preallocation policy, added a fast retained-KV capacity estimate in `Qwen35ParoResidentSession`: after resident weights load and before full-attention KV buffers are allocated, the runtime reads `hipMemGetInfo`, subtracts a configurable reserve (`HIPENGINE_KV_CAPACITY_RESERVE_MIB`, default 512 MiB), computes bytes/token for the selected KV dtype and for INT8 KV, emits structured progress records, and logs a warning when INT8 capacity is below the model's advertised max context. This gives startup logs such as "INT8 KV can fit about N tokens vs model max M" instead of discovering the limit via an OOMing `hipMalloc`.
+
+Wired the estimate into server policy: `ServerConfig`/CLI now have `--max-context-tokens` (default 131072), `--kv-storage`, `--kv-scale-dtype`, and `--kv-scale-granularity`. Eager startup calls `LLM.prepare(max_sequence_length=...)` before the warmup generation, so the resident PARO session/KV arena is preallocated for the configured context. Request admission checks `prompt_tokens + max_tokens + 1 <= max_context_tokens`, and per-request KV settings must match the server-wide preallocated policy; violations return OpenAI-style 400 errors instead of triggering a session resize/reload.
+
+Also made PARO resident session construction clean up partially-built buffers if preflight/allocation fails, so an over-large requested context fails cleanly instead of leaking resident weights/KV. Chat default coverage was restored to assert the intended 8192-token server default.
+
+Validation:
+
+```bash
+python -m pytest -q tests/test_server_api.py tests/test_generation_qwen35_paro.py tests/test_llm_generate.py
+# 24 passed
+python -m py_compile hipengine/runtime/qwen35_paro_runner.py hipengine/loading/qwen35_paro.py hipengine/server/api.py hipengine/server/__main__.py hipengine/llm.py hipengine/generation/qwen35_paro.py
+```
+
+## 2026-05-25 — server auto context sizing accounts for retained metadata
+
+Follow-up to the PARO server KV preallocation work: a real startup with default auto sizing attempted the model max 262144-token BF16 context and OOMed before the old KV estimate ran because `_allocate_common_buffers()` materialized a large persistent prefill block table (`prefill_capacity_rows x blocks`). Fixed the sizing path to load resident weights first, run the HIP free-memory estimate before context-dependent metadata/KV allocation, and include persistent context metadata in the allocatable-token calculation. Automatic server startup now selects `min(model_max_context_tokens, allocatable_context_tokens)` for the selected KV policy, while explicit `--max-context-tokens` still fails fast with guidance to lower the cap or use INT8 KV.
+
+Also made chat `max_tokens` dynamic when omitted: after the resident context is prepared, chat requests default to the remaining admitted context (`max_context_tokens - prompt_tokens - 1`) instead of a fixed 8192. Startup logs were compacted to `Config`, `KVCache`, `WARMUP`, and `hipEngine is ready.` lines, with KVCache reporting KV bytes plus retained metadata bytes.
+
+Validation:
+
+```bash
+python -m pytest -q tests/test_generation_qwen35_paro.py tests/test_server_api.py tests/test_llm_generate.py
+# 24 passed
+python -m py_compile hipengine/runtime/qwen35_paro_runner.py hipengine/server/api.py hipengine/generation/qwen35_paro.py hipengine/llm.py hipengine/server/__main__.py
+```
+
+## 2026-05-25 — suppress duplicate low-level KV capacity logs
+
+Kept the server `Config`/`KVCache` summary as the user-facing startup output and demoted the lower-level PARO runner KV-capacity info/warning lines to DEBUG. The resolved `Config` and `KVCache` lines intentionally remain after resident weights load because auto sizing uses actual `hipMemGetInfo` free memory after model materialization.
+
+Validation:
+
+```bash
+python -m pytest -q tests/test_server_api.py tests/test_generation_qwen35_paro.py
+python -m py_compile hipengine/runtime/qwen35_paro_runner.py hipengine/server/api.py
+```
+
+## 2026-05-26 — v0.2.1 release validation and artifact build
+
+Prepared v0.2.1 for publishing with the server KV preallocation improvements folded into the `CHANGELOG.md` v0.2.1 entry (leaving `Unreleased` empty). Used a clean detached worktree at `/tmp/hipengine-release-v021.QhOZ2d` to avoid the main checkout's unrelated untracked `uv.lock`.
+
+Validation and artifact commands:
+
+```bash
+git fetch --tags origin && git pull --ff-only
+python3 -m compileall -q hipengine scripts tests
+uv run --extra dev python -m pytest -q
+uv run --python 3.10 --extra dev python -m pytest -q
+uv run --extra dev hipengine-server --help
+rm -rf dist && python3 -m build
+unzip -p dist/hipengine-0.2.1-py3-none-manylinux_2_39_x86_64.whl hipengine-0.2.1.dist-info/WHEEL
+uvx --from twine twine check dist/*
+WHEEL=/tmp/hipengine-release-v021.QhOZ2d/dist/hipengine-0.2.1-py3-none-manylinux_2_39_x86_64.whl; \
+  (cd /tmp && uv run --isolated --with "${WHEEL}[server]" hipengine-server --help)
+```
+
+Results: compileall passed; full pytest passed on CPython 3.10.16 (including the explicit `--python 3.10` run); server CLI smoke passed; `python3 -m build` produced `hipengine-0.2.1.tar.gz` and `hipengine-0.2.1-py3-none-manylinux_2_39_x86_64.whl`; wheel metadata reports `Root-Is-Purelib: false` and tag `py3-none-manylinux_2_39_x86_64`; `twine check` passed for both artifacts; isolated wheel server CLI smoke passed.
+
+## 2026-05-26 — retarget server KV preallocation release to v0.2.2
+
+Confirmed `v0.2.1` was already immutable on PyPI and tagged on GitHub, so retargeted the server KV preallocation improvements to `v0.2.2`. Updated `pyproject.toml` to `0.2.2` and split `CHANGELOG.md` so `v0.2.1` retains the already-published session reuse/warmup/streaming notes while `v0.2.2` carries the resident context/KV preallocation controls, auto context sizing, retained-KV capacity estimate, `max_tokens=auto`, and partial-construction cleanup notes.
+
+## 2026-05-26 — v0.2.2 release validation and artifact build
+
+Built the server KV preallocation release as `v0.2.2` from clean detached worktree `/tmp/hipengine-release-v021.QhOZ2d` at commit `fc16570` before the release-validation log append.
+
+Validation and artifact commands:
+
+```bash
+python3 -m compileall -q hipengine scripts tests
+uv run --extra dev python -m pytest -q
+uv run --python 3.10 --extra dev python -m pytest -q
+uv run --extra dev hipengine-server --help
+rm -f uv.lock && rm -rf .venv dist
+python3 -m build
+unzip -p dist/hipengine-0.2.2-py3-none-manylinux_2_39_x86_64.whl hipengine-0.2.2.dist-info/WHEEL
+uvx --from twine twine check dist/*
+WHEEL=/tmp/hipengine-release-v021.QhOZ2d/dist/hipengine-0.2.2-py3-none-manylinux_2_39_x86_64.whl; \
+  (cd /tmp && uv run --isolated --with "${WHEEL}[server]" hipengine-server --help)
+```
+
+Results: compileall passed; full pytest passed on CPython 3.10.16 (including the explicit `--python 3.10` run); server CLI smoke passed; `python3 -m build` produced `hipengine-0.2.2.tar.gz` and `hipengine-0.2.2-py3-none-manylinux_2_39_x86_64.whl`; wheel metadata reports `Root-Is-Purelib: false` and tag `py3-none-manylinux_2_39_x86_64`; `twine check` passed for both artifacts; isolated wheel server CLI smoke passed. Rebuild artifacts after this log append before publishing so `dist/` matches the final commit.

{hipengine-0.2.0 → hipengine-0.2.2}/docs/API.md

@@ -1,6 +1,6 @@
 # OpenAI-Compatible Server API
 
-Last updated: 2026-05-18
+Last updated: 2026-05-25
 
 hipEngine ships a thin optional FastAPI layer that adapts OpenAI-style requests
 to the torch-free `hipengine.LLM.generate()` library API. It is installed only
@@ -17,13 +17,17 @@ pip install -e ".[server]"
 
 ```bash
 python -m hipengine.server \
-  --model /path/to/qwen-paro-model \
+  --model shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed \
   --quant w4_paro \
   --served-model-name qwen-paro \
   --host 127.0.0.1 \
   --port 8000
 ```
 
+`--model` accepts a local filesystem path or a Hugging Face model ID that is
+already present in the local HF cache. hipEngine resolves IDs with local cache
+lookups only; it does not download weights during server startup.
+
 After installation, the console script is equivalent:
 
 ```bash
@@ -36,6 +40,25 @@ select `cpu_reference` where a CPU implementation exists; nearby targets such as
 `gfx1101`/`gfx1102` can force a backend with `--backend hip_gfx1100` or
 `HIPENGINE_BACKEND=hip_gfx1100` after local validation.
 
+By default the server eagerly loads the model, loads resident weights, estimates
+remaining HIP memory for KV cache plus persistent context metadata, then
+preallocates `min(model max context, estimated allocatable context)`. Pass
+`--max-context-tokens` (or `HIPENGINE_MAX_CONTEXT_TOKENS`) to force a lower cap.
+Startup fails with a clear error if the requested cap cannot be allocated; lower
+`--max-context-tokens` or use `--kv-storage int8_per_token_head`. Disable eager
+startup with `--no-eager-load` or `HIPENGINE_EAGER_LOAD=0`. The warmup prompt and
+token count are configurable via `--eager-load-prompt` and
+`--eager-load-max-tokens`.
+
+The resident KV policy is server-wide: set `--kv-storage` (`auto`, `bf16`, or
+`int8_per_token_head`), `--kv-scale-dtype`, and `--kv-scale-granularity` at
+startup. Requests that ask for a different KV policy are rejected instead of
+rebuilding the resident model. Startup logs include a compact KVCache summary
+from current HIP free memory and warn when even INT8 KV is below the model's
+advertised max context. Chat requests that omit `max_tokens` use
+`max_tokens=auto`, meaning the remaining admitted context
+(`max_context_tokens - prompt_tokens - 1`).
+
 Set `HIPENGINE_API_KEY` or pass `--api-key` to require OpenAI-style bearer
 authentication:
 
@@ -50,8 +73,8 @@ curl -H 'Authorization: Bearer local-secret' http://127.0.0.1:8000/v1/models
 | --- | --- | --- |
 | `GET /health` | Built in | Unauthenticated health/model probe. |
 | `GET /v1/models` | Built in | Returns the single served model id. |
-| `POST /v1/completions` | Built in | Text prompt(s) to `LLM.generate()`. Supports `stream=true` as one server-sent event chunk plus `[DONE]`. |
-| `POST /v1/chat/completions` | Built in | Renders text-only messages to a Qwen-style prompt and calls `LLM.generate()`. Supports `stream=true` as one server-sent event chunk plus `[DONE]`. |
+| `POST /v1/completions` | Built in | Text prompt(s) to `LLM.generate()`. Supports `stream=true` (one SSE chunk plus `[DONE]`). |
+| `POST /v1/chat/completions` | Built in | Renders text-only messages to a Qwen-style prompt and calls `LLM.generate()`. Supports token-level `stream=true` SSE. `<think>` spans are separated into `reasoning_content` (non-streaming) or `delta.reasoning_content` chunks (streaming). |
 
 ## Examples