hipengine 0.2.1__tar.gz → 0.2.2__tar.gz

{hipengine-0.2.1 → hipengine-0.2.2}/CHANGELOG.md

@@ -6,6 +6,41 @@ This changelog is for package/API releases. Performance rollup history remains i
 [`benchmarks/CHANGELOG.md`](benchmarks/CHANGELOG.md), with detailed benchmark
 evidence under [`benchmarks/results/`](benchmarks/results/).
 
+## Unreleased
+
+No changes yet.
+
+## v0.2.2 - 2026-05-26
+
+Patch release improving server startup context preallocation, KV memory
+admission, and request defaults.
+
+### Added
+
+- Server-wide resident context/KV preallocation controls:
+  `--max-context-tokens`, `--kv-storage`, `--kv-scale-dtype`, and
+  `--kv-scale-granularity`. Eager startup prepares the resident PARO session for
+  the configured context, and requests beyond that context or with a different
+  KV policy are rejected instead of resizing/reloading the model.
+- Automatic server context sizing when `--max-context-tokens` is omitted: after
+  resident weights load, the runtime estimates the selected KV dtype plus
+  persistent context metadata and preallocates
+  `min(model_max_context_tokens, allocatable_context_tokens)`.
+- Fast PARO retained-KV capacity estimate during resident session build. The
+  runtime uses current `hipMemGetInfo` after model weights load to report the
+  estimated max context for the selected KV dtype and for INT8 KV, warning when
+  INT8 still falls below the model's advertised max context.
+
+### Changed
+
+- Chat requests that omit `max_tokens` now use `max_tokens=auto`, meaning the
+  remaining admitted context (`max_context_tokens - prompt_tokens - 1`).
+
+### Fixed
+
+- Clean up partially-built PARO resident sessions if capacity preflight or
+  allocation fails, avoiding leaked resident buffers on startup/admission OOM.
+
 ## v0.2.1 - 2026-05-25
 
 Patch release improving server session management, streaming, and

{hipengine-0.2.1 → hipengine-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hipengine
-Version: 0.2.1
+Version: 0.2.2
 Summary: ROCm-native local LLM inference engine with a torch-free runtime hot path
 Project-URL: Homepage, https://github.com/shisa-ai/hipEngine
 Project-URL: Repository, https://github.com/shisa-ai/hipEngine

{hipengine-0.2.1 → hipengine-0.2.2}/WORKLOG.md

@@ -26634,3 +26634,90 @@ uv run --extra dev python -m pytest -q tests/test_server_api.py tests/test_llm_g
 ```
 
 Restarted the LAN server from the local checkout with TheRock ROCm library paths. Eager startup completed after 31s, `/v1/models` returned `qwen-paro`, and VRAM was already resident at 19.54GB before serving requests. A subsequent streamed chat request returned the role chunk at 0.01s and reasoning chunks as `delta.reasoning_content` from 0.49s onward, with VRAM staying resident at 19.54GB. Server PID 12107, wrapper PID 12082, log `/tmp/hipengine-server-8000.log`.
+
+## 2026-05-25 — PARO KV capacity preflight for server sizing
+
+Revisited the first-request reload diagnosis. The chat default should remain `max_tokens=8192`; the real issue is that session sizing is request-driven, so a warmed 4096-token resident session can be discarded when a later request needs a larger KV arena. Before changing admission/preallocation policy, added a fast retained-KV capacity estimate in `Qwen35ParoResidentSession`: after resident weights load and before full-attention KV buffers are allocated, the runtime reads `hipMemGetInfo`, subtracts a configurable reserve (`HIPENGINE_KV_CAPACITY_RESERVE_MIB`, default 512 MiB), computes bytes/token for the selected KV dtype and for INT8 KV, emits structured progress records, and logs a warning when INT8 capacity is below the model's advertised max context. This gives startup logs such as "INT8 KV can fit about N tokens vs model max M" instead of discovering the limit via an OOMing `hipMalloc`.
+
+Wired the estimate into server policy: `ServerConfig`/CLI now have `--max-context-tokens` (default 131072), `--kv-storage`, `--kv-scale-dtype`, and `--kv-scale-granularity`. Eager startup calls `LLM.prepare(max_sequence_length=...)` before the warmup generation, so the resident PARO session/KV arena is preallocated for the configured context. Request admission checks `prompt_tokens + max_tokens + 1 <= max_context_tokens`, and per-request KV settings must match the server-wide preallocated policy; violations return OpenAI-style 400 errors instead of triggering a session resize/reload.
+
+Also made PARO resident session construction clean up partially-built buffers if preflight/allocation fails, so an over-large requested context fails cleanly instead of leaking resident weights/KV. Chat default coverage was restored to assert the intended 8192-token server default.
+
+Validation:
+
+```bash
+python -m pytest -q tests/test_server_api.py tests/test_generation_qwen35_paro.py tests/test_llm_generate.py
+# 24 passed
+python -m py_compile hipengine/runtime/qwen35_paro_runner.py hipengine/loading/qwen35_paro.py hipengine/server/api.py hipengine/server/__main__.py hipengine/llm.py hipengine/generation/qwen35_paro.py
+```
+
+## 2026-05-25 — server auto context sizing accounts for retained metadata
+
+Follow-up to the PARO server KV preallocation work: a real startup with default auto sizing attempted the model max 262144-token BF16 context and OOMed before the old KV estimate ran because `_allocate_common_buffers()` materialized a large persistent prefill block table (`prefill_capacity_rows x blocks`). Fixed the sizing path to load resident weights first, run the HIP free-memory estimate before context-dependent metadata/KV allocation, and include persistent context metadata in the allocatable-token calculation. Automatic server startup now selects `min(model_max_context_tokens, allocatable_context_tokens)` for the selected KV policy, while explicit `--max-context-tokens` still fails fast with guidance to lower the cap or use INT8 KV.
+
+Also made chat `max_tokens` dynamic when omitted: after the resident context is prepared, chat requests default to the remaining admitted context (`max_context_tokens - prompt_tokens - 1`) instead of a fixed 8192. Startup logs were compacted to `Config`, `KVCache`, `WARMUP`, and `hipEngine is ready.` lines, with KVCache reporting KV bytes plus retained metadata bytes.
+
+Validation:
+
+```bash
+python -m pytest -q tests/test_generation_qwen35_paro.py tests/test_server_api.py tests/test_llm_generate.py
+# 24 passed
+python -m py_compile hipengine/runtime/qwen35_paro_runner.py hipengine/server/api.py hipengine/generation/qwen35_paro.py hipengine/llm.py hipengine/server/__main__.py
+```
+
+## 2026-05-25 — suppress duplicate low-level KV capacity logs
+
+Kept the server `Config`/`KVCache` summary as the user-facing startup output and demoted the lower-level PARO runner KV-capacity info/warning lines to DEBUG. The resolved `Config` and `KVCache` lines intentionally remain after resident weights load because auto sizing uses actual `hipMemGetInfo` free memory after model materialization.
+
+Validation:
+
+```bash
+python -m pytest -q tests/test_server_api.py tests/test_generation_qwen35_paro.py
+python -m py_compile hipengine/runtime/qwen35_paro_runner.py hipengine/server/api.py
+```
+
+## 2026-05-26 — v0.2.1 release validation and artifact build
+
+Prepared v0.2.1 for publishing with the server KV preallocation improvements folded into the `CHANGELOG.md` v0.2.1 entry (leaving `Unreleased` empty). Used a clean detached worktree at `/tmp/hipengine-release-v021.QhOZ2d` to avoid the main checkout's unrelated untracked `uv.lock`.
+
+Validation and artifact commands:
+
+```bash
+git fetch --tags origin && git pull --ff-only
+python3 -m compileall -q hipengine scripts tests
+uv run --extra dev python -m pytest -q
+uv run --python 3.10 --extra dev python -m pytest -q
+uv run --extra dev hipengine-server --help
+rm -rf dist && python3 -m build
+unzip -p dist/hipengine-0.2.1-py3-none-manylinux_2_39_x86_64.whl hipengine-0.2.1.dist-info/WHEEL
+uvx --from twine twine check dist/*
+WHEEL=/tmp/hipengine-release-v021.QhOZ2d/dist/hipengine-0.2.1-py3-none-manylinux_2_39_x86_64.whl; \
+  (cd /tmp && uv run --isolated --with "${WHEEL}[server]" hipengine-server --help)
+```
+
+Results: compileall passed; full pytest passed on CPython 3.10.16 (including the explicit `--python 3.10` run); server CLI smoke passed; `python3 -m build` produced `hipengine-0.2.1.tar.gz` and `hipengine-0.2.1-py3-none-manylinux_2_39_x86_64.whl`; wheel metadata reports `Root-Is-Purelib: false` and tag `py3-none-manylinux_2_39_x86_64`; `twine check` passed for both artifacts; isolated wheel server CLI smoke passed.
+
+## 2026-05-26 — retarget server KV preallocation release to v0.2.2
+
+Confirmed `v0.2.1` was already immutable on PyPI and tagged on GitHub, so retargeted the server KV preallocation improvements to `v0.2.2`. Updated `pyproject.toml` to `0.2.2` and split `CHANGELOG.md` so `v0.2.1` retains the already-published session reuse/warmup/streaming notes while `v0.2.2` carries the resident context/KV preallocation controls, auto context sizing, retained-KV capacity estimate, `max_tokens=auto`, and partial-construction cleanup notes.
+
+## 2026-05-26 — v0.2.2 release validation and artifact build
+
+Built the server KV preallocation release as `v0.2.2` from clean detached worktree `/tmp/hipengine-release-v021.QhOZ2d` at commit `fc16570` before the release-validation log append.
+
+Validation and artifact commands:
+
+```bash
+python3 -m compileall -q hipengine scripts tests
+uv run --extra dev python -m pytest -q
+uv run --python 3.10 --extra dev python -m pytest -q
+uv run --extra dev hipengine-server --help
+rm -f uv.lock && rm -rf .venv dist
+python3 -m build
+unzip -p dist/hipengine-0.2.2-py3-none-manylinux_2_39_x86_64.whl hipengine-0.2.2.dist-info/WHEEL
+uvx --from twine twine check dist/*
+WHEEL=/tmp/hipengine-release-v021.QhOZ2d/dist/hipengine-0.2.2-py3-none-manylinux_2_39_x86_64.whl; \
+  (cd /tmp && uv run --isolated --with "${WHEEL}[server]" hipengine-server --help)
+```
+
+Results: compileall passed; full pytest passed on CPython 3.10.16 (including the explicit `--python 3.10` run); server CLI smoke passed; `python3 -m build` produced `hipengine-0.2.2.tar.gz` and `hipengine-0.2.2-py3-none-manylinux_2_39_x86_64.whl`; wheel metadata reports `Root-Is-Purelib: false` and tag `py3-none-manylinux_2_39_x86_64`; `twine check` passed for both artifacts; isolated wheel server CLI smoke passed. Rebuild artifacts after this log append before publishing so `dist/` matches the final commit.

{hipengine-0.2.1 → hipengine-0.2.2}/docs/API.md

@@ -40,12 +40,25 @@ select `cpu_reference` where a CPU implementation exists; nearby targets such as
 `gfx1101`/`gfx1102` can force a backend with `--backend hip_gfx1100` or
 `HIPENGINE_BACKEND=hip_gfx1100` after local validation.
 
-By default the server eagerly loads the model and runs a short warmup
-generation at startup so the first real request does not pay load/compile cost.
-Disable with `--no-eager-load` or `HIPENGINE_EAGER_LOAD=0`. The warmup prompt
-and token count are configurable via `--eager-load-prompt` and
+By default the server eagerly loads the model, loads resident weights, estimates
+remaining HIP memory for KV cache plus persistent context metadata, then
+preallocates `min(model max context, estimated allocatable context)`. Pass
+`--max-context-tokens` (or `HIPENGINE_MAX_CONTEXT_TOKENS`) to force a lower cap.
+Startup fails with a clear error if the requested cap cannot be allocated; lower
+`--max-context-tokens` or use `--kv-storage int8_per_token_head`. Disable eager
+startup with `--no-eager-load` or `HIPENGINE_EAGER_LOAD=0`. The warmup prompt and
+token count are configurable via `--eager-load-prompt` and
 `--eager-load-max-tokens`.
 
+The resident KV policy is server-wide: set `--kv-storage` (`auto`, `bf16`, or
+`int8_per_token_head`), `--kv-scale-dtype`, and `--kv-scale-granularity` at
+startup. Requests that ask for a different KV policy are rejected instead of
+rebuilding the resident model. Startup logs include a compact KVCache summary
+from current HIP free memory and warn when even INT8 KV is below the model's
+advertised max context. Chat requests that omit `max_tokens` use
+`max_tokens=auto`, meaning the remaining admitted context
+(`max_context_tokens - prompt_tokens - 1`).
+
 Set `HIPENGINE_API_KEY` or pass `--api-key` to require OpenAI-style bearer
 authentication:
 

{hipengine-0.2.1 → hipengine-0.2.2}/hipengine/generation/qwen35_paro.py

@@ -64,6 +64,36 @@ class Qwen35ParoOneTokenGenerator:
             for prompt in request.prompts
         ]
 
+    def prepare(self, *, max_sequence_length: int | None = None, sampling_params: Any | None = None) -> int:
+        params = sampling_params
+        runner = self._get_runner()
+        kv_policy = resolve_kv_policy(
+            getattr(params, "kv_storage", "auto"),
+            scale_dtype=getattr(params, "kv_scale_dtype", "fp16"),
+            scale_granularity=getattr(params, "kv_scale_granularity", "per_token_head"),
+        )
+        auto_context_length = max_sequence_length is None
+        if auto_context_length:
+            requested_length = int(getattr(runner.config, "max_position_embeddings", 0) or 0)
+            if requested_length <= 0:
+                requested_length = _session_capacity_for(1)
+        else:
+            if int(max_sequence_length) <= 0:
+                raise ValueError("max_sequence_length must be positive")
+            requested_length = int(max_sequence_length)
+        session_capacity = _session_capacity_for(requested_length)
+        session = self._get_session(
+            runner,
+            max_sequence_length=session_capacity,
+            kv_policy=kv_policy,
+            auto_context_length=auto_context_length,
+        )
+        return int(getattr(session, "max_sequence_length", self._session_capacity))
+
+    def count_tokens(self, text: str) -> int:
+        _last_token_id, prompt_ids = _select_token(Path(self.model_path), str(text), None)
+        return len(prompt_ids)
+
     def stream(self, request: GenerationRequest) -> Iterator[str]:
         if len(request.prompts) != 1:
             raise ValueError("streaming currently supports exactly one prompt")
@@ -183,6 +213,7 @@ class Qwen35ParoOneTokenGenerator:
         *,
         max_sequence_length: int,
         kv_policy,
+        auto_context_length: bool = False,
     ) -> Qwen35ParoResidentSession:
         kv_key = (
             kv_policy.storage_dtype.value,
@@ -190,20 +221,23 @@ class Qwen35ParoOneTokenGenerator:
             kv_policy.scale_granularity,
             int(kv_policy.block_size),
         )
+        capacity_ok = self._session_capacity >= max_sequence_length or bool(auto_context_length)
         if (
             self._session is None
-            or self._session_capacity < max_sequence_length
+            or not capacity_ok
             or self._session_kv_key != kv_key
         ):
             self.close()
-            self._session = Qwen35ParoResidentSession(
-                runner,
-                max_sequence_length=max_sequence_length,
-                kv_policy=kv_policy.create_policy(),
-                kv_scale_dtype=kv_policy.scale_dtype,
-                kv_scale_granularity=kv_policy.scale_granularity,
-            )
-            self._session_capacity = max_sequence_length
+            session_kwargs = {
+                "max_sequence_length": max_sequence_length,
+                "kv_policy": kv_policy.create_policy(),
+                "kv_scale_dtype": kv_policy.scale_dtype,
+                "kv_scale_granularity": kv_policy.scale_granularity,
+            }
+            if auto_context_length:
+                session_kwargs["auto_context_length"] = True
+            self._session = Qwen35ParoResidentSession(runner, **session_kwargs)
+            self._session_capacity = int(getattr(self._session, "max_sequence_length", max_sequence_length))
             self._session_kv_key = kv_key
         else:
             self._session.reset()

{hipengine-0.2.1 → hipengine-0.2.2}/hipengine/llm.py

@@ -71,6 +71,36 @@ class LLM:
         for text in generator.generate(request):
             yield text
 
+    def prepare(
+        self,
+        *,
+        max_sequence_length: int | None = None,
+        sampling_params: SamplingParams | None = None,
+    ) -> int | None:
+        """Eagerly prepare a resident session when the generator supports it.
+
+        Passing ``max_sequence_length=None`` lets generators choose the largest
+        context they can preallocate for the selected model/KV policy.
+        """
+
+        generator = self._get_text_generator()
+        preparer = getattr(generator, "prepare", None)
+        if not callable(preparer):
+            return None
+        return preparer(
+            max_sequence_length=None if max_sequence_length is None else int(max_sequence_length),
+            sampling_params=sampling_params or SamplingParams(),
+        )
+
+    def count_tokens(self, text: str) -> int:
+        """Return tokenizer token count when the resolved generator exposes one."""
+
+        generator = self._get_text_generator()
+        counter = getattr(generator, "count_tokens", None)
+        if not callable(counter):
+            raise NotImplementedError("token counting is not supported by this generator")
+        return int(counter(str(text)))
+
     def _get_text_generator(self) -> Any:
         if self._text_generator is not None:
             return self._text_generator

{hipengine-0.2.1 → hipengine-0.2.2}/hipengine/loading/qwen35_paro.py

@@ -46,6 +46,7 @@ class Qwen35ParoConfig:
     layer_types: tuple[str, ...]
     quant_method: str
     vocab_size: int = 0
+    max_position_embeddings: int = 0
     rms_norm_eps: float = 1.0e-6
     rope_theta: float = 1000000.0
     rotary_dim: int = 0
@@ -142,6 +143,7 @@ def qwen35_paro_config_from_hf(config: dict[str, Any]) -> Qwen35ParoConfig:
         layer_types=layer_types,
         quant_method=quant_method,
         vocab_size=int(text.get("vocab_size", 0) or 0),
+        max_position_embeddings=int(text.get("max_position_embeddings", config.get("max_position_embeddings", 0)) or 0),
         rms_norm_eps=float(text.get("rms_norm_eps", 1.0e-6) or 1.0e-6),
         rope_theta=float(text.get("rope_theta", rope_parameters.get("rope_theta", 1000000.0)) or 1000000.0),
         rotary_dim=rotary_dim,