hipengine 0.2.0__tar.gz → 0.2.1__tar.gz

{hipengine-0.2.0 → hipengine-0.2.1}/CHANGELOG.md

@@ -6,6 +6,47 @@ This changelog is for package/API releases. Performance rollup history remains i
 [`benchmarks/CHANGELOG.md`](benchmarks/CHANGELOG.md), with detailed benchmark
 evidence under [`benchmarks/results/`](benchmarks/results/).
 
+## v0.2.1 - 2026-05-25
+
+Patch release improving server session management, streaming, and
+OpenAI-compatible reasoning output.
+
+### Added
+
+- Eager model warmup on server startup: the configured model and a short
+  warmup generation run before uvicorn reports ready, so the first real
+  request does not pay load/compile cost. Controlled by `--eager-load` /
+  `--no-eager-load` (default: on), `--eager-load-prompt`, and
+  `--eager-load-max-tokens`, with `HIPENGINE_EAGER_LOAD`,
+  `HIPENGINE_EAGER_LOAD_PROMPT`, and `HIPENGINE_EAGER_LOAD_MAX_TOKENS`
+  environment variable equivalents.
+- `LLM.stream()` method for single-prompt token-by-token generation when
+  the underlying text generator supports it.
+- Reasoning-content splitting for chat completions: `<think>…</think>`
+  spans (Qwen/DeepSeek-style) are now separated into
+  `message.reasoning_content` (non-streaming) or `delta.reasoning_content`
+  chunks (streaming), matching the OpenAI reasoning-content convention.
+
+### Changed
+
+- PARO text generators and their resident sessions are now cached on the
+  `LLM` instance and reused across requests. Session capacity is bucketed
+  (floor 4 Ki tokens, configurable via `HIPENGINE_SESSION_MIN_TOKENS` and
+  `HIPENGINE_SESSION_BUCKET_TOKENS`) so normal chat-history growth does not
+  force reallocation every turn.
+- Chat `stream=true` now yields token-level SSE chunks from the resident
+  decode loop instead of buffering the full response and wrapping it in a
+  single SSE frame.
+- Chat completions default `max_tokens` raised from 16 to 8192 so clients
+  that omit the field get usable reply lengths, including verbose
+  chain-of-thought reasoning.
+
+### Fixed
+
+- Fixed `LLM.generate()` re-resolving the generation factory on every call,
+  which discarded generator-local caches and caused the PARO resident
+  session (layer weights, KV buffers) to be allocated and freed per request.
+
 ## v0.2.0 - 2026-05-25
 
 Minor release for the GGUF runtime path and W7900 benchmark refresh. GGUF is a

{hipengine-0.2.0 → hipengine-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hipengine
-Version: 0.2.0
+Version: 0.2.1
 Summary: ROCm-native local LLM inference engine with a torch-free runtime hot path
 Project-URL: Homepage, https://github.com/shisa-ai/hipEngine
 Project-URL: Repository, https://github.com/shisa-ai/hipEngine
@@ -75,7 +75,7 @@ supported GPUs and models.
 
 ## Status
 
-**v0.2.0 alpha.** The runtime hot path is torch-free by construction, and the
+**v0.2.1 alpha.** The runtime hot path is torch-free by construction, and the
 first two 35B-class model-loading surfaces are now available on gfx1100:
 [shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed](https://huggingface.co/shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed)
 (19.07 GiB, 4.68 bpw) in packed
@@ -336,13 +336,20 @@ Install the optional server extra and run the FastAPI layer:
 ```bash
 pip install -e ".[server]"
 python -m hipengine.server \
-  --model /path/to/model \
+  --model shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed \
   --quant w4_paro \
   --served-model-name qwen-paro
 ```
 
-Supported v0.1 endpoints: `GET /v1/models`, `POST /v1/completions`, and
-`POST /v1/chat/completions` (including one-chunk SSE for `stream=true`). See
+`--model` accepts either a local filesystem path or a Hugging Face model ID
+already present in the local HF cache; hipEngine resolves IDs locally and does
+not download weights during startup.
+
+Supported endpoints: `GET /v1/models`, `POST /v1/completions`, and
+`POST /v1/chat/completions` with token-level SSE streaming. Chat responses
+separate `<think>` reasoning into `reasoning_content` (matching the OpenAI
+reasoning-content convention). The server eagerly warms the model on startup
+by default so the first request does not pay load/compile cost. See
 [`docs/API.md`](docs/API.md) for request examples, bearer-token auth, and
 current limitations.
 

{hipengine-0.2.0 → hipengine-0.2.1}/README.md

@@ -37,7 +37,7 @@ supported GPUs and models.
 
 ## Status
 
-**v0.2.0 alpha.** The runtime hot path is torch-free by construction, and the
+**v0.2.1 alpha.** The runtime hot path is torch-free by construction, and the
 first two 35B-class model-loading surfaces are now available on gfx1100:
 [shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed](https://huggingface.co/shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed)
 (19.07 GiB, 4.68 bpw) in packed
@@ -298,13 +298,20 @@ Install the optional server extra and run the FastAPI layer:
 ```bash
 pip install -e ".[server]"
 python -m hipengine.server \
-  --model /path/to/model \
+  --model shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed \
   --quant w4_paro \
   --served-model-name qwen-paro
 ```
 
-Supported v0.1 endpoints: `GET /v1/models`, `POST /v1/completions`, and
-`POST /v1/chat/completions` (including one-chunk SSE for `stream=true`). See
+`--model` accepts either a local filesystem path or a Hugging Face model ID
+already present in the local HF cache; hipEngine resolves IDs locally and does
+not download weights during startup.
+
+Supported endpoints: `GET /v1/models`, `POST /v1/completions`, and
+`POST /v1/chat/completions` with token-level SSE streaming. Chat responses
+separate `<think>` reasoning into `reasoning_content` (matching the OpenAI
+reasoning-content convention). The server eagerly warms the model on startup
+by default so the first request does not pay load/compile cost. See
 [`docs/API.md`](docs/API.md) for request examples, bearer-token auth, and
 current limitations.
 

{hipengine-0.2.0 → hipengine-0.2.1}/WORKLOG.md

@@ -26555,3 +26555,82 @@ uv run --extra dev python -m pytest -q
 ```
 
 Next: commit, push `main`, move annotated tag `v0.2.0` to the fix commit, force-push the tag per user approval, and watch the publish workflow complete via trusted publishing.
+
+### v0.2.0 publish workflow result
+
+Committed and pushed the HIP guard fix (`36e3516`), force-moved annotated tag `v0.2.0` to that commit per user approval, and re-pushed the tag. The `Publish to PyPI` workflow reran as `26370452261` and completed successfully: build validation passed, artifact attestation was generated, and `Publish to PyPI (trusted publishing)` succeeded. Verified published install path:
+
+```bash
+uvx --refresh --from "hipengine[server]==0.2.0" hipengine-server --help
+# downloaded hipengine 0.2.0 from PyPI and printed CLI help
+```
+
+Final release pointers: GitHub release `https://github.com/shisa-ai/hipEngine/releases/tag/v0.2.0`, publish workflow `https://github.com/shisa-ai/hipEngine/actions/runs/26370452261`, release tag commit `36e351607955158e821b51f443bac042b140624f`.
+
+## 2026-05-25 — HF cache model id resolution for server/LLM
+
+Added local Hugging Face cache resolution for public model references so `hipengine-server --model` and `LLM(model=...)` can take either a filesystem path or a cached HF model id such as `shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed`. Resolution remains local-only: existing paths win, then `huggingface_hub.snapshot_download(..., local_files_only=True)` is used if available, then the standard HF cache layout is inspected directly; no weights are downloaded during startup. GGUF discovery now resolves the model reference first, so cached GGUF repos/directories work as well as direct `.gguf` file paths.
+
+Also updated README/API server examples to show the HF model id form and note that cache resolution is local-only. User's `uv run --extra server ...` command is the right shape; the `--no-build` error indicates a uv no-build mode/config/environment for the local editable package, not a hipEngine server argument issue. Workaround is to unset no-build for local editable runs or use the published wheel via `uvx --from "hipengine[server]==0.2.0"`.
+
+Validation:
+
+```bash
+python -m py_compile hipengine/loading/hf_cache.py hipengine/loading/safetensors.py hipengine/loading/gguf.py hipengine/llm.py
+python -m pytest -q tests/test_hf_cache.py tests/test_llm_generate.py tests/test_loading_safetensors.py tests/test_server_api.py
+# 19 passed
+uv run --extra dev python -m pytest -q
+# passed
+```
+## 2026-05-23 — llama.cpp wrapper metadata fix for RX 7900 XTX 5-run sweep
+
+Before rerunning `benchmarks/7900XTX.md` with `--repetitions 5`, fixed `scripts/llamacpp_bench_with_peak.py` artifact metadata so it no longer hardcodes W7900 or a single-shot `--repetitions 1` note. The artifact now records the selected amdgpu card name/PCI/VRAM in `hardware` and emits a dynamic repetitions note. Validation: `python3 -m py_compile scripts/llamacpp_bench_with_peak.py`.
+
+## 2026-05-24 — server resident-session reuse smoke
+
+Debugged the local OpenAI-compatible PARO server after LAN chat showed short `<think>` replies and apparent reloads. Root causes:
+
+- `LLM.generate()` re-resolved the generation factory and constructed a new text generator on every call, discarding generator-local caches.
+- `Qwen35ParoOneTokenGenerator` then constructed `Qwen35ParoResidentSession` inside each prompt call and closed it immediately, so resident layer weights/KV buffers were materialized and freed per request.
+- The OpenAI chat server default `max_tokens` was 16; clients that omit `max_tokens` saw very short replies. The Qwen/PARO model config advertises `max_position_embeddings=262144`, but the runtime session capacity was allocated as `len(prompt_ids) + max_tokens + 1` per request.
+- `stream=true` only returned an SSE wrapper around the completed response; it did not yield tokens while generation was running.
+
+Fixes made in-tree:
+
+- Cache the resolved text generator on `hipengine.LLM` so server `app.state.hipengine_llm` keeps its backend/model/quant generator across requests.
+- Cache/reuse `Qwen35ParoResidentSession` inside the PARO generator when the existing session capacity and KV policy cover the next request; reset it between prompts instead of closing it. Session capacity now floors/buckets at 4096 tokens by default (`HIPENGINE_SESSION_MIN_TOKENS`, `HIPENGINE_SESSION_BUCKET_TOKENS`) so normal chat history growth does not force reallocation every turn.
+- Add a single-prompt streaming path using resident `step()` per token, and route chat `stream=true` through it. Chat default `max_tokens` is now 256.
+
+Validation:
+
+```bash
+uv run --extra dev python -m pytest -q tests/test_llm_generate.py tests/test_generation_qwen35_paro.py tests/test_server_api.py
+# 17 passed
+```
+
+Restarted the running LAN server from the local checkout with TheRock ROCm library paths:
+
+```bash
+uv run --extra server hipengine-server --model shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed --quant w4_paro --served-model-name qwen-paro --host 0.0.0.0 --port 8000
+```
+
+Smoke evidence on the running server after restart from the local checkout: first streamed `max_tokens=4` request sent the role chunk immediately, then took 30.25s to warm/materialize and yielded token chunks (`<think>`, newline, `Here`, `'s`); VRAM reached 19.54GB. A second chat-history request reused the resident 4096-token session, yielded the first content chunk at 0.46s, completed at 0.49s, and VRAM stayed resident at 19.55GB. Server PID 9305, wrapper PID recorded in `/tmp/hipengine-server-8000.pid`, log `/tmp/hipengine-server-8000.log`.
+
+## 2026-05-24 — eager server warmup and reasoning-channel split
+
+Implemented eager server warmup and OpenAI-compatible reasoning segregation for the local PARO server.
+
+Changes:
+
+- Added `ServerConfig.eager_load` plus CLI `--eager-load/--no-eager-load` (default on), `--eager-load-prompt`, and `--eager-load-max-tokens`. Startup now constructs `LLM` and runs a one-token warmup before uvicorn reports startup complete, so `/v1/models` is only reachable after the model/session is resident.
+- Default warmup prompt is `one two three four`; the initial `hello` default failed native prefill because Qwen35/PARO requires at least `linear_conv_kernel_dim` (4) prompt tokens.
+- Split Qwen/DeepSeek-style `<think>...</think>` spans in chat responses. Non-stream responses put visible answer text in `message.content` and hidden reasoning text in `message.reasoning_content`. Streaming responses emit `delta.reasoning_content` for reasoning chunks and `delta.content` for final answer chunks.
+
+Validation:
+
+```bash
+uv run --extra dev python -m pytest -q tests/test_server_api.py tests/test_llm_generate.py tests/test_generation_qwen35_paro.py
+# 19 passed
+```
+
+Restarted the LAN server from the local checkout with TheRock ROCm library paths. Eager startup completed after 31s, `/v1/models` returned `qwen-paro`, and VRAM was already resident at 19.54GB before serving requests. A subsequent streamed chat request returned the role chunk at 0.01s and reasoning chunks as `delta.reasoning_content` from 0.49s onward, with VRAM staying resident at 19.54GB. Server PID 12107, wrapper PID 12082, log `/tmp/hipengine-server-8000.log`.

{hipengine-0.2.0 → hipengine-0.2.1}/docs/API.md

@@ -1,6 +1,6 @@
 # OpenAI-Compatible Server API
 
-Last updated: 2026-05-18
+Last updated: 2026-05-25
 
 hipEngine ships a thin optional FastAPI layer that adapts OpenAI-style requests
 to the torch-free `hipengine.LLM.generate()` library API. It is installed only
@@ -17,13 +17,17 @@ pip install -e ".[server]"
 
 ```bash
 python -m hipengine.server \
-  --model /path/to/qwen-paro-model \
+  --model shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed \
   --quant w4_paro \
   --served-model-name qwen-paro \
   --host 127.0.0.1 \
   --port 8000
 ```
 
+`--model` accepts a local filesystem path or a Hugging Face model ID that is
+already present in the local HF cache. hipEngine resolves IDs with local cache
+lookups only; it does not download weights during server startup.
+
 After installation, the console script is equivalent:
 
 ```bash
@@ -36,6 +40,12 @@ select `cpu_reference` where a CPU implementation exists; nearby targets such as
 `gfx1101`/`gfx1102` can force a backend with `--backend hip_gfx1100` or
 `HIPENGINE_BACKEND=hip_gfx1100` after local validation.
 
+By default the server eagerly loads the model and runs a short warmup
+generation at startup so the first real request does not pay load/compile cost.
+Disable with `--no-eager-load` or `HIPENGINE_EAGER_LOAD=0`. The warmup prompt
+and token count are configurable via `--eager-load-prompt` and
+`--eager-load-max-tokens`.
+
 Set `HIPENGINE_API_KEY` or pass `--api-key` to require OpenAI-style bearer
 authentication:
 
@@ -50,8 +60,8 @@ curl -H 'Authorization: Bearer local-secret' http://127.0.0.1:8000/v1/models
 | --- | --- | --- |
 | `GET /health` | Built in | Unauthenticated health/model probe. |
 | `GET /v1/models` | Built in | Returns the single served model id. |
-| `POST /v1/completions` | Built in | Text prompt(s) to `LLM.generate()`. Supports `stream=true` as one server-sent event chunk plus `[DONE]`. |
-| `POST /v1/chat/completions` | Built in | Renders text-only messages to a Qwen-style prompt and calls `LLM.generate()`. Supports `stream=true` as one server-sent event chunk plus `[DONE]`. |
+| `POST /v1/completions` | Built in | Text prompt(s) to `LLM.generate()`. Supports `stream=true` (one SSE chunk plus `[DONE]`). |
+| `POST /v1/chat/completions` | Built in | Renders text-only messages to a Qwen-style prompt and calls `LLM.generate()`. Supports token-level `stream=true` SSE. `<think>` spans are separated into `reasoning_content` (non-streaming) or `delta.reasoning_content` chunks (streaming). |
 
 ## Examples
 

hipengine-0.2.1/hipengine/generation/qwen35_paro.py

@@ -0,0 +1,298 @@
+"""Qwen3.5/PARO text generation bring-up path."""
+
+from __future__ import annotations
+
+from collections.abc import Iterator
+from dataclasses import dataclass, field
+import os
+from pathlib import Path
+from typing import Any
+
+from hipengine.generation.registry import GenerationRequest, register_text_generator
+from hipengine.kvcache import resolve_kv_policy
+from hipengine.loading import WeightIndex
+from hipengine.runtime.qwen35_paro_runner import (
+    Qwen35ParoNextTokenRunner,
+    Qwen35ParoResidentSession,
+    _decode_token_cached,
+    _select_token,
+)
+
+
+@dataclass
+class Qwen35ParoOneTokenGenerator:
+    """Greedy Qwen3.5/PARO generator backed by resident c=1 execution.
+
+    The implementation is still serial across prompts, but each prompt uses the
+    resident single-request native prefill path followed by multi-token
+    autoregressive decode using the resident HIP layer chain.
+    """
+
+    model_path: str | Path
+    weight_index: WeightIndex
+    model_plugin: Any
+    backend: str = "auto"
+    lm_head_chunk: int = 4096
+    _runner: Qwen35ParoNextTokenRunner | None = field(default=None, init=False, repr=False)
+    _session: Qwen35ParoResidentSession | None = field(default=None, init=False, repr=False)
+    _session_capacity: int = field(default=0, init=False, repr=False)
+    _session_kv_key: tuple[str, str, str, int] | None = field(default=None, init=False, repr=False)
+
+    def generate(self, request: GenerationRequest) -> list[str]:
+        if request.max_tokens < 0:
+            raise ValueError("max_tokens must be non-negative")
+        if request.temperature != 0.0 or request.top_p != 1.0:
+            raise NotImplementedError(
+                "Qwen3.5/PARO generator currently supports greedy sampling only"
+            )
+        if request.max_tokens == 0:
+            return ["" for _ in request.prompts]
+        runner = self._get_runner()
+        kv_policy = resolve_kv_policy(
+            request.kv_storage,
+            scale_dtype=request.kv_scale_dtype,
+            scale_granularity=request.kv_scale_granularity,
+        )
+        return [
+            self._generate_one(
+                runner,
+                prompt,
+                request.max_tokens,
+                ignore_eos=request.ignore_eos,
+                kv_policy=kv_policy,
+            )
+            for prompt in request.prompts
+        ]
+
+    def stream(self, request: GenerationRequest) -> Iterator[str]:
+        if len(request.prompts) != 1:
+            raise ValueError("streaming currently supports exactly one prompt")
+        if request.max_tokens < 0:
+            raise ValueError("max_tokens must be non-negative")
+        if request.temperature != 0.0 or request.top_p != 1.0:
+            raise NotImplementedError(
+                "Qwen3.5/PARO generator currently supports greedy sampling only"
+            )
+        if request.max_tokens == 0:
+            return
+        runner = self._get_runner()
+        kv_policy = resolve_kv_policy(
+            request.kv_storage,
+            scale_dtype=request.kv_scale_dtype,
+            scale_granularity=request.kv_scale_granularity,
+        )
+        yield from self._stream_one(
+            runner,
+            request.prompts[0],
+            request.max_tokens,
+            ignore_eos=request.ignore_eos,
+            kv_policy=kv_policy,
+        )
+
+    def _generate_one(
+        self,
+        runner: Qwen35ParoNextTokenRunner,
+        prompt: str,
+        max_tokens: int,
+        *,
+        ignore_eos: bool,
+        kv_policy,
+    ) -> str:
+        _last_token_id, prompt_ids = _select_token(Path(self.model_path), prompt, None)
+        if not prompt_ids:
+            raise ValueError("prompt produced no tokens")
+        required_sequence_length = len(prompt_ids) + max_tokens + 1
+        session_capacity = _session_capacity_for(required_sequence_length)
+        generated_text: list[str] = []
+        session = self._get_session(
+            runner,
+            max_sequence_length=session_capacity,
+            kv_policy=kv_policy,
+        )
+        next_result = session.prefill_native(prompt_ids, sample=True)
+        if next_result is None:
+            raise RuntimeError("native prefill did not produce next-token logits")
+        generated_text.append(next_result.token_text)
+        if not ignore_eos and _is_eos(session.tokenizer, next_result.token_id):
+            return "".join(generated_text)
+
+        remaining = max_tokens - 1
+        if remaining:
+            with session.capture_decode_graph(
+                position=len(prompt_ids),
+                steps_per_replay=1,
+                max_replay_steps=remaining,
+                record_steps=remaining,
+            ) as graph:
+                graph.replay(remaining)
+                token_ids = graph.read_generated_token_ids(remaining)
+            for token_id in token_ids:
+                generated_text.append(_decode_token_cached(session.tokenizer, token_id))
+                if not ignore_eos and _is_eos(session.tokenizer, token_id):
+                    break
+        return "".join(generated_text)
+
+    def _stream_one(
+        self,
+        runner: Qwen35ParoNextTokenRunner,
+        prompt: str,
+        max_tokens: int,
+        *,
+        ignore_eos: bool,
+        kv_policy,
+    ) -> Iterator[str]:
+        _last_token_id, prompt_ids = _select_token(Path(self.model_path), prompt, None)
+        if not prompt_ids:
+            raise ValueError("prompt produced no tokens")
+        required_sequence_length = len(prompt_ids) + max_tokens + 1
+        session_capacity = _session_capacity_for(required_sequence_length)
+        session = self._get_session(
+            runner,
+            max_sequence_length=session_capacity,
+            kv_policy=kv_policy,
+        )
+        next_result = session.prefill_native(prompt_ids, sample=True)
+        if next_result is None:
+            raise RuntimeError("native prefill did not produce next-token logits")
+        yield next_result.token_text
+        if not ignore_eos and _is_eos(session.tokenizer, next_result.token_id):
+            return
+
+        current_token_id = next_result.token_id
+        for position in range(len(prompt_ids), len(prompt_ids) + max_tokens - 1):
+            result = session.step(current_token_id, position=position, sample=True)
+            if result is None:
+                raise RuntimeError("decode step did not produce next-token logits")
+            yield result.token_text
+            current_token_id = result.token_id
+            if not ignore_eos and _is_eos(session.tokenizer, result.token_id):
+                return
+
+    def _get_runner(self) -> Qwen35ParoNextTokenRunner:
+        if self._runner is None:
+            self._runner = Qwen35ParoNextTokenRunner(
+                self.model_path,
+                index=self.weight_index,
+                backend=self.backend,
+            )
+        return self._runner
+
+    def _get_session(
+        self,
+        runner: Qwen35ParoNextTokenRunner,
+        *,
+        max_sequence_length: int,
+        kv_policy,
+    ) -> Qwen35ParoResidentSession:
+        kv_key = (
+            kv_policy.storage_dtype.value,
+            kv_policy.scale_dtype.value,
+            kv_policy.scale_granularity,
+            int(kv_policy.block_size),
+        )
+        if (
+            self._session is None
+            or self._session_capacity < max_sequence_length
+            or self._session_kv_key != kv_key
+        ):
+            self.close()
+            self._session = Qwen35ParoResidentSession(
+                runner,
+                max_sequence_length=max_sequence_length,
+                kv_policy=kv_policy.create_policy(),
+                kv_scale_dtype=kv_policy.scale_dtype,
+                kv_scale_granularity=kv_policy.scale_granularity,
+            )
+            self._session_capacity = max_sequence_length
+            self._session_kv_key = kv_key
+        else:
+            self._session.reset()
+        return self._session
+
+    def close(self) -> None:
+        if self._session is not None:
+            self._session.close()
+            self._session = None
+        self._session_capacity = 0
+        self._session_kv_key = None
+
+
+def _session_capacity_for(required_sequence_length: int) -> int:
+    """Return a reusable session capacity for a request.
+
+    Chat prompts grow after every turn, so allocating exactly the current
+    prompt+decode length forces resident weights/KV buffers to be torn down and
+    rebuilt on each request.  Keep a modest floor and bucket growth to preserve
+    the resident session across normal local chat turns while still allowing
+    larger explicit contexts to expand on demand.
+    """
+
+    required = int(required_sequence_length)
+    if required <= 0:
+        raise ValueError("required_sequence_length must be positive")
+    floor = max(1, _env_int("HIPENGINE_SESSION_MIN_TOKENS", 4096))
+    bucket = max(1, _env_int("HIPENGINE_SESSION_BUCKET_TOKENS", 1024))
+    capacity = max(required, floor)
+    return ((capacity + bucket - 1) // bucket) * bucket
+
+
+def _env_int(name: str, default: int) -> int:
+    raw = os.environ.get(name)
+    if raw is None or raw == "":
+        return default
+    try:
+        return int(raw)
+    except ValueError:
+        return default
+
+
+def _is_eos(tokenizer: Any | None, token_id: int) -> bool:
+    if tokenizer is None:
+        return False
+    try:
+        eos_id = getattr(tokenizer, "token_to_id")("<|endoftext|>")
+    except Exception:
+        eos_id = None
+    return eos_id is not None and int(token_id) == int(eos_id)
+
+
+def make_qwen35_paro_one_token_generator(
+    *,
+    model_path: str | Path,
+    weight_index: WeightIndex,
+    model_plugin: Any,
+) -> Qwen35ParoOneTokenGenerator:
+    return Qwen35ParoOneTokenGenerator(
+        model_path=model_path,
+        weight_index=weight_index,
+        model_plugin=model_plugin,
+        backend="hip_gfx1100",
+    )
+
+
+def make_qwen35_paro_one_token_generator_gfx1151(
+    *,
+    model_path: str | Path,
+    weight_index: WeightIndex,
+    model_plugin: Any,
+) -> Qwen35ParoOneTokenGenerator:
+    return Qwen35ParoOneTokenGenerator(
+        model_path=model_path,
+        weight_index=weight_index,
+        model_plugin=model_plugin,
+        backend="hip_gfx1151",
+    )
+
+
+register_text_generator(
+    model="qwen3_5_moe_paro",
+    backend="hip_gfx1100",
+    quant="w4_paro",
+    factory=make_qwen35_paro_one_token_generator,
+)
+register_text_generator(
+    model="qwen3_5_moe_paro",
+    backend="hip_gfx1151",
+    quant="w4_paro",
+    factory=make_qwen35_paro_one_token_generator_gfx1151,
+)

{hipengine-0.2.0 → hipengine-0.2.1}/hipengine/llm.py

@@ -6,7 +6,7 @@ through a registry at call time so backend/quant choices do not become engine br
 
 from __future__ import annotations
 
-from collections.abc import Iterable
+from collections.abc import Iterable, Iterator
 from dataclasses import dataclass
 from pathlib import Path
 from typing import Any
@@ -41,6 +41,7 @@ class LLM:
         self._resolved_backend: str | None = None
         self._weight_index: Any | None = None
         self._model_plugin: Any | None = None
+        self._text_generator: Any | None = None
 
     def generate(
         self,
@@ -50,13 +51,31 @@ class LLM:
         prompt_tuple = _normalize_prompts(prompts)
         if not prompt_tuple:
             return []
-        params = sampling_params or SamplingParams()
+        generator = self._get_text_generator()
+        request = _generation_request(prompt_tuple, sampling_params or SamplingParams())
+        return generator.generate(request)
 
-        from hipengine.generation import (
-            GenerationRequest,
-            register_builtin_generators,
-            resolve_text_generator,
-        )
+    def stream(
+        self,
+        prompt: str,
+        sampling_params: SamplingParams | None = None,
+    ) -> Iterator[str]:
+        """Yield generated text chunks for a single prompt when supported."""
+
+        generator = self._get_text_generator()
+        request = _generation_request((str(prompt),), sampling_params or SamplingParams())
+        streamer = getattr(generator, "stream", None)
+        if callable(streamer):
+            yield from streamer(request)
+            return
+        for text in generator.generate(request):
+            yield text
+
+    def _get_text_generator(self) -> Any:
+        if self._text_generator is not None:
+            return self._text_generator
+
+        from hipengine.generation import register_builtin_generators, resolve_text_generator
 
         register_builtin_generators()
         weight_index, model_plugin = self._load_model_metadata()
@@ -66,23 +85,12 @@ class LLM:
             backend=backend,
             quant=self.quant,
         )
-        generator = factory(
+        self._text_generator = factory(
             model_path=self.model,
             weight_index=weight_index,
             model_plugin=model_plugin,
         )
-        return generator.generate(
-            GenerationRequest(
-                prompts=prompt_tuple,
-                max_tokens=params.max_tokens,
-                temperature=params.temperature,
-                top_p=params.top_p,
-                ignore_eos=params.ignore_eos,
-                kv_storage=params.kv_storage,
-                kv_scale_dtype=params.kv_scale_dtype,
-                kv_scale_granularity=params.kv_scale_granularity,
-            )
-        )
+        return self._text_generator
 
     def _resolve_backend(self) -> str:
         if self._resolved_backend is not None:
@@ -97,10 +105,10 @@ class LLM:
         if self._weight_index is not None and self._model_plugin is not None:
             return self._weight_index, self._model_plugin
 
-        from hipengine.loading import discover_gguf_files, load_gguf_index, load_weight_index
+        from hipengine.loading import discover_gguf_files, load_gguf_index, load_weight_index, resolve_model_path
         from hipengine.models import resolve_model
 
-        model_path = Path(self.model).expanduser()
+        model_path = resolve_model_path(self.model)
         if _looks_like_gguf_path(model_path):
             index = load_gguf_index(discover_gguf_files(model_path)[0])
             self.model = str(index.path)
@@ -116,6 +124,21 @@ class LLM:
         return index, plugin
 
 
+def _generation_request(prompt_tuple: tuple[str, ...], params: SamplingParams):
+    from hipengine.generation import GenerationRequest
+
+    return GenerationRequest(
+        prompts=prompt_tuple,
+        max_tokens=params.max_tokens,
+        temperature=params.temperature,
+        top_p=params.top_p,
+        ignore_eos=params.ignore_eos,
+        kv_storage=params.kv_storage,
+        kv_scale_dtype=params.kv_scale_dtype,
+        kv_scale_granularity=params.kv_scale_granularity,
+    )
+
+
 def _looks_like_gguf_path(path: Path) -> bool:
     if path.is_file():
         return path.suffix.lower() == ".gguf"