hipengine 0.1.1__tar.gz → 0.2.0__tar.gz

hipengine-0.2.0/.gitattributes

@@ -0,0 +1,10 @@
+# WORKLOG.md is append-only: when two branches both add new entries, git's
+# built-in union driver concatenates both sides instead of emitting conflict
+# markers. For an append-only file this is the correct semantics (common
+# prefix + ours-tail + theirs-tail). See scripts/resolve_worklog_conflict.py
+# for the recovery path when conflict markers already exist in the file.
+WORKLOG.md merge=union
+
+hipengine/kernels/hip_gfx1100/attention/aotriton_runtime/** -whitespace
+hipengine/kernels/hip_gfx1100/attention/aotriton_runtime/**/*.aks2 filter=lfs diff=lfs merge=lfs -text
+hipengine/kernels/hip_gfx1100/attention/aotriton_runtime/**/*.so* filter=lfs diff=lfs merge=lfs -text

{hipengine-0.1.1 → hipengine-0.2.0}/AGENTS.md

@@ -64,6 +64,7 @@ Do not drift these casually. They define what hipEngine is.
 
 - Keep changes scoped to one logical unit (one kernel family, one plugin, one doc, one phase milestone).
 - Write or update the targeted test/fixture before implementation when behavior or math changes. If RED-first is impractical, record why in `WORKLOG.md`.
+- When adding tests that call HIP/ROCm runtime, `hipcc`, or GPU kernels, add an explicit HIP-availability guard (for example `ctypes.CDLL("libamdhip64.so")` + `pytest.skip`) so no-ROCm CI/publish runners skip them instead of failing release validation.
 - Log non-trivial decisions, measurements, and dependency additions in `WORKLOG.md` as they happen.
 - When profiling Python/ctypes JIT-built kernels with `rocprofv3`, prebuild the `.so` outside the profiler and run the profiled command with a precomputed compiler-version file plus `require_cached`; do not let the profiled process spawn `hipcc`/clang.
 - Do not silently add `import torch`, `flash_attn`, or other CUDA-only deps to hot-path modules.
@@ -144,6 +145,7 @@ Working tree is shared state. Other agents or the human may be editing concurren
 - **High-conflict files:** `AGENTS.md`, `CLAUDE.md`, `docs/PLAN.md`, `docs/BENCHMARK.md`, `docs/TESTING.md`, `docs/KERNELS.md`, `docs/IMPLEMENTATION.md`, `WORKLOG.md`, `pyproject.toml`, `hipengine/kernels/registry.py`, `hipengine/quant/registry.py`, `hipengine/models/registry.py`, `hipengine/dispatch/fusion.py`, `hipengine/core/*`.
 - Same-file contention: stop and coordinate. The designated agent stages and commits their scoped hunks first to unblock others.
 - `WORKLOG.md` appends are expected and not a conflict unless there are actual conflict markers or interleaved garbled lines. Re-read the live tail, append after it, commit with your logical unit.
+- `WORKLOG.md` is configured with git's built-in `merge=union` driver (see `.gitattributes`), so concurrent appends auto-resolve as `common prefix + ours-tail + theirs-tail` with no conflict markers. If markers do appear (e.g. a stash or a rebase started before this was configured), run `python3 scripts/resolve_worklog_conflict.py WORKLOG.md` (add `--sort-by-date` to re-order `## YYYY-MM-DD` sections in each resolved block; `--check` for a pre-commit gate). The script only touches conflict blocks; content outside markers is left exactly as-is.
 - Do not clean up another agent's benchmark outputs, staged files, or local artifacts unless the task explicitly asks for that cleanup.
 
 ## External Reference Repos

{hipengine-0.1.1 → hipengine-0.2.0}/CHANGELOG.md

@@ -6,6 +6,54 @@ This changelog is for package/API releases. Performance rollup history remains i
 [`benchmarks/CHANGELOG.md`](benchmarks/CHANGELOG.md), with detailed benchmark
 evidence under [`benchmarks/results/`](benchmarks/results/).
 
+## v0.2.0 - 2026-05-25
+
+Minor release for the GGUF runtime path and W7900 benchmark refresh. GGUF is a
+meaningful new model-loading surface rather than a patch-level fix, so this
+supersedes the previously planned v0.1.2 patch.
+
+### Added
+
+- Added Qwen3.6 35B MoE GGUF support for `Q4_K_M` and `Q4_K_S` model files,
+  including resident GGUF loading, bulk prefill, graph-replay decode,
+  decode-repacked T16 layouts, and WMMA/GEMV fast-path controls used by the
+  W7900 benchmark profile.
+- Added `docs/ENVS.md` as the canonical environment-variable reference, including
+  TheRock ROCm process setup, cached-build profiling guidance, and safe GGUF
+  benchmark profiles.
+- Added a persistent README sweep harness that loads each hipEngine model once
+  and runs repeated in-session workload measurements, matching llama-bench-style
+  repetition without multiplying model load/decode-repack time by every shape.
+
+### Changed
+
+- Refreshed W7900 README performance tables with 5-run persistent-session medians
+  for packed PARO and GGUF Q4_K_S while keeping the existing llama.cpp HIP/Vulkan
+  comparison rows unchanged.
+- Documented the current GGUF tradeoffs: higher one-time load cost and resident
+  memory from decode-repack, Q4_K_S preferred for tighter VRAM budgets, and
+  performance still behind PARO on some shapes while already competitive in the
+  broader W7900 comparison.
+
+### Fixed
+
+- Fixed the PARO resident prefill workspace-overlap regression that shipped in
+  v0.1.1: short and mid prompts now keep prefill workspaces resident through
+  32K tokens, restoring 512/128-class prefill throughput while retaining the
+  long-context memory-saving path for prompts above 32K when active chunking
+  splits the prompt.
+- Fixed GGUF non-split full-attention decode in max-context persistent sessions
+  by launching the context kernel with the active decode context instead of the
+  session's maximum allocation length.
+
+### Known limitations
+
+- GGUF support remains alpha: production correctness and performance coverage is
+  strongest for the documented Qwen3.6 35B MoE Q4_K_M/Q4_K_S paths on gfx1100,
+  and other GGUF quants/models require local validation.
+- GGUF model load is slower than packed PARO on the same host because current
+  decode-repack happens on load and is not yet cached on disk.
+
 ## v0.1.1 - 2026-05-19
 
 Patch release focused on long-context memory documentation and the INT8 KV cache

{hipengine-0.1.1 → hipengine-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hipengine
-Version: 0.1.1
+Version: 0.2.0
 Summary: ROCm-native local LLM inference engine with a torch-free runtime hot path
 Project-URL: Homepage, https://github.com/shisa-ai/hipEngine
 Project-URL: Repository, https://github.com/shisa-ai/hipEngine
@@ -75,15 +75,16 @@ supported GPUs and models.
 
 ## Status
 
-**v0.1.x.** The runtime hot path is torch-free by construction; kernel
-families and registry plumbing are landing under
-[`hipengine/kernels/hip_gfx1100/`](hipengine/kernels/hip_gfx1100/). Current
-single-model tuning targets
+**v0.2.0 alpha.** The runtime hot path is torch-free by construction, and the
+first two 35B-class model-loading surfaces are now available on gfx1100:
 [shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed](https://huggingface.co/shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed)
 (19.07 GiB, 4.68 bpw) in packed
-[ParoQuant](https://github.com/shisa-ai/paroquant) format.
+[ParoQuant](https://github.com/shisa-ai/paroquant) format, plus Qwen3.6 GGUF
+`Q4_K_M` / `Q4_K_S` files through the new resident GGUF path.
 
-- INT8 KV cache support has been added. Qwen 3 MoE's full 256K context window can fit in <24GB tracked memory; see [Memory Usage](#memory-usage).
+- INT8 KV cache support has been added for PARO. Qwen 3 MoE's full 256K context window can fit in <24GB tracked memory; see [Memory Usage](#memory-usage).
+- Qwen 3.6 [Q4_K_M](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF?show_file_info=Qwen3.6-35B-A3B-UD-Q4_K_M.gguf) and [Q4_K_S](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF?show_file_info=Qwen3.6-35B-A3B-UD-IQ4_XS.gguf) GGUF support has landed (W7900 Q4_K_S sweep is in [Performance](#performance) alongside packed PARO and llama.cpp Q4_K_M HIP/Vulkan baselines). GGUF uses a substantial GGUF-specific runtime path with bulk prefill, graph decode, and on-load decode-repack into T16 tile layouts. Q4_K_S is recommended on 24 GiB cards because Q4_K_M is bigger; on the 48 GiB W7900 Q4_K_S fits all the way to 128K context, while on 24 GiB cards expect roughly 64K. GGUF also has a higher per-session load cost (~60 s vs ~24 s for PARO packed on the same hardware) for the same decode-repack reason.
+- Current gfx1100 performance snapshots are summarized in [Performance](#performance) and compared against recent llama.cpp Q4_K_M baselines.
 
 
 ## Hardware targets
@@ -144,38 +145,65 @@ See
 and [`docs/KVCACHE.md`](docs/KVCACHE.md) for commands, artifacts, and the full
 no-shadow memory audit.
 
+### llama.cpp
+
+When run with `q8_0` kvcache, llama.cpp can also fit in 24GB:
+
+```bash
+--flash-attn on -ctk q8_0 -ctv q8_0 -c 262144 -b 128 -ub 128
+```
+
+Results:
+
+| Model | llama.cpp model buffer | KV cache | Compute buffer | rocm-smi VRAM used | Free VRAM |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Q4_K_M | 20583 MiB | 2720 MiB | 203 MiB | 24017 MiB / 23.45 GiB | ~543 MiB |
+| Q4_K_S | 19399 MiB | 2720 MiB | 203 MiB | 22832 MiB / 22.30 GiB | ~1728 MiB |
+
+With `-ub 512`:
+
+| Model | Compute buffer | rocm-smi VRAM used | Free VRAM |
+| --- | ---: | ---: | ---: |
+| Q4_K_M | 812 MiB | 24540 MiB | ~20 MiB |
+| Q4_K_S | 812 MiB | 23443 MiB | ~1117 MiB |
+
+- Note Q4_K_M is incredibly tight with only 20 MiB of headroom and you may either need to resize down or set `-b 512 -ub 128`.
+- Q4_K_S does not need small `-b`/`-ub`; `-ub 512` fits fine, and can even increase to `-b 2048` (but `-ub` is the more important VRAM knob that controls the physical microbatch / compute buffer size for llama.cpp).
+
 ## Performance
 
 ### gfx1100 (Radeon RX 7900 XTX / Radeon Pro W7900)
 
-While we are far from [gfx1100 roofline](https://github.com/shisa-ai/hipEngine/blob/main/docs/ROOFLINE.md), the current gfx1100 implementation does well compared to Q4_K_M quants of recent llama.cpp builds (`b9042`) on the same model family. The latest W7900 packed rows use the default prefill policy: 512-token prompts stay unchunked and prompts above 1K use `1024/1024/4096/1024/1024` chunks.
+While we are far from [gfx1100 roofline](https://github.com/shisa-ai/hipEngine/blob/main/docs/ROOFLINE.md), the current gfx1100 implementation does well compared to Q4_K_M quants of recent llama.cpp builds (`b9042`) on the same model family. The latest W7900 hipEngine rows use TheRock ROCm 7.13 and load each resident model once for 1 warmup + 5 measured in-session repetitions per shape. PARO uses the default prefill policy: 512-token prompts stay unchunked and prompts above 1K use `1024/1024/4096/1024/1024` chunks. The `hipEngine GGUF Q4_K_S` column uses the same chunked-prefill policy plus the WMMA prefill + GEMV decode fast paths and the persistent on-load decode-repack into T16 tile layouts.
 
 ### Prefill tok/s
 
-| Workload | hipEngine shisa Qwen3.6 packed PARO | llama.cpp HIP | llama.cpp Vulkan |
-| --- | ---: | ---: | ---: |
-| 512/128 | **2500.565** | 2436.049 | 1816.927 |
-| 4K/128 | **2899.685** | 2176.905 | 1705.093 |
-| 32K/128 | **2115.050** | 1496.409 | 1128.554 |
-| 128K/128 | **1054.291** | 710.213 | 480.539 |
+| Workload | hipEngine PARO | hipEngine GGUF Q4_K_S | llama.cpp HIP | llama.cpp Vulkan |
+| --- | ---: | ---: | ---: | ---: |
+| 512/128 | **2718.497** | 2258.847 | 2436.049 | 1816.927 |
+| 4K/128 | **2838.773** | 2576.673 | 2176.905 | 1705.093 |
+| 32K/128 | **2074.699** | 1893.967 | 1496.409 | 1128.554 |
+| 128K/128 | **1055.454** | 998.143 | 710.213 | 480.539 |
 
 ### Decode tok/s
 
-| Workload | hipEngine shisa Qwen3.6 packed PARO | llama.cpp HIP | llama.cpp Vulkan |
-| --- | ---: | ---: | ---: |
-| 512/128 | 111.516 | 85.487 | **127.515** |
-| 4K/128 | 113.094 | 87.375 | **120.163** |
-| 32K/128 | 97.594 | 76.994 | **98.073** |
-| 128K/128 | 62.027 | 57.341 | **64.478** |
+| Workload | hipEngine PARO | hipEngine GGUF Q4_K_S | llama.cpp HIP | llama.cpp Vulkan |
+| --- | ---: | ---: | ---: | ---: |
+| 512/128 | 103.460 | 109.152 | 85.487 | **127.515** |
+| 4K/128 | 101.964 | 100.048 | 87.375 | **120.163** |
+| 32K/128 | 90.438 | 86.774 | 76.994 | **98.073** |
+| 128K/128 | 59.598 | 57.954 | 57.341 | **64.478** |
 
 ### Peak GiB
 
-| Workload | hipEngine shisa Qwen3.6 packed PARO | llama.cpp HIP | llama.cpp Vulkan |
-| --- | ---: | ---: | ---: |
-| 512/128 | **18.123** | 21.125 | 20.844 |
-| 4K/128 | **19.455** | 21.197 | 20.969 |
-| 32K/128 | **20.267** | 21.738 | 21.533 |
-| 128K/128 | **23.235** | 23.605 | 23.596 |
+| Workload | hipEngine PARO | hipEngine GGUF Q4_K_S | llama.cpp HIP | llama.cpp Vulkan |
+| --- | ---: | ---: | ---: | ---: |
+| 512/128 | 20.962 | 25.108 | 21.125 | **20.844** |
+| 4K/128 | 21.906 | 25.108 | 21.197 | **20.969** |
+| 32K/128 | 22.016 | 25.108 | 21.738 | **21.533** |
+| 128K/128 | **22.122** | 25.108 | 23.605 | 23.596 |
+
+hipEngine W7900 row source: [`benchmarks/results/2026-05-25-w7900-hipengine-readme-persistent-5run-diagnostic.json`](benchmarks/results/2026-05-25-w7900-hipengine-readme-persistent-5run-diagnostic.json). Both hipEngine columns are 5-run medians from one resident session allocated for the maximum requested context (`128K/128`), so the peak-memory column is a max-context persistent-session high-water mark rather than each shape's minimum allocation. Existing W7900 llama.cpp HIP/Vulkan Q4_K_M rows are reused unchanged. The hipEngine GGUF Q4_K_S column is compared against the existing llama.cpp Q4_K_M baselines because that is the lineage of measured baselines we have on this host; cross-quant comparisons should be read as approximate.
 
 ### gfx1151 (AMD Ryzen AI MAX+ 395 / Radeon 8060S)
 
@@ -183,7 +211,7 @@ The gfx1151 backend is a native `--offload-arch=gfx1151` peer backend using the
 
 ### Prefill tok/s
 
-| Workload | hipEngine shisa Qwen3.6 packed PARO | llama.cpp HIP | llama.cpp Vulkan |
+| Workload | hipEngine PARO | llama.cpp HIP | llama.cpp Vulkan |
 | --- | ---: | ---: | ---: |
 | 512/128 | 983.206 | **1058.738** | 638.008 |
 | 4K/128 | **1029.402** | 1004.220 | 595.400 |
@@ -192,7 +220,7 @@ The gfx1151 backend is a native `--offload-arch=gfx1151` peer backend using the
 
 ### Decode tok/s
 
-| Workload | hipEngine shisa Qwen3.6 packed PARO | llama.cpp HIP | llama.cpp Vulkan |
+| Workload | hipEngine PARO | llama.cpp HIP | llama.cpp Vulkan |
 | --- | ---: | ---: | ---: |
 | 512/128 | **62.060** | 50.537 | 57.615 |
 | 4K/128 | **63.605** | 49.379 | 55.027 |
@@ -204,6 +232,27 @@ On Strix Halo, `rocm-smi` / sysfs expose only a 512 MiB VRAM aperture, so cross-
 See [`benchmarks/README.md`](benchmarks/README.md) for full protocol details,
 correctness status, source-lineage targets, and external comparison baselines.
 
+## GGUF Support
+
+As of v0.2.0, hipEngine includes resident Qwen3.6 GGUF support for `Q4_K_M` and
+`Q4_K_S` model files (with more formats planned). This is a major runtime path,
+not just a loader shim: GGUF has its own quant readers, bulk-prefill path,
+decode-repacked T16 layouts, and fast-path controls.
+
+Current caveats:
+
+- PARO models take ~24s to load on the W7900 test host; GGUF currently takes
+  about 60s because decode-repack happens on load. On-disk caching could reduce
+  startup time later, but would require additional storage for repacked layouts.
+- GGUF has higher resident memory than packed PARO. In the current W7900 README
+  sweep, the max-context Q4_K_S session peaks at ~25.1 GiB tracked, so 128K is
+  W7900/48 GiB territory; on 24 GiB cards, expect roughly 64K context with
+  Q4_K_S.
+- GGUF is close enough to PARO to share some high-level scheduling ideas, but in
+  practice it needs substantial GGUF-only kernels and dispatch. The goal for
+  future releases is to keep closing the remaining PARO/GGUF speed gap.
+
+
 ## Architecture at a glance
 
 ```
@@ -305,6 +354,7 @@ current limitations.
 | [`docs/BENCHMARK.md`](docs/BENCHMARK.md) | Benchmark protocols, baselines, correctness gate, artifact format |
 | [`docs/TESTING.md`](docs/TESTING.md) | RED/GREEN workflow, correctness oracles, fixture policy |
 | [`docs/KERNELS.md`](docs/KERNELS.md) | Kernel catalog, source-lineage drift workflow, JIT cache gotchas, build profiles |
+| [`docs/ENVS.md`](docs/ENVS.md) | Environment variables, TheRock setup, benchmark/profiling profiles |
 | [`docs/ROOFLINE.md`](docs/ROOFLINE.md) | RDNA3 / W7900 performance model and decision tree |
 | [`docs/IMPLEMENTATION.md`](docs/IMPLEMENTATION.md) | Implementation status and concrete milestones |
 | [`docs/API.md`](docs/API.md) | OpenAI-compatible server usage and endpoint support |

{hipengine-0.1.1 → hipengine-0.2.0}/README.md

@@ -37,15 +37,16 @@ supported GPUs and models.
 
 ## Status
 
-**v0.1.x.** The runtime hot path is torch-free by construction; kernel
-families and registry plumbing are landing under
-[`hipengine/kernels/hip_gfx1100/`](hipengine/kernels/hip_gfx1100/). Current
-single-model tuning targets
+**v0.2.0 alpha.** The runtime hot path is torch-free by construction, and the
+first two 35B-class model-loading surfaces are now available on gfx1100:
 [shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed](https://huggingface.co/shisa-ai/Qwen3.6-35B-A3B-PARO-full4096-e5-packed)
 (19.07 GiB, 4.68 bpw) in packed
-[ParoQuant](https://github.com/shisa-ai/paroquant) format.
+[ParoQuant](https://github.com/shisa-ai/paroquant) format, plus Qwen3.6 GGUF
+`Q4_K_M` / `Q4_K_S` files through the new resident GGUF path.
 
-- INT8 KV cache support has been added. Qwen 3 MoE's full 256K context window can fit in <24GB tracked memory; see [Memory Usage](#memory-usage).
+- INT8 KV cache support has been added for PARO. Qwen 3 MoE's full 256K context window can fit in <24GB tracked memory; see [Memory Usage](#memory-usage).
+- Qwen 3.6 [Q4_K_M](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF?show_file_info=Qwen3.6-35B-A3B-UD-Q4_K_M.gguf) and [Q4_K_S](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF?show_file_info=Qwen3.6-35B-A3B-UD-IQ4_XS.gguf) GGUF support has landed (W7900 Q4_K_S sweep is in [Performance](#performance) alongside packed PARO and llama.cpp Q4_K_M HIP/Vulkan baselines). GGUF uses a substantial GGUF-specific runtime path with bulk prefill, graph decode, and on-load decode-repack into T16 tile layouts. Q4_K_S is recommended on 24 GiB cards because Q4_K_M is bigger; on the 48 GiB W7900 Q4_K_S fits all the way to 128K context, while on 24 GiB cards expect roughly 64K. GGUF also has a higher per-session load cost (~60 s vs ~24 s for PARO packed on the same hardware) for the same decode-repack reason.
+- Current gfx1100 performance snapshots are summarized in [Performance](#performance) and compared against recent llama.cpp Q4_K_M baselines.
 
 
 ## Hardware targets
@@ -106,38 +107,65 @@ See
 and [`docs/KVCACHE.md`](docs/KVCACHE.md) for commands, artifacts, and the full
 no-shadow memory audit.
 
+### llama.cpp
+
+When run with `q8_0` kvcache, llama.cpp can also fit in 24GB:
+
+```bash
+--flash-attn on -ctk q8_0 -ctv q8_0 -c 262144 -b 128 -ub 128
+```
+
+Results:
+
+| Model | llama.cpp model buffer | KV cache | Compute buffer | rocm-smi VRAM used | Free VRAM |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Q4_K_M | 20583 MiB | 2720 MiB | 203 MiB | 24017 MiB / 23.45 GiB | ~543 MiB |
+| Q4_K_S | 19399 MiB | 2720 MiB | 203 MiB | 22832 MiB / 22.30 GiB | ~1728 MiB |
+
+With `-ub 512`:
+
+| Model | Compute buffer | rocm-smi VRAM used | Free VRAM |
+| --- | ---: | ---: | ---: |
+| Q4_K_M | 812 MiB | 24540 MiB | ~20 MiB |
+| Q4_K_S | 812 MiB | 23443 MiB | ~1117 MiB |
+
+- Note Q4_K_M is incredibly tight with only 20 MiB of headroom and you may either need to resize down or set `-b 512 -ub 128`.
+- Q4_K_S does not need small `-b`/`-ub`; `-ub 512` fits fine, and can even increase to `-b 2048` (but `-ub` is the more important VRAM knob that controls the physical microbatch / compute buffer size for llama.cpp).
+
 ## Performance
 
 ### gfx1100 (Radeon RX 7900 XTX / Radeon Pro W7900)
 
-While we are far from [gfx1100 roofline](https://github.com/shisa-ai/hipEngine/blob/main/docs/ROOFLINE.md), the current gfx1100 implementation does well compared to Q4_K_M quants of recent llama.cpp builds (`b9042`) on the same model family. The latest W7900 packed rows use the default prefill policy: 512-token prompts stay unchunked and prompts above 1K use `1024/1024/4096/1024/1024` chunks.
+While we are far from [gfx1100 roofline](https://github.com/shisa-ai/hipEngine/blob/main/docs/ROOFLINE.md), the current gfx1100 implementation does well compared to Q4_K_M quants of recent llama.cpp builds (`b9042`) on the same model family. The latest W7900 hipEngine rows use TheRock ROCm 7.13 and load each resident model once for 1 warmup + 5 measured in-session repetitions per shape. PARO uses the default prefill policy: 512-token prompts stay unchunked and prompts above 1K use `1024/1024/4096/1024/1024` chunks. The `hipEngine GGUF Q4_K_S` column uses the same chunked-prefill policy plus the WMMA prefill + GEMV decode fast paths and the persistent on-load decode-repack into T16 tile layouts.
 
 ### Prefill tok/s
 
-| Workload | hipEngine shisa Qwen3.6 packed PARO | llama.cpp HIP | llama.cpp Vulkan |
-| --- | ---: | ---: | ---: |
-| 512/128 | **2500.565** | 2436.049 | 1816.927 |
-| 4K/128 | **2899.685** | 2176.905 | 1705.093 |
-| 32K/128 | **2115.050** | 1496.409 | 1128.554 |
-| 128K/128 | **1054.291** | 710.213 | 480.539 |
+| Workload | hipEngine PARO | hipEngine GGUF Q4_K_S | llama.cpp HIP | llama.cpp Vulkan |
+| --- | ---: | ---: | ---: | ---: |
+| 512/128 | **2718.497** | 2258.847 | 2436.049 | 1816.927 |
+| 4K/128 | **2838.773** | 2576.673 | 2176.905 | 1705.093 |
+| 32K/128 | **2074.699** | 1893.967 | 1496.409 | 1128.554 |
+| 128K/128 | **1055.454** | 998.143 | 710.213 | 480.539 |
 
 ### Decode tok/s
 
-| Workload | hipEngine shisa Qwen3.6 packed PARO | llama.cpp HIP | llama.cpp Vulkan |
-| --- | ---: | ---: | ---: |
-| 512/128 | 111.516 | 85.487 | **127.515** |
-| 4K/128 | 113.094 | 87.375 | **120.163** |
-| 32K/128 | 97.594 | 76.994 | **98.073** |
-| 128K/128 | 62.027 | 57.341 | **64.478** |
+| Workload | hipEngine PARO | hipEngine GGUF Q4_K_S | llama.cpp HIP | llama.cpp Vulkan |
+| --- | ---: | ---: | ---: | ---: |
+| 512/128 | 103.460 | 109.152 | 85.487 | **127.515** |
+| 4K/128 | 101.964 | 100.048 | 87.375 | **120.163** |
+| 32K/128 | 90.438 | 86.774 | 76.994 | **98.073** |
+| 128K/128 | 59.598 | 57.954 | 57.341 | **64.478** |
 
 ### Peak GiB
 
-| Workload | hipEngine shisa Qwen3.6 packed PARO | llama.cpp HIP | llama.cpp Vulkan |
-| --- | ---: | ---: | ---: |
-| 512/128 | **18.123** | 21.125 | 20.844 |
-| 4K/128 | **19.455** | 21.197 | 20.969 |
-| 32K/128 | **20.267** | 21.738 | 21.533 |
-| 128K/128 | **23.235** | 23.605 | 23.596 |
+| Workload | hipEngine PARO | hipEngine GGUF Q4_K_S | llama.cpp HIP | llama.cpp Vulkan |
+| --- | ---: | ---: | ---: | ---: |
+| 512/128 | 20.962 | 25.108 | 21.125 | **20.844** |
+| 4K/128 | 21.906 | 25.108 | 21.197 | **20.969** |
+| 32K/128 | 22.016 | 25.108 | 21.738 | **21.533** |
+| 128K/128 | **22.122** | 25.108 | 23.605 | 23.596 |
+
+hipEngine W7900 row source: [`benchmarks/results/2026-05-25-w7900-hipengine-readme-persistent-5run-diagnostic.json`](benchmarks/results/2026-05-25-w7900-hipengine-readme-persistent-5run-diagnostic.json). Both hipEngine columns are 5-run medians from one resident session allocated for the maximum requested context (`128K/128`), so the peak-memory column is a max-context persistent-session high-water mark rather than each shape's minimum allocation. Existing W7900 llama.cpp HIP/Vulkan Q4_K_M rows are reused unchanged. The hipEngine GGUF Q4_K_S column is compared against the existing llama.cpp Q4_K_M baselines because that is the lineage of measured baselines we have on this host; cross-quant comparisons should be read as approximate.
 
 ### gfx1151 (AMD Ryzen AI MAX+ 395 / Radeon 8060S)
 
@@ -145,7 +173,7 @@ The gfx1151 backend is a native `--offload-arch=gfx1151` peer backend using the
 
 ### Prefill tok/s
 
-| Workload | hipEngine shisa Qwen3.6 packed PARO | llama.cpp HIP | llama.cpp Vulkan |
+| Workload | hipEngine PARO | llama.cpp HIP | llama.cpp Vulkan |
 | --- | ---: | ---: | ---: |
 | 512/128 | 983.206 | **1058.738** | 638.008 |
 | 4K/128 | **1029.402** | 1004.220 | 595.400 |
@@ -154,7 +182,7 @@ The gfx1151 backend is a native `--offload-arch=gfx1151` peer backend using the
 
 ### Decode tok/s
 
-| Workload | hipEngine shisa Qwen3.6 packed PARO | llama.cpp HIP | llama.cpp Vulkan |
+| Workload | hipEngine PARO | llama.cpp HIP | llama.cpp Vulkan |
 | --- | ---: | ---: | ---: |
 | 512/128 | **62.060** | 50.537 | 57.615 |
 | 4K/128 | **63.605** | 49.379 | 55.027 |
@@ -166,6 +194,27 @@ On Strix Halo, `rocm-smi` / sysfs expose only a 512 MiB VRAM aperture, so cross-
 See [`benchmarks/README.md`](benchmarks/README.md) for full protocol details,
 correctness status, source-lineage targets, and external comparison baselines.
 
+## GGUF Support
+
+As of v0.2.0, hipEngine includes resident Qwen3.6 GGUF support for `Q4_K_M` and
+`Q4_K_S` model files (with more formats planned). This is a major runtime path,
+not just a loader shim: GGUF has its own quant readers, bulk-prefill path,
+decode-repacked T16 layouts, and fast-path controls.
+
+Current caveats:
+
+- PARO models take ~24s to load on the W7900 test host; GGUF currently takes
+  about 60s because decode-repack happens on load. On-disk caching could reduce
+  startup time later, but would require additional storage for repacked layouts.
+- GGUF has higher resident memory than packed PARO. In the current W7900 README
+  sweep, the max-context Q4_K_S session peaks at ~25.1 GiB tracked, so 128K is
+  W7900/48 GiB territory; on 24 GiB cards, expect roughly 64K context with
+  Q4_K_S.
+- GGUF is close enough to PARO to share some high-level scheduling ideas, but in
+  practice it needs substantial GGUF-only kernels and dispatch. The goal for
+  future releases is to keep closing the remaining PARO/GGUF speed gap.
+
+
 ## Architecture at a glance
 
 ```
@@ -267,6 +316,7 @@ current limitations.
 | [`docs/BENCHMARK.md`](docs/BENCHMARK.md) | Benchmark protocols, baselines, correctness gate, artifact format |
 | [`docs/TESTING.md`](docs/TESTING.md) | RED/GREEN workflow, correctness oracles, fixture policy |
 | [`docs/KERNELS.md`](docs/KERNELS.md) | Kernel catalog, source-lineage drift workflow, JIT cache gotchas, build profiles |
+| [`docs/ENVS.md`](docs/ENVS.md) | Environment variables, TheRock setup, benchmark/profiling profiles |
 | [`docs/ROOFLINE.md`](docs/ROOFLINE.md) | RDNA3 / W7900 performance model and decision tree |
 | [`docs/IMPLEMENTATION.md`](docs/IMPLEMENTATION.md) | Implementation status and concrete milestones |
 | [`docs/API.md`](docs/API.md) | OpenAI-compatible server usage and endpoint support |