autumn-kvcache 0.1.0__tar.gz

autumn_kvcache-0.1.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

autumn_kvcache-0.1.0/PKG-INFO

@@ -0,0 +1,323 @@
+Metadata-Version: 2.4
+Name: autumn-kvcache
+Version: 0.1.0
+Summary: sglang HiCache L3 and vLLM KV-connector backends on autumn-rs
+Author: autumn-rs contributors
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/journeymidnight/autumn_kvcache
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: autumn-rs==0.1.0
+Requires-Dist: numpy
+Provides-Extra: sglang
+Dynamic: license-file
+
+# autumn-kvcache
+
+Daemon-less KV-cache **L3 storage backends** built on the autumn-rs partition
+layer. Two adapters, one data plane (`autumn.BatchClient` + the sync/async
+`_bridge`):
+
+| Adapter | Engine | Contract | Module |
+|---------|--------|----------|--------|
+| `AutumnKVCacheStorage` | **sglang** | HiCache `HiCacheStorage` (L3) | `autumn_kvcache.sglang_backend` |
+| `AutumnKVConnector` | **vLLM** | `KVConnectorBase_V1` (external KV transfer) | `autumn_kvcache.vllm_connector` |
+
+## Which autumn-rs this tracks
+
+`AUTUMN_REV` names the autumn-rs commit this tree was last verified against.
+It matters more than a normal version pin: the adapters ride autumn's rkyv
+client, and a client/cluster mismatch does **not** fail the handshake — it can
+decode garbage. FreeToken's `docker/Dockerfile.freetoken-l3` carries the same
+pin as `AUTUMN_IMAGE_PIN` for exactly this reason.
+
+The PyO3 extension (`import autumn`) is the `autumn-rs` distribution on PyPI:
+one manylinux abi3 wheel per architecture for every CPython >= 3.11. Its
+version must match the cluster's autumn-rs release; for a cluster built from
+another commit, build the extension from that commit (below).
+
+There is **no daemon and no local DRAM cache** — the autumn partition layer is
+the only persistence path. KV pages are content-addressed
+(`kvc/{h}/{model_scope}/{pool}/{hash}/{layer}`, `h` = the hash's first hex
+char), so there is no invalidation. The cluster serves them once the `kvc`
+namespace exists: `autumn-op namespace-create --name kvc --parts 16` registers
+it and cuts it at `kvc/<hex>` (autumn-rs `cluster.sh` does this on its first
+start). `h` comes first so those 16 partitions take every model's pages with
+no per-model step.
+
+## Install
+
+```bash
+pip install autumn-kvcache          # pulls autumn-rs, the native `autumn` client
+pip install "git+https://github.com/journeymidnight/autumn_kvcache@<rev>"   # or from git
+```
+
+The native client from source, for a cluster at another commit or for RDMA
+(an autumn-rs checkout at that commit):
+
+```bash
+cd autumn-rs/python
+maturin build --release --interpreter /path/to/venv/bin/python
+#    for UCX: add `--features ucx --auditwheel skip`; the wheel must link the
+#    system UCX, since auditwheel's bundled copy has no mlx5 transport modules
+#    and the client then cannot connect over RDMA
+/path/to/venv/bin/pip install --force-reinstall target/wheels/autumn_rs-*.whl
+```
+
+Both adapter modules import **without** the engine (sglang / vLLM) installed —
+the engine import is defensive — so `python -c "from
+autumn_kvcache.vllm_connector import AutumnKVConnector"` works for tooling and
+the data-plane smoke test regardless.
+
+## Faster reads: the partition server's memory tier
+
+Start the partition servers with `--mem-tier-bytes <budget> --mem-tier-prefix kvc/`
+and KV pages are held in their DRAM and acknowledged without a log write; reads
+come from memory. Pages held there are lost on a PS restart (a cache miss, then
+recompute). Measured with SGLang + Qwen3-VL-32B: per-request p50 1344 → 980 ms,
+2 GiB fetch 999 → 699 ms. See autumn-rs `docs/ops.md` ("Memory tier for KV cache").
+
+The tier evicts least recently used first, and among pages used at about the
+same time the deeper ones first, so a prefix loses its tail before its head
+(a hit is a run of pages from the start). The adapters pass each page's
+position in its prefix with the write (`evict_rank`, autumn wire 87): vLLM the
+prefix's length; sglang `(pages before the batch + page index) × page_size`.
+sglang tells the backend the pages before a batch only with
+`"hicache_storage_pass_prefix_keys": true` in
+`--hicache-storage-backend-extra-config`; set it. Without it every batch ranks
+from 0, and even with it a node written from the root over several 128-page
+batches gets an empty list after the first (sglang only extends a non-empty
+one): those batches rank from 0 too, which orders them by write time alone.
+
+The tier can evict a prefix's layers while its vLLM `__present__` marker stays.
+A load that then finds a layer missing injects nothing, and the connector
+deletes that marker (the worker its own rank's, the scheduler the one it
+probes), so the next request for the prefix saves it again. Run vLLM with
+`kv_load_failure_policy="recompute"`: under the default `"fail"` the request
+whose load failed ends with an error instead of being recomputed.
+
+## Using autumn-kvcache as a vLLM L3 (cross-instance prefix cache)
+
+`AutumnKVConnector` is a native vLLM V1 KV connector (Phase 3a — CPU-offload
+path). It offloads each request's block-aligned prompt-prefix KV into autumn on
+the producing instance, and loads it back on **any** instance that later sees
+the same prefix — a cross-instance / cross-restart prefix cache backed by the
+partition layer, with no extra service to run.
+
+> **Pinned vLLM:** verified against **vLLM 0.23.0** (torch 2.11). The
+> `KVConnectorBase_V1` signatures drift across vLLM releases (e.g. the external
+> connector constructor gained a 3rd `kv_cache_config` argument); the adapter
+> forwards it when present and falls back to the 2-arg form, but pin a known
+> version in production.
+
+Point both engines at a running autumn cluster's **manager** endpoint and give
+each `vllm serve` the connector config:
+
+```bash
+KVCFG='{
+  "kv_connector":"AutumnKVConnector",
+  "kv_connector_module_path":"autumn_kvcache.vllm_connector",
+  "kv_role":"kv_both",
+  "kv_connector_extra_config":{"endpoint":"127.0.0.1:9001","transport":"tcp"}
+}'
+
+# instance 1 (e.g. GPU 0)
+CUDA_VISIBLE_DEVICES=0 vllm serve /models/Qwen3-8B --served-model-name qwen3 \
+    --port 8101 --enforce-eager --kv-transfer-config "$KVCFG"
+
+# instance 2 (e.g. GPU 1) — a SEPARATE engine, empty local prefix cache
+CUDA_VISIBLE_DEVICES=1 vllm serve /models/Qwen3-8B --served-model-name qwen3 \
+    --port 8102 --enforce-eager --kv-transfer-config "$KVCFG"
+```
+
+`kv_connector_extra_config` keys: `endpoint` (autumn manager `host:port`,
+required), `transport` (`tcp` default, or `ucx`), `max_inflight`, `ttl_secs`,
+`model_id` (optional explicit model identity — see below). The client reads
+with one Rust worker: ~5 GB/s per rank on TCP, ~42 GB/s on UCX (rendezvous
+into the page); `client_workers` is no longer read.
+
+### What the external cache does (and does not) speed up
+
+The connector is an **L3** behind vLLM's own local prefix cache (GPU + host
+RAM). vLLM matches the local cache first and asks the connector only for the
+tokens *beyond* the local match, because remote storage is slower than a local
+hit — you would never want to prefer autumn over GPU cache. Concretely:
+
+- **Same engine, repeated prompt** → the local prefix cache serves it, so the
+  external cache is (correctly) never *loaded* and its hit rate is ~0%. This is
+  expected, not a failure.
+- **A *different* engine / a *restarted* engine / after local eviction** → the
+  local cache is cold, so the connector loads the prefix from autumn and skips
+  prefill. This is the whole point (measured: a 1.3 k-token prefix loads in
+  ~0.3 s vs. ~2 s of recompute — a ~3–4× TTFT win on the cold-local path).
+
+So judge the connector by **cross-instance / post-restart** hit rate, not by
+same-instance repeats. It also **never re-saves a prefix that is already durable
+in autumn** (BUG-KVC-NO-HIT): a repeat whose KV the local cache already served
+writes nothing, so it costs neither storage nor prefill time. (It issues one
+lightweight presence probe per new request to make that de-dup decision.)
+
+### Saves are asynchronous (almost nothing on the prefill critical path)
+
+On the forward pass, `save_kv_layer` does **only** the cheap GPU-side gather
+(`_extract_layer` → a *standalone* tensor, decoupled from the paged blocks — no
+CPU sync). Everything expensive runs on a background thread: the D2H `.cpu()`
+copy, the store-dedup probe, the durable batched `put_from`, and the marker. A
+CUDA event lets the background thread wait for the gathers before the D2H, and
+the `__present__` marker is published only *after* every layer ACKs (so a reader
+that sees the marker always finds a complete, correct prefix — verified: an
+external-load reproduces vLLM's local-cache output token-for-token).
+
+Measured no-hit overhead (Qwen3-VL-32B, 1 k-token prompt, H200): **TTFT +≈6 ms,
+TPOT ≈0** vs. no connector — i.e. a prefill that never gets reused costs almost
+nothing. (Before moving the D2H off the path it was +≈148 ms.) When the prefix
+*is* reused cross-instance, the connector skips the whole prefill instead.
+
+A prompt longer than `max_num_batched_tokens` is prefilled in chunks over
+several steps. Each step gathers only the tokens it computed (so the GPU staging
+stays within vLLM's token budget), the background thread copies them into the
+prefix's host buffer, and the prefix is written once its last chunk is in. A
+layer's value larger than the partition server's 64 MiB cap (Qwen3-VL-4B: a
+prefix over 16K tokens) is stored as equal pieces `{layer}/{i}`.
+
+At most `_MAX_INFLIGHT_SAVES` saves are in flight, from a prefix's first chunk to
+its write; each holds a registered host buffer the size of its prefix's KV, and
+the buffers grow to the longest prefix seen (at most 2 × `max_model_len` ×
+per-token KV). Over that a save is dropped rather than blocking prefill — it is
+a pure cache, so a later request re-saves. The **store-dedup also lives in the
+background**: a prefix already durable in autumn is neither re-copied nor
+re-written, so a repeated prompt (served by the local GPU cache) costs no storage
+and no extra latency.
+
+### Model-scope isolation: the model's real identity is part of the key
+
+The model-scope segment of every key is
+`{model}_{fingerprint}_{tp_rank}_{tp_size}[...]`, where `fingerprint` is a
+short hash of the model's **identity**: architecture shape
+(layers / hidden / kv-heads / head-size / vocab / dtype / quantization / MLA)
+plus the weights source (`load_format`, and for `--load-format autumn` the
+autumn weights `path` from `model_loader_extra_config`), plus `model_id` if
+you set one, plus the **layout versions** — the running **vLLM version**
+(full `x.y.z`: the KV page layout is a vLLM-internal detail with no stability
+contract, patch releases included) and the connector's own storage-format
+version (`VLLM_KV_STORAGE_FORMAT`, bumped whenever the extract/inject byte
+layout changes).
+
+This exists because the model *path* is not an identity: with
+a shared config dir, every model is served from the same fixed local config
+dir, so two different models used to share one model scope and cross-read each
+other's KV (observed live: Qwen2.5-7B and 32B under one model scope; a same-shape
+pair would have been silently wrong). Set
+`kv_connector_extra_config["model_id"]` when even the fingerprint can't
+distinguish your deployments — e.g. two finetunes with identical architecture
+loaded from the **same** path, or weights overwritten in place at one autumn
+path (don't do that — store new weights at a new path).
+
+> **Upgrade note:** any change to the fingerprint inputs — **including every
+> vLLM version change, patch releases too** — moves the model scope, so the whole
+> vLLM pool goes cold and rebuilds on first use (pure cache, content-addressed
+> — no migration needed). This is deliberate: an upgrade already restarts the
+> pods (GPU cache is lost anyway), so the re-warm is a one-time, predictable
+> cost, whereas carrying KV across a layout-incompatible vLLM would be silent
+> garbage. Keys written by older connectors stay behind under the old model scope;
+> with `ttl_secs=0` they never expire, so reclaim them manually if you care
+> about the space: `client.batch_delete(f"kvc/{h}/<old-model-scope>/vllm/".encode())`
+> for each `h` in `0123456789abcdef` (keys from before the hash-bucket layout:
+> the single prefix `kvc/<old-model-scope>/`; the old model scope is in the
+> previous deployment's startup log).
+
+**`ttl_secs`** (default `0` = no expiry) is the relative TTL after which an
+offloaded prefix stops being *served*. Content-addressed keys never invalidate,
+so a TTL is the only reclamation knob for a long-running cluster. Two caveats on
+what exactly it bounds:
+
+- The connector writes the per-prefix completion marker with `ttl_secs` and the
+  layer pages with `ttl_secs + grace` (grace = 300 s), so the marker always
+  expires *before* its layers. The scheduler admits a load on the marker; a
+  marker that outlived its layers would let the worker silently load
+  uninitialised KV (a correctness bug). So `ttl_secs` bounds **admissibility**
+  (when the prefix can still be hit); the layer **data** lingers ~`grace`
+  longer before its own lazy expiry.
+- Reclamation is **lazy** (keys are dropped on read after they expire, plus
+  background compaction), not a hard wall-clock free — size capacity planning
+  accordingly.
+
+### Verifying the cross-instance hit
+
+Send a long prompt (longer than the cache block size) to **instance 1**, then
+the **same** prompt to **instance 2**. Instance 2 has never seen it locally, so
+any external-cache hit is served from autumn:
+
+```bash
+curl -s localhost:8101/v1/completions -H 'Content-Type: application/json' \
+  -d '{"model":"qwen3","prompt":"<long prompt>","max_tokens":1}' >/dev/null
+curl -s localhost:8102/v1/completions -H 'Content-Type: application/json' \
+  -d '{"model":"qwen3","prompt":"<same long prompt>","max_tokens":1}' >/dev/null
+
+# instance 2's connector metrics now show the cross-instance hit:
+curl -s localhost:8102/metrics | grep -E \
+  'vllm:external_prefix_cache_hits_total|external_kv_transfer'
+# vllm:external_prefix_cache_hits_total{...}                 272.0
+# vllm:prompt_tokens_by_source_total{source="external_kv_transfer"} 272.0
+```
+
+(vLLM 0.23's OpenAI `usage` does **not** surface `cached_tokens` for the
+connector path — use the `vllm:external_prefix_cache_*` /
+`prompt_tokens_by_source{source="external_kv_transfer"}` metrics, which are the
+authoritative external-hit signal.)
+
+A scripted version of this check is `tests/` + the e2e in the repo.
+
+## Data-plane smoke test (no engine required)
+
+Exercises the autumn-facing core (`_AutumnKVStore`) against a real 1-node
+cluster — key format, byte-exact page store/load round-trip, existence probe:
+
+```bash
+AUTUMN_KVCACHE_ENDPOINT=127.0.0.1:9001 AUTUMN_KVCACHE_TRANSPORT=tcp \
+  python -m tests.test_vllm_dataplane    # run from this repo root
+```
+
+## Architecture notes
+
+- **Stateless adapter / content-addressed keys** — no invalidation, partition
+  is the only persistence path.
+- **TTL reclamation (`ttl_secs`)** — both adapters accept `ttl_secs` in their
+  extra-config (vLLM `kv_connector_extra_config`, sglang
+  `hicache-storage-backend-extra-config`); default `0` = no expiry. The
+  partition layer expires keys lazily on read. The sglang backend just stamps
+  every page (its own L1/L2 + hash manage existence, so an expired L3 key is a
+  clean miss); the vLLM connector additionally keeps the completion marker's TTL
+  shorter than its layers' (see above) to preserve load correctness.
+- **sglang host-pool layouts** — a page is read into, and written from, the
+  host pool itself. Where the page (MLA `page_first*`) or each of its K/V
+  halves (MHA `page_first*`) is a window into the pool, it moves with no copy;
+  otherwise (`layer_first`, the default, and pools that assemble a page, such
+  as Mamba) it is copied out to write, and a read lands in a scratch buffer
+  that `set_from_flat_data_page` puts into the pool. Stored bytes follow the
+  pool's own order, so every layout but `layer_first` carries its name in the
+  key: instances with different layouts never read each other's pages. For the
+  copy-free path, run sglang with `--hicache-mem-layout page_first_direct`; the
+  copy path keeps a scratch buffer the size of one batch (128 pages) on each of
+  sglang's backup and prefetch threads.
+  These layouts need the v1 interface (`"interface_v1": 1`); the v0 wrappers
+  use the untagged key.
+- **No duplicate writes** — a page already stored (all of its values present)
+  is not written again; one existence probe per batch. Its `ttl_secs`
+  therefore counts from the first write: backing a hot page up again does not
+  extend it.
+- **Return fast** — never block past the engine's step budget; the load path is
+  synchronous in Phase 3a (per-layer overlap is Phase 3b).
+- **Model-scope isolation** — keys carry a model scope derived from model +
+  model-identity fingerprint (`_identity.py`) + TP/PP, so different models /
+  parallel layouts never alias. The sglang model-scope format is unchanged
+  (`model_name` is a real identity there); its escape hatch is
+  `extra_config["model_id"]`.
+
+## Offline unit tests (no cluster, no engine, no native module)
+
+```bash
+uv run --with pytest python -m pytest tests/test_model_identity.py -q   # from this repo root
+python -m pytest tests/test_page_layouts.py -q   # needs torch; every sglang host-pool layout
+```