lmcache 0.4.8.dev24__tar.gz → 0.4.8.dev31__tar.gz

lmcache-0.4.8.dev31/.claude/skills/hybrid-benchmarking/SKILL.md

@@ -0,0 +1,176 @@
+---
+name: hybrid-benchmarking
+description: Benchmark a hybrid-attention model across the LMCache performance ladder (vLLM no-hybrid-allocator → hybrid allocator + prefix caching → hybrid allocator + LMCache) and produce a decode-throughput / TTFT / cache-hit-rate comparison. Use when asked to benchmark, demo, or quantify the value of LMCache caching for a hybrid model.
+allowed-tools: Bash, Read, Grep, Glob, Write
+argument-hint: "<hf-model-id> [--gpus N] [--tp N] [--output-dir DIR]"
+---
+
+# Hybrid-Model Benchmarking — the LMCache performance ladder
+
+Given a hybrid-attention model + hardware, calibrate a `long-doc-qa` workload, run the vLLM
+config ladder, and write `report.md` (launch commands + decode-throughput / TTFT / TPOT /
+hit-rate table + a rationale for every tuned number). Validated on `google/gemma-4-31B-it`
+(sliding-window), `Qwen/Qwen3.6-27B` (Mamba/GDN), and `deepseek-ai/DeepSeek-V4-Flash`
+(sparse-MLA + fp8).
+
+> **Run fully autonomously.** Log each decision and keep going through all runs and the
+> report — do **not** pause for approval/calibration confirmation. Only stop on a hard
+> blocker (model not given, gated weights missing). The user interrupts if something's off.
+
+## Inputs
+
+`$ARGUMENTS`: positional model id (required, e.g. `Qwen/Qwen3.6-27B`); `--gpus N` / `--tp N`
+(default: detect via `nvidia-smi -L`, tp = gpus); `--output-dir DIR` (default
+`./hybrid-bench-<slug>/`). On a shared box, pick free GPUs / non-default ports and never kill
+processes you didn't start.
+
+## The ladder
+
+Same `long-doc-qa` workload, four vLLM configs — each isolates one variable:
+
+| Run | config | what it shows |
+|--|--|--|
+| **A** | allocator off, prefix caching off | baseline: smallest batch → lowest decode tput, worst TTFT |
+| **B** | allocator on, prefix caching off | isolates allocator: packs ~8× more tokens → much higher decode tput |
+| **C** | + `--enable-prefix-caching` (no LMCache) | GPU prefix cache ~0 % hit under batch saturation → ≈ Run B |
+| **D** | + `LMCacheMPConnector` + `lmcache server` | CPU pool serves prefixes → ~100 % hit → lowest TTFT, highest decode tput |
+
+Two counterintuitive facts: **(1)** for sliding-window hybrids the allocator gives the
+*larger* batch (sliding layers keep only a window-sized KV slice → ~`num_layers /
+num_full_layers`× more tokens, ≈8× for Gemma-4), so Run A (all layers full) is the
+*smallest*/slowest. **(2)** Run C's GPU prefix cache is ~0 % hit because the active batch's KV
+(`batch × (doc+output)`) fills the whole pool — no blocks left to *retain* prefixes. That
+saturation gap is exactly what LMCache's CPU pool fills.
+
+> **Mandatory-allocator models → 3 runs (A ≡ B).** Mamba/GDN (Qwen3.5/3.6, Qwen3-Next) and
+> DeepSeek-V4 sparse-MLA have no allocator-off baseline — `--disable-hybrid-kv-cache-manager`
+> is rejected (Mamba) or fails at runtime (DeepSeek: CuTe sparse-attn needs C128/block_size=8).
+> Drop Run A; the story is the cache (C→D). The full 4-run structure applies only to
+> sliding-window hybrids (Gemma 3/4, gpt-oss).
+
+## Make the criteria visible
+
+Default `long-doc-qa` is prefill-bound, which *hides* the decode gap. **Always** pass
+`--ignore-eos --ldqa-max-output-length <N>` (≈2048 at L=24 000) → decode-bound *and* deterministic
+(identical total output tokens/run, so decode-throughput numbers are reproducible). Size the
+working set to **overflow the GPU pool but fit the LMCache pool with margin** (step 3).
+
+## Procedure
+
+**0 — Versions + env.** Record for the report (benchmarks are meaningless without them):
+`vllm --version` (→ commit); LMCache `git rev-parse --short HEAD` + `git status --short` (note
+any local patch, e.g. the `--ignore-eos` / `--ldqa-max-output-length` bench patch → record "base
+commit + with `<patch>`"). Export `HF_HOME` if weights live off-default — for **every** command
+(env doesn't persist across tool calls). Read the model's `docs/source/recipes/*.rst` for
+validated flags + quirks (else `docs/source/mp/hybrid_models.rst`, and warn it's unvalidated).
+Confirm weights are present.
+
+**1 — Classify → decide Run A.**
+
+| Family | Examples | Run A |
+|--|--|--|
+| Sliding-window + full | Gemma 3/4, gpt-oss | `--disable-hybrid-kv-cache-manager --no-enable-prefix-caching` |
+| Mamba / GDN | Qwen3.5/3.6, Qwen3-Next | allocator mandatory → no Run A; caching is non-bit-exact (GDN) |
+| Sparse-MLA + fp8 | DeepSeek-V4 | allocator mandatory (CuTe needs C128/block_size=8) → no Run A |
+| Dense | Llama, Mistral | not a hybrid; allocator distinction is moot |
+
+Classify via the recipe, else HF `config.json` `architectures`/`layer_types`.
+
+**2 — Probe pools.** Launch with `--max-model-len auto` (**never pin small** — it shrinks the
+sliding-window pool ~5×). From the startup log record GPU pool tokens for hybrid (`P_B`) and
+hybrid-off (`P_A`); their ratio is the allocator gain. For LMCache, once the server+connector
+engine is up read `cache_size_per_token` from `/status` → `tokens_per_gb = 1024³ / it`; pass it
+via `--tokens-per-gb-kvcache` so `num_documents` is identical across runs. (Real on-pool storage
+≈ 1.4× this — size against that, not the raw number.)
+
+**3 — Calibrate.** Util < 0.9; lower it so GPU memory is the binding constraint (0.50 on H200;
+**0.35 for fp8 DeepSeek** — fp8 makes the GPU pool huge). Pick `L` (≤ `P_A`, a sane hybrid batch
+`P_B/(L+output)`, a meaningful prefix). `num_documents = floor(kv_cache_volume_gb × tokens_per_gb
+/ L)` chosen so it **overflows GPU** (`num_docs × L > P_B`) and **fits LMCache with margin**
+(real storage ≈ 1.4×(tokens / tokens_per_gb) ≤ ~0.6 × `l1_size_gb`, watermark 0.95). `l1_size_gb`
+≤ host RAM and /dev/shm. Set `--ldqa-max-output-length` so decode dominates (≈2048 @ L=24 000). Log
+the operating point and proceed.
+
+**4 — Freeze one shared config.**
+```bash
+lmcache bench engine --engine-url http://localhost:8000 --workload long-doc-qa \
+    --model <model> --tokens-per-gb-kvcache <N> --kv-cache-volume <GB> \
+    --ldqa-document-length <L> --ldqa-query-per-document 1 \
+    --ldqa-num-inflight-requests <≥ hybrid batch> --ldqa-max-output-length <out> --ignore-eos \
+    --no-interactive --export-config "$OUT/shared.json"
+```
+Replay every run with `--config "$OUT/shared.json"` (so `num_documents` is byte-identical, even in A/B with no server).
+
+**5 — Run.** Start `lmcache server` **first** so its pinned pool pre-expands (lazy ~80 GiB/min) before D:
+```bash
+lmcache server --port 5560 --http-port 8090 --prometheus-port 9099 \
+    --l1-size-gb <GB> --eviction-policy LRU --eviction-trigger-watermark 0.95 [--chunk-size N]
+```
+Each run: launch engine → poll `curl -sf :8000/health` → snapshot metrics (step 6) → bench → teardown. `<base>` = recipe flags (`--tensor-parallel-size`, `--trust-remote-code`, …).
+```bash
+# Run-flags appended to <base>:
+#   A  --disable-hybrid-kv-cache-manager --no-enable-prefix-caching   (skip for mandatory-allocator families)
+#   B  --no-enable-prefix-caching
+#   C  --enable-prefix-caching
+#   D  C + --kv-transfer-config '{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.port":5560,"lmcache.mp.mq_timeout":900}}'
+vllm serve <model> <base> --gpu-memory-utilization <U> --max-model-len auto --port 8000 <run-flags>
+lmcache bench engine --engine-url http://localhost:8000 --config "$OUT/shared.json" \
+    --no-interactive --json --output-dir "$OUT/run-<A|B|C|D>"
+```
+**Run D at steady state:** bench **twice** — run #1 primes, run #2 measures; report run #2's delta. First confirm the pool is pinned (`/status memory_total_bytes ≈ l1`) and the engine registered (`cache_context_meta` non-empty).
+**Teardown by PID, not pattern:** `pkill -f "vllm serve …"` self-matches (exit 144) and orphans EngineCore/Workers that keep holding GPU memory. Kill the main PID + worker PIDs from `nvidia-smi --query-compute-apps`; kill `lmcache server` cleanly; verify GPUs return to ~0 MiB.
+
+**6 — Metrics.** Bench JSON `results`: `mean_ttft_ms`, `p90/p99_ttft_ms`, `output_throughput`
+(decode), `input_throughput` (prefill), `mean_decode_speed` (TPOT = 1000/it),
+`total_output_tokens`. **Hit rate from vLLM `:8000/metrics`** (LMCache Prometheus is often
+disabled in MP builds): `prefix_cache_*` (GPU-local), `external_prefix_cache_*` (LMCache).
+Snapshot before/after each bench; for D report the measure-run delta and prefer the
+`external_prefix_cache_*` token counters.
+
+**7 — Report `$OUT/report.md`.** Header: versions, hardware, TP. **(a)** exact launch commands
+(server + runs + bench; note D's prime+measure double-run). **(b)** table: decode tput,
+mean/p90/p99 TTFT, TPOT, prefill tput, hit rate, wall-clock, GPU pool tokens, effective batch.
+**(c)** rationale + findings, validating the criteria explicitly: A→B decode jumps (≈8× batch);
+B→C ≈ no change (0 % hit, saturated); C→D TTFT collapses + decode rises (hit ~100 %).
+
+## Family turnkey configs
+
+**Mamba/GDN** (Qwen3.5/3.6, …): every run uses `--mamba-cache-mode align
+--max-num-batched-tokens 2N-1`, where N is the unified block size (vLLM logs `Setting attention
+block size to N tokens`; also set server `--chunk-size N`). **Use 2N-1, not N** — at exactly N a
+decoding request consumes the per-step budget, so a new request's block-aligned prefill rounds
+to 0 tokens → serial execution (Running:1, ~7× slower; verified by single-variable A/B). Also
+`--max-num-seqs ≤ #Mamba cache blocks` vLLM reports, or CUDA-graph capture fails.
+
+**DeepSeek-V4 sparse-MLA + fp8**: `export CUDA_HOME=/usr/local/cuda-<ver>
+PATH=$CUDA_HOME/bin:$PATH` (sparse-attn JIT fails on the default toolchain) for **every** launch;
+flags `--kv-cache-dtype fp8_ds_mla --trust-remote-code --tokenizer-mode deepseek_v4
+--enable-expert-parallel`. **Util ≈0.35** (at 0.80 the fp8 GPU pool swallows the whole working
+set → nothing for LMCache to serve); large `--l1-size-gb` (full per-token state ~62 KB,
+`tokens_per_gb ≈ 16 800`).
+
+## Troubleshooting
+
+| Symptom | Fix |
+|--|--|
+| B decode ≈ A (not ≫) | run is prefill-bound — add `--ignore-eos`, raise `--ldqa-max-output-length` |
+| D hit ≪ 100 % | working set hit the watermark mid-fill — pre-pin before D; raise `--l1-size-gb`; shrink working set (real ≈1.4× est.) |
+| D aborts "register_kv_caches within 300s" | big pinned pool → set `lmcache.mp.mq_timeout` ≥ 900 |
+| reg hangs "Wrapping N KV tensors for IPC" | server/connector transfer-mode mismatch — use default (auto) on both |
+| pool ~5× too small | `--max-model-len` pinned — use `auto` |
+| `exit 144` / leaked GPU after kill | `pkill` self-matched / orphaned workers — kill by PID (main + `nvidia-smi` workers) |
+| "CUDA compiler … headers are incompatible" (DeepSeek) | set `CUDA_HOME` matching torch's CUDA, every launch |
+| "CuTe DSL … only supports C128 block_size=8" | tried to disable the allocator on DeepSeek — don't; it's mandatory (3 runs) |
+| fp8: D ~100 % but no saturation / nothing to serve | GPU pool fits the whole working set — lower util (≈0.35) |
+
+## Flag cheat-sheet
+
+- Allocator off: `--disable-hybrid-kv-cache-manager` (sliding-window/dense only).
+- Prefix caching: `--no-enable-prefix-caching` / `--enable-prefix-caching`.
+- Deterministic decode-bound: `--ignore-eos --ldqa-max-output-length <N>`.
+- Pools: `--max-model-len auto`; `lmcache server --l1-size-gb <GB> --eviction-trigger-watermark 0.95` (start early to pre-pin).
+- LMCache wiring: `--kv-transfer-config '{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.port":5560,"lmcache.mp.mq_timeout":900}}'`.
+- Mamba/GDN: `--mamba-cache-mode align --max-num-batched-tokens 2N-1`, server `--chunk-size N`.
+- DeepSeek sparse-MLA: `CUDA_HOME` set; `--kv-cache-dtype fp8_ds_mla --tokenizer-mode deepseek_v4 --enable-expert-parallel`; util ≈0.35.
+- Hit rate: vLLM `:8000/metrics` (`prefix_cache_*` GPU-local, `external_prefix_cache_*` LMCache).
+- Teardown: by PID; verify GPUs ~0 MiB.

lmcache-0.4.8.dev31/.github/scripts/download_opt125m_github.sh

@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: Apache-2.0
+#
+# Download facebook/opt-125m from the LMCache/opt-125m GitHub release
+# and place it into the HuggingFace hub cache structure so vLLM can
+# find it via ``facebook/opt-125m`` without any network call to HF.
+#
+# Environment:
+#   OPT125M_GH_RELEASE  release tag (default: v1.0)
+#   OPT125M_SNAPSHOT    HF snapshot hash (default: 27dcfa...)
+
+set -euo pipefail
+
+GH_RELEASE="${OPT125M_GH_RELEASE:-v1.0}"
+SNAPSHOT="${OPT125M_SNAPSHOT:-27dcfa74d334bc871f3234de431e71c6eeba5dd6}"
+
+HF_CACHE="${HF_HOME:-$HOME/.cache/huggingface}/hub"
+MODEL_DIR="${HF_CACHE}/models--facebook--opt-125m"
+SNAPSHOT_DIR="${MODEL_DIR}/snapshots/${SNAPSHOT}"
+
+if [ -f "${SNAPSHOT_DIR}/pytorch_model.bin" ]; then
+    echo "opt-125m already cached in ${SNAPSHOT_DIR}, skipping download"
+    exit 0
+fi
+
+echo "Downloading opt-125m from GitHub release ${GH_RELEASE}..."
+mkdir -p "${SNAPSHOT_DIR}"
+
+TMP_DIR="$(mktemp -d)"
+trap 'rm -rf "${TMP_DIR}"' EXIT
+
+TARBALL="${TMP_DIR}/opt-125m.tar.gz"
+DOWNLOAD_URL="https://github.com/LMCache/opt-125m/releases/download/${GH_RELEASE}/opt-125m.tar.gz"
+
+for i in $(seq 1 5); do
+    if curl -fsSL --retry 3 -o "${TARBALL}" "${DOWNLOAD_URL}"; then
+        break
+    fi
+    if [ "${i}" -eq 5 ]; then
+        echo "!! Failed to download after 5 attempts: ${DOWNLOAD_URL}"
+        exit 1
+    fi
+    sleep $((10 * i))
+done
+
+# Extract directly – the tarball contains opt-125m/ at top level.
+EXTRACT_DIR="${TMP_DIR}/extract"
+mkdir -p "${EXTRACT_DIR}"
+tar -xzf "${TARBALL}" -C "${EXTRACT_DIR}"
+mv "${EXTRACT_DIR}"/opt-125m/* "${SNAPSHOT_DIR}/"
+
+# Create the refs/main pointer so snapshot_download(local_files_only=True)
+# can resolve the snapshot hash.
+mkdir -p "${MODEL_DIR}/refs"
+echo "${SNAPSHOT}" > "${MODEL_DIR}/refs/main"
+
+echo "opt-125m cached at ${SNAPSHOT_DIR}"
+ls -lah "${SNAPSHOT_DIR}/"

{lmcache-0.4.8.dev24 → lmcache-0.4.8.dev31}/.github/scripts/run-cpu-e2e-validation.sh

@@ -345,9 +345,8 @@ else
   echo "✅ numpy<2 installed"
 fi
 
-echo "[Phase 2 / Step 2] Downloading facebook/opt-125m model (cache-aware)"
-HF_DOWNLOAD_FAIL_ON_ERROR=1 \
-  bash "${SHARED_SCRIPTS_DIR}/download_model.sh" facebook/opt-125m
+echo "[Phase 2 / Step 2] Downloading facebook/opt-125m model (GitHub release)"
+bash "${SHARED_SCRIPTS_DIR}/download_opt125m_github.sh"
 echo "✅ Model download/check complete"
 
 echo "[Phase 2 / Step 3] Starting LMCache server"

{lmcache-0.4.8.dev24 → lmcache-0.4.8.dev31}/.github/workflows/cpu_device.yml

@@ -94,11 +94,11 @@ jobs:
             pyproject.toml
             requirements/*.txt
 
-      - name: Cache HuggingFace models
+      - name: Cache opt-125m model (GitHub release)
         uses: actions/cache@v4
         with:
           path: ~/.cache/huggingface
-          key: hf-opt-125m-v1
+          key: opt-125m-gh-v2
 
       - name: Install vLLM CPU (prebuilt nightly from PyPI)
         run: bash .github/scripts/install_vllm_cpu.sh
@@ -106,11 +106,8 @@ jobs:
       - name: Install lmcache (CPU-only, no vLLM)
         run: bash .github/scripts/install_lmcache_cpu.sh
 
-      - name: Download facebook/opt-125m
-        env:
-          HF_TOKEN: ${{ secrets.HF_TOKEN }}
-        run: |
-          bash .github/scripts/download_model.sh facebook/opt-125m
+      - name: Download facebook/opt-125m (GitHub release)
+        run: bash .github/scripts/download_opt125m_github.sh
 
       - name: Server bench — LMCache-driven
         run: |

{lmcache-0.4.8.dev24 → lmcache-0.4.8.dev31}/.github/workflows/publish.yml

@@ -305,13 +305,14 @@ jobs:
                     dist/*.whl
 
     # Build container image and push to DockerHub when:
-    # - a new GitHub release is created
+    # - a new stable GitHub release is created (non-RC)
     publish-image:
         name: Publish image to DockerHub
         if: |
             needs.publish-pypi.result == 'success' &&
             !cancelled() &&
-            github.repository_owner == 'LMCache' && github.event.action == 'published'
+            github.repository_owner == 'LMCache' && github.event.action == 'published' &&
+            !contains(github.ref_name, 'rc')
 
         runs-on: ubuntu-latest
         needs: [publish-pypi, publish-cu129-github-release]
@@ -465,3 +466,64 @@ jobs:
               run: |
                 docker push lmcache/standalone:latest-cu129
                 docker push lmcache/standalone:${{ env.LATEST_TAG }}-cu129
+
+    # Build RC container image and push to DockerHub when:
+    # - a new RC GitHub release is created (e.g. v0.4.8rc1)
+    # Only pushes lmcache/vllm-openai:<tag> — does not touch latest or other image variants.
+    publish-image-rc:
+        name: Publish RC image to DockerHub
+        if: |
+            needs.publish-pypi.result == 'success' &&
+            !cancelled() &&
+            github.repository_owner == 'LMCache' && github.event.action == 'published' &&
+            contains(github.ref_name, 'rc')
+
+        runs-on: ubuntu-latest
+        needs: [publish-pypi]
+
+        steps:
+            - name: "Harden Runner"
+              uses: step-security/harden-runner@20cf305ff2072d973412fa9b1e3a4f227bda3c76 # v2.14.0
+              with:
+                disable-sudo-and-containers: false
+                egress-policy: audit
+
+            - name: Login to DockerHub
+              uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
+              with:
+                username: ${{ vars.DOCKERHUB_USERNAME }}
+                password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+            - name: Set up Docker Buildx
+              uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
+
+            - name: Checkout code
+              uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
+              with:
+                fetch-depth: 0
+
+            - name: Free disk space
+              uses: ./.github/actions/free-disk-space
+
+            - name: Get the release tag
+              run: |
+                REF="${{ github.ref_name }}"
+                [[ "$REF" =~ ^v[0-9]+\.[0-9]+\.[0-9]+rc[0-9]+$ ]] && echo "LATEST_RC_TAG=$REF" >> "$GITHUB_ENV"
+
+            - name: Build lmcache/vllm-openai container image (cu13)
+              run: |
+                docker build \
+                --build-arg BASE_IMAGE=nvcr.io/nvidia/cuda-dl-base:25.09-cuda13.0-devel-ubuntu24.04 \
+                --build-arg CUDA_VERSION=13.0 \
+                --build-arg torch_cuda_arch_list='7.5 8.0 8.6 8.9 9.0 10.0 12.0+PTX' \
+                --target image-release \
+                --tag lmcache/vllm-openai:${{ env.LATEST_RC_TAG }} \
+                --file docker/Dockerfile .
+
+            - name: Smoke test lmcache/vllm-openai (cu13)
+              run: |
+                docker run --rm --entrypoint lmcache lmcache/vllm-openai:${{ env.LATEST_RC_TAG }} --help
+
+            - name: Push lmcache/vllm-openai RC image to DockerHub
+              run: |
+                docker push lmcache/vllm-openai:${{ env.LATEST_RC_TAG }}

{lmcache-0.4.8.dev24 → lmcache-0.4.8.dev31}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: lmcache
-Version: 0.4.8.dev24
+Version: 0.4.8.dev31
 Summary: A LLM serving engine extension to reduce TTFT and increase throughput, especially under long-context scenarios.
 Author-email: LMCache Team <lmcacheteam@gmail.com>
 License-Expression: Apache-2.0

lmcache-0.4.8.dev31/docs/design/v1/distributed/l2_adapters/fault_inject.md

@@ -0,0 +1,100 @@
+# Fault-Injecting L2 Adapter
+
+`lmcache/v1/distributed/l2_adapters/fault_inject_l2_adapter.py`
+
+A **test/diagnostic-only** L2 adapter that wraps a real inner adapter (e.g.
+`fs_native`) and deterministically drops keys on the load read, simulating
+partial L2 retrieve failures. Its purpose is to exercise the **segmented**
+code paths — gapped found-set → segmented prefetch → segmented scatter /
+attention — that a healthy cache never produces, so CacheBlend's
+segmented-prefix handling can be tested without an actual faulty L2.
+
+## Fault model
+
+It faults the **load read primitive only**. A dropped key is reported
+**present at lookup** but its **load fails** — the faithful "L2 retrieve
+error". The lookup-miss case is *intentionally not modeled*: a key absent at
+lookup merely shortens the found-set, which the `PREFIX` trim policy
+(`count_leading_ones`) already covers. Modeling load-failure-after-lookup is
+what produces a *gapped* found-set (hit … miss … hit), the input the
+segmented path is built to handle.
+
+When a load fails, **no unlock is issued here** — the prefetch controller
+releases the load-failed read locks itself via the trim mask.
+
+## Drop selection
+
+A load position is dropped (`_drop_positions` / `_should_drop_key`) if **any**
+of the following hold:
+
+| knob | meaning | use |
+|---|---|---|
+| `gap_tail_ratios: list[float]` | positions given as **distance-from-tail ÷ load-length** in `[0,1]` (`0.0`=last, `0.5`=middle, `1.0`=first); the dropped slot `round((1-ratio)·(n-1))` is computed from the `n`-key load batch the server actually receives. **Workload-agnostic and self-scaling** — the server needs no advance knowledge of the stored content, and the same fraction picks the right chunk at any context length. | primary lever for segmented repros (drop the mid chunk → a stable gap at any length) |
+| `gap_indices: list[int]` | exact head-relative task-positions to always drop | precise single-gap unit tests |
+| `rate: float` + `seed: int` | per-key drop probability in `[0,1]` via a stable seeded hash of the key, bucketed by `rate` | randomized resilience sweeps; deterministic given the seed |
+
+Defaults are pass-through (`rate=0.0`, empty `gap_indices`/`gap_tail_ratios`).
+
+> **Why a ratio and not a chunk hash?** An earlier `drop_chunk_hashes` knob keyed
+> drops on `ObjectKey.chunk_hash` — content-addressed, so the *same* chunk failed
+> on every leg. But it forced the test author (and thus the server config) to know
+> a hash computed from a not-yet-stored workload, which inverted the dependency
+> (the server "knowing" a future workload). `gap_tail_ratios` is a positional rule
+> the server evaluates from the load batch alone, so no content needs to be known
+> in advance. Trade-off: the drop is **per-load-batch**, not content-identical
+> across legs — for the segmented-prefix repro the gap lives in the prefix leg's
+> load, which is exactly what's exercised.
+
+## Configuration
+
+Activated through `--l2-adapter` (or the engine L2 spec). The required
+`inner` sub-dict is a full adapter spec — dispatched through the L2 adapter
+config registry — so any inner adapter type works and keeps its own
+eviction/persist/serde config:
+
+```json
+{
+  "type": "fault_inject",
+  "inner": { "type": "fs_native", "base_path": "/dev/shm/cb_l2" },
+  "gap_tail_ratios": [0.5],
+  "rate": 0.0,
+  "seed": 0,
+  "gap_indices": []
+}
+```
+
+`"gap_tail_ratios": [0.5]` drops the middle chunk of every load — the stable
+mid-prefix gap the segmented repro needs, at any context length.
+
+At startup it logs `FaultInjectL2Adapter ACTIVE (rate=… seed=… gap_indices=…
+gap_tail_ratios=[…]) wrapping <inner> -- test/diagnostic use only.`
+
+## Data flow
+
+```
+submit_load_task(keys)         → delegate to inner; stash keys under task_id
+query_load_result(task_id)     → bitmap = inner.query_load_result(task_id)
+                                 for i in _drop_positions(keys): bitmap.clear(i)
+                                 return bitmap        # dropped bits now read "miss"
+```
+
+`submit_load_task` records the per-task key list so `query_load_result` can map
+the dropped bit positions back to keys; the result query is **non-idempotent**
+(a non-`None` bitmap is returned once per task), matching the inner contract.
+
+## Delegation
+
+Everything other than the load-result query passes straight through to the
+inner adapter: store, lookup-and-lock, unlock, delete, the event fds,
+listener registration, usage, and global-eviction support. The fault layer
+holds no data of its own. `report_status()` returns the inner status annotated
+with a `fault_inject` sub-dict (`rate`, `seed`, `gap_indices`,
+`gap_tail_ratios`) so the active fault config is visible in diagnostics.
+
+## Scope and safety
+
+Test/diagnostic only — never enable in production. It silently makes reads
+fail, so any served KV is intentionally incomplete. The primary consumer is
+the CacheBlend **segmented-prefix** validation (the `ci_v3_l2_hit`
+`segmented_prefix` workload sets `gap_tail_ratios=[0.5]` to force a gapped
+prefix, so V3 loads prefix+tail and recomputes only the gap).

{lmcache-0.4.8.dev24 → lmcache-0.4.8.dev31}/docs/design/v1/multiprocess/worker_liveness.md

@@ -5,12 +5,12 @@ the vLLM multiprocess adapter.
 
 ## 1. Problem
 
-When a vLLM worker dies without sending `UNREGISTER_KV_CACHE` (SIGKILL, OOM-kill,
-node loss), its per-instance state leaks on the MP server forever: the GPU
-`ContextEntry` (a `GPUCacheContext` holding CUDA IPC handles), the
-`NonGPUContextEntry` + `TransferStrategy` pair, and any blend-mode per-instance
-state (e.g. CB rope caches). Nothing observes worker death; on a shared server,
-leaked contexts accumulate until device memory is exhausted.
+When an engine worker dies without sending `UNREGISTER_KV_CACHE` (SIGKILL,
+OOM-kill, node loss), its per-instance state leaks on the MP server forever: the
+LMCache-driven `ContextEntry` (a `GPUCacheContext` holding CUDA IPC handles), the
+`EngineDrivenContextEntry` + `TransferStrategy` pair, and any blend-mode
+per-instance state (e.g. CB rope caches). Nothing observes worker death; on a
+shared server, leaked contexts accumulate until device memory is exhausted.
 
 Worse, `instance_id` is `os.getpid()`: containerized pods reuse small PIDs, so a
 new worker can register with a dead worker's id, and the idempotent register
@@ -31,7 +31,7 @@ when the server returns the unhealthy-to-healthy edge fires the recover callback
 which re-registers — a noop when the entry survived.
 
 ```
-vLLM worker adapter                              MP server
+engine worker adapter                            MP server
 +---------------------------+                   +--------------------------------------+
 | HeartbeatThread           |   PING [id]       | ManagementModule                     |
 |  (instance_id, 10s)  -----+------------------>|  ping(id) -> touch_instance(id)      |
@@ -39,11 +39,12 @@ vLLM worker adapter                              MP server
 |   (starts healthy)        |                   |   -> reap_stale_instances(           |
 |  unhealthy->healthy edge  |                   |        timeout, registration_grace)  |
 |   -> re-register callback |   REGISTER        |   -> drop_instance_state(id) fan-out |
-|  freeze-gap detection ->  +------------------>|        |                |            |
-|   force 1 unhealthy cycle |   STORE/RETRIEVE  |        v                v            |
-|                           |   (refresh too)   | GPU / NonGPU            BlendV3      |
-| register_kv_caches        |                   |  TransferModule         (CB rope     |
-|  (no pings until traffic) |                   |  Entry{.., last_seen,    dropped)    |
+|                           +------------------>|        |                |            |
+| register_kv_caches        |   STORE/RETRIEVE  |        v                v            |
+|  (no pings until traffic) |   (refresh too)   | LMCacheDriven/          BlendV3      |
+|                           |                   |  EngineDriven           (CB rope     |
+|                           |                   |  TransferModule          dropped)    |
+|                           |                   |  Entry{.., last_seen,               |
 |                           |                   |   has_liveness_signal}               |
 +---------------------------+                   |  _lock (leaf); pop -> cleanup        |
                                                 +--------------------------------------+
@@ -73,13 +74,13 @@ Every id-carrying request reads the same field, so no other payloads change.
 
 ### 5.1 Liveness state and two-tier windows
 
-`ContextEntry` (GPU) and `NonGPUContextEntry` gain `last_seen: float`
-(`time.monotonic()`) and `has_liveness_signal: bool`, latched only by PING. The
-flag selects the staleness window: an entry that has pinged provably runs the
-heartbeat protocol and is judged on the reap timeout; one that never pinged (e.g.
-still warming under lazy start) is judged on the generous registration grace.
-`last_seen` is refreshed by PING (`touch_instance`: refresh if present, never
-insert), register (create and NOOP paths), and every GPU and non-GPU transfer
+`ContextEntry` (LMCache-driven) and `EngineDrivenContextEntry` gain
+`last_seen: float` (`time.monotonic()`) and `has_liveness_signal: bool`, latched
+only by PING. The flag selects the staleness window: an entry that has pinged
+provably runs the heartbeat protocol and is judged on the reap timeout; one that
+never pinged (e.g. still warming under lazy start) is judged on the generous
+registration grace. `last_seen` is refreshed by PING (`touch_instance`: refresh
+if present, never insert), register (create and NOOP paths), and every transfer
 path, so a worker mid-transfer is never reaped; traffic never latches the flag.
 
 ### 5.2 Locking
@@ -88,11 +89,11 @@ The reaper runs on its own thread, so the per-instance dicts are now mutated off
 the MQ handler threads. Each transfer module gains one `threading.Lock` so the
 reaper's scan-and-pop cannot race a concurrent register/unregister/transfer (which
 would otherwise corrupt the dict or hand out a half-removed entry). In
-`NonGPUTransferModule` the context and strategy dicts mutate as a pair under that
-lock, so a reap racing a re-register can never strand a fresh context without its
+`EngineDrivenTransferModule` the context and strategy dicts mutate as a pair under
+that lock, so a reap racing a re-register can never strand a fresh context without its
 strategy. It is a leaf lock — never held across context construction, storage
 calls, or any other component — so no thread ever holds two locks. External
-readers use the locked accessors `get_context_entry` (get-and-refresh) and
+readers use the locked accessors `get_and_touch_context_entry` (get-and-refresh) and
 `context_entries_snapshot` instead of touching the dict directly.
 
 ### 5.3 Reaper
@@ -101,37 +102,40 @@ readers use the locked accessors `get_context_entry` (get-and-refresh) and
 `reap_timeout / 4`. Each scan calls `reap_stale_instances` on every target: under
 the module lock, collect ids whose staleness exceeds their window and pop them;
 outside the lock, run the same cleanup as a client unregister and log a WARNING
-per instance (repeated reaps of one id signal a too-small timeout). Reaped ids fan
-out to every `InstanceReapListener.drop_instance_state(id)`; `BlendV3Module` drops
+per instance (repeated reaps of one id signal a too-small timeout). Reaped ids are
+then passed to `drop_instance_state(id)` on every target; `BlendV3Module` drops
 the reaped instance's per-instance CB state (e.g. rope state) there. (It no longer
 mirrors the GPU cache context — that mirror was removed upstream, so reaping the
 GPU entry now frees the context directly.) Collect+pop shares the module lock with
 register's refresh, serializing every register-vs-reap race; on close, the reaper
 is stopped and joined before any module clears state.
 
-### 5.4 Public protocols and config
+### 5.4 Public protocol and config
 
 ```python
 class InstanceLivenessTarget(Protocol):
+    # All methods default to a no-op; an implementer overrides only its role.
     def touch_instance(self, instance_id: int) -> None: ...
     def reap_stale_instances(
         self, reap_timeout_s: float, registration_grace_s: float
     ) -> list[int]: ...
     def tracked_instance_count(self) -> int: ...
-
-class InstanceReapListener(Protocol):
     def drop_instance_state(self, instance_id: int) -> None: ...
 ```
 
-Both transfer modules implement `InstanceLivenessTarget`; `BlendV3Module` is an
-`InstanceReapListener`; `ManagementModule` receives both by constructor injection.
+One protocol covers both reaper-driven roles. The transfer modules override the
+liveness methods (`touch`/`reap`/`count`); `BlendV3Module` overrides only
+`drop_instance_state` to drop its mirrored CB state. `ManagementModule` receives
+all targets in a single injected list. (Earlier a separate one-method
+`InstanceReapListener` held `drop_instance_state`; it was folded in since only
+`BlendV3Module` ever implemented it.)
 
 Config: `worker_reap_timeout_seconds` (default `120.0`; `0` disables, otherwise
 `>= 30.0`) and `worker_registration_grace_seconds` (default `3600.0`; `>=` the
 reap timeout — a tighter grace would reap warming workers faster than crashed
-ones), with matching CLI flags. Freeze-gap recovery additionally requires the
-timeout `>= 3 x` the client's `lmcache.mp.heartbeat_interval`; the worker adapter
-warns at startup when `3 x interval` exceeds the 30 s config floor.
+ones), with matching CLI flags. Keep the timeout `>= 3 x` the client's
+`lmcache.mp.heartbeat_interval` so a few missed pings never reap a live worker;
+the worker adapter warns at startup when `3 x interval` exceeds the 30 s floor.
 
 ## 6. Adapter Side
 
@@ -142,23 +146,11 @@ warmup; the registration grace covers that window. It starts healthy (the event
 is set at construction), so the first store/retrieve is not gated. A live worker
 then pings every interval, refreshing its server-side `last_seen`, so it is never
 reaped while alive — no re-registration is needed at start. The recover callback
-re-registers only on a genuine recovery edge (Section 6.2/6.3). A retrieve dropped
+re-registers only on a genuine recovery edge (Section 6.2). A retrieve dropped
 while the server is unhealthy is still reported via `get_finished` so async loads
 cannot hang.
 
-### 6.2 Freeze-gap detection
-
-A whole-process freeze longer than the reap window (SIGSTOP, cgroup freezer, VM
-migration) produces no ping failures and no traffic: the server reaps while the
-client's `health_event` stays set, so on thaw no edge would fire and every store
-would hit a missing context. `HeartbeatThread` therefore tracks the time of its
-last successful cycle; when a ping succeeds while considered healthy and the gap
-exceeds `3 x interval`, it clears `health_event` and returns, forcing one
-unhealthy cycle — the next ping takes the normal edge and re-registers. Forcing a
-cycle keeps the recover callback from racing live submissions on `transfer_ctx`
-and drains futures parked between thaw and detection.
-
-### 6.3 Recovery after a reap
+### 6.2 Recovery after a reap
 
 ```
 T0        outage begins; pings time out -> health_event cleared, traffic stops
@@ -172,7 +164,7 @@ T1        recover callback re-registers (id absent -> fresh context) before
 A shorter outage hits the NOOP register path, which refreshes `last_seen` and
 builds nothing — the server never asks a worker to re-register.
 
-### 6.4 Shutdown
+### 6.3 Shutdown
 
 `shutdown()` stops the heartbeat before sending UNREGISTER, so no stray ping
 lands on a closing client. The heartbeat cycle skips the recover callback and
@@ -188,6 +180,5 @@ stop is already requested — a straggling cycle cannot re-create a ghost contex
 | Worker alive but never pinged, idle past the grace | Reaped while alive only if it never pinged (heartbeat never started). Once the heartbeat is running, pings refresh `last_seen` every interval, so a live worker is never reaped regardless of traffic. |
 | Heartbeat thread starved, worker transferring | Store/retrieve/prepare/commit refresh `last_seen`; never reaped. |
 | Partition shorter than the reap window | No reap. On heal, the recover callback re-registers; the NOOP path refreshes `last_seen`; zero context churn. |
-| Whole-process freeze past the reap window | Reaped during the freeze. On thaw, freeze-gap detection forces one unhealthy cycle (parked futures drained), then edge → re-register. Worst case ~2 heartbeat intervals of degraded traffic. |
 | Worker crash + restart | The new process gets a fresh uuid-derived id and a fresh entry; the dead id is reaped independently. No PID-reuse aliasing. |
 | Mixed client/server versions | Every PING fails the payload-count check; the client sits permanently unhealthy. Loud, never silent corruption; upgrade both sides together. |