hotwire-vllm 0.0.2__tar.gz

hotwire_vllm-0.0.2/.github/workflows/test.yml

@@ -0,0 +1,22 @@
+name: tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  unit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install (CPU torch + triton)
+        run: |
+          pip install torch --index-url https://download.pytorch.org/whl/cpu
+          pip install triton pytest
+          pip install -e .
+      - name: Unit tests (GPU + integration tests auto-skip)
+        run: pytest -q

hotwire_vllm-0.0.2/.gitignore

@@ -0,0 +1,11 @@
+__pycache__/
+*.egg-info/
+.pytest_cache/
+.venv/
+dist/
+notes/
+
+virtualenvs/
+.idea/
+dist/
+*.egg-info/

hotwire_vllm-0.0.2/AGENTS.md

@@ -0,0 +1,42 @@
+# Using hotwire-vllm (guide for coding agents)
+
+You are helping someone add per-request activation steering to a vLLM server
+with `hotwire-vllm`. It is a vLLM plugin — small surface, but the details
+below are the ones agents get wrong.
+
+## Install and load
+
+    pip install hotwire-vllm
+
+It auto-registers via the `vllm.general_plugins` entry point when vLLM
+starts — nothing to import in user code. If steering silently does nothing,
+the plugin did not load: confirm vLLM sees the entry point.
+
+## Steer one request
+
+Steering rides on each request as a `vllm_xargs` field. The `hotwire` value is
+a JSON **string** (stringified spec), not a nested object:
+
+    {"messages": [...],
+     "vllm_xargs": {"hotwire": "{\"id\": \"NAME\", \"layer\": 20, \"scale\": 1.5}"}}
+
+Multiple layers: pass a JSON list of such specs. Add `"decode_only": true` to
+steer only generated tokens.
+
+## The gotchas that actually bite
+
+- **Slots are a fixed budget.** Each distinct steering config lives in a
+  fixed-size GPU slot bank; distinct (layer, scale) combos each consume one
+  persistent slot. Don't sweep hundreds of combos on a live server — you will
+  exhaust the bank. Reuse configs; size the sweep to the slot budget.
+- **Vectors must be registered** with the server (wire format is JSON /
+  safetensors, no pickle). An unknown id degrades to *unsteered*, never a
+  failed request — so "no effect" usually means "id not registered" or
+  "plugin not loaded", not "steering broke".
+- **Malformed specs degrade to unsteered**, silently. Validate your spec JSON.
+- Unsteered requests (including batchmates of steered ones) are untouched, and
+  idle steering is zero-cost — CUDA graphs stay intact.
+
+Calibrate the vector first with
+[hidden-directions](https://github.com/moudrkat/hidden-directions) (it produces
+the `{id, layer, scale, decode_only}` spec you paste here).

hotwire_vllm-0.0.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Katerina Fajmanova
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

hotwire_vllm-0.0.2/PKG-INFO

@@ -0,0 +1,267 @@
+Metadata-Version: 2.4
+Name: hotwire-vllm
+Version: 0.0.2
+Summary: CUDA-graph-safe activation steering plugin for vLLM
+License-Expression: MIT
+License-File: LICENSE
+Requires-Python: >=3.10
+Requires-Dist: numpy
+Provides-Extra: dev
+Requires-Dist: pytest; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# hotwire
+
+[![PyPI](https://img.shields.io/pypi/v/hotwire-vllm)](https://pypi.org/project/hotwire-vllm/)
+
+```
+pip install hotwire-vllm
+```
+
+**Activation steering for vLLM that doesn't turn off the engine.**
+
+For teams running vLLM in production who need per-request activation
+steering at native speed — CUDA graphs and torch.compile intact.
+
+Every existing steering tool for vLLM ([vllm-lens](https://github.com/UKGovernmentBEIS/vllm-lens),
+[EasySteer](https://arxiv.org/abs/2509.25175), IBM's vLLM Hook) forces
+`enforce_eager=True`: PyTorch forward hooks don't survive CUDA graph capture, so
+they disable CUDA graphs and torch.compile for the whole server — every request
+pays, steered or not. Fine for research, a non-starter for production.
+
+hotwire keeps the graphs. The steering addition is a custom torch op (Triton
+kernel) that gets baked *into* the captured graph; per-request routing happens
+by updating the contents of persistent GPU buffers between graph replays —
+the graph reads fresh data at the same addresses.
+
+The technique was proven viable in [RhizoNymph's vLLM fork](https://github.com/RhizoNymph/vllm)
+(see [RFC #36998](https://github.com/vllm-project/vllm/issues/36998), where
+in-flight steering is explicitly deferred to "Phase 2"). hotwire packages it as
+an out-of-tree plugin: `pip install`, no fork, registered via vLLM's official
+`general_plugins` entry point.
+
+## ⚡ Run in 30 s
+
+```bash
+pip install -e .          # registers the vllm.general_plugins entry point
+export HOTWIRE_VECTORS=/path/to/vectors   # dir of .pt files, (n_layers, hidden) each
+vllm serve Qwen/Qwen3-4B-Instruct-2507    # CUDA graphs stay ON
+```
+
+Steer any request by id + layer + scale:
+
+```python
+# offline
+SamplingParams(extra_args={"hotwire": '{"id": "tesla_car", "layer": 20, "scale": 1.5}'})
+```
+
+Optional per-entry flag `"decode_only": true` steers generated tokens only,
+never the prompt — use it for vectors calibrated on generation-only steering
+(research rigs typically don't steer the prefill; applying such a vector to a
+long prompt as well multiplies the effective dose and can wreck coherence).
+
+Lab twin: [brainscope](https://github.com/moudrkat/brainscope) accepts this
+exact spec and wire format — calibrate a vector under its lenses (its
+`export_hotwire` ships `.pt` files with a regime passport), deploy it here
+unchanged, and replay production conversations back under the lens when a
+vector misbehaves.
+
+```bash
+# OpenAI API
+curl .../v1/chat/completions -d '{..., "vllm_xargs":
+  {"hotwire": "{\"id\": \"tesla_car\", \"layer\": 20, \"scale\": 1.5}"}}'
+```
+
+Unsteered requests — including batchmates of steered ones — are untouched.
+Malformed specs and unknown vector ids degrade to "unsteered", never to a
+failed request.
+
+## Design
+
+Three persistent GPU tensors, allocated at model-load time:
+
+| buffer | shape | role |
+|---|---|---|
+| `bank` | `(n_slots, hidden)` | steering vectors, one per active slot |
+| `scales` | `(n_slots,)` | per-slot multiplier |
+| `slot_map` | `(n_layers, max_tokens)` | token → slot per layer, `-1` = untouched |
+
+The op, called at the end of each decoder layer's forward:
+
+```
+hidden[tok] += scales[slot] * bank[slot]   where slot = slot_map[layer, tok] >= 0
+```
+
+- **Graph-safe:** the op is `torch.library.custom_op` with a fake impl —
+  opaque to torch.compile, captured into CUDA graphs as a fixed kernel on
+  fixed addresses. Steering on/off/vector changes are buffer *content*
+  updates between replays (host-side copy), never a re-capture.
+- **Per-request:** a pre-forward hook reads `forward_context`
+  (`query_start_loc` + `req_ids`, same bookkeeping vllm-lens validated)
+  and fills `slot_map` for the step.
+- **No pickle:** vectors enter as safetensors files or base64 JSON via a
+  registration endpoint (`POST /steer/vectors`), requests reference them by
+  id + scale in `vllm_xargs`. Nothing executable crosses the wire.
+- **Zero cost when idle:** `slot_map` all `-1` → kernel early-exits per token.
+  (Benchmark target: unmeasurable vs baseline; RhizoNymph reported minimal
+  overhead on H100.)
+
+## Layout
+
+- `hotwire/_kernel.py` — Triton kernel + `hotwire::steer` custom op (working)
+- `hotwire/_bank.py` — slot allocation, vector registration (working)
+- `hotwire/_patch.py` — decoder-layer wrapping + pre-forward slot fill (WIP:
+  integration points against vLLM 0.25.x)
+- `hotwire/wire.py` — JSON/safetensors vector wire format, no pickle (working)
+
+## Verify on your hardware
+
+Two commands, ~3 minutes on any CUDA box with vLLM installed:
+
+```bash
+pip install git+https://github.com/moudrkat/hotwire-vllm
+python -m hotwire.verify --model Qwen/Qwen3-0.6B   # any HF model id works
+```
+
+It generates a throwaway steering vector for the model, checks that steering
+fires, that unsteered requests (including batchmates) are untouched, and
+compares decode cost idle vs all-steered — then prints a report block.
+**Please paste the report into an issue**, especially from hardware, model
+families, or configs (TP > 1, 7B+, H100s) the tables below don't cover yet —
+that's currently the most useful contribution this project can receive.
+
+## Status
+
+Working end-to-end on vLLM 0.25.1, **both model runners** (the classic
+`GPUModelRunner` and the new V2 runner that 0.25.1 selects by default for
+dense generate models), with CUDA graphs captured (PIECEWISE + FULL) and
+torch.compile on. Verified on Qwen3-0.6B / Qwen3-4B on a single 16 GB GPU:
+solo steering, mixed batches, decode-phase graph replays.
+
+hotwire also salts vLLM's torch.compile/AOT cache key (`VllmConfig.compute_hash`)
+— the op is traced into the compiled model, and vLLM's cache doesn't know about
+plugins, so without the salt a stale cache silently serves a model with no
+steering op in it.
+
+Tests: `pytest` (unit, CPU-safe), `pytest -m integration` (real engine, GPU).
+
+Verified architectures (chaos-vector A/B + batchmate-isolation check, both
+model runners exercised):
+
+| model | steering works | unsteered untouched | TPOT idle → all-steered |
+|---|---|---|---|
+| Qwen3-14B-AWQ (4-bit) | ✓ | ✓ | 1.91 → 1.91 ms/tok |
+| Llama-3.1-8B-Instruct-AWQ | ✓ | ✓ | 1.10 → 1.10 ms/tok |
+| Qwen3-8B-FP8 | ✓ | ✓ | 2.27 → 2.28 ms/tok |
+| Mistral-7B-Instruct-v0.2-AWQ | ✓ | ✓ | 0.94 → 0.94 ms/tok |
+| Qwen3-4B-Instruct-2507 | ✓ | ✓ | 1.78 → 1.78 ms/tok |
+| Qwen3-0.6B | ✓ | ✓ | — |
+| Qwen2.5-1.5B-Instruct | ✓ | ✓ | 0.77 → 0.77 ms/tok |
+| Phi-3.5-mini-instruct | ✓ | ✓ | 1.73 → 1.73 ms/tok |
+| tiny-aya-water (Cohere) | ✓ | ✓ | 1.54 → 1.54 ms/tok |
+
+Quantized checkpoints work — steering touches the residual stream, not the
+weights, and the AWQ / FP8 rows above confirm it end-to-end, CUDA graphs
+captured (PIECEWISE + FULL).
+
+Models that still OOM on the 16 GB test GPU before the plugin engages:
+OLMo-2-7B, command-r7b, Qwen3.5-4B, gpt-oss-20b (13.8 GiB weight load
+succeeds, engine init doesn't), gemma-4-E4B-it (vision tower). No
+architecture failure observed yet; reports from bigger cards welcome. The
+layer patch targets any `*DecoderLayer` module with the standard
+`(positions, hidden_states, residual)` signature.
+
+## Numbers
+
+Qwen3-4B-Instruct-2507, bf16, RTX 4070 Ti SUPER 16 GB, 8 concurrent requests,
+256 decode tokens each, medians of 3 (`benchmarks/bench_decode.py`):
+
+| condition | TTFT | decode TPOT |
+|---|---|---|
+| vanilla vLLM (plugin not installed) | 4.9 ms | 1.78 ms/tok |
+| hotwire installed, no request steered | 4.9 ms | 1.78 ms/tok |
+| hotwire, **all 8 requests steered** | 4.6 ms | 1.78 ms/tok |
+| vLLM `enforce_eager` (no plugin) | 5.0 ms | 1.88 ms/tok |
+
+Idle and fully-steered are both within noise of vanilla. The eager row is what
+hook-based steering tools pay *before* their Python hooks even run.
+
+Batch sweep (same model/GPU): the eager tax grows with batch pressure —
++2.3% at 1 request (batch-1 decode is weight-streaming-bound, which hides
+launch overhead), +4.7% at 2, +5.6% at 8. hotwire's idle == steered holds at
+every batch size, to the second decimal.
+
+| batch | graphs idle | graphs all-steered | eager |
+|---|---|---|---|
+| 1 | 13.69 ms/tok | 13.69 | 14.00 |
+| 2 | 6.97 ms/tok | 6.97 | 7.30 |
+| 8 | 1.78 ms/tok | 1.78 | 1.88 |
+
+Untested configurations (no known issues, but nobody has run them — treat as
+unsupported until someone does): tensor parallel > 1, pipeline parallel,
+speculative decoding, LoRA, GPTQ and MXFP4 quantization (AWQ and FP8 are
+verified — see the table). Issues welcome.
+
+Known limitation: one vector per (layer, token) — multiple spec entries
+targeting the **same layer** don't stack; the last one wins. Different layers
+compose fine. Workaround: pre-combine same-layer vectors into one .pt
+(`a*v1 + b*v2`) and register the combo; native stacking is on the roadmap.
+
+Known limitation: the slot budget. Steering configs live in a fixed-size GPU
+table allocated before graph capture — CUDA graphs read fixed addresses, so
+it can never grow at runtime. Size it with `HOTWIRE_SLOTS` (default 16;
+a slot is one vector row, ~5 KB on a 4B model, so 256 costs ~1.3 MB and
+nothing per token). Each distinct **(vector, layer, scale)** combo occupies
+one slot **permanently** — nothing frees slots when requests finish. A fixed
+catalog of vectors at fixed scales therefore runs forever, but continuously
+varying scales (0.80, 0.83, 0.87, …) mint a fresh slot each and exhaust the
+table; once full, requests with an unregistrable combo run unsteered (logged)
+while already-registered combos keep working, batchmates included.
+Workaround today: round scales to a small fixed
+palette and set `HOTWIRE_SLOTS` generously at startup. The real fixes are on
+the roadmap below — slots *can* recycle (the scale isn't baked into the
+stored vector; the kernel reads it separately at replay), it's bookkeeping,
+not graph physics.
+
+Roadmap:
+- HTTP vector registration at runtime (via `vllm.endpoint_plugins`), replacing
+  startup-only `$HOTWIRE_VECTORS`.
+- Slot eviction: refcount slots per in-flight request and `release()` when the
+  last user of a combo finishes, so the table recycles instead of filling.
+- Per-token scales: key slots by (vector, layer) only and move scale into a
+  per-token buffer — continuous intensities without minting new slots.
+- Norm-matched and position-targeted steering modes.
+- Tracking the RFC vllm-project/vllm#36998 Phase 2 interface as it lands.
+## Where this sits in the lab
+
+```mermaid
+flowchart LR
+    hd["🧭 hidden-directions<br/>behavior → vector"]
+    bs(["🧠 brainscope<br/>watch the model think"])
+    hw["🔥 hotwire-vllm<br/>steering in production"]
+    st["🕹️ steeropathy<br/>agents talk via activations"]
+    tm["⚖️ in-two-minds<br/>agent hesitating between tools"]
+    sm["🧪 steering-mechanics<br/>how steering actually works"]
+
+    hd -->|vectors| bs
+    hd -->|vector + passport| hw
+    bs --> st
+    bs --> tm
+    bs -->|causal replay| sm
+    hw -.->|vector under study| sm
+
+    click hd "https://github.com/moudrkat/hidden-directions"
+    click bs "https://github.com/moudrkat/brainscope"
+    click hw "https://github.com/moudrkat/hotwire-vllm"
+    click st "https://github.com/moudrkat/steeropathy"
+    click tm "https://github.com/moudrkat/in-two-minds"
+    click sm "https://github.com/moudrkat/steering-mechanics"
+
+    classDef dim fill:#f6f8fa,stroke:#d0d7de,color:#57606a;
+    classDef here fill:#8957e5,stroke:#6e40c9,color:#ffffff;
+    class hd,bs,hw,st,tm,sm dim;
+    class hw here;
+```
+
+*Highlighted = this repo. The full lab map (with the two other repos' stories) lives on [moudrkat](https://github.com/moudrkat).*
+

hotwire_vllm-0.0.2/README.md

@@ -0,0 +1,255 @@
+# hotwire
+
+[![PyPI](https://img.shields.io/pypi/v/hotwire-vllm)](https://pypi.org/project/hotwire-vllm/)
+
+```
+pip install hotwire-vllm
+```
+
+**Activation steering for vLLM that doesn't turn off the engine.**
+
+For teams running vLLM in production who need per-request activation
+steering at native speed — CUDA graphs and torch.compile intact.
+
+Every existing steering tool for vLLM ([vllm-lens](https://github.com/UKGovernmentBEIS/vllm-lens),
+[EasySteer](https://arxiv.org/abs/2509.25175), IBM's vLLM Hook) forces
+`enforce_eager=True`: PyTorch forward hooks don't survive CUDA graph capture, so
+they disable CUDA graphs and torch.compile for the whole server — every request
+pays, steered or not. Fine for research, a non-starter for production.
+
+hotwire keeps the graphs. The steering addition is a custom torch op (Triton
+kernel) that gets baked *into* the captured graph; per-request routing happens
+by updating the contents of persistent GPU buffers between graph replays —
+the graph reads fresh data at the same addresses.
+
+The technique was proven viable in [RhizoNymph's vLLM fork](https://github.com/RhizoNymph/vllm)
+(see [RFC #36998](https://github.com/vllm-project/vllm/issues/36998), where
+in-flight steering is explicitly deferred to "Phase 2"). hotwire packages it as
+an out-of-tree plugin: `pip install`, no fork, registered via vLLM's official
+`general_plugins` entry point.
+
+## ⚡ Run in 30 s
+
+```bash
+pip install -e .          # registers the vllm.general_plugins entry point
+export HOTWIRE_VECTORS=/path/to/vectors   # dir of .pt files, (n_layers, hidden) each
+vllm serve Qwen/Qwen3-4B-Instruct-2507    # CUDA graphs stay ON
+```
+
+Steer any request by id + layer + scale:
+
+```python
+# offline
+SamplingParams(extra_args={"hotwire": '{"id": "tesla_car", "layer": 20, "scale": 1.5}'})
+```
+
+Optional per-entry flag `"decode_only": true` steers generated tokens only,
+never the prompt — use it for vectors calibrated on generation-only steering
+(research rigs typically don't steer the prefill; applying such a vector to a
+long prompt as well multiplies the effective dose and can wreck coherence).
+
+Lab twin: [brainscope](https://github.com/moudrkat/brainscope) accepts this
+exact spec and wire format — calibrate a vector under its lenses (its
+`export_hotwire` ships `.pt` files with a regime passport), deploy it here
+unchanged, and replay production conversations back under the lens when a
+vector misbehaves.
+
+```bash
+# OpenAI API
+curl .../v1/chat/completions -d '{..., "vllm_xargs":
+  {"hotwire": "{\"id\": \"tesla_car\", \"layer\": 20, \"scale\": 1.5}"}}'
+```
+
+Unsteered requests — including batchmates of steered ones — are untouched.
+Malformed specs and unknown vector ids degrade to "unsteered", never to a
+failed request.
+
+## Design
+
+Three persistent GPU tensors, allocated at model-load time:
+
+| buffer | shape | role |
+|---|---|---|
+| `bank` | `(n_slots, hidden)` | steering vectors, one per active slot |
+| `scales` | `(n_slots,)` | per-slot multiplier |
+| `slot_map` | `(n_layers, max_tokens)` | token → slot per layer, `-1` = untouched |
+
+The op, called at the end of each decoder layer's forward:
+
+```
+hidden[tok] += scales[slot] * bank[slot]   where slot = slot_map[layer, tok] >= 0
+```
+
+- **Graph-safe:** the op is `torch.library.custom_op` with a fake impl —
+  opaque to torch.compile, captured into CUDA graphs as a fixed kernel on
+  fixed addresses. Steering on/off/vector changes are buffer *content*
+  updates between replays (host-side copy), never a re-capture.
+- **Per-request:** a pre-forward hook reads `forward_context`
+  (`query_start_loc` + `req_ids`, same bookkeeping vllm-lens validated)
+  and fills `slot_map` for the step.
+- **No pickle:** vectors enter as safetensors files or base64 JSON via a
+  registration endpoint (`POST /steer/vectors`), requests reference them by
+  id + scale in `vllm_xargs`. Nothing executable crosses the wire.
+- **Zero cost when idle:** `slot_map` all `-1` → kernel early-exits per token.
+  (Benchmark target: unmeasurable vs baseline; RhizoNymph reported minimal
+  overhead on H100.)
+
+## Layout
+
+- `hotwire/_kernel.py` — Triton kernel + `hotwire::steer` custom op (working)
+- `hotwire/_bank.py` — slot allocation, vector registration (working)
+- `hotwire/_patch.py` — decoder-layer wrapping + pre-forward slot fill (WIP:
+  integration points against vLLM 0.25.x)
+- `hotwire/wire.py` — JSON/safetensors vector wire format, no pickle (working)
+
+## Verify on your hardware
+
+Two commands, ~3 minutes on any CUDA box with vLLM installed:
+
+```bash
+pip install git+https://github.com/moudrkat/hotwire-vllm
+python -m hotwire.verify --model Qwen/Qwen3-0.6B   # any HF model id works
+```
+
+It generates a throwaway steering vector for the model, checks that steering
+fires, that unsteered requests (including batchmates) are untouched, and
+compares decode cost idle vs all-steered — then prints a report block.
+**Please paste the report into an issue**, especially from hardware, model
+families, or configs (TP > 1, 7B+, H100s) the tables below don't cover yet —
+that's currently the most useful contribution this project can receive.
+
+## Status
+
+Working end-to-end on vLLM 0.25.1, **both model runners** (the classic
+`GPUModelRunner` and the new V2 runner that 0.25.1 selects by default for
+dense generate models), with CUDA graphs captured (PIECEWISE + FULL) and
+torch.compile on. Verified on Qwen3-0.6B / Qwen3-4B on a single 16 GB GPU:
+solo steering, mixed batches, decode-phase graph replays.
+
+hotwire also salts vLLM's torch.compile/AOT cache key (`VllmConfig.compute_hash`)
+— the op is traced into the compiled model, and vLLM's cache doesn't know about
+plugins, so without the salt a stale cache silently serves a model with no
+steering op in it.
+
+Tests: `pytest` (unit, CPU-safe), `pytest -m integration` (real engine, GPU).
+
+Verified architectures (chaos-vector A/B + batchmate-isolation check, both
+model runners exercised):
+
+| model | steering works | unsteered untouched | TPOT idle → all-steered |
+|---|---|---|---|
+| Qwen3-14B-AWQ (4-bit) | ✓ | ✓ | 1.91 → 1.91 ms/tok |
+| Llama-3.1-8B-Instruct-AWQ | ✓ | ✓ | 1.10 → 1.10 ms/tok |
+| Qwen3-8B-FP8 | ✓ | ✓ | 2.27 → 2.28 ms/tok |
+| Mistral-7B-Instruct-v0.2-AWQ | ✓ | ✓ | 0.94 → 0.94 ms/tok |
+| Qwen3-4B-Instruct-2507 | ✓ | ✓ | 1.78 → 1.78 ms/tok |
+| Qwen3-0.6B | ✓ | ✓ | — |
+| Qwen2.5-1.5B-Instruct | ✓ | ✓ | 0.77 → 0.77 ms/tok |
+| Phi-3.5-mini-instruct | ✓ | ✓ | 1.73 → 1.73 ms/tok |
+| tiny-aya-water (Cohere) | ✓ | ✓ | 1.54 → 1.54 ms/tok |
+
+Quantized checkpoints work — steering touches the residual stream, not the
+weights, and the AWQ / FP8 rows above confirm it end-to-end, CUDA graphs
+captured (PIECEWISE + FULL).
+
+Models that still OOM on the 16 GB test GPU before the plugin engages:
+OLMo-2-7B, command-r7b, Qwen3.5-4B, gpt-oss-20b (13.8 GiB weight load
+succeeds, engine init doesn't), gemma-4-E4B-it (vision tower). No
+architecture failure observed yet; reports from bigger cards welcome. The
+layer patch targets any `*DecoderLayer` module with the standard
+`(positions, hidden_states, residual)` signature.
+
+## Numbers
+
+Qwen3-4B-Instruct-2507, bf16, RTX 4070 Ti SUPER 16 GB, 8 concurrent requests,
+256 decode tokens each, medians of 3 (`benchmarks/bench_decode.py`):
+
+| condition | TTFT | decode TPOT |
+|---|---|---|
+| vanilla vLLM (plugin not installed) | 4.9 ms | 1.78 ms/tok |
+| hotwire installed, no request steered | 4.9 ms | 1.78 ms/tok |
+| hotwire, **all 8 requests steered** | 4.6 ms | 1.78 ms/tok |
+| vLLM `enforce_eager` (no plugin) | 5.0 ms | 1.88 ms/tok |
+
+Idle and fully-steered are both within noise of vanilla. The eager row is what
+hook-based steering tools pay *before* their Python hooks even run.
+
+Batch sweep (same model/GPU): the eager tax grows with batch pressure —
++2.3% at 1 request (batch-1 decode is weight-streaming-bound, which hides
+launch overhead), +4.7% at 2, +5.6% at 8. hotwire's idle == steered holds at
+every batch size, to the second decimal.
+
+| batch | graphs idle | graphs all-steered | eager |
+|---|---|---|---|
+| 1 | 13.69 ms/tok | 13.69 | 14.00 |
+| 2 | 6.97 ms/tok | 6.97 | 7.30 |
+| 8 | 1.78 ms/tok | 1.78 | 1.88 |
+
+Untested configurations (no known issues, but nobody has run them — treat as
+unsupported until someone does): tensor parallel > 1, pipeline parallel,
+speculative decoding, LoRA, GPTQ and MXFP4 quantization (AWQ and FP8 are
+verified — see the table). Issues welcome.
+
+Known limitation: one vector per (layer, token) — multiple spec entries
+targeting the **same layer** don't stack; the last one wins. Different layers
+compose fine. Workaround: pre-combine same-layer vectors into one .pt
+(`a*v1 + b*v2`) and register the combo; native stacking is on the roadmap.
+
+Known limitation: the slot budget. Steering configs live in a fixed-size GPU
+table allocated before graph capture — CUDA graphs read fixed addresses, so
+it can never grow at runtime. Size it with `HOTWIRE_SLOTS` (default 16;
+a slot is one vector row, ~5 KB on a 4B model, so 256 costs ~1.3 MB and
+nothing per token). Each distinct **(vector, layer, scale)** combo occupies
+one slot **permanently** — nothing frees slots when requests finish. A fixed
+catalog of vectors at fixed scales therefore runs forever, but continuously
+varying scales (0.80, 0.83, 0.87, …) mint a fresh slot each and exhaust the
+table; once full, requests with an unregistrable combo run unsteered (logged)
+while already-registered combos keep working, batchmates included.
+Workaround today: round scales to a small fixed
+palette and set `HOTWIRE_SLOTS` generously at startup. The real fixes are on
+the roadmap below — slots *can* recycle (the scale isn't baked into the
+stored vector; the kernel reads it separately at replay), it's bookkeeping,
+not graph physics.
+
+Roadmap:
+- HTTP vector registration at runtime (via `vllm.endpoint_plugins`), replacing
+  startup-only `$HOTWIRE_VECTORS`.
+- Slot eviction: refcount slots per in-flight request and `release()` when the
+  last user of a combo finishes, so the table recycles instead of filling.
+- Per-token scales: key slots by (vector, layer) only and move scale into a
+  per-token buffer — continuous intensities without minting new slots.
+- Norm-matched and position-targeted steering modes.
+- Tracking the RFC vllm-project/vllm#36998 Phase 2 interface as it lands.
+## Where this sits in the lab
+
+```mermaid
+flowchart LR
+    hd["🧭 hidden-directions<br/>behavior → vector"]
+    bs(["🧠 brainscope<br/>watch the model think"])
+    hw["🔥 hotwire-vllm<br/>steering in production"]
+    st["🕹️ steeropathy<br/>agents talk via activations"]
+    tm["⚖️ in-two-minds<br/>agent hesitating between tools"]
+    sm["🧪 steering-mechanics<br/>how steering actually works"]
+
+    hd -->|vectors| bs
+    hd -->|vector + passport| hw
+    bs --> st
+    bs --> tm
+    bs -->|causal replay| sm
+    hw -.->|vector under study| sm
+
+    click hd "https://github.com/moudrkat/hidden-directions"
+    click bs "https://github.com/moudrkat/brainscope"
+    click hw "https://github.com/moudrkat/hotwire-vllm"
+    click st "https://github.com/moudrkat/steeropathy"
+    click tm "https://github.com/moudrkat/in-two-minds"
+    click sm "https://github.com/moudrkat/steering-mechanics"
+
+    classDef dim fill:#f6f8fa,stroke:#d0d7de,color:#57606a;
+    classDef here fill:#8957e5,stroke:#6e40c9,color:#ffffff;
+    class hd,bs,hw,st,tm,sm dim;
+    class hw here;
+```
+
+*Highlighted = this repo. The full lab map (with the two other repos' stories) lives on [moudrkat](https://github.com/moudrkat).*
+