@ericrisco/rsc 0.1.39 → 0.1.41

package/skills/unsloth/references/masking-and-templates.md

@@ -0,0 +1,91 @@
+# Chat templates + response-only masking
+
+Facts here track docs.unsloth.ai/basics/chat-templates (accessed 2026-07). Template names and turn
+markers change with new model releases — **verify the pair for your exact base model.**
+
+## Why a helper and not a string
+
+A chat model was pretrained with a specific set of turn markers (`<|start_header_id|>…`,
+`<start_of_turn>…`, `<|im_start|>…`). If you concatenate messages with the wrong markers, the model
+never sees its own format and the fine-tune underperforms or degenerates. `get_chat_template` writes
+the *correct* template onto your tokenizer; `apply_chat_template` then renders your message list into
+the single `text` string the trainer consumes.
+
+```python
+from unsloth.chat_templates import get_chat_template
+
+tokenizer = get_chat_template(
+    tokenizer,
+    chat_template = "llama-3.1",   # match the base model
+    # mapping = {"role":"from","content":"value","user":"human","assistant":"gpt"},  # if your
+    #   dataset uses ShareGPT-style keys instead of role/content
+    # map_eos_token = True,        # map e.g. <|im_end|> -> the model's EOS
+)
+
+def to_text(example):
+    example["text"] = tokenizer.apply_chat_template(
+        example["messages"], tokenize = False, add_generation_prompt = False,
+    )
+    return example
+
+dataset = dataset.map(to_text)
+```
+
+Supported template names include (non-exhaustive; verify live): `llama-3`, `llama-3.1`, `chatml`,
+`mistral`, `phi-4`, `gemma-3`, `qwen`, `zephyr`, `vicuna`, `alpaca`, `unsloth`. Use the one that
+matches your checkpoint.
+
+Thinking / reasoning models expose a toggle at render time:
+
+```python
+tokenizer.apply_chat_template(msgs, tokenize=False,
+                              add_generation_prompt=True, enable_thinking=True)  # or False
+```
+
+## `train_on_responses_only` — the mask
+
+Supervised fine-tuning of a chat model should compute loss on the **assistant** tokens only. Without
+this, the model is also trained to predict the user's words; loss can even collapse to ~0 on some
+templates (the "zero-loss trap"). Wrap the trainer AFTER constructing it:
+
+```python
+from unsloth.chat_templates import train_on_responses_only
+
+trainer = train_on_responses_only(
+    trainer,
+    instruction_part = "<|start_header_id|>user<|end_header_id|>\n\n",
+    response_part    = "<|start_header_id|>assistant<|end_header_id|>\n\n",
+)
+```
+
+`instruction_part` / `response_part` are the template's own turn-opening markers. Everything from an
+`instruction_part` up to the next `response_part` is masked (label = `-100`); the assistant span keeps
+its labels. **The strings must match the template you applied.** Common pairs (verify per model):
+
+| Base family      | `instruction_part`                              | `response_part`                                    |
+| ---------------- | ----------------------------------------------- | -------------------------------------------------- |
+| Llama-3 / 3.1    | `<\|start_header_id\|>user<\|end_header_id\|>\n\n`  | `<\|start_header_id\|>assistant<\|end_header_id\|>\n\n` |
+| Gemma-3 / vision | `<start_of_turn>user\n`                         | `<start_of_turn>model\n`                           |
+| ChatML (Qwen…)   | `<\|im_start\|>user\n`                            | `<\|im_start\|>assistant\n`                          |
+
+## Sanity-check the mask (always do this once)
+
+```python
+row = trainer.train_dataset[0]
+# 1) full sequence — should include both user + assistant turns
+print(tokenizer.decode(row["input_ids"]))
+# 2) UNMASKED labels only — should be the ASSISTANT answer and nothing else
+print(tokenizer.decode(
+    [tokenizer.pad_token_id if x == -100 else x for x in row["labels"]]
+).replace(tokenizer.pad_token, " "))
+```
+
+If step 2 prints the user's question, your `instruction_part`/`response_part` don't match the
+template — fix them before training. Spending GPU-hours on an unmasked run is the classic waste.
+
+## Vision (VLM)
+
+Vision fine-tunes use `FastVisionModel.from_pretrained(...)` and a processor/tokenizer pair; the
+template and mask concepts carry over but the collator handles image tokens. See
+docs.unsloth.ai for the current VLM notebook — the exact API for multimodal collation moves faster
+than the text path, so pull it live rather than trusting a frozen snippet.

package/skills/vllm/SKILL.md

@@ -0,0 +1,235 @@
+---
+name: vllm
+description: "Use when self-hosting an open-weight LLM for high-throughput, concurrent serving with vLLM — running `vllm serve` as an OpenAI-compatible endpoint, splitting a model across GPUs with tensor/pipeline parallelism, loading AWQ/GPTQ/FP8/compressed-tensors quants, serving one or many LoRA adapters, or debugging KV-cache OOM from --gpu-memory-utilization / --max-model-len. Triggers: 'serve a 70B to many users at once', 'point my OpenAI client at a self-hosted model', 'split a model across 4 GPUs with tensor parallel', 'why does vllm OOM at long context', 'serve my fine-tuned LoRA adapter', 'health check my vllm server', 'servir un modelo abierto con alta concurrencia en mis propias GPUs'. NOT renting/provisioning the GPU box itself (that is runpod / modal), NOT single-user laptop inference (that is ollama), NOT a hosted inference API you do not operate (that is together-fireworks / huggingface)."
+tags: [vllm, llm-serving, inference-server, tensor-parallel, quantization]
+recommends: [open-weights, finetuning, runpod, modal, ollama]
+origin: risco
+---
+
+# vLLM — high-throughput serving of open-weight models
+
+vLLM is the inference **engine** you put in front of an open-weight model when many requests hit it at
+once. Its job — and this skill's — is throughput under concurrency: keep the GPU busy across dozens of
+simultaneous requests, not squeeze one prompt out fast. You own the `vllm serve` flags; the box those
+flags run on is a `runpod`/`modal` concern.
+
+**Why not just loop a `transformers` `generate()`?** Naive serving runs one request at a time and pads
+every batch to the longest sequence, so the GPU idles. vLLM fixes both:
+
+- **PagedAttention** stores the KV cache in non-contiguous fixed-size blocks (like OS virtual-memory
+  paging), so there is almost no padding/reservation waste and long contexts pack tightly.
+- **Continuous batching** admits and retires requests token-by-token instead of per-batch, so a new
+  request joins the running batch immediately rather than waiting for the slowest one to finish.
+
+Net effect: an order-of-magnitude more concurrent throughput than single-request serving. If you only
+ever have one user on a laptop, that machinery is wasted — that is `ollama`, not this.
+
+## Version & setup reality (verify at author time — vLLM ships ~weekly)
+
+- **Latest stable is ~0.25.x (July 2026)** — releases land roughly weekly. Check
+  [PyPI](https://pypi.org/project/vllm/) / [releases](https://github.com/vllm-project/vllm/releases);
+  do not pin to a number you read here.
+- **The V1 engine is the default since v0.8.0** and recent releases have removed the legacy V0 path,
+  so treat V1 as the only engine. It runs the scheduler + core loop in a separate process and turns on
+  chunked prefill and prefix caching out of the box, so you rarely tune the scheduler by hand
+  ([V1 guide](https://docs.vllm.ai/en/latest/usage/v1_guide.html), accessed 2026-07). `VLLM_USE_V1`
+  historically toggled it — if you see it referenced, it is legacy.
+- **Install:** `pip install vllm` (default build is CUDA/NVIDIA). ROCm, CPU, TPU and other backends
+  have separate install paths — see the docs' installation matrix. Needs Python + a supported GPU.
+- **Auth is OFF by default.** A bare `vllm serve` is an *open* endpoint on `0.0.0.0:8000`. To require a
+  bearer token, pass `--api-key <KEY>` (or set `VLLM_API_KEY`); clients then send
+  `Authorization: Bearer <KEY>`. Do not expose an unauthenticated server publicly.
+
+## `vllm serve` → an OpenAI-compatible server
+
+```bash
+vllm serve Qwen/Qwen3-8B                 # download from HF (or a local path) and serve on :8000
+vllm serve /models/qwen3-8b --api-key sk-local-xyz --port 8000 --host 0.0.0.0
+```
+
+The server speaks the OpenAI wire protocol, so **any OpenAI client works unchanged** — just repoint
+`base_url` and use a dummy (or your `--api-key`) key. Endpoints
+([online serving docs](https://docs.vllm.ai/en/latest/serving/openai_compatible_server/), accessed 2026-07):
+
+| Endpoint | Purpose |
+| --- | --- |
+| `POST /v1/chat/completions` | chat protocol (messages array) — the usual path |
+| `POST /v1/completions` | raw text completion (single prompt string) |
+| `GET  /v1/models` | list the served model + any loaded LoRA adapters |
+| `POST /v1/embeddings` | only for an embedding model (`--task embed`) |
+| `GET  /health` | liveness — 200 when ready, **no auth, no body** |
+| `GET  /metrics` | Prometheus metrics (queue depth, throughput, cache usage) |
+
+```python
+from openai import OpenAI
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="sk-local-xyz")  # key = your --api-key
+r = client.chat.completions.create(
+    model="Qwen/Qwen3-8B",               # the id `/v1/models` reports (or a LoRA adapter name)
+    messages=[{"role": "user", "content": "Name three primes."}],
+)
+print(r.choices[0].message.content)
+```
+
+The `model` field must match what `/v1/models` returns — the served model id or a LoRA adapter name
+(below), not an arbitrary string.
+
+## Parallelism: fit the model, then scale out
+
+Two orthogonal knobs. Reach for parallelism only when the model does **not** fit one GPU — a model that
+fits should stay on a single GPU (no split), because every split adds communication overhead.
+
+- **`--tensor-parallel-size N`** — shard each layer's weights *across N GPUs on one node*. Use this
+  first: it needs fast intra-node links (NVLink / PCIe) because GPUs sync every layer. `N` must divide
+  the model's attention-head count. This is how you serve a model too big for one GPU but fitting the
+  node.
+- **`--pipeline-parallel-size M`** — split the model *by layer stages across M nodes*. Tolerates slower
+  inter-node network. Use it when the model does not fit even a full node.
+
+Rule of thumb from the docs
+([parallelism & scaling](https://docs.vllm.ai/en/latest/serving/distributed_serving.html), accessed 2026-07):
+set **tensor-parallel = GPUs per node**, **pipeline-parallel = number of nodes**.
+
+```bash
+vllm serve meta-llama/Llama-3.3-70B-Instruct --tensor-parallel-size 4          # 1 node, 4 GPUs
+vllm serve <huge-model> --tensor-parallel-size 8 --pipeline-parallel-size 2    # 2 nodes × 8 GPUs
+```
+
+Multi-node needs a Ray cluster wired up first — that orchestration is a `runpod`/`modal` concern, not a
+vLLM flag.
+
+## Quantization: smaller weights ≠ automatic speedup
+
+Quantization shrinks the weights so a model fits fewer/smaller GPUs and frees VRAM for KV cache. Select
+a method with `--quantization` (often auto-detected from the checkpoint's config). vLLM supports
+**AWQ, GPTQ/GPTQModel, FP8 (W8A8), compressed-tensors (LLM Compressor), INT4/INT8, bitsandbytes** and
+more ([quantization docs](https://docs.vllm.ai/en/latest/features/quantization/), accessed 2026-07).
+
+```bash
+vllm serve TheModel/Qwen3-8B-AWQ --quantization awq
+vllm serve neuralmagic/Model-FP8   --quantization compressed-tensors   # FP8 needs Ada/Hopper+
+```
+
+The honest tradeoffs:
+
+- **Support is GPU-arch-specific** — FP8 W8A8 wants Ada/Hopper-class (or AMD) hardware; a method that
+  is fast on one GPU may be unsupported or emulated on another. Check the compatibility table.
+- **Quant is a memory win, not a guaranteed throughput win.** At **low batch / low concurrency** the
+  workload is memory-bandwidth-bound and a quant can help; but dequant overhead can *cost* latency, and
+  at high batch you may be compute-bound where a weight-only quant does little. Benchmark your quant on
+  your hardware at your real concurrency before assuming it is faster.
+- **Quality drops** — usually small for 8-bit/well-calibrated 4-bit, larger for aggressive 4-bit. Serve
+  a quant to save VRAM, not as a free lunch. Picking *which* quantized checkpoint to pull is an
+  `open-weights` decision.
+
+## LoRA adapters: serve the fine-tuning output
+
+vLLM serves LoRA adapters on top of one loaded base model, so your `finetuning`/`unsloth` output goes
+live without merging or a second server
+([LoRA docs](https://docs.vllm.ai/en/latest/features/lora.html), accessed 2026-07).
+
+```bash
+vllm serve meta-llama/Llama-3.2-3B-Instruct \
+  --enable-lora \
+  --lora-modules sql=/adapters/sql-lora legal=/adapters/legal-lora \
+  --max-loras 2 \        # how many adapters resident at once
+  --max-lora-rank 16     # must be >= the rank the adapter was trained at
+```
+
+Route to an adapter by naming it in the request `model` field: `"model": "sql"` hits the SQL adapter,
+`"model": "meta-llama/Llama-3.2-3B-Instruct"` hits the untuned base — same server, no reload.
+
+Load/unload at runtime with `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`, then
+`POST /v1/load_lora_adapter` / `POST /v1/unload_lora_adapter`. Base + adapter must match (same family
+and dims), and `--max-lora-rank` must be ≥ the trained rank or load fails.
+
+## Memory & throughput: where OOM comes from
+
+The two knobs that cause (and cure) most OOM
+([engine args](https://docs.vllm.ai/en/latest/configuration/engine_args.html), accessed 2026-07):
+
+- **`--gpu-memory-utilization`** (default ~**0.92**) — fraction of each GPU vLLM may claim. Weights are
+  loaded, then the **rest of this budget becomes the KV-cache pool**. Raising it toward 1.0 buys more
+  concurrent sequences but risks OOM from activation/CUDA-graph spikes; lower it if you get OOM at load
+  or under burst.
+- **`--max-model-len`** — max context (prompt + output) per request; auto-derived from the model config
+  if unset. This is the single biggest OOM lever: KV cache scales with `max-model-len × concurrent
+  sequences`. A model whose weights fit will still OOM if you leave the full 128K context on and let
+  many long requests batch. **Cap `--max-model-len` to what you actually need.**
+
+The mental model: `KV_pool = gpu_memory_utilization × VRAM − weights`, and
+`concurrent_sequences ≈ KV_pool ÷ (bytes_per_token × context_len)`. Too-long context or too-high
+utilization eats the pool. `--max-num-seqs` and `--max-num-batched-tokens` cap the batch to trade
+latency vs throughput. Full KV math + an OOM playbook: [references/memory-and-throughput.md](references/memory-and-throughput.md).
+
+## Self-host health check (no credential needed)
+
+`/health` requires no API key, so it is the safe first probe on any endpoint you did not just start:
+
+```bash
+export VLLM_BASE_URL=http://localhost:8000
+curl -fsS "$VLLM_BASE_URL/health" && echo "  up"     # 200, empty body when ready
+curl -fsS "$VLLM_BASE_URL/v1/models" \
+  -H "Authorization: Bearer ${VLLM_API_KEY:-sk-local}"  # confirms WHICH model/adapters are served
+```
+
+`/health` says "the server is alive"; `/v1/models` says "and it is serving the model you expect" (plus
+any LoRA adapters). If `--api-key` is set, `/v1/models` needs the bearer header but `/health` never
+does. During load a slow first `/health` is normal — big models take a while to page in.
+
+## Honest alternatives — when to prefer another engine
+
+- **Text Generation Inference (TGI)** — Hugging Face's server; reach for it if you are all-in on the HF
+  ecosystem/tooling and want their supported stack.
+- **SGLang** — competitive throughput with strong RadixAttention prefix-cache reuse; prefer it for
+  heavy shared-prefix workloads (agents, long system prompts, structured generation).
+- **TensorRT-LLM** — squeezes maximum latency/throughput on NVIDIA GPUs via compiled engines, at the
+  cost of a heavier build/ahead-of-time compile step; prefer it when you must wring out every last ms
+  on NVIDIA hardware and can pay the ops complexity.
+- **Ollama / llama.cpp** — single-box / laptop / CPU-or-Metal, one or few users, GGUF quants,
+  zero-ceremony. Prefer it when there is no concurrency to exploit — that is the `ollama` skill.
+
+## Guardrails
+
+- **Cap `--max-model-len`.** Leaving the full context window on is the top OOM cause; KV cache scales
+  with context × concurrency ([engine args](https://docs.vllm.ai/en/latest/configuration/engine_args.html)).
+- **Tune `--gpu-memory-utilization` for OOM, not throughput first.** Lower it if OOM at load/burst;
+  raise it (carefully) for more concurrency. It is a fraction of *one* GPU's memory.
+- **A model that fits one GPU should stay on one GPU.** Tensor parallelism adds sync overhead — use it
+  to *fit*, not to speed up an already-fitting model.
+- **Quant is a VRAM win, not free speed**, and its quality cost is real — verify throughput at your
+  concurrency and quality on your task ([quantization](https://docs.vllm.ai/en/latest/features/quantization/)).
+- **Never expose an unauthenticated server.** No `--api-key` = open endpoint on `0.0.0.0:8000`.
+- **`--max-lora-rank` must be ≥ the adapter's trained rank**, and base + adapter must match, or the LoRA
+  fails to load.
+- **This is not a laptop tool.** No GPU / one user → `ollama`. The GPU box + autoscaling → `runpod` /
+  `modal`.
+
+## Related skills
+
+- **`open-weights`** — choose the model/size/license/quant to serve. vLLM runs it; it does not pick it.
+- **`finetuning`** (and single-GPU **`unsloth`**) — produce the LoRA adapter or merged weights that
+  `--enable-lora`/`vllm serve` then hosts. They train; vLLM serves.
+- **`runpod`** — rent/provision the GPU box vLLM runs on (bring-your-own-container GPU rental).
+- **`modal`** — serverless GPU containers + autoscaling around a vLLM process. The *box and scaling*,
+  vs vLLM the *engine*.
+- **`ollama`** — the single-user / laptop counterpart. No concurrency to exploit → use it, not vLLM.
+- Hosted inference you do not operate → `together-fireworks` / `huggingface` (you call an API; here you
+  run the server).
+
+## Checklist
+
+- [ ] Chose vLLM because there is **concurrency** to exploit (else `ollama` / a hosted API).
+- [ ] `vllm serve <model>` up; `/health` returns 200 and `/v1/models` shows the expected model.
+- [ ] `--api-key` (or `VLLM_API_KEY`) set if the endpoint is reachable beyond localhost.
+- [ ] Parallelism only if the model does not fit one GPU: TP = GPUs/node, PP = nodes.
+- [ ] `--max-model-len` capped to real need; `--gpu-memory-utilization` set with OOM headroom.
+- [ ] Quant chosen for VRAM fit and benchmarked at real concurrency (not assumed faster).
+- [ ] LoRA: `--max-lora-rank` ≥ trained rank; adapter reachable by name via the `model` field.
+- [ ] Client points at `base_url=<server>/v1`; `model` matches a `/v1/models` id.
+
+## References
+
+- [references/flags-and-endpoints.md](references/flags-and-endpoints.md) — the load-bearing `vllm serve`
+  flags, the full endpoint catalog with example requests, parallelism sizing, and the LoRA runtime API.
+- [references/memory-and-throughput.md](references/memory-and-throughput.md) — KV-cache math, the OOM
+  playbook, quantization tradeoffs at different batch sizes, and throughput-tuning knobs.

package/skills/vllm/evals/README.md

@@ -0,0 +1,9 @@
+# vllm evals
+
+- `should_trigger`: concurrent self-hosted serving jobs vLLM owns — `vllm serve` as an OpenAI-compatible
+  endpoint, tensor parallelism, serving a LoRA adapter, and KV-cache OOM debugging (incl. a Spanish phrasing).
+- `should_not_trigger`: routes the box/provisioning to `runpod`, autoscaling to `modal`, laptop/single-user
+  to `ollama`, model/quant selection to `open-weights`, hosted APIs to `together-fireworks`, and training to `finetuning`.
+- `capability`: asserts a full serve command surfaces the OpenAI-compat endpoints, tensor parallelism for a
+  node-sized model, LoRA flags with the rank constraint, the max-model-len/gpu-memory-utilization OOM guardrail,
+  auth + health check, and the boundary/version hedge.

package/skills/vllm/evals/cases.yaml

@@ -0,0 +1,47 @@
+skill: vllm
+
+should_trigger:
+  - prompt: "I need to serve Llama 3.3 70B to hundreds of concurrent users on our own GPUs — what should I run?"
+    why: Core niche — high-throughput concurrent self-hosted serving is exactly what vLLM (PagedAttention + continuous batching) is for.
+  - prompt: "Split a model across 4 A100s with vLLM tensor parallelism and expose an OpenAI-compatible endpoint."
+    why: --tensor-parallel-size plus the OpenAI-compatible server, both owned here.
+  - prompt: "My vllm serve keeps OOMing once requests get to long context — how do I fix it?"
+    why: Non-obvious — KV cache scales with max-model-len × concurrency; fix is --max-model-len / --gpu-memory-utilization, the load-bearing guardrail.
+  - prompt: "How do I serve my fine-tuned LoRA adapter on top of the base model without merging it?"
+    why: --enable-lora + --lora-modules, routing by the request model field — serving the finetuning output.
+  - prompt: "Point my existing OpenAI Python client at a self-hosted vLLM server instead of api.openai.com."
+    why: The /v1 OpenAI-compatible surface with a repointed base_url — a defining vLLM path.
+  - prompt: "Is serving an AWQ quant on vLLM actually faster, or just smaller in VRAM?"
+    why: Quantization tradeoff (memory vs throughput by batch regime), an explicit section here.
+  - prompt: "Servir un modelo abierto con alta concurrencia en mis propias GPUs con endpoint compatible con OpenAI."
+    why: Spanish; the concurrent self-hosted OpenAI-compatible serving framing that defines vLLM.
+
+should_not_trigger:
+  - prompt: "Rent an A100 box in the cloud so I have somewhere to run my inference server."
+    route_to: runpod
+    why: Provisioning/renting the GPU box, not the serving engine flags that run on it.
+  - prompt: "Wrap my model in a serverless container that autoscales GPUs with traffic."
+    route_to: modal
+    why: The box + autoscaling around the process; vLLM is the engine inside, not the scaler.
+  - prompt: "I just want to run a 7B locally on my laptop for myself, no concurrency."
+    route_to: ollama
+    why: Single-user / laptop inference — no concurrency to exploit, so vLLM's machinery is wasted.
+  - prompt: "Which open model and quant should I even pick for a 24GB GPU under a permissive license?"
+    route_to: open-weights
+    why: Model/size/license/quant selection is the knowledge layer feeding vLLM, not a serving-flag question.
+  - prompt: "Just call a hosted inference API for Llama so I don't run any servers."
+    route_to: together-fireworks
+    why: A hosted API the user does not operate; vLLM is for a server you run yourself.
+  - prompt: "Fine-tune Llama 3 on my dataset to produce a LoRA adapter."
+    route_to: finetuning
+    why: Producing the adapter is training; vLLM serves the result, it does not train it.
+
+capability:
+  - scenario: "We have one node with 4 A100 80GB GPUs and need to serve Llama-3.3-70B to many concurrent users behind an OpenAI-compatible endpoint, including one fine-tuned LoRA adapter, without OOMing at long context."
+    must_include:
+      - Starts the server with `vllm serve <model>` and notes it is OpenAI-compatible (/v1/chat/completions, /v1/models) so an OpenAI client works by repointing base_url.
+      - Uses `--tensor-parallel-size 4` because the 70B does not fit one GPU but fits the node, and notes a model that fits one GPU should not be split.
+      - Serves the adapter with `--enable-lora` + `--lora-modules name=path`, routing by the request `model` field, and requires `--max-lora-rank` >= the trained rank.
+      - Caps `--max-model-len` and sets `--gpu-memory-utilization` (default ~0.92) because KV cache scales with context × concurrency — the top OOM cause.
+      - Sets `--api-key` (or VLLM_API_KEY) so the endpoint is not left open, and checks `/health` (no auth) then `/v1/models` to confirm what is served.
+      - Names the boundary — renting/provisioning the box is runpod/modal, single-user is ollama — and hedges that vLLM ships ~weekly so versions/defaults should be verified.

package/skills/vllm/references/flags-and-endpoints.md

@@ -0,0 +1,105 @@
+# vLLM — flags, endpoints, and the LoRA runtime API
+
+Verify flag names/defaults at author time — vLLM ships ~weekly and engine args drift. Authoritative:
+[engine args](https://docs.vllm.ai/en/latest/configuration/engine_args.html),
+[online serving](https://docs.vllm.ai/en/latest/serving/openai_compatible_server/),
+[CLI reference](https://docs.vllm.ai/en/latest/cli/) (all accessed 2026-07).
+
+## The `vllm serve` flags you actually reach for
+
+| Flag | What it does | Note |
+| --- | --- | --- |
+| `<model>` (positional) | HF repo id **or** a local path | first download can be large; pre-stage weights on the box |
+| `--host` / `--port` | bind address / port | default `0.0.0.0:8000` |
+| `--api-key` | require `Authorization: Bearer <key>` | or env `VLLM_API_KEY`; **unset = open endpoint** |
+| `--served-model-name` | override the id clients pass in `model` | otherwise the repo id / path is the id |
+| `--tensor-parallel-size` | shard layers across GPUs on one node | must divide attention-head count |
+| `--pipeline-parallel-size` | split layer stages across nodes | needs a Ray cluster for multi-node |
+| `--quantization` | select quant method (`awq`, `gptq`, `fp8`, `compressed-tensors`, `bitsandbytes`, …) | often auto-detected from the checkpoint |
+| `--dtype` | compute dtype (`auto`, `bfloat16`, `float16`) | `auto` follows the model config |
+| `--max-model-len` | max context (prompt+output) per request | auto-derived from config if unset; **top OOM lever** |
+| `--gpu-memory-utilization` | fraction of each GPU vLLM may use | default ~`0.92`; the rest-of-budget becomes KV pool |
+| `--max-num-seqs` | max concurrent sequences in a batch | throughput vs latency cap |
+| `--max-num-batched-tokens` | token budget per engine step | works with chunked prefill (V1) |
+| `--enable-lora` | turn on LoRA serving | pair with `--lora-modules` |
+| `--lora-modules` | `name=path [name=path …]` adapters to preload | route by `name` in the request `model` |
+| `--max-loras` | adapters resident at once | |
+| `--max-lora-rank` | max adapter rank the server accepts | **must be ≥ the trained rank** |
+| `--task` | `generate` (default) / `embed` / `reward` / … | `embed` enables `/v1/embeddings` |
+| `--chat-template` | override the model's chat template | needed if the checkpoint ships none |
+| `--tensor-parallel-size` + `--enforce-eager` | disable CUDA-graph capture | lower memory / easier debug, slower |
+
+`vllm serve --help` prints the full, version-correct list. Do not treat this table as exhaustive.
+
+## Endpoint catalog
+
+OpenAI-compatible surface (default `http://localhost:8000`):
+
+- `POST /v1/chat/completions` — messages array; supports streaming (`"stream": true`), `tools`,
+  `response_format` (JSON / guided decoding), `logprobs`.
+- `POST /v1/completions` — single prompt string; legacy completion protocol.
+- `GET  /v1/models` — the served model id + any loaded LoRA adapter names.
+- `POST /v1/embeddings` — embedding vectors (server must be started with an embedding `--task`).
+
+vLLM-native / operational:
+
+- `GET  /health` — 200 + empty body when ready. **No auth.** First probe on any endpoint.
+- `GET  /metrics` — Prometheus text: queue depth, running/waiting seqs, tokens/s, KV-cache usage,
+  prefix-cache hit rate. Scrape this for capacity/throughput monitoring.
+- `GET  /version` — server version.
+- `POST /tokenize` · `POST /detokenize` — tokenizer round-trips.
+- `POST /v1/load_lora_adapter` · `POST /v1/unload_lora_adapter` — runtime LoRA (see below).
+
+### Example requests
+
+```bash
+# Chat, streaming, with an API key
+curl -N http://localhost:8000/v1/chat/completions \
+  -H "Authorization: Bearer $VLLM_API_KEY" -H 'Content-Type: application/json' \
+  -d '{"model":"Qwen/Qwen3-8B",
+       "messages":[{"role":"user","content":"Two facts about paging."}],
+       "stream":true, "max_tokens":200, "temperature":0.2}'
+
+# Guided JSON (structured output) via response_format
+curl http://localhost:8000/v1/chat/completions \
+  -H "Authorization: Bearer $VLLM_API_KEY" -H 'Content-Type: application/json' \
+  -d '{"model":"Qwen/Qwen3-8B",
+       "messages":[{"role":"user","content":"Extract name and age from: Ana is 30."}],
+       "response_format":{"type":"json_object"}}'
+```
+
+Field names track the OpenAI spec; vLLM adds `extra_body` params (e.g. `guided_json`, `guided_regex`,
+`top_k`) documented under [sampling params](https://docs.vllm.ai/en/latest/api/inference_params.html).
+
+## Parallelism sizing
+
+- **Fits one GPU** → no parallelism flag. Every split costs sync overhead.
+- **Too big for one GPU, fits a node** → `--tensor-parallel-size = GPUs on the node`. Wants fast
+  intra-node interconnect (NVLink ≫ PCIe). Head count must be divisible by TP size.
+- **Too big for a node** → add `--pipeline-parallel-size = number of nodes`; keep TP = GPUs/node.
+  Example: 2 nodes × 8 GPUs → `--tensor-parallel-size 8 --pipeline-parallel-size 2`. Multi-node needs
+  Ray started across the nodes first — that provisioning belongs to `runpod`/`modal`.
+- Rough VRAM: `weights_bytes ≈ params × bytes_per_param` (bf16 = 2, FP8/INT8 ≈ 1, 4-bit ≈ 0.5), split
+  across TP GPUs, **plus** the KV pool per GPU. Sizing which model fits which box = `open-weights`.
+
+## LoRA runtime API
+
+Preload at start with `--lora-modules name=path`, or manage live:
+
+```bash
+# start with runtime updates allowed
+VLLM_ALLOW_RUNTIME_LORA_UPDATING=True vllm serve <base-model> --enable-lora --max-lora-rank 32
+
+# load an adapter without restarting
+curl -X POST http://localhost:8000/v1/load_lora_adapter \
+  -H 'Content-Type: application/json' \
+  -d '{"lora_name":"sql","lora_path":"/adapters/sql-lora"}'
+
+# it now shows in /v1/models and is routable as "model":"sql"; unload to free it
+curl -X POST http://localhost:8000/v1/unload_lora_adapter \
+  -H 'Content-Type: application/json' -d '{"lora_name":"sql"}'
+```
+
+Constraints: the adapter must be for the loaded base model (same family/dims); `--max-lora-rank` must
+be ≥ the rank the adapter was trained at; `--max-loras` bounds how many are resident simultaneously.
+A `LoRAResolver` plugin can auto-resolve adapters from a local dir or the HF Hub — see the LoRA docs.

package/skills/vllm/references/memory-and-throughput.md

@@ -0,0 +1,78 @@
+# vLLM — memory, OOM, and throughput tuning
+
+Grounded in [engine args](https://docs.vllm.ai/en/latest/configuration/engine_args.html),
+[optimization/conserving memory](https://docs.vllm.ai/en/latest/configuration/optimization.html), and
+the [V1 guide](https://docs.vllm.ai/en/latest/usage/v1_guide.html) (accessed 2026-07). Defaults drift
+between weekly releases — verify.
+
+## Where the VRAM goes
+
+For each GPU:
+
+```
+usable          = gpu_memory_utilization × total_VRAM       # default utilization ≈ 0.92
+KV_cache_pool   = usable − weights − activation/CUDA-graph overhead
+max_concurrency ≈ KV_cache_pool ÷ (bytes_per_token × max_model_len)
+```
+
+- **Weights** are fixed once the model (and quant) is chosen; TP splits them across GPUs.
+- **KV cache** is the elastic part — PagedAttention blocks allocated per active token. It grows with
+  `max_model_len × concurrent sequences`. This is what runs you out of memory, not the weights.
+- **Overhead** — activations, CUDA graphs, the profiling reservation vLLM makes at startup. `--enforce-eager`
+  drops CUDA-graph memory (and some speed); it is a lever when you are just over the line.
+
+`bytes_per_token` depends on layers × KV heads × head_dim × 2 (K and V) × dtype bytes. You do not
+usually compute it — you observe `/metrics` (`vllm:gpu_cache_usage_perc`) and adjust.
+
+## OOM playbook (in order)
+
+1. **Cap `--max-model-len`** to the real max prompt+output. Going from 128K to, say, 8K multiplies how
+   many sequences fit. This is the highest-leverage fix.
+2. **Lower `--gpu-memory-utilization`** if OOM happens *at load* or on bursty spikes (e.g. 0.92 → 0.85).
+   Counterintuitively, a lower number can stop crashes by leaving headroom for activation spikes.
+3. **Lower `--max-num-seqs`** to bound concurrent sequences (and thus peak KV usage).
+4. **Quantize the weights** (AWQ/GPTQ/FP8/compressed-tensors) to free VRAM for the KV pool — a memory
+   win, not necessarily a speed win.
+5. **Add GPUs via `--tensor-parallel-size`** so weights + KV split across devices.
+6. **`--enforce-eager`** or **`--kv-cache-dtype fp8`** to trim graph/cache memory when marginally over.
+
+If none fit, the box is too small — that is a `runpod`/`modal` sizing decision, not a vLLM flag.
+
+## Quantization tradeoffs by regime
+
+| Method | Weights | Typical use | Watch out |
+| --- | --- | --- | --- |
+| FP8 (W8A8) | 8-bit | near-lossless, good on Ada/Hopper+ | needs recent NVIDIA/AMD arch; older GPUs unsupported |
+| INT8 / W8A8 | 8-bit | broad support, small quality hit | activation quant sensitive to calibration |
+| AWQ / GPTQ (4-bit) | ~4-bit | max VRAM savings | quality cost real; dequant can add latency |
+| compressed-tensors | varies | LLM Compressor output, mixed schemes | method-specific kernel/arch support |
+| bitsandbytes | 4/8-bit | convenient, load-time quant | generally slower than dedicated kernels |
+
+Regime that decides whether a quant *helps speed*:
+
+- **Low batch / low concurrency** — memory-bandwidth-bound; a quant that shrinks weights can raise
+  tokens/s. But dequant overhead can eat the win — measure.
+- **High batch / high concurrency** — often compute-bound; a *weight-only* quant does little for
+  throughput there (its win is the freed VRAM → bigger KV pool → more concurrency).
+
+So: quantize primarily to **fit** and to **grow the KV pool**, and benchmark before claiming it is
+faster. Choosing *which* quantized checkpoint to pull is `open-weights`.
+
+## Throughput knobs (once it fits)
+
+- **Prefix caching** (on by default in V1) reuses the KV of shared prompt prefixes — big win for repeated
+  system prompts / few-shot / agent loops. Watch `vllm:prefix_cache_hit_rate` in `/metrics`.
+- **Chunked prefill** (V1 default) interleaves prefill and decode so long prompts do not stall decoding.
+  `--max-num-batched-tokens` sets the per-step budget.
+- **`--max-num-seqs`** raises the concurrency ceiling if you have KV headroom; lowering it cuts latency.
+- **Speculative decoding** (`--speculative-config …`) can raise tokens/s for a request when a draft model
+  or n-gram method fits — verify current syntax in the docs; the flag shape changes.
+
+## Measuring, not guessing
+
+- Scrape `/metrics` for `gpu_cache_usage_perc`, `num_requests_running`/`_waiting`, `prompt`/`generation`
+  throughput, and prefix-cache hit rate.
+- Load-test with the bundled benchmark (`vllm bench serve` / `benchmarks/benchmark_serving.py` in the
+  repo — name has moved across versions; check `vllm bench --help`) at your real concurrency and prompt
+  mix. Tune `--max-model-len`, `--gpu-memory-utilization`, `--max-num-seqs` against those numbers, not a
+  single-request feel.