@damiangilgonzalez/harness 1.3.7

package/skills/viral-score/SKILL.md

@@ -0,0 +1,122 @@
+---
+name: viral-score
+description: "Use to judge whether a short-form clip is worth publishing — scores a reel, short or podcast excerpt 0-100 on ten weighted criteria with automatic penalties, then ranks a batch and says where the cut-off falls. Run it on transcript candidates before rendering. NOT writing the hook or caption (that is `shortform-packaging`), NOT inventing the idea (that is `shortform-ideation`), NOT executing the edit (that is `shortform-editing`)."
+tags: [viral, score, rubric, shortform, reels, shorts, tiktok, ranking, evaluation, retention]
+recommends: [shortform-packaging, shortform-ideation, shortform-editing, shortform-strategy, video-shorts]
+profiles: []
+origin: risco
+---
+
+# viral-score — is this clip worth publishing?
+
+A fixed rubric for scoring a vertical clip's chance of travelling. Its value is that it is
+**fixed**: the same clip scores the same on Monday and Friday, and two clips are comparable
+because they were measured with the same instrument.
+
+So **do not substitute your own judgment for the rubric and do not adjust the weights.** If a clip
+has a quality the rubric does not capture, say so in the reasons — but score it with this.
+
+Score candidates **before rendering** whenever you can. Judging a transcript excerpt costs seconds;
+editing a clip that gets discarded costs an afternoon.
+
+## The ten criteria
+
+Each is scored **0-10**.
+
+| # | Criterion | Weight | What earns the points |
+|---|---|---|---|
+| 1 | **Opening hook** | 20 % | Do the first 3 seconds stop the scroll? Curiosity, surprise, emotion, a strong claim, a striking open |
+| 2 | **Expected retention** | 20 % | Is there a reason to reach the end? Sustained curiosity, pace, resolution, no dead air |
+| 3 | **Emotional impact** | 15 % | Intensity: inspiration, awe, fear, anger, joy, nostalgia, humour, surprise. **Indifference scores very low** |
+| 4 | **Shareability** | 10 % | Would someone send this to a friend? |
+| 5 | **Comment-worthiness** | 10 % | Does it invite debate or a reply? |
+| 6 | **Message clarity** | 5 % | Does the idea fit in one sentence? |
+| 7 | **Value density** | 5 % | Does every second earn its place? |
+| 8 | **Originality** | 5 % | Is it different or unexpected? |
+| 9 | **Context independence** | 5 % | Does it land without having heard the rest of the episode? |
+| 10 | **Storytelling** | 5 % | Is there a story, a lesson, or a transformation? |
+
+## The calculation
+
+```text
+Viral Score = ( Hook×0.20 + Retention×0.20 + Emotion×0.15 + Shareability×0.10
+              + Comments×0.10 + Clarity×0.05 + Density×0.05
+              + Originality×0.05 + Context×0.05 + Storytelling×0.05 ) × 10
+```
+
+## Automatic penalties
+
+Subtracted **after** the formula, and they stack.
+
+| Penalty | Points |
+|---|---|
+| Weak hook | −10 |
+| Slow intro | −8 |
+| Needs too much context | −10 |
+| Long pauses | −5 |
+| Idea too generic | −8 |
+| Provokes no emotion | −10 |
+| Unsatisfying ending | −5 |
+
+Clamp the final result to **[0, 100]**.
+
+## Reading the score
+
+| Range | Verdict |
+|---|---|
+| 95-100 | Exceptional. Publish immediately |
+| 90-94 | Very likely to perform |
+| 80-89 | Good candidate. Worth editing and publishing |
+| 70-79 | Acceptable, but there are better clips |
+| 60-69 | Weak. Publish only if there is no alternative |
+| 0-59 | Discard |
+
+## Required output, per clip
+
+- **Start** and **end timestamp**
+- **Viral Score** (0-100)
+- A **table** with all ten criterion scores
+- **Reasons** behind the scoring
+- **Main strengths**
+- **Main weaknesses**
+- **Verdict**: Publish / Review / Discard
+
+Scoring several clips adds two things: a **ranking from highest to lowest**, and **where the
+cut-off falls** — the point below which publishing stops being worth it.
+
+## Applying it well
+
+**Score the clip, not the topic.** A fascinating episode routinely yields a clip that scores 45.
+
+**Judge the hook on the actual first three seconds**, not on the idea in the abstract. A clip that
+opens with a filler word or halfway through a sentence has a weak hook: −10. That is also why the
+excerpt should start on a sentence boundary — where you cut decides the hook.
+
+**Be strict about context independence.** If the viewer needs to know who the guest is, or what the
+previous question was, that is −10.
+
+**Do not inflate the mean.** If twenty clips all come out above 80, the rubric is not being applied.
+A healthy batch spreads widely, with few above 85.
+
+**Justify every low score with what happens in the clip**, at a timestamp — not with a generality.
+
+## Anti-patterns
+
+| Anti-pattern | Do instead |
+|---|---|
+| Scoring how interesting the subject is | Score this clip's execution; the topic is not the artifact |
+| Grading the hook from the full idea | Read only the first three seconds as the viewer receives them |
+| Compressing a batch into 78-85 | Use the range; a flat distribution means the instrument is not being used |
+| "Weak hook" with no evidence | Quote the opening words and the timestamp |
+| Adjusting weights for a clip you like | The weights are the instrument. Argue in the reasons, score with the rubric |
+| Rendering first, scoring after | Score transcript candidates, then edit only what survives |
+| Skipping the penalties because the total looks low | They are automatic and they stack; a 72 with three penalties is a 49 |
+
+## See Also
+
+- `../shortform-packaging/SKILL.md` — the hook line, caption, hashtags and cover for a clip that
+  passed this bar.
+- `../shortform-ideation/SKILL.md` — inventing the angle, upstream of anything scoreable.
+- `../shortform-editing/SKILL.md` — executing the edit once a clip is chosen.
+- `../shortform-strategy/SKILL.md` — what the channel should publish over time, which this feeds
+  with per-clip evidence.

package/skills/viral-score/evals/README.md

@@ -0,0 +1,16 @@
+# Evals — viral-score
+
+These are routing and capability checks, not an automated harness. Run them by hand:
+read each `should_trigger` prompt and confirm the skill fires on it — including the two
+non-obvious ones ("this clip feels flat but I can't say why" and the retrospective "why
+did this short underperform"), which carry no scoring keyword, and the Spanish prompt;
+read each `should_not_trigger` prompt and confirm the description's NOT-boundary sends it
+to the named `route_to` sibling instead. The neighbours are close together, so the
+packaging / ideation / editing / strategy split is the part most worth checking.
+
+For the `capability` cases, score the described clip for real and check the answer hits
+every `must_include` line. The first case exists because penalties are the step most often
+skipped: an opener mid-sentence, a 4s pause and an unresolved ending must each subtract,
+and the total must be clamped after they do. The second checks that batch mode produces
+what a single-clip run does not — a ranking and an explicit cut-off. No network, no
+fixtures.

package/skills/viral-score/evals/cases.yaml

@@ -0,0 +1,50 @@
+skill: viral-score
+
+should_trigger:
+  - prompt: "Score this reel — is it worth publishing?"
+    why: Core ask — a single clip judged against the rubric, verdict required.
+  - prompt: "I cut 20 clips from yesterday's episode. Rank them and tell me which ones to post."
+    why: Batch mode — ranking plus the cut-off line, which only this skill produces.
+  - prompt: "Order these shorts from best to worst."
+    why: Comparison with no scoring keyword; ordering clips requires a shared instrument.
+  - prompt: "Here's the transcript of a 2h podcast. Which passages would make good shorts?"
+    why: The pre-render case the skill exists for — score candidates before anything is edited.
+  - prompt: "Valora este reel y dime si lo publico o lo descarto."
+    why: Spanish — same core ask, publish/discard verdict.
+  - prompt: "This clip feels flat to me but I can't say why. Is it me?"
+    why: Non-obvious — a vague quality doubt resolves into criterion scores and named weaknesses.
+  - prompt: "Why did this short underperform? It seemed strong when I made it."
+    why: Non-obvious — retrospective diagnosis against the same criteria, especially hook and context independence.
+
+should_not_trigger:
+  - prompt: "Write the caption and hashtags for this Reel."
+    route_to: shortform-packaging
+    why: Upload-form copy for a clip already chosen, not the decision to choose it.
+  - prompt: "Give me ten ideas for shorts about discipline."
+    route_to: shortform-ideation
+    why: Inventing angles, upstream of anything that can be scored.
+  - prompt: "Burn in the subtitles and add B-roll to this clip."
+    route_to: shortform-editing
+    why: Executing the edit; scoring is what decides whether the edit is worth doing.
+  - prompt: "How often should I post on this channel, and at what times?"
+    route_to: shortform-strategy
+    why: Channel cadence over time, not the merit of one artifact.
+  - prompt: "Design a thumbnail for my YouTube video."
+    route_to: youtube-thumbnails
+    why: A different surface and a different artifact; this rubric scores the clip itself.
+
+capability:
+  - prompt: "Score this 18s clip: it opens mid-sentence with 'o sea, entonces lo que pasa es que…', has a 4s pause in the middle, and ends without resolving the point."
+    must_include:
+      - "a score for all ten criteria"
+      - "the weak-hook penalty applied for opening mid-sentence"
+      - "the long-pause penalty"
+      - "the unsatisfying-ending penalty"
+      - "a final score clamped to 0-100 after penalties"
+      - "a Publish / Review / Discard verdict"
+  - prompt: "Here are 6 clips from one episode with their transcripts. Rank them."
+    must_include:
+      - "a Viral Score for every clip"
+      - "a ranking from highest to lowest"
+      - "an explicit cut-off point below which publishing is not worth it"
+      - "strengths and weaknesses per clip, tied to what happens in it"

package/skills/vllm/SKILL.md

@@ -0,0 +1,235 @@
+---
+name: vllm
+description: "Use when self-hosting an open-weight LLM for high-throughput concurrent serving with vLLM — running an OpenAI-compatible endpoint, splitting a model across GPUs with tensor or pipeline parallelism, loading quantized weights, serving one or many LoRA adapters, and debugging KV-cache OOM from memory-utilisation and context-length flags. NOT renting or provisioning the GPU box (that is `runpod` or `modal`), NOT single-user laptop inference (that is `ollama`), NOT a hosted inference API you do not operate (that is `together-fireworks` or `huggingface`)."
+tags: [vllm, llm-serving, inference-server, tensor-parallel, quantization]
+recommends: [open-weights, finetuning, runpod, modal, ollama]
+origin: risco
+---
+
+# vLLM — high-throughput serving of open-weight models
+
+vLLM is the inference **engine** you put in front of an open-weight model when many requests hit it at
+once. Its job — and this skill's — is throughput under concurrency: keep the GPU busy across dozens of
+simultaneous requests, not squeeze one prompt out fast. You own the `vllm serve` flags; the box those
+flags run on is a `runpod`/`modal` concern.
+
+**Why not just loop a `transformers` `generate()`?** Naive serving runs one request at a time and pads
+every batch to the longest sequence, so the GPU idles. vLLM fixes both:
+
+- **PagedAttention** stores the KV cache in non-contiguous fixed-size blocks (like OS virtual-memory
+  paging), so there is almost no padding/reservation waste and long contexts pack tightly.
+- **Continuous batching** admits and retires requests token-by-token instead of per-batch, so a new
+  request joins the running batch immediately rather than waiting for the slowest one to finish.
+
+Net effect: an order-of-magnitude more concurrent throughput than single-request serving. If you only
+ever have one user on a laptop, that machinery is wasted — that is `ollama`, not this.
+
+## Version & setup reality (verify at author time — vLLM ships ~weekly)
+
+- **Latest stable is ~0.25.x (July 2026)** — releases land roughly weekly. Check
+  [PyPI](https://pypi.org/project/vllm/) / [releases](https://github.com/vllm-project/vllm/releases);
+  do not pin to a number you read here.
+- **The V1 engine is the default since v0.8.0** and recent releases have removed the legacy V0 path,
+  so treat V1 as the only engine. It runs the scheduler + core loop in a separate process and turns on
+  chunked prefill and prefix caching out of the box, so you rarely tune the scheduler by hand
+  ([V1 guide](https://docs.vllm.ai/en/latest/usage/v1_guide.html), accessed 2026-07). `VLLM_USE_V1`
+  historically toggled it — if you see it referenced, it is legacy.
+- **Install:** `pip install vllm` (default build is CUDA/NVIDIA). ROCm, CPU, TPU and other backends
+  have separate install paths — see the docs' installation matrix. Needs Python + a supported GPU.
+- **Auth is OFF by default.** A bare `vllm serve` is an *open* endpoint on `0.0.0.0:8000`. To require a
+  bearer token, pass `--api-key <KEY>` (or set `VLLM_API_KEY`); clients then send
+  `Authorization: Bearer <KEY>`. Do not expose an unauthenticated server publicly.
+
+## `vllm serve` → an OpenAI-compatible server
+
+```bash
+vllm serve Qwen/Qwen3-8B                 # download from HF (or a local path) and serve on :8000
+vllm serve /models/qwen3-8b --api-key sk-local-xyz --port 8000 --host 0.0.0.0
+```
+
+The server speaks the OpenAI wire protocol, so **any OpenAI client works unchanged** — just repoint
+`base_url` and use a dummy (or your `--api-key`) key. Endpoints
+([online serving docs](https://docs.vllm.ai/en/latest/serving/openai_compatible_server/), accessed 2026-07):
+
+| Endpoint | Purpose |
+| --- | --- |
+| `POST /v1/chat/completions` | chat protocol (messages array) — the usual path |
+| `POST /v1/completions` | raw text completion (single prompt string) |
+| `GET  /v1/models` | list the served model + any loaded LoRA adapters |
+| `POST /v1/embeddings` | only for an embedding model (`--task embed`) |
+| `GET  /health` | liveness — 200 when ready, **no auth, no body** |
+| `GET  /metrics` | Prometheus metrics (queue depth, throughput, cache usage) |
+
+```python
+from openai import OpenAI
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="sk-local-xyz")  # key = your --api-key
+r = client.chat.completions.create(
+    model="Qwen/Qwen3-8B",               # the id `/v1/models` reports (or a LoRA adapter name)
+    messages=[{"role": "user", "content": "Name three primes."}],
+)
+print(r.choices[0].message.content)
+```
+
+The `model` field must match what `/v1/models` returns — the served model id or a LoRA adapter name
+(below), not an arbitrary string.
+
+## Parallelism: fit the model, then scale out
+
+Two orthogonal knobs. Reach for parallelism only when the model does **not** fit one GPU — a model that
+fits should stay on a single GPU (no split), because every split adds communication overhead.
+
+- **`--tensor-parallel-size N`** — shard each layer's weights *across N GPUs on one node*. Use this
+  first: it needs fast intra-node links (NVLink / PCIe) because GPUs sync every layer. `N` must divide
+  the model's attention-head count. This is how you serve a model too big for one GPU but fitting the
+  node.
+- **`--pipeline-parallel-size M`** — split the model *by layer stages across M nodes*. Tolerates slower
+  inter-node network. Use it when the model does not fit even a full node.
+
+Rule of thumb from the docs
+([parallelism & scaling](https://docs.vllm.ai/en/latest/serving/distributed_serving.html), accessed 2026-07):
+set **tensor-parallel = GPUs per node**, **pipeline-parallel = number of nodes**.
+
+```bash
+vllm serve meta-llama/Llama-3.3-70B-Instruct --tensor-parallel-size 4          # 1 node, 4 GPUs
+vllm serve <huge-model> --tensor-parallel-size 8 --pipeline-parallel-size 2    # 2 nodes × 8 GPUs
+```
+
+Multi-node needs a Ray cluster wired up first — that orchestration is a `runpod`/`modal` concern, not a
+vLLM flag.
+
+## Quantization: smaller weights ≠ automatic speedup
+
+Quantization shrinks the weights so a model fits fewer/smaller GPUs and frees VRAM for KV cache. Select
+a method with `--quantization` (often auto-detected from the checkpoint's config). vLLM supports
+**AWQ, GPTQ/GPTQModel, FP8 (W8A8), compressed-tensors (LLM Compressor), INT4/INT8, bitsandbytes** and
+more ([quantization docs](https://docs.vllm.ai/en/latest/features/quantization/), accessed 2026-07).
+
+```bash
+vllm serve TheModel/Qwen3-8B-AWQ --quantization awq
+vllm serve neuralmagic/Model-FP8   --quantization compressed-tensors   # FP8 needs Ada/Hopper+
+```
+
+The honest tradeoffs:
+
+- **Support is GPU-arch-specific** — FP8 W8A8 wants Ada/Hopper-class (or AMD) hardware; a method that
+  is fast on one GPU may be unsupported or emulated on another. Check the compatibility table.
+- **Quant is a memory win, not a guaranteed throughput win.** At **low batch / low concurrency** the
+  workload is memory-bandwidth-bound and a quant can help; but dequant overhead can *cost* latency, and
+  at high batch you may be compute-bound where a weight-only quant does little. Benchmark your quant on
+  your hardware at your real concurrency before assuming it is faster.
+- **Quality drops** — usually small for 8-bit/well-calibrated 4-bit, larger for aggressive 4-bit. Serve
+  a quant to save VRAM, not as a free lunch. Picking *which* quantized checkpoint to pull is an
+  `open-weights` decision.
+
+## LoRA adapters: serve the fine-tuning output
+
+vLLM serves LoRA adapters on top of one loaded base model, so your `finetuning`/`unsloth` output goes
+live without merging or a second server
+([LoRA docs](https://docs.vllm.ai/en/latest/features/lora.html), accessed 2026-07).
+
+```bash
+vllm serve meta-llama/Llama-3.2-3B-Instruct \
+  --enable-lora \
+  --lora-modules sql=/adapters/sql-lora legal=/adapters/legal-lora \
+  --max-loras 2 \        # how many adapters resident at once
+  --max-lora-rank 16     # must be >= the rank the adapter was trained at
+```
+
+Route to an adapter by naming it in the request `model` field: `"model": "sql"` hits the SQL adapter,
+`"model": "meta-llama/Llama-3.2-3B-Instruct"` hits the untuned base — same server, no reload.
+
+Load/unload at runtime with `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`, then
+`POST /v1/load_lora_adapter` / `POST /v1/unload_lora_adapter`. Base + adapter must match (same family
+and dims), and `--max-lora-rank` must be ≥ the trained rank or load fails.
+
+## Memory & throughput: where OOM comes from
+
+The two knobs that cause (and cure) most OOM
+([engine args](https://docs.vllm.ai/en/latest/configuration/engine_args.html), accessed 2026-07):
+
+- **`--gpu-memory-utilization`** (default ~**0.92**) — fraction of each GPU vLLM may claim. Weights are
+  loaded, then the **rest of this budget becomes the KV-cache pool**. Raising it toward 1.0 buys more
+  concurrent sequences but risks OOM from activation/CUDA-graph spikes; lower it if you get OOM at load
+  or under burst.
+- **`--max-model-len`** — max context (prompt + output) per request; auto-derived from the model config
+  if unset. This is the single biggest OOM lever: KV cache scales with `max-model-len × concurrent
+  sequences`. A model whose weights fit will still OOM if you leave the full 128K context on and let
+  many long requests batch. **Cap `--max-model-len` to what you actually need.**
+
+The mental model: `KV_pool = gpu_memory_utilization × VRAM − weights`, and
+`concurrent_sequences ≈ KV_pool ÷ (bytes_per_token × context_len)`. Too-long context or too-high
+utilization eats the pool. `--max-num-seqs` and `--max-num-batched-tokens` cap the batch to trade
+latency vs throughput. Full KV math + an OOM playbook: [references/memory-and-throughput.md](references/memory-and-throughput.md).
+
+## Self-host health check (no credential needed)
+
+`/health` requires no API key, so it is the safe first probe on any endpoint you did not just start:
+
+```bash
+export VLLM_BASE_URL=http://localhost:8000
+curl -fsS "$VLLM_BASE_URL/health" && echo "  up"     # 200, empty body when ready
+curl -fsS "$VLLM_BASE_URL/v1/models" \
+  -H "Authorization: Bearer ${VLLM_API_KEY:-sk-local}"  # confirms WHICH model/adapters are served
+```
+
+`/health` says "the server is alive"; `/v1/models` says "and it is serving the model you expect" (plus
+any LoRA adapters). If `--api-key` is set, `/v1/models` needs the bearer header but `/health` never
+does. During load a slow first `/health` is normal — big models take a while to page in.
+
+## Honest alternatives — when to prefer another engine
+
+- **Text Generation Inference (TGI)** — Hugging Face's server; reach for it if you are all-in on the HF
+  ecosystem/tooling and want their supported stack.
+- **SGLang** — competitive throughput with strong RadixAttention prefix-cache reuse; prefer it for
+  heavy shared-prefix workloads (agents, long system prompts, structured generation).
+- **TensorRT-LLM** — squeezes maximum latency/throughput on NVIDIA GPUs via compiled engines, at the
+  cost of a heavier build/ahead-of-time compile step; prefer it when you must wring out every last ms
+  on NVIDIA hardware and can pay the ops complexity.
+- **Ollama / llama.cpp** — single-box / laptop / CPU-or-Metal, one or few users, GGUF quants,
+  zero-ceremony. Prefer it when there is no concurrency to exploit — that is the `ollama` skill.
+
+## Guardrails
+
+- **Cap `--max-model-len`.** Leaving the full context window on is the top OOM cause; KV cache scales
+  with context × concurrency ([engine args](https://docs.vllm.ai/en/latest/configuration/engine_args.html)).
+- **Tune `--gpu-memory-utilization` for OOM, not throughput first.** Lower it if OOM at load/burst;
+  raise it (carefully) for more concurrency. It is a fraction of *one* GPU's memory.
+- **A model that fits one GPU should stay on one GPU.** Tensor parallelism adds sync overhead — use it
+  to *fit*, not to speed up an already-fitting model.
+- **Quant is a VRAM win, not free speed**, and its quality cost is real — verify throughput at your
+  concurrency and quality on your task ([quantization](https://docs.vllm.ai/en/latest/features/quantization/)).
+- **Never expose an unauthenticated server.** No `--api-key` = open endpoint on `0.0.0.0:8000`.
+- **`--max-lora-rank` must be ≥ the adapter's trained rank**, and base + adapter must match, or the LoRA
+  fails to load.
+- **This is not a laptop tool.** No GPU / one user → `ollama`. The GPU box + autoscaling → `runpod` /
+  `modal`.
+
+## Related skills
+
+- **`open-weights`** — choose the model/size/license/quant to serve. vLLM runs it; it does not pick it.
+- **`finetuning`** (and single-GPU **`unsloth`**) — produce the LoRA adapter or merged weights that
+  `--enable-lora`/`vllm serve` then hosts. They train; vLLM serves.
+- **`runpod`** — rent/provision the GPU box vLLM runs on (bring-your-own-container GPU rental).
+- **`modal`** — serverless GPU containers + autoscaling around a vLLM process. The *box and scaling*,
+  vs vLLM the *engine*.
+- **`ollama`** — the single-user / laptop counterpart. No concurrency to exploit → use it, not vLLM.
+- Hosted inference you do not operate → `together-fireworks` / `huggingface` (you call an API; here you
+  run the server).
+
+## Checklist
+
+- [ ] Chose vLLM because there is **concurrency** to exploit (else `ollama` / a hosted API).
+- [ ] `vllm serve <model>` up; `/health` returns 200 and `/v1/models` shows the expected model.
+- [ ] `--api-key` (or `VLLM_API_KEY`) set if the endpoint is reachable beyond localhost.
+- [ ] Parallelism only if the model does not fit one GPU: TP = GPUs/node, PP = nodes.
+- [ ] `--max-model-len` capped to real need; `--gpu-memory-utilization` set with OOM headroom.
+- [ ] Quant chosen for VRAM fit and benchmarked at real concurrency (not assumed faster).
+- [ ] LoRA: `--max-lora-rank` ≥ trained rank; adapter reachable by name via the `model` field.
+- [ ] Client points at `base_url=<server>/v1`; `model` matches a `/v1/models` id.
+
+## References
+
+- [references/flags-and-endpoints.md](references/flags-and-endpoints.md) — the load-bearing `vllm serve`
+  flags, the full endpoint catalog with example requests, parallelism sizing, and the LoRA runtime API.
+- [references/memory-and-throughput.md](references/memory-and-throughput.md) — KV-cache math, the OOM
+  playbook, quantization tradeoffs at different batch sizes, and throughput-tuning knobs.

package/skills/vllm/evals/README.md

@@ -0,0 +1,9 @@
+# vllm evals
+
+- `should_trigger`: concurrent self-hosted serving jobs vLLM owns — `vllm serve` as an OpenAI-compatible
+  endpoint, tensor parallelism, serving a LoRA adapter, and KV-cache OOM debugging (incl. a Spanish phrasing).
+- `should_not_trigger`: routes the box/provisioning to `runpod`, autoscaling to `modal`, laptop/single-user
+  to `ollama`, model/quant selection to `open-weights`, hosted APIs to `together-fireworks`, and training to `finetuning`.
+- `capability`: asserts a full serve command surfaces the OpenAI-compat endpoints, tensor parallelism for a
+  node-sized model, LoRA flags with the rank constraint, the max-model-len/gpu-memory-utilization OOM guardrail,
+  auth + health check, and the boundary/version hedge.

package/skills/vllm/evals/cases.yaml

@@ -0,0 +1,47 @@
+skill: vllm
+
+should_trigger:
+  - prompt: "I need to serve Llama 3.3 70B to hundreds of concurrent users on our own GPUs — what should I run?"
+    why: Core niche — high-throughput concurrent self-hosted serving is exactly what vLLM (PagedAttention + continuous batching) is for.
+  - prompt: "Split a model across 4 A100s with vLLM tensor parallelism and expose an OpenAI-compatible endpoint."
+    why: --tensor-parallel-size plus the OpenAI-compatible server, both owned here.
+  - prompt: "My vllm serve keeps OOMing once requests get to long context — how do I fix it?"
+    why: Non-obvious — KV cache scales with max-model-len × concurrency; fix is --max-model-len / --gpu-memory-utilization, the load-bearing guardrail.
+  - prompt: "How do I serve my fine-tuned LoRA adapter on top of the base model without merging it?"
+    why: --enable-lora + --lora-modules, routing by the request model field — serving the finetuning output.
+  - prompt: "Point my existing OpenAI Python client at a self-hosted vLLM server instead of api.openai.com."
+    why: The /v1 OpenAI-compatible surface with a repointed base_url — a defining vLLM path.
+  - prompt: "Is serving an AWQ quant on vLLM actually faster, or just smaller in VRAM?"
+    why: Quantization tradeoff (memory vs throughput by batch regime), an explicit section here.
+  - prompt: "Servir un modelo abierto con alta concurrencia en mis propias GPUs con endpoint compatible con OpenAI."
+    why: Spanish; the concurrent self-hosted OpenAI-compatible serving framing that defines vLLM.
+
+should_not_trigger:
+  - prompt: "Rent an A100 box in the cloud so I have somewhere to run my inference server."
+    route_to: runpod
+    why: Provisioning/renting the GPU box, not the serving engine flags that run on it.
+  - prompt: "Wrap my model in a serverless container that autoscales GPUs with traffic."
+    route_to: modal
+    why: The box + autoscaling around the process; vLLM is the engine inside, not the scaler.
+  - prompt: "I just want to run a 7B locally on my laptop for myself, no concurrency."
+    route_to: ollama
+    why: Single-user / laptop inference — no concurrency to exploit, so vLLM's machinery is wasted.
+  - prompt: "Which open model and quant should I even pick for a 24GB GPU under a permissive license?"
+    route_to: open-weights
+    why: Model/size/license/quant selection is the knowledge layer feeding vLLM, not a serving-flag question.
+  - prompt: "Just call a hosted inference API for Llama so I don't run any servers."
+    route_to: together-fireworks
+    why: A hosted API the user does not operate; vLLM is for a server you run yourself.
+  - prompt: "Fine-tune Llama 3 on my dataset to produce a LoRA adapter."
+    route_to: finetuning
+    why: Producing the adapter is training; vLLM serves the result, it does not train it.
+
+capability:
+  - scenario: "We have one node with 4 A100 80GB GPUs and need to serve Llama-3.3-70B to many concurrent users behind an OpenAI-compatible endpoint, including one fine-tuned LoRA adapter, without OOMing at long context."
+    must_include:
+      - Starts the server with `vllm serve <model>` and notes it is OpenAI-compatible (/v1/chat/completions, /v1/models) so an OpenAI client works by repointing base_url.
+      - Uses `--tensor-parallel-size 4` because the 70B does not fit one GPU but fits the node, and notes a model that fits one GPU should not be split.
+      - Serves the adapter with `--enable-lora` + `--lora-modules name=path`, routing by the request `model` field, and requires `--max-lora-rank` >= the trained rank.
+      - Caps `--max-model-len` and sets `--gpu-memory-utilization` (default ~0.92) because KV cache scales with context × concurrency — the top OOM cause.
+      - Sets `--api-key` (or VLLM_API_KEY) so the endpoint is not left open, and checks `/health` (no auth) then `/v1/models` to confirm what is served.
+      - Names the boundary — renting/provisioning the box is runpod/modal, single-user is ollama — and hedges that vLLM ships ~weekly so versions/defaults should be verified.

package/skills/vllm/references/flags-and-endpoints.md

@@ -0,0 +1,105 @@
+# vLLM — flags, endpoints, and the LoRA runtime API
+
+Verify flag names/defaults at author time — vLLM ships ~weekly and engine args drift. Authoritative:
+[engine args](https://docs.vllm.ai/en/latest/configuration/engine_args.html),
+[online serving](https://docs.vllm.ai/en/latest/serving/openai_compatible_server/),
+[CLI reference](https://docs.vllm.ai/en/latest/cli/) (all accessed 2026-07).
+
+## The `vllm serve` flags you actually reach for
+
+| Flag | What it does | Note |
+| --- | --- | --- |
+| `<model>` (positional) | HF repo id **or** a local path | first download can be large; pre-stage weights on the box |
+| `--host` / `--port` | bind address / port | default `0.0.0.0:8000` |
+| `--api-key` | require `Authorization: Bearer <key>` | or env `VLLM_API_KEY`; **unset = open endpoint** |
+| `--served-model-name` | override the id clients pass in `model` | otherwise the repo id / path is the id |
+| `--tensor-parallel-size` | shard layers across GPUs on one node | must divide attention-head count |
+| `--pipeline-parallel-size` | split layer stages across nodes | needs a Ray cluster for multi-node |
+| `--quantization` | select quant method (`awq`, `gptq`, `fp8`, `compressed-tensors`, `bitsandbytes`, …) | often auto-detected from the checkpoint |
+| `--dtype` | compute dtype (`auto`, `bfloat16`, `float16`) | `auto` follows the model config |
+| `--max-model-len` | max context (prompt+output) per request | auto-derived from config if unset; **top OOM lever** |
+| `--gpu-memory-utilization` | fraction of each GPU vLLM may use | default ~`0.92`; the rest-of-budget becomes KV pool |
+| `--max-num-seqs` | max concurrent sequences in a batch | throughput vs latency cap |
+| `--max-num-batched-tokens` | token budget per engine step | works with chunked prefill (V1) |
+| `--enable-lora` | turn on LoRA serving | pair with `--lora-modules` |
+| `--lora-modules` | `name=path [name=path …]` adapters to preload | route by `name` in the request `model` |
+| `--max-loras` | adapters resident at once | |
+| `--max-lora-rank` | max adapter rank the server accepts | **must be ≥ the trained rank** |
+| `--task` | `generate` (default) / `embed` / `reward` / … | `embed` enables `/v1/embeddings` |
+| `--chat-template` | override the model's chat template | needed if the checkpoint ships none |
+| `--tensor-parallel-size` + `--enforce-eager` | disable CUDA-graph capture | lower memory / easier debug, slower |
+
+`vllm serve --help` prints the full, version-correct list. Do not treat this table as exhaustive.
+
+## Endpoint catalog
+
+OpenAI-compatible surface (default `http://localhost:8000`):
+
+- `POST /v1/chat/completions` — messages array; supports streaming (`"stream": true`), `tools`,
+  `response_format` (JSON / guided decoding), `logprobs`.
+- `POST /v1/completions` — single prompt string; legacy completion protocol.
+- `GET  /v1/models` — the served model id + any loaded LoRA adapter names.
+- `POST /v1/embeddings` — embedding vectors (server must be started with an embedding `--task`).
+
+vLLM-native / operational:
+
+- `GET  /health` — 200 + empty body when ready. **No auth.** First probe on any endpoint.
+- `GET  /metrics` — Prometheus text: queue depth, running/waiting seqs, tokens/s, KV-cache usage,
+  prefix-cache hit rate. Scrape this for capacity/throughput monitoring.
+- `GET  /version` — server version.
+- `POST /tokenize` · `POST /detokenize` — tokenizer round-trips.
+- `POST /v1/load_lora_adapter` · `POST /v1/unload_lora_adapter` — runtime LoRA (see below).
+
+### Example requests
+
+```bash
+# Chat, streaming, with an API key
+curl -N http://localhost:8000/v1/chat/completions \
+  -H "Authorization: Bearer $VLLM_API_KEY" -H 'Content-Type: application/json' \
+  -d '{"model":"Qwen/Qwen3-8B",
+       "messages":[{"role":"user","content":"Two facts about paging."}],
+       "stream":true, "max_tokens":200, "temperature":0.2}'
+
+# Guided JSON (structured output) via response_format
+curl http://localhost:8000/v1/chat/completions \
+  -H "Authorization: Bearer $VLLM_API_KEY" -H 'Content-Type: application/json' \
+  -d '{"model":"Qwen/Qwen3-8B",
+       "messages":[{"role":"user","content":"Extract name and age from: Ana is 30."}],
+       "response_format":{"type":"json_object"}}'
+```
+
+Field names track the OpenAI spec; vLLM adds `extra_body` params (e.g. `guided_json`, `guided_regex`,
+`top_k`) documented under [sampling params](https://docs.vllm.ai/en/latest/api/inference_params.html).
+
+## Parallelism sizing
+
+- **Fits one GPU** → no parallelism flag. Every split costs sync overhead.
+- **Too big for one GPU, fits a node** → `--tensor-parallel-size = GPUs on the node`. Wants fast
+  intra-node interconnect (NVLink ≫ PCIe). Head count must be divisible by TP size.
+- **Too big for a node** → add `--pipeline-parallel-size = number of nodes`; keep TP = GPUs/node.
+  Example: 2 nodes × 8 GPUs → `--tensor-parallel-size 8 --pipeline-parallel-size 2`. Multi-node needs
+  Ray started across the nodes first — that provisioning belongs to `runpod`/`modal`.
+- Rough VRAM: `weights_bytes ≈ params × bytes_per_param` (bf16 = 2, FP8/INT8 ≈ 1, 4-bit ≈ 0.5), split
+  across TP GPUs, **plus** the KV pool per GPU. Sizing which model fits which box = `open-weights`.
+
+## LoRA runtime API
+
+Preload at start with `--lora-modules name=path`, or manage live:
+
+```bash
+# start with runtime updates allowed
+VLLM_ALLOW_RUNTIME_LORA_UPDATING=True vllm serve <base-model> --enable-lora --max-lora-rank 32
+
+# load an adapter without restarting
+curl -X POST http://localhost:8000/v1/load_lora_adapter \
+  -H 'Content-Type: application/json' \
+  -d '{"lora_name":"sql","lora_path":"/adapters/sql-lora"}'
+
+# it now shows in /v1/models and is routable as "model":"sql"; unload to free it
+curl -X POST http://localhost:8000/v1/unload_lora_adapter \
+  -H 'Content-Type: application/json' -d '{"lora_name":"sql"}'
+```
+
+Constraints: the adapter must be for the loaded base model (same family/dims); `--max-lora-rank` must
+be ≥ the rank the adapter was trained at; `--max-loras` bounds how many are resident simultaneously.
+A `LoRAResolver` plugin can auto-resolve adapters from a local dir or the HF Hub — see the LoRA docs.