@dsalazardev/ai-hub-cli 0.0.0-stage → 0.2.0

package/skills/shared/verify/evals/README.md

@@ -0,0 +1,78 @@
+# Eval harness — `verify` skill
+
+`verify` is the ai-hub Spec-Kit post-implementation **gate**. These evals check two
+things: that it **triggers** at the right moment (someone is about to claim
+"done"/"green"/"ready to merge") and stays quiet on near-misses, and that it
+**measurably changes the behavior** — running the real stack gate, walking the
+done-checks and acceptance criteria, and refusing to claim done without
+evidence. Cases live in `cases.yaml`. Grading is done by an **agent harness** (a
+Claude Code agent with skills loaded) plus a human spot-check, because both
+routing and "did it actually demand evidence" are judgment calls — there is no
+pure shell runner.
+
+## What's in `cases.yaml`
+
+- `should_trigger` — prompts that MUST load `verify` (8 cases, incl. non-obvious
+  phrasings like "are we ready for review yet?" and "prove the fix works").
+- `should_not_trigger` — near-misses that must route to a real sibling
+  (`route_to`): writing tests (`fastapi`), debugging a flaky test (`go`),
+  building the gate script (`nextjs`), security audit (`secure-coding`),
+  migration/constraint checks (`postgresdb`). The recurring trap: anything that
+  WRITES or RUNS code, or OWNS the tooling, is a stack skill — `verify` only
+  orchestrates and judges.
+- `capability` — one end-to-end scenario with a `must_include` rubric to grade
+  WITH vs WITHOUT the skill.
+
+## Triggering eval
+
+Goal: the skill fires when a "done" claim is imminent and never on a near-miss.
+
+1. Configure an agent with the **full catalog of skill descriptions** available
+   for routing (verify + the ai-hub Spec-Kit phase skills it chains to — implement,
+   debug, review, clarify, ship — plus the stack/process skills: fastapi, go,
+   nextjs, flutter, postgresdb, secure-coding, deployment, harness, init) so
+   routing competes realistically.
+2. For each `should_trigger` prompt: feed it cold, record whether `verify` is
+   the skill the agent loads. Run **3–5 trials** per prompt (fresh context).
+3. For each `should_not_trigger` prompt: confirm `verify` does NOT load and that
+   the chosen skill matches `route_to`. Same 3–5 trials.
+4. Score: `triggered_correctly / total_trials` across both lists.
+
+**Pass bar: ≥ 90% trigger accuracy** over all prompts and trials, with **zero
+systematic false-positives** on the "write the tests" / "debug the failing
+test" / "build the gate script" near-misses — those are the known traps. A
+prompt about *producing* tests or *fixing* a failure must not pull in `verify`.
+
+## Capability eval
+
+Goal: prove the skill changes the answer, not just the routing.
+
+1. For the `capability` scenario, run it **twice**:
+   - **WITHOUT** the skill (base agent, no `verify` loaded).
+   - **WITH** the `verify` skill loaded.
+2. Grade each output against the scenario's `must_include` checklist — one point
+   per checkable item covered. A human or grading agent marks each present /
+   absent.
+3. Compute coverage = `items_covered / total_items` for each run.
+
+**Pass bar: WITH the skill covers ≥ 80% of `must_include`; WITHOUT clearly
+lower** (target a ≥ 30-point gap). The discriminating behaviors are the ones a
+base agent skips: actually *running* the stack `verify.sh` (not trusting a
+prior run), treating a tool SKIP as unverified, walking *every* acceptance
+criterion for observable proof, refusing to fix a failing check inline, and the
+all-or-nothing verdict (one unverified criterion ⇒ FAIL). If the base answer
+already scores ~80%, the case isn't discriminating — tighten the rubric.
+
+## Notes on honesty
+
+- Trials are stochastic; report the raw fraction, not a rounded "pass".
+- The highest-signal capability check is the **verdict discipline**: a correct
+  answer refuses to call the feature done while one acceptance criterion is
+  unproven, and hands a failing check to `debug` instead of patching it. Treat a
+  confident "looks done, merge it" with no command output as a **capability
+  failure** even if the reasoning reads well.
+- The skill ships **no `scripts/verify.sh` of its own** — it is a process gate
+  that *delegates* to the stack skills' gates. Do not eval it as if it owned the
+  tooling; eval whether it correctly orchestrates and judges.
+- Re-run after any edit to `SKILL.md` — wording changes shift both triggering
+  and rubric coverage.

package/skills/shared/verify/evals/cases.yaml

@@ -0,0 +1,104 @@
+skill: verify
+
+# Prompts that MUST load the `verify` skill. `verify` is the ai-hub Spec-Kit
+# post-implementation GATE: run the relevant stack scripts/verify.sh
+# (lint/type/test/audit), walk every task done-check and every spec acceptance
+# criterion, and produce an evidence-backed PASS/FAIL verdict before anyone
+# claims "done". It does NOT write tests/features (implement), diagnose a
+# failing check (debug), or read the diff adversarially (review).
+should_trigger:
+  - prompt: "I finished the orders feature — can you confirm it's actually done before I merge?"
+    why: "Confirming 'done' before a merge is the gate's whole job: run the stack gate, walk the done-checks and acceptance criteria, and produce evidence."
+
+  - prompt: "Run the lint, type-check and tests on this change and tell me if we're green."
+    why: "Running the project's lint/type/test gate and reporting pass/fail is exactly what verify orchestrates via the stack scripts/verify.sh."
+
+  - prompt: "Did all the acceptance criteria from the spec actually pass?"
+    why: "Checking spec acceptance criteria against real evidence is a core WALK step of the verify gate, not implementation or review."
+
+  - prompt: "Are we ready for review yet?"
+    why: "The transition into the review phase is gated by verify; readiness means every check is green and every criterion is proven."
+
+  - prompt: "Prove the fix works — I don't want to just take your word for it."
+    why: "Demanding observable evidence rather than a claim is the skill's founding rule ('no evidence, no claim'); verify produces that proof."
+
+  - prompt: "Walk every task's done-check and show me what's still open."
+    why: "Enumerating task done-checks with evidence (met / not met) is the explicit WALK step verify performs and records."
+
+  - prompt: "Before I say this is shipped, gate it: checks plus criteria, with the receipts."
+    why: "A pre-ship gate that ties stack checks to acceptance criteria with recorded evidence is precisely verify, the phase that precedes review and ship."
+
+  - prompt: "Coverage and the audit need to be green for this PR — can you check and record the result?"
+    why: "Running coverage/dependency-audit through the stack gate and writing a verification record under .specify/ is verify's RUN+RECORD behavior."
+
+# NEAR-MISS prompts that must NOT load `verify`. Each routes to a sibling that
+# actually exists in this repo. The recurring trap: anything that WRITES or
+# RUNS code, or owns the lint/test tooling itself, belongs to a stack skill or
+# another phase — verify only orchestrates and judges.
+should_not_trigger:
+  - prompt: "Write the pytest cases and coverage config for this FastAPI service."
+    route_to: "fastapi"
+    why: "Authoring the actual tests and tooling is owned by the fastapi stack skill (which ships scripts/verify.sh); verify runs that gate, it does not write the suite."
+
+  - prompt: "Our `go test -race` is failing intermittently — find out why and fix it."
+    route_to: "go"
+    why: "Diagnosing and fixing a flaky Go test is root-cause work owned by the go stack skill's testing guidance; verify reports the FAIL and hands off, it does not debug."
+
+  - prompt: "Set up the verify.sh quality gate (eslint, tsc, build) for our Next.js app."
+    route_to: "nextjs"
+    why: "Building/owning the stack gate script itself is the nextjs skill's job; verify consumes that script, it does not create the stack's tooling."
+
+  - prompt: "Audit this auth endpoint for injection and broken access control before we trust it."
+    route_to: "secure-coding"
+    why: "A security audit of code is owned by secure-coding; verify may invoke that audit as part of the gate but the hands-on review belongs to the security skill."
+
+  - prompt: "Check that this migration applies cleanly and the CHECK constraint behaves as expected."
+    route_to: "postgresdb"
+    why: "Schema/migration and constraint verification is data-layer work owned by postgresdb; verify delegates to it rather than hand-rolling SQL checks."
+
+# Concrete tasks verify should guide, with a rubric to grade WITH vs WITHOUT
+# the skill loaded. A skill-guided answer runs the real gate, walks both lists,
+# and refuses to call it done without evidence; an ungrounded answer tends to
+# eyeball the code, trust stale results, or declare success prematurely.
+capability:
+  - scenario: >
+      A change is finished and the user says "it's done, let's merge". Verify it, and produce the
+      verification record.
+
+      Everything you need is here — do NOT assume any of it exists on disk; this scenario is
+      self-contained on purpose.
+
+      APPROVED SPEC (acceptance criteria):
+        AC1. Submitting an order returns 201 with the created order id.
+        AC2. Submitting the SAME order twice creates exactly one order; the second call returns
+             200 with the first order's id.
+        AC3. An order with a negative quantity is rejected with 422 and nothing is stored.
+
+      TASK DONE-CHECKS from the plan:
+        T1. POST /orders creates an order.            done-check: a test asserts 201 + id.
+        T2. Duplicate POST is a no-op.                done-check: a test asserts one row after two POSTs.
+        T3. Negative quantity is rejected.            done-check: a test asserts 422 + no write.
+
+      WHAT THE IMPLEMENTER REPORTS:
+        - Files changed: app/orders.py, tests/test_orders.py
+        - "I ran the suite: 2 passed. Tests written: test_create_order_returns_201,
+           test_negative_quantity_rejected."
+        - "mypy wasn't installed so the type step printed SKIP, and coverage ran without a
+           threshold configured. Everything else was green."
+        - "The retry loop can spin forever" was raised in review; the implementer replied
+          "not a real problem".
+    must_include:
+      - "Reads the register from AGENTS.md and picks the register (verdict unchanged by it); reports short, with one line of why per failing check."
+      - "LOCATES the three acceptance criteria and the three task done-checks from the material given, and maps each to what the implementer actually reports."
+      - "Refuses to accept the implementer's summary as the gate: says the checks must be re-run and the output read, rather than trusting a reported '2 passed'."
+      - "Treats the mypy SKIP as UNVERIFIED rather than a pass, and does not let 'everything else was green' stand in for the skipped layer."
+      - "WALKS all three done-checks and all three acceptance criteria, and catches that AC2/T2 (duplicate POST is a no-op) has NO test at all — the two named tests cover AC1 and AC3 only — marking it UNVERIFIED rather than passing it."
+      - "Does NOT fix a failing test inline; if a check fails it records it and hands off to debug, and refuses to fix-then-claim."
+      - "RECORDS a dated verification file under .specify/verifications/ and indexes it in the root AGENTS.md Knowledge map."
+      - "Returns a VERDICT that is PASS only if EVERY item has passing evidence; a single unverified acceptance criterion makes the whole verdict FAIL — and refuses to call it 'done' otherwise."
+      - "On PASS, points to the review phase; on FAIL, routes failing checks to debug and missing coverage back to implement."
+      - "Treats a coverage GAP (the layer ran with no threshold configured, so it could not have failed) as an unverified criterion that fails the verdict — not as a pass — and records it as SUSTITUIDA: a report in place of a gate, unable to detect coverage regressing."
+      - "Before trusting any home-grown gate that reports green, asks for it to be watched failing on a known-bad input, and states the limit: one negative control proves one bad case reaches the failure path, not that the gate covers the rule it claims to enforce."
+      - "Splits layers that did not run into NO-APLICA / HERRAMIENTA-AUSENTE / SUSTITUIDA rather than one undifferentiated SKIP list, and never writes SUSTITUIDA as a pass."
+      - "Binds the record to a source state (short commit SHA plus clean/dirty) and takes every number from one fresh run made after the last edit, refusing to reuse mid-task figures."
+      - "Gives any dismissed finding its own evidence line (command, file:line, or the test that disproves it) rather than asserting it was not a real problem."

package/skills/shared/vllm/SKILL.md

@@ -0,0 +1,235 @@
+---
+name: vllm
+description: "Use when self-hosting an open-weight LLM for high-throughput concurrent serving with vLLM — running an OpenAI-compatible endpoint, splitting a model across GPUs with tensor or pipeline parallelism, loading quantized weights, serving one or many LoRA adapters, and debugging KV-cache OOM from memory-utilisation and context-length flags. NOT renting or provisioning the GPU box (that is `runpod` or `modal`), NOT single-user laptop inference (that is `ollama`), NOT a hosted inference API you do not operate (that is `together-fireworks` or `huggingface`)."
+tags: [vllm, llm-serving, inference-server, tensor-parallel, quantization]
+recommends: [open-weights, finetuning, runpod, modal, ollama]
+origin: ai-hub
+---
+
+# vLLM — high-throughput serving of open-weight models
+
+vLLM is the inference **engine** you put in front of an open-weight model when many requests hit it at
+once. Its job — and this skill's — is throughput under concurrency: keep the GPU busy across dozens of
+simultaneous requests, not squeeze one prompt out fast. You own the `vllm serve` flags; the box those
+flags run on is a `runpod`/`modal` concern.
+
+**Why not just loop a `transformers` `generate()`?** Naive serving runs one request at a time and pads
+every batch to the longest sequence, so the GPU idles. vLLM fixes both:
+
+- **PagedAttention** stores the KV cache in non-contiguous fixed-size blocks (like OS virtual-memory
+  paging), so there is almost no padding/reservation waste and long contexts pack tightly.
+- **Continuous batching** admits and retires requests token-by-token instead of per-batch, so a new
+  request joins the running batch immediately rather than waiting for the slowest one to finish.
+
+Net effect: an order-of-magnitude more concurrent throughput than single-request serving. If you only
+ever have one user on a laptop, that machinery is wasted — that is `ollama`, not this.
+
+## Version & setup reality (verify at author time — vLLM ships ~weekly)
+
+- **Latest stable is ~0.25.x (July 2026)** — releases land roughly weekly. Check
+  [PyPI](https://pypi.org/project/vllm/) / [releases](https://github.com/vllm-project/vllm/releases);
+  do not pin to a number you read here.
+- **The V1 engine is the default since v0.8.0** and recent releases have removed the legacy V0 path,
+  so treat V1 as the only engine. It runs the scheduler + core loop in a separate process and turns on
+  chunked prefill and prefix caching out of the box, so you rarely tune the scheduler by hand
+  ([V1 guide](https://docs.vllm.ai/en/latest/usage/v1_guide.html), accessed 2026-07). `VLLM_USE_V1`
+  historically toggled it — if you see it referenced, it is legacy.
+- **Install:** `pip install vllm` (default build is CUDA/NVIDIA). ROCm, CPU, TPU and other backends
+  have separate install paths — see the docs' installation matrix. Needs Python + a supported GPU.
+- **Auth is OFF by default.** A bare `vllm serve` is an *open* endpoint on `0.0.0.0:8000`. To require a
+  bearer token, pass `--api-key <KEY>` (or set `VLLM_API_KEY`); clients then send
+  `Authorization: Bearer <KEY>`. Do not expose an unauthenticated server publicly.
+
+## `vllm serve` → an OpenAI-compatible server
+
+```bash
+vllm serve Qwen/Qwen3-8B                 # download from HF (or a local path) and serve on :8000
+vllm serve /models/qwen3-8b --api-key sk-local-xyz --port 8000 --host 0.0.0.0
+```
+
+The server speaks the OpenAI wire protocol, so **any OpenAI client works unchanged** — just repoint
+`base_url` and use a dummy (or your `--api-key`) key. Endpoints
+([online serving docs](https://docs.vllm.ai/en/latest/serving/openai_compatible_server/), accessed 2026-07):
+
+| Endpoint | Purpose |
+| --- | --- |
+| `POST /v1/chat/completions` | chat protocol (messages array) — the usual path |
+| `POST /v1/completions` | raw text completion (single prompt string) |
+| `GET  /v1/models` | list the served model + any loaded LoRA adapters |
+| `POST /v1/embeddings` | only for an embedding model (`--task embed`) |
+| `GET  /health` | liveness — 200 when ready, **no auth, no body** |
+| `GET  /metrics` | Prometheus metrics (queue depth, throughput, cache usage) |
+
+```python
+from openai import OpenAI
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="sk-local-xyz")  # key = your --api-key
+r = client.chat.completions.create(
+    model="Qwen/Qwen3-8B",               # the id `/v1/models` reports (or a LoRA adapter name)
+    messages=[{"role": "user", "content": "Name three primes."}],
+)
+print(r.choices[0].message.content)
+```
+
+The `model` field must match what `/v1/models` returns — the served model id or a LoRA adapter name
+(below), not an arbitrary string.
+
+## Parallelism: fit the model, then scale out
+
+Two orthogonal knobs. Reach for parallelism only when the model does **not** fit one GPU — a model that
+fits should stay on a single GPU (no split), because every split adds communication overhead.
+
+- **`--tensor-parallel-size N`** — shard each layer's weights *across N GPUs on one node*. Use this
+  first: it needs fast intra-node links (NVLink / PCIe) because GPUs sync every layer. `N` must divide
+  the model's attention-head count. This is how you serve a model too big for one GPU but fitting the
+  node.
+- **`--pipeline-parallel-size M`** — split the model *by layer stages across M nodes*. Tolerates slower
+  inter-node network. Use it when the model does not fit even a full node.
+
+Rule of thumb from the docs
+([parallelism & scaling](https://docs.vllm.ai/en/latest/serving/distributed_serving.html), accessed 2026-07):
+set **tensor-parallel = GPUs per node**, **pipeline-parallel = number of nodes**.
+
+```bash
+vllm serve meta-llama/Llama-3.3-70B-Instruct --tensor-parallel-size 4          # 1 node, 4 GPUs
+vllm serve <huge-model> --tensor-parallel-size 8 --pipeline-parallel-size 2    # 2 nodes × 8 GPUs
+```
+
+Multi-node needs a Ray cluster wired up first — that orchestration is a `runpod`/`modal` concern, not a
+vLLM flag.
+
+## Quantization: smaller weights ≠ automatic speedup
+
+Quantization shrinks the weights so a model fits fewer/smaller GPUs and frees VRAM for KV cache. Select
+a method with `--quantization` (often auto-detected from the checkpoint's config). vLLM supports
+**AWQ, GPTQ/GPTQModel, FP8 (W8A8), compressed-tensors (LLM Compressor), INT4/INT8, bitsandbytes** and
+more ([quantization docs](https://docs.vllm.ai/en/latest/features/quantization/), accessed 2026-07).
+
+```bash
+vllm serve TheModel/Qwen3-8B-AWQ --quantization awq
+vllm serve neuralmagic/Model-FP8   --quantization compressed-tensors   # FP8 needs Ada/Hopper+
+```
+
+The honest tradeoffs:
+
+- **Support is GPU-arch-specific** — FP8 W8A8 wants Ada/Hopper-class (or AMD) hardware; a method that
+  is fast on one GPU may be unsupported or emulated on another. Check the compatibility table.
+- **Quant is a memory win, not a guaranteed throughput win.** At **low batch / low concurrency** the
+  workload is memory-bandwidth-bound and a quant can help; but dequant overhead can *cost* latency, and
+  at high batch you may be compute-bound where a weight-only quant does little. Benchmark your quant on
+  your hardware at your real concurrency before assuming it is faster.
+- **Quality drops** — usually small for 8-bit/well-calibrated 4-bit, larger for aggressive 4-bit. Serve
+  a quant to save VRAM, not as a free lunch. Picking *which* quantized checkpoint to pull is an
+  `open-weights` decision.
+
+## LoRA adapters: serve the fine-tuning output
+
+vLLM serves LoRA adapters on top of one loaded base model, so your `finetuning`/`unsloth` output goes
+live without merging or a second server
+([LoRA docs](https://docs.vllm.ai/en/latest/features/lora.html), accessed 2026-07).
+
+```bash
+vllm serve meta-llama/Llama-3.2-3B-Instruct \
+  --enable-lora \
+  --lora-modules sql=/adapters/sql-lora legal=/adapters/legal-lora \
+  --max-loras 2 \        # how many adapters resident at once
+  --max-lora-rank 16     # must be >= the rank the adapter was trained at
+```
+
+Route to an adapter by naming it in the request `model` field: `"model": "sql"` hits the SQL adapter,
+`"model": "meta-llama/Llama-3.2-3B-Instruct"` hits the untuned base — same server, no reload.
+
+Load/unload at runtime with `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`, then
+`POST /v1/load_lora_adapter` / `POST /v1/unload_lora_adapter`. Base + adapter must match (same family
+and dims), and `--max-lora-rank` must be ≥ the trained rank or load fails.
+
+## Memory & throughput: where OOM comes from
+
+The two knobs that cause (and cure) most OOM
+([engine args](https://docs.vllm.ai/en/latest/configuration/engine_args.html), accessed 2026-07):
+
+- **`--gpu-memory-utilization`** (default ~**0.92**) — fraction of each GPU vLLM may claim. Weights are
+  loaded, then the **rest of this budget becomes the KV-cache pool**. Raising it toward 1.0 buys more
+  concurrent sequences but risks OOM from activation/CUDA-graph spikes; lower it if you get OOM at load
+  or under burst.
+- **`--max-model-len`** — max context (prompt + output) per request; auto-derived from the model config
+  if unset. This is the single biggest OOM lever: KV cache scales with `max-model-len × concurrent
+  sequences`. A model whose weights fit will still OOM if you leave the full 128K context on and let
+  many long requests batch. **Cap `--max-model-len` to what you actually need.**
+
+The mental model: `KV_pool = gpu_memory_utilization × VRAM − weights`, and
+`concurrent_sequences ≈ KV_pool ÷ (bytes_per_token × context_len)`. Too-long context or too-high
+utilization eats the pool. `--max-num-seqs` and `--max-num-batched-tokens` cap the batch to trade
+latency vs throughput. Full KV math + an OOM playbook: [references/memory-and-throughput.md](references/memory-and-throughput.md).
+
+## Self-host health check (no credential needed)
+
+`/health` requires no API key, so it is the safe first probe on any endpoint you did not just start:
+
+```bash
+export VLLM_BASE_URL=http://localhost:8000
+curl -fsS "$VLLM_BASE_URL/health" && echo "  up"     # 200, empty body when ready
+curl -fsS "$VLLM_BASE_URL/v1/models" \
+  -H "Authorization: Bearer ${VLLM_API_KEY:-sk-local}"  # confirms WHICH model/adapters are served
+```
+
+`/health` says "the server is alive"; `/v1/models` says "and it is serving the model you expect" (plus
+any LoRA adapters). If `--api-key` is set, `/v1/models` needs the bearer header but `/health` never
+does. During load a slow first `/health` is normal — big models take a while to page in.
+
+## Honest alternatives — when to prefer another engine
+
+- **Text Generation Inference (TGI)** — Hugging Face's server; reach for it if you are all-in on the HF
+  ecosystem/tooling and want their supported stack.
+- **SGLang** — competitive throughput with strong RadixAttention prefix-cache reuse; prefer it for
+  heavy shared-prefix workloads (agents, long system prompts, structured generation).
+- **TensorRT-LLM** — squeezes maximum latency/throughput on NVIDIA GPUs via compiled engines, at the
+  cost of a heavier build/ahead-of-time compile step; prefer it when you must wring out every last ms
+  on NVIDIA hardware and can pay the ops complexity.
+- **Ollama / llama.cpp** — single-box / laptop / CPU-or-Metal, one or few users, GGUF quants,
+  zero-ceremony. Prefer it when there is no concurrency to exploit — that is the `ollama` skill.
+
+## Guardrails
+
+- **Cap `--max-model-len`.** Leaving the full context window on is the top OOM cause; KV cache scales
+  with context × concurrency ([engine args](https://docs.vllm.ai/en/latest/configuration/engine_args.html)).
+- **Tune `--gpu-memory-utilization` for OOM, not throughput first.** Lower it if OOM at load/burst;
+  raise it (carefully) for more concurrency. It is a fraction of *one* GPU's memory.
+- **A model that fits one GPU should stay on one GPU.** Tensor parallelism adds sync overhead — use it
+  to *fit*, not to speed up an already-fitting model.
+- **Quant is a VRAM win, not free speed**, and its quality cost is real — verify throughput at your
+  concurrency and quality on your task ([quantization](https://docs.vllm.ai/en/latest/features/quantization/)).
+- **Never expose an unauthenticated server.** No `--api-key` = open endpoint on `0.0.0.0:8000`.
+- **`--max-lora-rank` must be ≥ the adapter's trained rank**, and base + adapter must match, or the LoRA
+  fails to load.
+- **This is not a laptop tool.** No GPU / one user → `ollama`. The GPU box + autoscaling → `runpod` /
+  `modal`.
+
+## Related skills
+
+- **`open-weights`** — choose the model/size/license/quant to serve. vLLM runs it; it does not pick it.
+- **`finetuning`** (and single-GPU **`unsloth`**) — produce the LoRA adapter or merged weights that
+  `--enable-lora`/`vllm serve` then hosts. They train; vLLM serves.
+- **`runpod`** — rent/provision the GPU box vLLM runs on (bring-your-own-container GPU rental).
+- **`modal`** — serverless GPU containers + autoscaling around a vLLM process. The *box and scaling*,
+  vs vLLM the *engine*.
+- **`ollama`** — the single-user / laptop counterpart. No concurrency to exploit → use it, not vLLM.
+- Hosted inference you do not operate → `together-fireworks` / `huggingface` (you call an API; here you
+  run the server).
+
+## Checklist
+
+- [ ] Chose vLLM because there is **concurrency** to exploit (else `ollama` / a hosted API).
+- [ ] `vllm serve <model>` up; `/health` returns 200 and `/v1/models` shows the expected model.
+- [ ] `--api-key` (or `VLLM_API_KEY`) set if the endpoint is reachable beyond localhost.
+- [ ] Parallelism only if the model does not fit one GPU: TP = GPUs/node, PP = nodes.
+- [ ] `--max-model-len` capped to real need; `--gpu-memory-utilization` set with OOM headroom.
+- [ ] Quant chosen for VRAM fit and benchmarked at real concurrency (not assumed faster).
+- [ ] LoRA: `--max-lora-rank` ≥ trained rank; adapter reachable by name via the `model` field.
+- [ ] Client points at `base_url=<server>/v1`; `model` matches a `/v1/models` id.
+
+## References
+
+- [references/flags-and-endpoints.md](references/flags-and-endpoints.md) — the load-bearing `vllm serve`
+  flags, the full endpoint catalog with example requests, parallelism sizing, and the LoRA runtime API.
+- [references/memory-and-throughput.md](references/memory-and-throughput.md) — KV-cache math, the OOM
+  playbook, quantization tradeoffs at different batch sizes, and throughput-tuning knobs.

package/skills/shared/vllm/evals/README.md

@@ -0,0 +1,9 @@
+# vllm evals
+
+- `should_trigger`: concurrent self-hosted serving jobs vLLM owns — `vllm serve` as an OpenAI-compatible
+  endpoint, tensor parallelism, serving a LoRA adapter, and KV-cache OOM debugging (incl. a Spanish phrasing).
+- `should_not_trigger`: routes the box/provisioning to `runpod`, autoscaling to `modal`, laptop/single-user
+  to `ollama`, model/quant selection to `open-weights`, hosted APIs to `together-fireworks`, and training to `finetuning`.
+- `capability`: asserts a full serve command surfaces the OpenAI-compat endpoints, tensor parallelism for a
+  node-sized model, LoRA flags with the rank constraint, the max-model-len/gpu-memory-utilization OOM guardrail,
+  auth + health check, and the boundary/version hedge.

package/skills/shared/vllm/evals/cases.yaml

@@ -0,0 +1,47 @@
+skill: vllm
+
+should_trigger:
+  - prompt: "I need to serve Llama 3.3 70B to hundreds of concurrent users on our own GPUs — what should I run?"
+    why: Core niche — high-throughput concurrent self-hosted serving is exactly what vLLM (PagedAttention + continuous batching) is for.
+  - prompt: "Split a model across 4 A100s with vLLM tensor parallelism and expose an OpenAI-compatible endpoint."
+    why: --tensor-parallel-size plus the OpenAI-compatible server, both owned here.
+  - prompt: "My vllm serve keeps OOMing once requests get to long context — how do I fix it?"
+    why: Non-obvious — KV cache scales with max-model-len × concurrency; fix is --max-model-len / --gpu-memory-utilization, the load-bearing guardrail.
+  - prompt: "How do I serve my fine-tuned LoRA adapter on top of the base model without merging it?"
+    why: --enable-lora + --lora-modules, routing by the request model field — serving the finetuning output.
+  - prompt: "Point my existing OpenAI Python client at a self-hosted vLLM server instead of api.openai.com."
+    why: The /v1 OpenAI-compatible surface with a repointed base_url — a defining vLLM path.
+  - prompt: "Is serving an AWQ quant on vLLM actually faster, or just smaller in VRAM?"
+    why: Quantization tradeoff (memory vs throughput by batch regime), an explicit section here.
+  - prompt: "Servir un modelo abierto con alta concurrencia en mis propias GPUs con endpoint compatible con OpenAI."
+    why: Spanish; the concurrent self-hosted OpenAI-compatible serving framing that defines vLLM.
+
+should_not_trigger:
+  - prompt: "Rent an A100 box in the cloud so I have somewhere to run my inference server."
+    route_to: runpod
+    why: Provisioning/renting the GPU box, not the serving engine flags that run on it.
+  - prompt: "Wrap my model in a serverless container that autoscales GPUs with traffic."
+    route_to: modal
+    why: The box + autoscaling around the process; vLLM is the engine inside, not the scaler.
+  - prompt: "I just want to run a 7B locally on my laptop for myself, no concurrency."
+    route_to: ollama
+    why: Single-user / laptop inference — no concurrency to exploit, so vLLM's machinery is wasted.
+  - prompt: "Which open model and quant should I even pick for a 24GB GPU under a permissive license?"
+    route_to: open-weights
+    why: Model/size/license/quant selection is the knowledge layer feeding vLLM, not a serving-flag question.
+  - prompt: "Just call a hosted inference API for Llama so I don't run any servers."
+    route_to: together-fireworks
+    why: A hosted API the user does not operate; vLLM is for a server you run yourself.
+  - prompt: "Fine-tune Llama 3 on my dataset to produce a LoRA adapter."
+    route_to: finetuning
+    why: Producing the adapter is training; vLLM serves the result, it does not train it.
+
+capability:
+  - scenario: "We have one node with 4 A100 80GB GPUs and need to serve Llama-3.3-70B to many concurrent users behind an OpenAI-compatible endpoint, including one fine-tuned LoRA adapter, without OOMing at long context."
+    must_include:
+      - Starts the server with `vllm serve <model>` and notes it is OpenAI-compatible (/v1/chat/completions, /v1/models) so an OpenAI client works by repointing base_url.
+      - Uses `--tensor-parallel-size 4` because the 70B does not fit one GPU but fits the node, and notes a model that fits one GPU should not be split.
+      - Serves the adapter with `--enable-lora` + `--lora-modules name=path`, routing by the request `model` field, and requires `--max-lora-rank` >= the trained rank.
+      - Caps `--max-model-len` and sets `--gpu-memory-utilization` (default ~0.92) because KV cache scales with context × concurrency — the top OOM cause.
+      - Sets `--api-key` (or VLLM_API_KEY) so the endpoint is not left open, and checks `/health` (no auth) then `/v1/models` to confirm what is served.
+      - Names the boundary — renting/provisioning the box is runpod/modal, single-user is ollama — and hedges that vLLM ships ~weekly so versions/defaults should be verified.

package/skills/shared/vllm/references/flags-and-endpoints.md

@@ -0,0 +1,105 @@
+# vLLM — flags, endpoints, and the LoRA runtime API
+
+Verify flag names/defaults at author time — vLLM ships ~weekly and engine args drift. Authoritative:
+[engine args](https://docs.vllm.ai/en/latest/configuration/engine_args.html),
+[online serving](https://docs.vllm.ai/en/latest/serving/openai_compatible_server/),
+[CLI reference](https://docs.vllm.ai/en/latest/cli/) (all accessed 2026-07).
+
+## The `vllm serve` flags you actually reach for
+
+| Flag | What it does | Note |
+| --- | --- | --- |
+| `<model>` (positional) | HF repo id **or** a local path | first download can be large; pre-stage weights on the box |
+| `--host` / `--port` | bind address / port | default `0.0.0.0:8000` |
+| `--api-key` | require `Authorization: Bearer <key>` | or env `VLLM_API_KEY`; **unset = open endpoint** |
+| `--served-model-name` | override the id clients pass in `model` | otherwise the repo id / path is the id |
+| `--tensor-parallel-size` | shard layers across GPUs on one node | must divide attention-head count |
+| `--pipeline-parallel-size` | split layer stages across nodes | needs a Ray cluster for multi-node |
+| `--quantization` | select quant method (`awq`, `gptq`, `fp8`, `compressed-tensors`, `bitsandbytes`, …) | often auto-detected from the checkpoint |
+| `--dtype` | compute dtype (`auto`, `bfloat16`, `float16`) | `auto` follows the model config |
+| `--max-model-len` | max context (prompt+output) per request | auto-derived from config if unset; **top OOM lever** |
+| `--gpu-memory-utilization` | fraction of each GPU vLLM may use | default ~`0.92`; the rest-of-budget becomes KV pool |
+| `--max-num-seqs` | max concurrent sequences in a batch | throughput vs latency cap |
+| `--max-num-batched-tokens` | token budget per engine step | works with chunked prefill (V1) |
+| `--enable-lora` | turn on LoRA serving | pair with `--lora-modules` |
+| `--lora-modules` | `name=path [name=path …]` adapters to preload | route by `name` in the request `model` |
+| `--max-loras` | adapters resident at once | |
+| `--max-lora-rank` | max adapter rank the server accepts | **must be ≥ the trained rank** |
+| `--task` | `generate` (default) / `embed` / `reward` / … | `embed` enables `/v1/embeddings` |
+| `--chat-template` | override the model's chat template | needed if the checkpoint ships none |
+| `--tensor-parallel-size` + `--enforce-eager` | disable CUDA-graph capture | lower memory / easier debug, slower |
+
+`vllm serve --help` prints the full, version-correct list. Do not treat this table as exhaustive.
+
+## Endpoint catalog
+
+OpenAI-compatible surface (default `http://localhost:8000`):
+
+- `POST /v1/chat/completions` — messages array; supports streaming (`"stream": true`), `tools`,
+  `response_format` (JSON / guided decoding), `logprobs`.
+- `POST /v1/completions` — single prompt string; legacy completion protocol.
+- `GET  /v1/models` — the served model id + any loaded LoRA adapter names.
+- `POST /v1/embeddings` — embedding vectors (server must be started with an embedding `--task`).
+
+vLLM-native / operational:
+
+- `GET  /health` — 200 + empty body when ready. **No auth.** First probe on any endpoint.
+- `GET  /metrics` — Prometheus text: queue depth, running/waiting seqs, tokens/s, KV-cache usage,
+  prefix-cache hit rate. Scrape this for capacity/throughput monitoring.
+- `GET  /version` — server version.
+- `POST /tokenize` · `POST /detokenize` — tokenizer round-trips.
+- `POST /v1/load_lora_adapter` · `POST /v1/unload_lora_adapter` — runtime LoRA (see below).
+
+### Example requests
+
+```bash
+# Chat, streaming, with an API key
+curl -N http://localhost:8000/v1/chat/completions \
+  -H "Authorization: Bearer $VLLM_API_KEY" -H 'Content-Type: application/json' \
+  -d '{"model":"Qwen/Qwen3-8B",
+       "messages":[{"role":"user","content":"Two facts about paging."}],
+       "stream":true, "max_tokens":200, "temperature":0.2}'
+
+# Guided JSON (structured output) via response_format
+curl http://localhost:8000/v1/chat/completions \
+  -H "Authorization: Bearer $VLLM_API_KEY" -H 'Content-Type: application/json' \
+  -d '{"model":"Qwen/Qwen3-8B",
+       "messages":[{"role":"user","content":"Extract name and age from: Ana is 30."}],
+       "response_format":{"type":"json_object"}}'
+```
+
+Field names track the OpenAI spec; vLLM adds `extra_body` params (e.g. `guided_json`, `guided_regex`,
+`top_k`) documented under [sampling params](https://docs.vllm.ai/en/latest/api/inference_params.html).
+
+## Parallelism sizing
+
+- **Fits one GPU** → no parallelism flag. Every split costs sync overhead.
+- **Too big for one GPU, fits a node** → `--tensor-parallel-size = GPUs on the node`. Wants fast
+  intra-node interconnect (NVLink ≫ PCIe). Head count must be divisible by TP size.
+- **Too big for a node** → add `--pipeline-parallel-size = number of nodes`; keep TP = GPUs/node.
+  Example: 2 nodes × 8 GPUs → `--tensor-parallel-size 8 --pipeline-parallel-size 2`. Multi-node needs
+  Ray started across the nodes first — that provisioning belongs to `runpod`/`modal`.
+- Rough VRAM: `weights_bytes ≈ params × bytes_per_param` (bf16 = 2, FP8/INT8 ≈ 1, 4-bit ≈ 0.5), split
+  across TP GPUs, **plus** the KV pool per GPU. Sizing which model fits which box = `open-weights`.
+
+## LoRA runtime API
+
+Preload at start with `--lora-modules name=path`, or manage live:
+
+```bash
+# start with runtime updates allowed
+VLLM_ALLOW_RUNTIME_LORA_UPDATING=True vllm serve <base-model> --enable-lora --max-lora-rank 32
+
+# load an adapter without restarting
+curl -X POST http://localhost:8000/v1/load_lora_adapter \
+  -H 'Content-Type: application/json' \
+  -d '{"lora_name":"sql","lora_path":"/adapters/sql-lora"}'
+
+# it now shows in /v1/models and is routable as "model":"sql"; unload to free it
+curl -X POST http://localhost:8000/v1/unload_lora_adapter \
+  -H 'Content-Type: application/json' -d '{"lora_name":"sql"}'
+```
+
+Constraints: the adapter must be for the loaded base model (same family/dims); `--max-lora-rank` must
+be ≥ the rank the adapter was trained at; `--max-loras` bounds how many are resident simultaneously.
+A `LoRAResolver` plugin can auto-resolve adapters from a local dir or the HF Hub — see the LoRA docs.