@ericrisco/rsc 0.1.39 → 0.1.41

package/skills/deep-learning/evals/cases.yaml

@@ -0,0 +1,48 @@
+skill: deep-learning
+
+should_trigger:
+  - prompt: "My PyTorch training loss isn't going down at all — the loop looks fine, what am I missing?"
+    why: "Core training-loop debugging in Torch; the diagnosis lands on the five-step loop and its silent bugs, this skill's center."
+  - prompt: "Loss becomes NaN after a few hundred steps when I turn on float16. How do I fix it?"
+    why: "AMP fp16 underflow / missing GradScaler and the bf16 alternative — the mixed-precision section owns this."
+  - prompt: "How do I use torch.autocast and GradScaler, and should I pick bf16 or fp16?"
+    why: "Explicit AMP API + the bf16-vs-fp16 decision this skill teaches."
+  - prompt: "I need to train this model across 4 GPUs — DDP or FSDP, and how do I launch it?"
+    why: "Distributed choice by memory-fit plus torchrun launch is a named section (and references/distributed.md)."
+  - prompt: "My validation accuracy is way worse than training and I think I forgot model.eval()."
+    why: "Non-obvious guardrail: eval() vs train() flips Dropout/BatchNorm; squarely a load-bearing gotcha here."
+  - prompt: "Gradients seem to add up across batches and training blows up — is my loop wrong?"
+    why: "Non-obvious: the symptom is a missing optimizer.zero_grad(), the flagship silent bug of this skill; no API named."
+  - prompt: "Build a CNN from scratch in PyTorch and write the training loop to fit it on my image dataset."
+    why: "From-scratch net + custom loop in Torch — inside the boundary vs finetuning/ML."
+  - prompt: "vull entrenar una xarxa neuronal des de zero amb PyTorch i que sigui reproduïble"
+    why: "Catalan trigger: from-scratch neural net + reproducibility, both owned here."
+
+should_not_trigger:
+  - prompt: "Fine-tune Llama with QLoRA on my instruction dataset using trl and peft."
+    route_to: finetuning
+    why: "Adapting a pretrained LLM with LoRA/QLoRA is the finetuning method skill, not general net training."
+  - prompt: "Should I use XGBoost or a random forest for this tabular churn dataset?"
+    route_to: machine-learning
+    why: "Classic/tabular GBDT modeling — no autograd, no PyTorch loop."
+  - prompt: "Which tokenizer and how do I compute F1/BLEU for my text classification task?"
+    route_to: nlp
+    why: "Tokenization and NLP task metrics live in nlp; only the raw training loop routes back here."
+  - prompt: "Serve my trained model with vLLM and set the right batching flags for throughput."
+    route_to: vllm
+    why: "Inference/serving engine, not training."
+  - prompt: "Set up a clean Python project with uv, pytest, and type checking for my model code."
+    route_to: python
+    why: "Env/packaging/testing hygiene, not the neural-net training itself."
+
+capability:
+  - scenario: "A user is training a from-scratch image classifier in PyTorch on one A100. Training loss is unstable and sometimes NaNs, validation numbers look wrong, and they want it faster and reproducible. Write the training setup and explain the fixes."
+    must_include:
+      - "Correct five-step loop with optimizer.zero_grad(set_to_none=True) EVERY step, explaining that PyTorch accumulates .grad so skipping zero_grad silently sums gradients across batches"
+      - "AMP via torch.autocast on forward+loss only with torch.amp.GradScaler; recommends bf16 on A100 (no scaler needed) and notes fp16 REQUIRES the scaler to avoid underflow NaNs"
+      - "Gradient clipping done after scaler.unscale_(optimizer), and CrossEntropyLoss fed raw logits (no Softmax before it)"
+      - "model.eval() plus torch.no_grad() for validation, stating it changes Dropout/BatchNorm behavior and prevents graph build, and model.train() to switch back — as the cause of bad val numbers"
+      - "AdamW with weight decay excluded from bias/Norm params, plus a warmup→cosine schedule (LinearLR + CosineAnnealingLR via SequentialLR) with step() cadence matching the scheduler units"
+      - "torch.compile(model) offered as the speedup, noting first steps are slow due to compilation"
+      - "Reproducibility: seed random/numpy/torch, torch.use_deterministic_algorithms(True), cudnn.deterministic/benchmark=False, CUBLAS_WORKSPACE_CONFIG, and the honest caveat that results are NOT guaranteed across PyTorch releases/platforms or CPU-vs-GPU even with identical seeds"
+      - "Checkpoints save model+optimizer+scheduler+scaler state_dicts (not the model object), resuming model first; version hedged to PyTorch 2.x (~2.13, verify)"

package/skills/deep-learning/references/distributed.md

@@ -0,0 +1,105 @@
+# Distributed training — DDP, FSDP2, DeepSpeed ZeRO (mechanics)
+
+Grounded in `pytorch.org/docs/stable/` (DDP: `notes/ddp.html`, `generated/torch.nn.parallel.DistributedDataParallel.html`;
+FSDP2: `distributed.fsdp.fully_shard.html`). Verify API against the installed version — distributed
+APIs move faster than the core loop.
+
+## The decision, restated
+
+| Situation | Use | Why |
+|---|---|---|
+| Model fits on one GPU, want more throughput / bigger effective batch | **DDP** | Simplest; replicate + all-reduce grads |
+| Model (params+grads+optimizer state) does NOT fit on one GPU | **FSDP2** | Shards the three memory hogs across ranks |
+| Need CPU/NVMe offload, or config-first scaling, or already using Accelerate/DeepSpeed | **DeepSpeed ZeRO** | Stage 1/2/3 + offload, external library |
+
+Data parallel of any kind assumes each rank sees a **different shard of the batch** — use
+`DistributedSampler` so ranks don't all train on the same rows.
+
+## DDP — one process per GPU
+
+```python
+import os, torch, torch.distributed as dist
+from torch.nn.parallel import DistributedDataParallel as DDP
+
+dist.init_process_group("nccl")                        # NCCL for CUDA; "gloo" for CPU
+local_rank = int(os.environ["LOCAL_RANK"])             # set by torchrun
+torch.cuda.set_device(local_rank)
+
+model = MyModel().to(local_rank)
+model = DDP(model, device_ids=[local_rank])            # grads all-reduced automatically in backward
+
+from torch.utils.data import DataLoader
+from torch.utils.data.distributed import DistributedSampler
+sampler = DistributedSampler(train_ds)                 # disjoint shard per rank
+loader  = DataLoader(train_ds, sampler=sampler, batch_size=bs)
+
+for epoch in range(epochs):
+    sampler.set_epoch(epoch)                           # REQUIRED: reshuffles per epoch across ranks
+    for x, y in loader:
+        ...                                            # normal 5-step loop; DDP syncs grads for you
+
+if local_rank == 0:
+    torch.save(model.module.state_dict(), "ckpt.pt")   # .module unwraps DDP; save on rank 0 only
+dist.destroy_process_group()
+```
+
+Launch: `torchrun --standalone --nproc_per_node=4 train.py` (4 GPUs, one node). Multi-node adds
+`--nnodes`, `--node_rank`, `--rdzv_endpoint`.
+
+DDP gotchas:
+- **`sampler.set_epoch(epoch)`** or every epoch sees the same order — a silent correctness bug.
+- **`find_unused_parameters=True`** only if some params get no grad each step (costs a graph
+  traversal); leaving it True unnecessarily slows training.
+- Save/load on **rank 0** and unwrap with `.module`; barrier before others load.
+- Effective batch = `per_gpu_batch * world_size` — scale LR accordingly (linear-scaling rule).
+
+## FSDP2 — shard when it doesn't fit
+
+FSDP2 (`fully_shard`) is the current API: DTensor-based, per-parameter dim-0 sharding. Apply it
+**bottom-up** — wrap the repeated blocks first, then the root module — so each `fully_shard` call
+forms a communication group and gathers params just-in-time.
+
+```python
+from torch.distributed.fsdp import fully_shard
+
+for block in model.transformer_blocks:       # wrap leaves/blocks first
+    fully_shard(block)
+fully_shard(model)                            # then the root
+
+# Mixed precision is a policy here, not GradScaler:
+from torch.distributed.fsdp import MixedPrecisionPolicy
+for block in model.transformer_blocks:
+    fully_shard(block, mp_policy=MixedPrecisionPolicy(param_dtype=torch.bfloat16))
+```
+
+- `reshard_after_forward=True` (default for non-root) frees gathered params after forward and
+  re-gathers in backward — saves memory, costs an all-gather. `False` (default for root) keeps them.
+- Wrapping **only** the root makes one giant communication group with no compute/comm overlap —
+  wrap blocks to overlap.
+- FSDP2 uses AMP via `MixedPrecisionPolicy`, not `torch.amp.GradScaler`.
+- Full-state-dict save needs `DTensor.full_tensor()` or Distributed Checkpoint (DCP) APIs; prefer
+  sharded checkpoints for large models.
+
+## DeepSpeed ZeRO — external, config-driven
+
+DeepSpeed is a **separate library** (Microsoft), commonly driven through a JSON config and/or HF
+Accelerate. Verify the current version and config schema — it evolves independently of Torch.
+
+| Stage | Shards | Rough memory win vs DDP | Comm cost |
+|---|---|---|---|
+| ZeRO-1 | optimizer states | ~4× on optimizer state | ~DDP |
+| ZeRO-2 | + gradients | more | ~DDP |
+| ZeRO-3 | + parameters (≈ FSDP) | largest | higher (gather params) |
+| + offload | move state to CPU/NVMe | fit far bigger models | slowest (PCIe/disk bound) |
+
+Choose ZeRO over FSDP2 when you specifically want **offload** or a config-first workflow, or you're
+already in the DeepSpeed/Accelerate stack. For pure Torch, FSDP2 covers the same ZeRO-3 territory
+natively.
+
+## Cross-cutting
+
+- **Gradient accumulation** (micro-batching) simulates a bigger batch on limited memory: run N
+  forward/backward passes, then one `optimizer.step()` + `zero_grad()`. Under DDP, wrap the
+  non-final micro-steps in `model.no_sync()` to skip redundant all-reduces.
+- **`torch.compile` composes** with DDP and FSDP2, but compile after wrapping and expect a longer
+  warmup.

package/skills/deep-learning/references/training-recipes.md

@@ -0,0 +1,135 @@
+# Training recipes — a full loop, reproducibility harness, and debug ladder
+
+Concrete, runnable single-GPU scaffolding to lift into a project. APIs verified against
+`pytorch.org/docs/stable/` (amp_examples, optim, notes/randomness). Hedge versions to "PyTorch 2.x".
+
+## A complete single-GPU training loop (AMP + AdamW + warmup→cosine + checkpoint)
+
+```python
+import torch, torch.nn as nn
+from torch.optim.lr_scheduler import LinearLR, CosineAnnealingLR, SequentialLR
+
+device = "cuda" if torch.cuda.is_available() else "cpu"
+model = MyModel().to(device)
+model = torch.compile(model)                 # optional speedup; first steps compile (slow)
+
+# No weight decay on bias/Norm params — two groups.
+decay, no_decay = [], []
+for n, p in model.named_parameters():
+    if not p.requires_grad:
+        continue
+    (no_decay if p.ndim <= 1 or n.endswith(".bias") else decay).append(p)
+opt = torch.optim.AdamW(
+    [{"params": decay, "weight_decay": 0.01},
+     {"params": no_decay, "weight_decay": 0.0}],
+    lr=3e-4, betas=(0.9, 0.999),
+)
+
+total_steps = len(train_loader) * epochs
+warmup = LinearLR(opt, start_factor=0.1, total_iters=min(500, total_steps // 20))
+cosine = CosineAnnealingLR(opt, T_max=total_steps - warmup.total_iters, eta_min=3e-5)
+sched  = SequentialLR(opt, [warmup, cosine], milestones=[warmup.total_iters])
+
+loss_fn = nn.CrossEntropyLoss(label_smoothing=0.1)     # takes RAW logits
+scaler  = torch.amp.GradScaler("cuda", enabled=(device == "cuda"))
+USE_BF16 = torch.cuda.is_bf16_supported() if device == "cuda" else False
+amp_dtype = torch.bfloat16 if USE_BF16 else torch.float16
+# NOTE: with bf16 the scaler is a no-op; keep enabled only for fp16.
+
+best_val = float("inf")
+for epoch in range(epochs):
+    model.train()
+    for x, y in train_loader:
+        x, y = x.to(device), y.to(device)
+        opt.zero_grad(set_to_none=True)
+        with torch.autocast(device_type=device, dtype=amp_dtype, enabled=(device == "cuda")):
+            loss = loss_fn(model(x), y)
+        scaler.scale(loss).backward()
+        scaler.unscale_(opt)                                    # unscale before clipping
+        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
+        scaler.step(opt)
+        scaler.update()
+        sched.step()                                            # per-step: units are steps
+
+    # ---- validation ----
+    model.eval()
+    val_loss, n = 0.0, 0
+    with torch.no_grad():                                       # no graph, no Dropout/BN train stats
+        for x, y in val_loader:
+            x, y = x.to(device), y.to(device)
+            val_loss += loss_fn(model(x), y).item() * x.size(0) # .item() -> no graph leak
+            n += x.size(0)
+    val_loss /= n
+
+    if val_loss < best_val:                                     # early-stopping signal + best ckpt
+        best_val = val_loss
+        torch.save({"epoch": epoch,
+                    "model": model.state_dict(),
+                    "optimizer": opt.state_dict(),
+                    "scheduler": sched.state_dict(),
+                    "scaler": scaler.state_dict(),
+                    "best_val": best_val}, "best.pt")
+```
+
+Resume:
+
+```python
+ckpt = torch.load("best.pt", map_location=device)
+model.load_state_dict(ckpt["model"])          # model FIRST
+opt.load_state_dict(ckpt["optimizer"])
+sched.load_state_dict(ckpt["scheduler"])
+scaler.load_state_dict(ckpt["scaler"])
+start_epoch = ckpt["epoch"] + 1
+```
+
+## Reproducibility harness (with the honest caveat)
+
+`pytorch.org/docs/stable/notes/randomness.html`: *"Completely reproducible results are not
+guaranteed across PyTorch releases, individual commits, or different platforms … results may not be
+reproducible between CPU and GPU executions, even when using identical seeds."* Seed for
+debugging/regression, never promise bit-exactness across machines.
+
+```python
+import os, random, numpy as np, torch
+
+def make_reproducible(seed=42, strict=False):
+    random.seed(seed); np.random.seed(seed); torch.manual_seed(seed)  # CPU + all CUDA devices
+    if strict:
+        torch.use_deterministic_algorithms(True)   # raises on ops lacking a deterministic path
+        torch.backends.cudnn.deterministic = True
+        torch.backends.cudnn.benchmark = False      # benchmark=True autotunes -> nondeterministic
+        os.environ.setdefault("CUBLAS_WORKSPACE_CONFIG", ":4096:8")  # CUDA>=10.2; set before launch
+
+# DataLoader workers fork their own RNG — seed each one, plus a seeded generator.
+def seed_worker(worker_id):
+    ws = torch.initial_seed() % 2**32
+    np.random.seed(ws); random.seed(ws)
+
+g = torch.Generator(); g.manual_seed(42)
+loader = torch.utils.data.DataLoader(ds, batch_size=64, num_workers=4,
+                                     worker_init_fn=seed_worker, generator=g)
+```
+
+`strict=True` (deterministic algorithms + `benchmark=False`) trades speed for reproducibility —
+use while debugging, drop for throughput runs.
+
+## Debug ladder — "loss won't go down" / "loss is NaN"
+
+Work top-down; each rules out a class of bug.
+
+1. **Overfit one batch.** Loop on a single mini-batch; loss must reach ~0. If it can't, the bug is
+   in the model/loss/data wiring, not optimization.
+2. **Check the loss input.** `CrossEntropyLoss`/`BCEWithLogitsLoss` want **raw logits** — a Softmax
+   before them silently stalls learning. Confirm target shape/dtype (class indices, not one-hot,
+   for `CrossEntropyLoss`).
+3. **Confirm grads flow.** After `backward()`, inspect a param's `.grad`: all `None`/zero means the
+   graph is detached (a stray `.detach()`, `.item()`, `.numpy()`, or `no_grad` in the path) or you
+   forgot `zero_grad` and are reading stale state.
+4. **LR sweep.** NaN early → LR too high (drop 10×) or missing fp16 scaler. No movement → LR too low
+   or a dead activation (all-zero ReLU). Add `clip_grad_norm_(…, 1.0)`.
+5. **Precision.** fp16 NaN/Inf → ensure `GradScaler` is active, or switch to **bf16** (wider range,
+   no scaler). Check for `log(0)`/`sqrt(neg)`/`0/0` in any custom loss.
+6. **Data.** NaNs/Infs in inputs, unnormalized features, or shuffled labels. Verify a few batches by
+   eye.
+7. **eval vs train mismatch.** Great train loss, bad val → you likely forgot `model.eval()` (Dropout
+   still on, BatchNorm using batch stats) or you're leaking train data into val.

package/skills/finetuning/SKILL.md

@@ -0,0 +1,244 @@
+---
+name: finetuning
+description: "Use when adapting an open-weight model to a target FORM or BEHAVIOR — tone, output format, domain style, a reasoning/tool-calling pattern — via LoRA/QLoRA or full fine-tuning with TRL SFTTrainer and then preference optimization (DPO/ORPO/KTO/GRPO); when choosing fine-tune vs prompt vs RAG vs longer context; when setting LoRA r/alpha/target_modules; when eval loss climbs while train loss keeps falling; when the adapter seems to have learned nothing; when inference outputs garbage after a run. Triggers: 'fine-tune Llama/Qwen on our data', 'LoRA vs QLoRA vs full fine-tune', 'SFT then DPO/ORPO/KTO/GRPO', 'set target_modules / rank / alpha', 'my fine-tune overfits', 'the model forgot everything after training', 'write a GRPO reward function', 'ajustar un modelo amb LoRA amb les nostres dades'. NOT adding facts or fresh knowledge to a model (that is rag); NOT the fast single-GPU Unsloth backend or GGUF export (that is unsloth); NOT serving the trained model (that is vllm)."
+tags: [finetuning, lora, qlora, sft, dpo, grpo, peft, trl, preference-optimization]
+recommends: [training-data, unsloth, open-weights, huggingface, vllm, rag]
+origin: risco
+---
+
+# finetuning — teach an open model a form or behavior, not a fact
+
+You own the discipline of **adapting an open-weight model**: deciding whether to fine-tune at all,
+then running SFT and (optionally) preference optimization with `trl` + `peft`, backend-agnostic.
+You are judged by whether the tuned model reliably produces the target **form/behavior** on a
+held-out set — not by train loss, and not by vibes.
+
+The one sentence that routes half of all "should I fine-tune?" questions correctly:
+**fine-tuning teaches *form and behavior*; RAG supplies *facts*.** If the ask is "know our latest
+prices / docs / tickets," that is retrieval (`../rag/SKILL.md`), not training. If the ask is "sound
+like us, always emit this JSON, follow this reasoning pattern," that is here.
+
+## Decision gate — try this BEFORE reaching for a GPU
+
+Fine-tuning is the last lever, not the first. Exhaust the cheaper, reversible options first; each
+row below is a real off-ramp.
+
+| If the goal is… | Do this first | Fine-tune only when… |
+|---|---|---|
+| The model should *know* current/company facts | **RAG** (`../rag/SKILL.md`) — retrieve + ground | never for facts; facts go stale, weights don't update |
+| One-off format/tone, small volume | **Prompt + few-shot** (`prompt-engineering`) | the prompt is huge, brittle, or you pay for it every call |
+| Behavior depends on a long document | **Longer context** / put it in the prompt | context won't fit, or per-call token cost is the bottleneck |
+| Consistent *form/behavior* at scale, latency/cost sensitive | — | prompting plateaus AND you have (or can build) good examples |
+| A capability the base model just can't do | — | you have a reward signal or demonstration data for it |
+
+Route out explicitly. Facts / freshness / citations → `../rag/SKILL.md`. Squeezing a prompt before
+spending money → `prompt-engineering`. Picking *which* base model (size/license/task) → `open-weights`.
+Building the JSONL/preference corpus → `training-data`. A fast single-GPU run + GGUF export →
+`../unsloth/SKILL.md`. Serving the result → `../vllm/SKILL.md`.
+
+> The cheapest fine-tune is the one you didn't need. Prompt + RAG solves most "make it behave"
+> asks at zero training cost and updates instantly. Fine-tune when that ceiling is real, measured,
+> and you can afford to re-run it every time the base model or data changes.
+
+## Version reality (verify at author time — this stack moves monthly)
+
+- **`trl`** consolidated into a **v1.x** line (v1.0 landed ~2026; docs at author time referenced
+  ~v1.8). Every method has a `Trainer` + a `Config` dataclass that inherits
+  `transformers.TrainingArguments` (`SFTTrainer`/`SFTConfig`, `DPOTrainer`/`DPOConfig`, …). Confirm
+  the current major before pinning: `pip show trl` / the [TRL docs](https://huggingface.co/docs/trl).
+- **`transformers`** is on a **v5.x** line; `peft`, `bitsandbytes`, `accelerate`, `datasets`
+  round out the stack. Do not freeze a pin as "the version" — say "current major is ~X, verify."
+- Some methods migrated to `trl.experimental.*` (e.g. `from trl.experimental.orpo import ORPOTrainer`
+  at author time). Import paths churn — check the method's doc page before copying an import.
+- **Model licenses are not facts to memorize.** Llama ships under the Meta *Community* license
+  (usage caps, not OSI-open); Gemma under custom Google terms; Qwen/Mistral vary per size and often
+  Apache-2.0 — but **read the specific model card, licenses change.** License/size selection is
+  `open-weights`.
+
+## LoRA / QLoRA vs full fine-tuning
+
+Three options on one memory↔quality axis. Default to **QLoRA** unless you have a proven reason not to.
+
+| Method | What trains | Rough VRAM (7–8B) | Use when |
+|---|---|---|---|
+| **Full FT** | every weight, fp16/bf16 | very high (needs multi-GPU / offload) | you have the hardware and a large, high-quality corpus and adapters underfit |
+| **LoRA** | small low-rank adapter matrices; base frozen (fp16) | high | base fits in fp16 and you want adapter portability + speed |
+| **QLoRA** | LoRA adapters over a **4-bit NF4** frozen base | lowest — single consumer GPU for 7–13B | the default; fine-tune big models on one GPU with ~no quality loss |
+
+**QLoRA** (Dettmers et al., [arXiv:2305.14314](https://arxiv.org/abs/2305.14314)): load the base
+in 4-bit **NF4** with double quantization, keep it frozen, and train LoRA adapters in bf16 on top.
+It made single-GPU fine-tuning of large models practical at near-full-FT quality.
+
+```python
+import torch
+from transformers import BitsAndBytesConfig
+from peft import LoraConfig
+from trl import SFTTrainer, SFTConfig
+
+# 4-bit NF4 base (QLoRA). Verify arg names against current bitsandbytes/transformers.
+bnb = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_use_double_quant=True,
+    bnb_4bit_compute_dtype=torch.bfloat16,
+)
+
+# LoRA over ALL linear layers — the safe target for QLoRA (PEFT quantization guide).
+peft_config = LoraConfig(
+    r=16, lora_alpha=32, lora_dropout=0.05,
+    bias="none", task_type="CAUSAL_LM",
+    target_modules="all-linear",   # or explicit ["q_proj","k_proj","v_proj","o_proj",...]
+)
+
+trainer = SFTTrainer(
+    model="Qwen/Qwen2.5-7B-Instruct",       # instruct base → chat template already present
+    args=SFTConfig(output_dir="out", max_length=2048, packing=True,
+                   learning_rate=2e-4, num_train_epochs=2, bf16=True),
+    train_dataset=dataset,                    # conversational: TRL applies the chat template
+    peft_config=peft_config,
+    quantization_config=bnb,                  # SFTTrainer + peft_config + this == QLoRA
+)
+trainer.train()
+trainer.save_model("out")                     # saves the ADAPTER, not a merged model
+```
+
+`peft_config` + `quantization_config` on `SFTTrainer` is the current one-liner for QLoRA — no manual
+`get_peft_model` / `prepare_model_for_kbit_training` wiring needed. `save_model` writes the small
+adapter; merge to a standalone model only when serving requires it (see references).
+
+## Pipeline: SFT first, then (maybe) preference optimization
+
+`SFT → preference optimization` is the standard post-training arc. **SFT** teaches the model the
+format and gives it the behavior by imitation. **Preference optimization** then sharpens *which* of
+several plausible outputs is better. Most projects need only SFT; add a preference stage when "the
+outputs are fine but I want the *good* one preferred" is the remaining gap.
+
+| Method | Data it needs | Stage | Pick it when |
+|---|---|---|---|
+| **SFT** | demonstrations (chat/`messages` or prompt→completion) | base of everything | always first (except ORPO) |
+| **DPO** ([2305.18290](https://arxiv.org/abs/2305.18290)) | paired `chosen`/`rejected` | after SFT | you have pairwise preferences; the workhorse aligner |
+| **ORPO** ([2403.07691](https://arxiv.org/abs/2403.07691)) | paired preferences | **replaces** SFT+DPO (single stage, ref-free) | you want one pass from a base model and have pairs |
+| **KTO** ([2402.01306](https://arxiv.org/abs/2402.01306)) | *unpaired* binary good/bad labels | after SFT | you have thumbs-up/down, not matched pairs |
+| **GRPO** ([2402.03300](https://arxiv.org/abs/2402.03300); DeepSeek-R1 [2501.12948](https://arxiv.org/abs/2501.12948)) | a **reward function** (verifier), no pairs | after SFT | correctness is checkable (math/code/format) → RL for reasoning |
+
+Rule of thumb: **have pairs → DPO** (or ORPO to fuse the two stages); **have only up/down votes → KTO**;
+**can score an answer programmatically → GRPO**. Preference optimization uses a *tiny* learning rate.
+
+```python
+# DPO after SFT — dataset has prompt / chosen / rejected columns.
+from trl import DPOTrainer, DPOConfig
+trainer = DPOTrainer(
+    model="out",                              # your SFT checkpoint (or SFT+adapter)
+    args=DPOConfig(output_dir="dpo-out", beta=0.1,   # beta = KL strength to the ref model
+                   learning_rate=5e-7, max_length=1024,
+                   precompute_ref_log_probs=True),   # saves memory; ref model auto-created
+    train_dataset=pref_dataset,
+    peft_config=peft_config,                  # LoRA works for preference stages too
+)
+trainer.train()
+```
+
+```python
+# GRPO — no preference pairs, a reward FUNCTION that returns a score per completion.
+from trl import GRPOTrainer, GRPOConfig
+def format_reward(completions, **kwargs):     # signature: gets completions (+ dataset cols via kwargs)
+    return [1.0 if "\\boxed{" in c[0]["content"] else 0.0 for c in completions]
+
+trainer = GRPOTrainer(
+    model="out",
+    reward_funcs=[format_reward],             # one or many; GRPOConfig.reward_weights to combine
+    args=GRPOConfig(output_dir="grpo-out", num_generations=8,  # group size per prompt
+                    beta=0.04, learning_rate=1e-6, use_vllm=True),  # vLLM speeds rollouts
+    train_dataset=prompts_dataset,
+)
+trainer.train()
+```
+
+Full runnable SFT→DPO and GRPO scripts, ORPO/KTO variants, dataset schemas, and adapter-merge steps
+are in `references/methods.md`.
+
+## Data — quality over quantity (LIMA)
+
+More rows is not the win. **LIMA** ([arXiv:2305.11206](https://arxiv.org/abs/2305.11206)) got strong
+instruction-following from **~1,000** carefully curated examples — "less is more for alignment."
+A thousand clean, on-distribution, correctly-templated examples beat 100k scraped noisy ones, which
+actively teach the model bad form. Building and validating that corpus (JSONL `messages`, preference
+pairs, dedup, contamination checks) is `training-data` — bring it here already clean.
+
+## Hyperparameters — the few that move the needle
+
+- **LoRA `r` and `lora_alpha`**: `r` = adapter rank (capacity); effective scaling = `lora_alpha / r`.
+  Common heuristic **`alpha ≈ 2·r`** (e.g. r=16→alpha=32) so scaling ≈ 2; then adjust LR, not both.
+  Start `r=8–16` for style/format, higher (32–64+) for harder behavior. (Newer "LoRA-without-regret"
+  guidance favors `target_modules="all-linear"` + higher rank + tuned LR — verify current advice.)
+- **`target_modules`**: `"all-linear"` is the safe default. Targeting too few / wrong-named modules
+  is a top silent failure — the run "succeeds," loss barely moves, the adapter learned ~nothing.
+- **Learning rate**: LoRA/QLoRA SFT ~**1e-4–2e-4** (adapters tolerate higher LR than full FT, whose
+  ~2e-5 is the `SFTConfig` default). Preference optimization is far lower — **DPO ~5e-7, GRPO ~1e-6**.
+- **Epochs**: usually **1–3**. This is the overfitting knob — see below. Watch **eval** loss.
+- **Batch × grad-accum**: raise the *effective* batch with `gradient_accumulation_steps` when VRAM
+  caps `per_device_train_batch_size`. Enable `packing=True` + `gradient_checkpointing` to fit more.
+- **Warmup**: a short `warmup_ratio` (~0.03–0.1) stabilizes the early, high-gradient steps.
+
+## Catastrophic forgetting
+
+Fine-tuning on a narrow task can degrade general ability the base model had. Three mitigations,
+cheapest first: **use LoRA/QLoRA** (base weights frozen — inherently gentler than full FT); **keep
+the LR low and epochs few**; and **replay** — mix a slice of general instruction data into your
+task data so the model doesn't forget how to be a general assistant. If a tuned model suddenly
+"got dumber" at everything else, this is the usual cause.
+
+## Evaluation — a held-out set and a task metric, not a vibe-check
+
+A vibe-check is **not** an eval. Before training, split off a **held-out** set the model never sees,
+and define a concrete task metric (exact-match / JSON-valid rate / rubric score / a task-specific
+score). Judge the run on that, plus **eval loss**.
+
+- **Watch eval loss, not train loss.** Train loss falling while **eval loss rises** = overfitting →
+  fewer epochs, lower LR, more/cleaner data, or earlier checkpoint. Train loss always keeps falling;
+  it tells you nothing about generalization.
+- **Contamination**: if eval examples leaked into training, your metric is a lie. De-dup train vs
+  eval; keep the held-out set quarantined. (Corpus-side hygiene is `training-data`.)
+- General LLM/agent eval harness and judging → `agent-eval`. Bring your task metric here.
+
+## When NOT to fine-tune (anti-patterns)
+
+| Anti-pattern | Why it breaks | Do instead |
+|---|---|---|
+| Fine-tune to add facts / fresh knowledge | Weights memorize poorly and go stale; hallucinations | `../rag/SKILL.md` — retrieve + ground |
+| Fine-tune before trying prompt + few-shot | Slow, costly, irreversible for a prompt-solvable ask | `prompt-engineering` first |
+| Wrong / too-few `target_modules` | Adapter has no capacity where it matters → learns ~nothing | `"all-linear"` (or correct proj names) |
+| Judge success by train loss | Falls even while the model overfits | Held-out eval set + task metric + **eval** loss |
+| Crank epochs "to learn it better" | Overfits, forgets, memorizes noise | 1–3 epochs; stop when eval loss turns up |
+| Train with a wrong/absent chat template | Inference emits garbage / never stops | Match train template to serve; align `eos_token` |
+| A few dozen examples for full FT | Not enough signal; unstable | Curate ~hundreds–thousands (LIMA) or use LoRA |
+| Fine-tune a model you can't legally deploy | License blocks your use case | Check the model card first → `open-weights` |
+
+## Related skills
+
+- **`../rag/SKILL.md`** — the fork of the decision gate: facts/freshness/citations live there;
+  form/behavior lives here. Most "make it behave" asks are RAG or prompting, not training.
+- **`prompt-engineering`** — the cheaper lever to exhaust before spending a GPU.
+- **`training-data`** — builds/validates the JSONL `messages` and preference-pair corpus you feed
+  here (LLM corpora, NOT tabular cleaning — that is `data-cleaning`).
+- **`open-weights`** — chooses *which* base model by task/size/license; you consume that choice.
+- **`../unsloth/SKILL.md`** — a fast single-GPU training backend + GGUF export; same LoRA/QLoRA
+  concepts, one optimized implementation. This skill stays backend-agnostic.
+- **`huggingface`** — download the base, push the adapter/merged model, host inference (NOT training).
+- **`../vllm/SKILL.md`** — serves the trained/merged model (adapter hot-swap, throughput flags).
+
+## Checklist
+
+- [ ] Ran the decision gate: confirmed this is a form/behavior need, not a facts need (else → rag).
+- [ ] Chose base model by license + size (→ `open-weights`); read the actual model card.
+- [ ] Picked method: QLoRA by default; full FT only with the hardware + a reason.
+- [ ] Data is clean, on-distribution, correctly templated (→ `training-data`); dedup vs eval.
+- [ ] Chat template applied at train time and it **matches** the serving template; `eos_token` aligned.
+- [ ] `target_modules="all-linear"` (or verified names); `alpha ≈ 2·r`; LoRA LR ~1e-4–2e-4.
+- [ ] Held-out eval set + a concrete task metric defined **before** training.
+- [ ] Trained 1–3 epochs; watched **eval** loss (not train) for the overfitting turn.
+- [ ] Preference stage only if needed; correct method for the data (pairs→DPO/ORPO, votes→KTO, verifier→GRPO); tiny LR.
+- [ ] Verified `trl`/`peft`/`transformers` current majors and import paths at author time.
+
+See `references/methods.md` for full SFT→DPO/GRPO/ORPO/KTO scripts, dataset schemas, and adapter
+merging; `references/hyperparameters-and-eval.md` for the tuning + forgetting + evaluation playbook.

package/skills/finetuning/evals/README.md

@@ -0,0 +1,13 @@
+# finetuning evals — how to run
+
+Routing and capability checks, not unit tests. Run the `should_trigger` prompts against your skill
+router and confirm `finetuning` fires (form/behavior adaptation, method choice, pipeline, and the
+overfit/forgetting/target_modules diagnostics). Run `should_not_trigger` and confirm the router
+defers to the named sibling instead (`rag` for facts, `prompt-engineering` for the cheaper lever,
+`open-weights` for base-model choice, `training-data` for corpus building, `vllm` for serving,
+`unsloth` for the fast single-GPU backend) — a miss there means the boundary in the description
+needs sharpening. For the `capability` case, hand the scenario to an agent with this skill loaded
+and grade the answer against each `must_include` bullet pass/fail: the decision gate (form vs
+facts), QLoRA + TRL/PEFT current API, the alpha≈2r and LR/epoch hyperparameters, the chat-template
+guardrail, LIMA quality-over-quantity, held-out eval watching eval loss, KTO for unpaired feedback,
+and the version/license hedge.

package/skills/finetuning/evals/cases.yaml

@@ -0,0 +1,50 @@
+skill: finetuning
+
+should_trigger:
+  - prompt: "We want Qwen to always answer in our brand voice and emit a fixed JSON schema — fine-tune it on ~2k of our examples."
+    why: A form/behavior adaptation (tone + output format) with demonstrations — the core SFT use case finetuning owns.
+  - prompt: "Should I use LoRA, QLoRA, or full fine-tuning to adapt an 8B model on a single 24GB GPU?"
+    why: The memory/quality method choice (LoRA vs QLoRA vs full FT) is squarely this skill's decision.
+  - prompt: "I ran SFT and now want to prefer the better of two answers — set up DPO after it with our chosen/rejected pairs."
+    why: SFT → preference optimization pipeline; picking DPO for paired preference data is finetuning's job.
+  - prompt: "My fine-tune's train loss keeps dropping but eval loss is climbing and it forgot general chat — what's wrong?"
+    why: Overfitting (watch eval not train loss) plus catastrophic forgetting — both are finetuning diagnostics.
+  - prompt: "My LoRA run finished but the adapter barely changed anything — did I set target_modules wrong?"
+    why: The classic silent failure (wrong/too-few target_modules → adapter learns ~nothing) lives here.
+  - prompt: "Correctness is checkable for our math tasks — write a GRPO reward function to RL the model toward right answers."
+    why: GRPO with a reward function for a verifiable task is a finetuning preference-optimization method.
+  - prompt: "Ajustar un model Qwen amb LoRA amb les nostres dades perquè segueixi el nostre estil."
+    why: Catalan — LoRA fine-tuning to teach a house style is a form/behavior adaptation this skill owns.
+
+should_not_trigger:
+  - prompt: "The bot needs to answer from our latest policy PDFs and cite them — it keeps giving stale info."
+    route_to: rag
+    why: Facts/freshness/citations are supplied by retrieval, not baked into weights — fine-tuning teaches form, not facts.
+  - prompt: "The prompt works sometimes; can I get consistent JSON with a few-shot example instead of training?"
+    route_to: prompt-engineering
+    why: The cheaper, reversible lever to exhaust before spending a GPU; no training needed.
+  - prompt: "Which open model should I pick by license and size for commercial use — Llama vs Qwen vs Mistral?"
+    route_to: open-weights
+    why: Base-model selection by license/size/task is the choice layer feeding finetuning, not the training method.
+  - prompt: "Build and dedup our instruction JSONL and preference pairs corpus and check for contamination."
+    route_to: training-data
+    why: Constructing/validating the training corpus is training-data; finetuning consumes an already-clean corpus.
+  - prompt: "Run our fine-tuned model as a high-throughput API with batching and adapter hot-swapping."
+    route_to: vllm
+    why: Serving the trained model (throughput, adapter hot-swap) is vllm, not the training discipline.
+  - prompt: "Give me the fastest single-GPU way to fine-tune and export a GGUF for llama.cpp."
+    route_to: unsloth
+    why: A specific fast single-GPU backend plus GGUF export is unsloth; finetuning stays backend-agnostic.
+
+capability:
+  - scenario: "A team wants an 8B open model to consistently reply in their support tone and always return a fixed JSON envelope, on one 24GB GPU. They have ~1,500 curated examples and later some thumbs-up/down feedback. Design the approach end to end."
+    must_include:
+      - Applies the decision gate and confirms this is a form/behavior task (tone + format), explicitly contrasting it with RAG (facts) and prompting.
+      - Chooses QLoRA (4-bit NF4 base + LoRA adapters, arXiv 2305.14314) as the memory-efficient default for a single 24GB GPU.
+      - Uses TRL SFTTrainer with an SFTConfig and a PEFT LoraConfig, passing peft_config (and quantization_config for QLoRA); target_modules="all-linear".
+      - States LoRA hyperparameters: r/alpha with the alpha≈2r heuristic and a LoRA-appropriate learning rate (~1e-4–2e-4), 1–3 epochs.
+      - Warns that the chat template at train time must match serving and eos_token must be aligned, or inference outputs garbage.
+      - Notes the ~1,500 curated examples fit LIMA's quality-over-quantity finding (2305.11206) and routes corpus building to training-data.
+      - Defines evaluation on a held-out set with a task metric and says to watch EVAL loss (not train loss) for overfitting, plus contamination check.
+      - For the later thumbs-up/down feedback, recommends KTO (unpaired binary preference) rather than DPO (which needs paired chosen/rejected).
+      - Hedges that trl/peft/transformers versions and import paths (and model licenses) must be verified at author time as they change frequently.