@dsalazardev/ai-hub-cli 0.0.0-stage → 0.2.0

package/skills/shared/deep-learning/SKILL.md

@@ -0,0 +1,226 @@
+---
+name: deep-learning
+description: "Use when training or debugging a neural net in PyTorch — the forward/loss/backward/step loop and its silent bugs, mixed precision (AMP), AdamW/LR schedules, DDP/FSDP/ZeRO, checkpoints and seeds. NOT LoRA/QLoRA on a pretrained LLM (that is `finetuning`), NOT tabular sklearn/XGBoost (that is `machine-learning`), NOT tokenization or NLP metrics (that is `nlp`)."
+tags: [pytorch, deep-learning, neural-networks, training-loop, autograd, mixed-precision, amp, ddp, fsdp, optimizer, lr-scheduler, reproducibility, torch-compile]
+recommends: [machine-learning, finetuning, nlp, python]
+origin: ai-hub
+---
+
+# deep-learning — train a neural net in PyTorch without the silent bugs
+
+You own **training and understanding neural networks in PyTorch**: the loop, autograd, mixed
+precision, optimizers/schedulers, multi-GPU, and the reproducibility/checkpoint hygiene that
+separates a real result from a lucky one. Nets from scratch, vision, custom architectures — all
+here. This is PyTorch-first by design: JAX and TensorFlow are real and fine, but the patterns,
+APIs, and gotchas below are Torch's.
+
+**Version reality (verify at author time).** Current stable is PyTorch **2.x** — ~**2.13** as of
+mid-2026 (`pytorch.org/get-started`, releases move fast; don't hard-pin a minor). Everything below
+is stable 2.x API. The one namespace shift to know: AMP now lives under `torch.amp`
+(`torch.amp.GradScaler("cuda")`), not the old `torch.cuda.amp.*`.
+
+## Am I in the right skill?
+
+| You are doing… | Skill |
+|---|---|
+| Training/debugging a net in PyTorch (from scratch, vision, custom loop, AMP, multi-GPU) | **deep-learning (here)** |
+| Adapting a *pretrained LLM* — LoRA/QLoRA, SFT, trl/peft | [`finetuning`](../finetuning/SKILL.md) |
+| Classic/tabular — sklearn, XGBoost/LightGBM, feature engineering | [`machine-learning`](../machine-learning/SKILL.md) |
+| Tokenization, NLP task modeling, task metrics (F1/BLEU/ROUGE) | [`nlp`](../nlp/SKILL.md) |
+| Envs, packaging, tests and hygiene around the model code | [`python`](../python/SKILL.md) |
+
+## 1. PyTorch essentials
+
+Three objects carry everything.
+
+- **Tensor** — an n-d array on a `device` (`cpu`/`cuda`/`mps`) with a `dtype`. `requires_grad=True`
+  makes autograd track ops on it. `.to(device)` / `.detach()` / `.item()` are the moves you use
+  constantly; `.item()` pulls a Python scalar and **drops the graph** (see the loop bugs below).
+- **autograd** — a tape. Every op on a `requires_grad` tensor records a node; `loss.backward()`
+  walks it and *accumulates* into each leaf's `.grad`. "Accumulates" is the word that bites people
+  (§2). Wrap read-only regions in `torch.no_grad()` to skip taping.
+- **`nn.Module`** — the model container. `__init__` registers submodules/params; `forward` defines
+  compute. `model.parameters()` feeds the optimizer; `model.train()` / `model.eval()` flip
+  train-vs-eval behavior for Dropout and BatchNorm.
+
+```python
+import torch, torch.nn as nn
+device = "cuda" if torch.cuda.is_available() else "cpu"
+
+class MLP(nn.Module):
+    def __init__(self, d_in, d_h, d_out, p=0.1):
+        super().__init__()
+        self.net = nn.Sequential(nn.Linear(d_in, d_h), nn.ReLU(),
+                                 nn.Dropout(p), nn.Linear(d_h, d_out))
+    def forward(self, x):
+        return self.net(x)
+
+model = MLP(784, 256, 10).to(device)
+```
+
+**`torch.compile`** — wrap the model once and get a JIT-optimized (Dynamo→Inductor) graph, often a
+real speedup with no code change: `model = torch.compile(model)`. Modes: `"default"` (balanced),
+`"reduce-overhead"` (CUDA graphs, best for small batches), `"max-autotune"` (Triton autotuning,
+slowest to warm up). First step(s) are slow — that's compilation, not your model. Keep it *after*
+`.to(device)` and before the loop. (`pytorch.org/docs/stable/generated/torch.compile.html`)
+
+## 2. The training loop — and its classic bugs
+
+The whole game is five ordered steps. Get the order or the resets wrong and it fails *silently* —
+no exception, just a model that won't learn.
+
+```python
+model.train()                                    # Dropout/BN in TRAIN mode
+for epoch in range(epochs):
+    for x, y in train_loader:
+        x, y = x.to(device), y.to(device)
+        optimizer.zero_grad(set_to_none=True)    # clear LAST step's grads
+        out  = model(x)                          # forward
+        loss = loss_fn(out, y)                    # loss  (raw logits in, e.g. CrossEntropyLoss)
+        loss.backward()                          # autograd fills .grad (ACCUMULATES)
+        optimizer.step()                         # update weights from .grad
+        scheduler.step()                         # if a per-step scheduler
+```
+
+The load-bearing traps, stated next to the lines they govern:
+
+- **Missing `optimizer.zero_grad()` → gradients silently add up across batches.** PyTorch
+  *accumulates* `.grad` by design (so you can split a batch); if you never zero it, every step
+  updates on the sum of all prior batches' grads and training destabilizes. Zero every step.
+  (`pytorch.org/tutorials` optimization tutorial; `torch.optim` docs.) `set_to_none=True` is the
+  default in modern Torch and is slightly faster.
+- **`CrossEntropyLoss` eats raw logits.** It applies `log_softmax` internally — do **not** put a
+  `Softmax`/`Sigmoid` before it or you double-activate and learning stalls. Same for
+  `BCEWithLogitsLoss`.
+- **Accumulating `loss` without `.item()` leaks memory.** `total += loss` keeps the whole autograd
+  graph alive across the epoch. Use `total += loss.item()` (or `.detach()`).
+- **`backward()` twice on one graph** raises unless you meant it (`retain_graph=True`) — usually a
+  sign you forgot to zero or re-ran forward.
+
+## 3. Mixed precision (AMP)
+
+Run forward/loss in low precision for speed and memory, keep a master copy for the update. Two
+pieces: `torch.autocast` (picks per-op precision) + `torch.amp.GradScaler` (rescues fp16 gradients
+from underflow). Verified against `pytorch.org/docs/stable/notes/amp_examples.html`.
+
+```python
+scaler = torch.amp.GradScaler("cuda")            # current namespace (not torch.cuda.amp)
+for x, y in train_loader:
+    x, y = x.to(device), y.to(device)
+    optimizer.zero_grad(set_to_none=True)
+    with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
+        out  = model(x)                          # forward + loss ONLY under autocast
+        loss = loss_fn(out, y)
+    scaler.scale(loss).backward()                # backward OUTSIDE autocast, on scaled loss
+    scaler.step(optimizer)                        # unscales, skips step if inf/NaN grads
+    scaler.update()                               # adjusts the scale for next step
+```
+
+**bf16 vs fp16 — the choice that matters:**
+
+| dtype | Exponent range | Needs GradScaler? | Use when |
+|---|---|---|---|
+| **bf16** (`bfloat16`) | same as fp32 | **No** (range can't underflow) | Ampere+ (A100/H100/…) or CPU — **prefer this**, it's the robust default |
+| **fp16** (`float16`) | narrow → underflows | **Yes** — scaler is mandatory | older GPUs without bf16; squeeze extra range with the scaler |
+
+- With **bf16 you can skip `GradScaler`** entirely (`GradScaler(enabled=False)` or just don't use
+  it) — its whole job is fp16 underflow. With **fp16 the scaler is not optional.**
+- **Gradient clipping under AMP:** `scaler.unscale_(optimizer)` first, *then*
+  `torch.nn.utils.clip_grad_norm_(...)`, then `scaler.step`. Clipping scaled grads clips the wrong
+  magnitude.
+- Autocast wraps **forward + loss only**. Never wrap `backward()`.
+
+## 4. Optimizers and LR schedules
+
+- **`torch.optim.AdamW`** is the default for most modern nets (transformers, and increasingly
+  vision): Adam with *decoupled* weight decay (correct L2, unlike plain `Adam(weight_decay=)`).
+  **SGD + momentum + Nesterov** still wins on many CNNs and generalizes well — worth trying.
+- **Don't weight-decay everything.** Biases and Norm (LayerNorm/BatchNorm) weights should get
+  `weight_decay=0`; use two param groups. Decaying them hurts with no upside.
+- **The LR is the hyperparameter.** Too high → loss NaNs or oscillates; too low → crawls. A warmup
+  then cosine decay is the reliable modern recipe.
+
+```python
+opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01, betas=(0.9, 0.999))
+
+# Warmup (linear ramp) THEN cosine decay, composed with SequentialLR — verified 2.x API.
+from torch.optim.lr_scheduler import LinearLR, CosineAnnealingLR, SequentialLR
+warmup = LinearLR(opt, start_factor=0.1, total_iters=500)          # 500 steps ramp-in
+cosine = CosineAnnealingLR(opt, T_max=total_steps - 500, eta_min=3e-5)
+scheduler = SequentialLR(opt, [warmup, cosine], milestones=[500])  # switch at step 500
+# call scheduler.step() every STEP here (T_max/total_iters are in steps).
+```
+
+Alternatively `OneCycleLR(opt, max_lr=..., total_steps=...)` bakes warmup+anneal into one
+scheduler. **Match `step()` cadence to the scheduler's units** — per-step schedulers stepped once
+per epoch (or vice-versa) silently give the wrong curve. (`pytorch.org/docs/stable/optim.html`)
+
+## 5. Distributed training — pick by what doesn't fit
+
+One line each on *when*; mechanics and launch commands in `references/distributed.md`.
+
+- **DDP** (`torch.nn.parallel.DistributedDataParallel`) — replicate the **whole model** on each GPU,
+  all-reduce grads every step. Use when **the model fits on one GPU** and you just want more
+  throughput / a larger effective batch. Default choice; one process per GPU via `torchrun`.
+- **FSDP2** (`torch.distributed.fsdp.fully_shard`) — **shard** parameters, gradients, and optimizer
+  states across GPUs (gather just-in-time). Use when **the model does NOT fit on one GPU.** FSDP2 is
+  the current DTensor-based, per-parameter API; apply it bottom-up (wrap layers, then the root).
+- **DeepSpeed ZeRO** (external Microsoft library, not in core Torch) — config-driven sharding:
+  **ZeRO-1** shards optimizer states, **ZeRO-2** adds gradients, **ZeRO-3** adds parameters
+  (≈ FSDP), plus CPU/NVMe **offload**. Use when you want offload or config-first scaling, or already
+  live in the DeepSpeed/Accelerate ecosystem. Verify the current version — it moves independently.
+
+Rule of thumb: fits on one GPU → **DDP**; too big → **FSDP2** (or ZeRO-3); need to offload to
+CPU/NVMe → **ZeRO**.
+
+## 6. Regularization, checkpointing, reproducibility
+
+**Regularization** (reach for these when val loss diverges from train — i.e. overfitting):
+weight decay (AdamW), `nn.Dropout`, data augmentation, early stopping on a val metric, label
+smoothing (`CrossEntropyLoss(label_smoothing=0.1)`), and gradient clipping for stability (not
+generalization).
+
+**Checkpointing — save `state_dict`s, not the model object.** Pickling the module ties the file to
+your code layout and breaks on refactor.
+
+```python
+torch.save({"epoch": epoch,
+            "model": model.state_dict(),          # DDP/FSDP: model.module.state_dict(), rank 0 only
+            "optimizer": opt.state_dict(),
+            "scheduler": scheduler.state_dict(),
+            "scaler": scaler.state_dict()},        # AMP scale must survive a resume
+           "ckpt.pt")
+# Resume: load model FIRST, then optimizer/scheduler; move optimizer state to device if needed.
+```
+
+Save optimizer + scheduler + scaler too, or a resumed run restarts its LR schedule and loss scale.
+
+**Reproducibility — seed everything, and know the ceiling.** From
+`pytorch.org/docs/stable/notes/randomness.html`, quoted: *"Completely reproducible results are not
+guaranteed across PyTorch releases, individual commits, or different platforms. Furthermore,
+results may not be reproducible between CPU and GPU executions, even when using identical seeds."*
+
+```python
+import random, numpy as np, torch
+seed = 42
+random.seed(seed); np.random.seed(seed); torch.manual_seed(seed)   # covers CPU + all CUDA devices
+torch.use_deterministic_algorithms(True)          # errors on ops with no deterministic impl
+torch.backends.cudnn.deterministic = True
+torch.backends.cudnn.benchmark = False            # benchmark=True picks fastest algo → nondeterministic
+# For CUDA >= 10.2, deterministic matmul also needs, set BEFORE launch:
+#   export CUBLAS_WORKSPACE_CONFIG=:4096:8
+```
+
+DataLoader workers need their own seeding (each worker forks its RNG), via `worker_init_fn` +
+a seeded `generator` — full snippet in `references/training-recipes.md`. Determinism costs speed
+(`benchmark=False` disables autotuned convs).
+
+## Anti-patterns (the silent killers)
+
+| Anti-pattern | What actually happens | Instead |
+|---|---|---|
+| Validating with the model left in `train()` mode, or without `torch.no_grad()` | Dropout keeps dropping and BatchNorm uses *batch* stats instead of running stats, so val numbers are wrong; the graph is still built and can OOM | `model.eval()` + `torch.no_grad()` around val/inference, `model.train()` after (`nn.Module.eval` / autograd docs, pytorch.org) |
+| Reading a NaN loss as a modeling problem | It is almost always LR too high, fp16 with no `GradScaler`, unclipped exploding grads, or a `log(0)`/`0/0` in a custom loss | Bisect: drop LR 10×, clip grads, switch fp16→bf16 |
+| fp16 with the scaler left off | Gradients underflow to zero → no learning, and no error to tell you | Keep `GradScaler` on fp16, or move to bf16 where underflow can't happen |
+| A stray CPU tensor in a CUDA graph | Hard device-mismatch error mid-step, usually on the targets or a hand-built mask rather than the inputs | `.to(device)` inputs, targets **and** model |
+| Promising bit-exact reproducibility from seeds | Not guaranteed across releases, commits, platforms, or CPU-vs-GPU even with identical seeds | Seed for debugging/regression and say so; state the caveat with the result (randomness note, pytorch.org) |

package/skills/shared/deep-learning/evals/README.md

@@ -0,0 +1,11 @@
+# Evals — deep-learning
+
+Routing and capability checks for a human or LLM grader, not an automated test suite. To run them:
+read each `should_trigger` prompt and confirm an agent would reach for `deep-learning` (PyTorch
+training/loop/AMP/distributed/reproducibility); read each `should_not_trigger` prompt and confirm it
+routes to the named sibling (`finetuning`, `machine-learning`, `nlp`, `vllm`, `python`) for the
+stated reason. Then grade the `capability` scenario against every `must_include` bullet — the
+correct five-step loop with per-step `zero_grad`, AMP with the bf16/fp16 + scaler rule, `model.eval()`
++ `no_grad()` at validation, AdamW + warmup→cosine, checkpoint hygiene, and the "determinism is not
+guaranteed across releases/platforms/devices" caveat. A pass covers all routing cases plus full
+rubric coverage on the capability scenario.

package/skills/shared/deep-learning/evals/cases.yaml

@@ -0,0 +1,48 @@
+skill: deep-learning
+
+should_trigger:
+  - prompt: "My PyTorch training loss isn't going down at all — the loop looks fine, what am I missing?"
+    why: "Core training-loop debugging in Torch; the diagnosis lands on the five-step loop and its silent bugs, this skill's center."
+  - prompt: "Loss becomes NaN after a few hundred steps when I turn on float16. How do I fix it?"
+    why: "AMP fp16 underflow / missing GradScaler and the bf16 alternative — the mixed-precision section owns this."
+  - prompt: "How do I use torch.autocast and GradScaler, and should I pick bf16 or fp16?"
+    why: "Explicit AMP API + the bf16-vs-fp16 decision this skill teaches."
+  - prompt: "I need to train this model across 4 GPUs — DDP or FSDP, and how do I launch it?"
+    why: "Distributed choice by memory-fit plus torchrun launch is a named section (and references/distributed.md)."
+  - prompt: "My validation accuracy is way worse than training and I think I forgot model.eval()."
+    why: "Non-obvious guardrail: eval() vs train() flips Dropout/BatchNorm; squarely a load-bearing gotcha here."
+  - prompt: "Gradients seem to add up across batches and training blows up — is my loop wrong?"
+    why: "Non-obvious: the symptom is a missing optimizer.zero_grad(), the flagship silent bug of this skill; no API named."
+  - prompt: "Build a CNN from scratch in PyTorch and write the training loop to fit it on my image dataset."
+    why: "From-scratch net + custom loop in Torch — inside the boundary vs finetuning/ML."
+  - prompt: "vull entrenar una xarxa neuronal des de zero amb PyTorch i que sigui reproduïble"
+    why: "Catalan trigger: from-scratch neural net + reproducibility, both owned here."
+
+should_not_trigger:
+  - prompt: "Fine-tune Llama with QLoRA on my instruction dataset using trl and peft."
+    route_to: finetuning
+    why: "Adapting a pretrained LLM with LoRA/QLoRA is the finetuning method skill, not general net training."
+  - prompt: "Should I use XGBoost or a random forest for this tabular churn dataset?"
+    route_to: machine-learning
+    why: "Classic/tabular GBDT modeling — no autograd, no PyTorch loop."
+  - prompt: "Which tokenizer and how do I compute F1/BLEU for my text classification task?"
+    route_to: nlp
+    why: "Tokenization and NLP task metrics live in nlp; only the raw training loop routes back here."
+  - prompt: "Serve my trained model with vLLM and set the right batching flags for throughput."
+    route_to: vllm
+    why: "Inference/serving engine, not training."
+  - prompt: "Set up a clean Python project with uv, pytest, and type checking for my model code."
+    route_to: python
+    why: "Env/packaging/testing hygiene, not the neural-net training itself."
+
+capability:
+  - scenario: "A user is training a from-scratch image classifier in PyTorch on one A100. Training loss is unstable and sometimes NaNs, validation numbers look wrong, and they want it faster and reproducible. Write the training setup and explain the fixes."
+    must_include:
+      - "Correct five-step loop with optimizer.zero_grad(set_to_none=True) EVERY step, explaining that PyTorch accumulates .grad so skipping zero_grad silently sums gradients across batches"
+      - "AMP via torch.autocast on forward+loss only with torch.amp.GradScaler; recommends bf16 on A100 (no scaler needed) and notes fp16 REQUIRES the scaler to avoid underflow NaNs"
+      - "Gradient clipping done after scaler.unscale_(optimizer), and CrossEntropyLoss fed raw logits (no Softmax before it)"
+      - "model.eval() plus torch.no_grad() for validation, stating it changes Dropout/BatchNorm behavior and prevents graph build, and model.train() to switch back — as the cause of bad val numbers"
+      - "AdamW with weight decay excluded from bias/Norm params, plus a warmup→cosine schedule (LinearLR + CosineAnnealingLR via SequentialLR) with step() cadence matching the scheduler units"
+      - "torch.compile(model) offered as the speedup, noting first steps are slow due to compilation"
+      - "Reproducibility: seed random/numpy/torch, torch.use_deterministic_algorithms(True), cudnn.deterministic/benchmark=False, CUBLAS_WORKSPACE_CONFIG, and the honest caveat that results are NOT guaranteed across PyTorch releases/platforms or CPU-vs-GPU even with identical seeds"
+      - "Checkpoints save model+optimizer+scheduler+scaler state_dicts (not the model object), resuming model first; version hedged to PyTorch 2.x (~2.13, verify)"

package/skills/shared/deep-learning/references/distributed.md

@@ -0,0 +1,105 @@
+# Distributed training — DDP, FSDP2, DeepSpeed ZeRO (mechanics)
+
+Grounded in `pytorch.org/docs/stable/` (DDP: `notes/ddp.html`, `generated/torch.nn.parallel.DistributedDataParallel.html`;
+FSDP2: `distributed.fsdp.fully_shard.html`). Verify API against the installed version — distributed
+APIs move faster than the core loop.
+
+## The decision, restated
+
+| Situation | Use | Why |
+|---|---|---|
+| Model fits on one GPU, want more throughput / bigger effective batch | **DDP** | Simplest; replicate + all-reduce grads |
+| Model (params+grads+optimizer state) does NOT fit on one GPU | **FSDP2** | Shards the three memory hogs across ranks |
+| Need CPU/NVMe offload, or config-first scaling, or already using Accelerate/DeepSpeed | **DeepSpeed ZeRO** | Stage 1/2/3 + offload, external library |
+
+Data parallel of any kind assumes each rank sees a **different shard of the batch** — use
+`DistributedSampler` so ranks don't all train on the same rows.
+
+## DDP — one process per GPU
+
+```python
+import os, torch, torch.distributed as dist
+from torch.nn.parallel import DistributedDataParallel as DDP
+
+dist.init_process_group("nccl")                        # NCCL for CUDA; "gloo" for CPU
+local_rank = int(os.environ["LOCAL_RANK"])             # set by torchrun
+torch.cuda.set_device(local_rank)
+
+model = MyModel().to(local_rank)
+model = DDP(model, device_ids=[local_rank])            # grads all-reduced automatically in backward
+
+from torch.utils.data import DataLoader
+from torch.utils.data.distributed import DistributedSampler
+sampler = DistributedSampler(train_ds)                 # disjoint shard per rank
+loader  = DataLoader(train_ds, sampler=sampler, batch_size=bs)
+
+for epoch in range(epochs):
+    sampler.set_epoch(epoch)                           # REQUIRED: reshuffles per epoch across ranks
+    for x, y in loader:
+        ...                                            # normal 5-step loop; DDP syncs grads for you
+
+if local_rank == 0:
+    torch.save(model.module.state_dict(), "ckpt.pt")   # .module unwraps DDP; save on rank 0 only
+dist.destroy_process_group()
+```
+
+Launch: `torchrun --standalone --nproc_per_node=4 train.py` (4 GPUs, one node). Multi-node adds
+`--nnodes`, `--node_rank`, `--rdzv_endpoint`.
+
+DDP gotchas:
+- **`sampler.set_epoch(epoch)`** or every epoch sees the same order — a silent correctness bug.
+- **`find_unused_parameters=True`** only if some params get no grad each step (costs a graph
+  traversal); leaving it True unnecessarily slows training.
+- Save/load on **rank 0** and unwrap with `.module`; barrier before others load.
+- Effective batch = `per_gpu_batch * world_size` — scale LR accordingly (linear-scaling rule).
+
+## FSDP2 — shard when it doesn't fit
+
+FSDP2 (`fully_shard`) is the current API: DTensor-based, per-parameter dim-0 sharding. Apply it
+**bottom-up** — wrap the repeated blocks first, then the root module — so each `fully_shard` call
+forms a communication group and gathers params just-in-time.
+
+```python
+from torch.distributed.fsdp import fully_shard
+
+for block in model.transformer_blocks:       # wrap leaves/blocks first
+    fully_shard(block)
+fully_shard(model)                            # then the root
+
+# Mixed precision is a policy here, not GradScaler:
+from torch.distributed.fsdp import MixedPrecisionPolicy
+for block in model.transformer_blocks:
+    fully_shard(block, mp_policy=MixedPrecisionPolicy(param_dtype=torch.bfloat16))
+```
+
+- `reshard_after_forward=True` (default for non-root) frees gathered params after forward and
+  re-gathers in backward — saves memory, costs an all-gather. `False` (default for root) keeps them.
+- Wrapping **only** the root makes one giant communication group with no compute/comm overlap —
+  wrap blocks to overlap.
+- FSDP2 uses AMP via `MixedPrecisionPolicy`, not `torch.amp.GradScaler`.
+- Full-state-dict save needs `DTensor.full_tensor()` or Distributed Checkpoint (DCP) APIs; prefer
+  sharded checkpoints for large models.
+
+## DeepSpeed ZeRO — external, config-driven
+
+DeepSpeed is a **separate library** (Microsoft), commonly driven through a JSON config and/or HF
+Accelerate. Verify the current version and config schema — it evolves independently of Torch.
+
+| Stage | Shards | Rough memory win vs DDP | Comm cost |
+|---|---|---|---|
+| ZeRO-1 | optimizer states | ~4× on optimizer state | ~DDP |
+| ZeRO-2 | + gradients | more | ~DDP |
+| ZeRO-3 | + parameters (≈ FSDP) | largest | higher (gather params) |
+| + offload | move state to CPU/NVMe | fit far bigger models | slowest (PCIe/disk bound) |
+
+Choose ZeRO over FSDP2 when you specifically want **offload** or a config-first workflow, or you're
+already in the DeepSpeed/Accelerate stack. For pure Torch, FSDP2 covers the same ZeRO-3 territory
+natively.
+
+## Cross-cutting
+
+- **Gradient accumulation** (micro-batching) simulates a bigger batch on limited memory: run N
+  forward/backward passes, then one `optimizer.step()` + `zero_grad()`. Under DDP, wrap the
+  non-final micro-steps in `model.no_sync()` to skip redundant all-reduces.
+- **`torch.compile` composes** with DDP and FSDP2, but compile after wrapping and expect a longer
+  warmup.

package/skills/shared/deep-learning/references/training-recipes.md

@@ -0,0 +1,135 @@
+# Training recipes — a full loop, reproducibility harness, and debug ladder
+
+Concrete, runnable single-GPU scaffolding to lift into a project. APIs verified against
+`pytorch.org/docs/stable/` (amp_examples, optim, notes/randomness). Hedge versions to "PyTorch 2.x".
+
+## A complete single-GPU training loop (AMP + AdamW + warmup→cosine + checkpoint)
+
+```python
+import torch, torch.nn as nn
+from torch.optim.lr_scheduler import LinearLR, CosineAnnealingLR, SequentialLR
+
+device = "cuda" if torch.cuda.is_available() else "cpu"
+model = MyModel().to(device)
+model = torch.compile(model)                 # optional speedup; first steps compile (slow)
+
+# No weight decay on bias/Norm params — two groups.
+decay, no_decay = [], []
+for n, p in model.named_parameters():
+    if not p.requires_grad:
+        continue
+    (no_decay if p.ndim <= 1 or n.endswith(".bias") else decay).append(p)
+opt = torch.optim.AdamW(
+    [{"params": decay, "weight_decay": 0.01},
+     {"params": no_decay, "weight_decay": 0.0}],
+    lr=3e-4, betas=(0.9, 0.999),
+)
+
+total_steps = len(train_loader) * epochs
+warmup = LinearLR(opt, start_factor=0.1, total_iters=min(500, total_steps // 20))
+cosine = CosineAnnealingLR(opt, T_max=total_steps - warmup.total_iters, eta_min=3e-5)
+sched  = SequentialLR(opt, [warmup, cosine], milestones=[warmup.total_iters])
+
+loss_fn = nn.CrossEntropyLoss(label_smoothing=0.1)     # takes RAW logits
+scaler  = torch.amp.GradScaler("cuda", enabled=(device == "cuda"))
+USE_BF16 = torch.cuda.is_bf16_supported() if device == "cuda" else False
+amp_dtype = torch.bfloat16 if USE_BF16 else torch.float16
+# NOTE: with bf16 the scaler is a no-op; keep enabled only for fp16.
+
+best_val = float("inf")
+for epoch in range(epochs):
+    model.train()
+    for x, y in train_loader:
+        x, y = x.to(device), y.to(device)
+        opt.zero_grad(set_to_none=True)
+        with torch.autocast(device_type=device, dtype=amp_dtype, enabled=(device == "cuda")):
+            loss = loss_fn(model(x), y)
+        scaler.scale(loss).backward()
+        scaler.unscale_(opt)                                    # unscale before clipping
+        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
+        scaler.step(opt)
+        scaler.update()
+        sched.step()                                            # per-step: units are steps
+
+    # ---- validation ----
+    model.eval()
+    val_loss, n = 0.0, 0
+    with torch.no_grad():                                       # no graph, no Dropout/BN train stats
+        for x, y in val_loader:
+            x, y = x.to(device), y.to(device)
+            val_loss += loss_fn(model(x), y).item() * x.size(0) # .item() -> no graph leak
+            n += x.size(0)
+    val_loss /= n
+
+    if val_loss < best_val:                                     # early-stopping signal + best ckpt
+        best_val = val_loss
+        torch.save({"epoch": epoch,
+                    "model": model.state_dict(),
+                    "optimizer": opt.state_dict(),
+                    "scheduler": sched.state_dict(),
+                    "scaler": scaler.state_dict(),
+                    "best_val": best_val}, "best.pt")
+```
+
+Resume:
+
+```python
+ckpt = torch.load("best.pt", map_location=device)
+model.load_state_dict(ckpt["model"])          # model FIRST
+opt.load_state_dict(ckpt["optimizer"])
+sched.load_state_dict(ckpt["scheduler"])
+scaler.load_state_dict(ckpt["scaler"])
+start_epoch = ckpt["epoch"] + 1
+```
+
+## Reproducibility harness (with the honest caveat)
+
+`pytorch.org/docs/stable/notes/randomness.html`: *"Completely reproducible results are not
+guaranteed across PyTorch releases, individual commits, or different platforms … results may not be
+reproducible between CPU and GPU executions, even when using identical seeds."* Seed for
+debugging/regression, never promise bit-exactness across machines.
+
+```python
+import os, random, numpy as np, torch
+
+def make_reproducible(seed=42, strict=False):
+    random.seed(seed); np.random.seed(seed); torch.manual_seed(seed)  # CPU + all CUDA devices
+    if strict:
+        torch.use_deterministic_algorithms(True)   # raises on ops lacking a deterministic path
+        torch.backends.cudnn.deterministic = True
+        torch.backends.cudnn.benchmark = False      # benchmark=True autotunes -> nondeterministic
+        os.environ.setdefault("CUBLAS_WORKSPACE_CONFIG", ":4096:8")  # CUDA>=10.2; set before launch
+
+# DataLoader workers fork their own RNG — seed each one, plus a seeded generator.
+def seed_worker(worker_id):
+    ws = torch.initial_seed() % 2**32
+    np.random.seed(ws); random.seed(ws)
+
+g = torch.Generator(); g.manual_seed(42)
+loader = torch.utils.data.DataLoader(ds, batch_size=64, num_workers=4,
+                                     worker_init_fn=seed_worker, generator=g)
+```
+
+`strict=True` (deterministic algorithms + `benchmark=False`) trades speed for reproducibility —
+use while debugging, drop for throughput runs.
+
+## Debug ladder — "loss won't go down" / "loss is NaN"
+
+Work top-down; each rules out a class of bug.
+
+1. **Overfit one batch.** Loop on a single mini-batch; loss must reach ~0. If it can't, the bug is
+   in the model/loss/data wiring, not optimization.
+2. **Check the loss input.** `CrossEntropyLoss`/`BCEWithLogitsLoss` want **raw logits** — a Softmax
+   before them silently stalls learning. Confirm target shape/dtype (class indices, not one-hot,
+   for `CrossEntropyLoss`).
+3. **Confirm grads flow.** After `backward()`, inspect a param's `.grad`: all `None`/zero means the
+   graph is detached (a stray `.detach()`, `.item()`, `.numpy()`, or `no_grad` in the path) or you
+   forgot `zero_grad` and are reading stale state.
+4. **LR sweep.** NaN early → LR too high (drop 10×) or missing fp16 scaler. No movement → LR too low
+   or a dead activation (all-zero ReLU). Add `clip_grad_norm_(…, 1.0)`.
+5. **Precision.** fp16 NaN/Inf → ensure `GradScaler` is active, or switch to **bf16** (wider range,
+   no scaler). Check for `log(0)`/`sqrt(neg)`/`0/0` in any custom loss.
+6. **Data.** NaNs/Infs in inputs, unnormalized features, or shuffled labels. Verify a few batches by
+   eye.
+7. **eval vs train mismatch.** Great train loss, bad val → you likely forgot `model.eval()` (Dropout
+   still on, BatchNorm using batch stats) or you're leaking train data into val.