@ericrisco/rsc 0.1.38 → 0.1.40

package/skills/finetuning/references/hyperparameters-and-eval.md

@@ -0,0 +1,74 @@
+# hyperparameters, forgetting & evaluation — the tuning playbook
+
+Depth overflow for `SKILL.md`. Defaults are starting points, not laws; the eval set is the judge.
+
+## LoRA capacity: r, alpha, target_modules
+
+The adapter adds two low-rank matrices per targeted layer; its output is scaled by
+**`lora_alpha / r`**. So `r` sets *capacity* and `alpha` sets *how strongly* the adapter speaks.
+
+- **`r`** — rank / capacity. `8–16` for style, tone, format nudges; `32–64+` for harder new
+  behavior or larger models. Bigger `r` = more trainable params = more VRAM and more overfit risk.
+- **`alpha`** — common heuristic **`alpha ≈ 2·r`** (scaling ≈ 2). Fix that ratio and tune the LR
+  rather than juggling both knobs at once.
+- **`target_modules`** — `"all-linear"` is the safe default and is the recommended setting for
+  QLoRA (LoRA on every linear layer). Naming only a couple of projections, or misspelling them, is
+  a top silent failure: the run completes, loss barely moves, the adapter learned ~nothing. Newer
+  "LoRA-without-regret" guidance favors all-linear + higher rank + a tuned LR — verify current advice.
+- **`lora_dropout`** ~`0.05`, **`bias="none"`**, **`task_type="CAUSAL_LM"`** are the usual defaults.
+
+## Learning rate, epochs, batch, warmup
+
+| Knob | SFT (LoRA/QLoRA) | Full FT | Preference (DPO/KTO/GRPO) |
+|---|---|---|---|
+| Learning rate | ~1e-4 – 2e-4 | ~2e-5 (SFTConfig default) | DPO/KTO ~5e-7 · GRPO ~1e-6 |
+| Epochs | 1–3 | 1–3 | usually 1 |
+| Warmup ratio | 0.03 – 0.1 | 0.03 – 0.1 | 0.03 – 0.1 |
+
+- **Effective batch = `per_device_train_batch_size × gradient_accumulation_steps × world_size`.**
+  When VRAM caps the per-device batch, raise `gradient_accumulation_steps` to reach a stable
+  effective batch (commonly 16–64) without more memory.
+- **Fit-more levers**: `packing=True` (fills sequences to `max_length`, less padding waste),
+  `gradient_checkpointing=True` (trade compute for memory), QLoRA 4-bit base, `activation_offloading`.
+- **Epochs is the overfitting dial.** Adapters memorize a small corpus fast; 1–3 passes is normal.
+  Prefer more/cleaner data over more epochs.
+
+## Catastrophic forgetting
+
+Narrow fine-tuning can erode general ability. Mitigations, cheapest first:
+
+1. **LoRA/QLoRA over full FT** — base weights stay frozen, so the model can't overwrite what it
+   knew; the adapter is a small, removable delta.
+2. **Low LR, few epochs** — big updates on a narrow task are what erase general skills.
+3. **Replay / data mixing** — blend a slice of general instruction data into the task data so the
+   model keeps its assistant behavior. A common mix is a minority fraction of general data.
+
+Symptom: the tuned model nails the target task but "got dumber" everywhere else. That is forgetting,
+not a data bug — pull LR/epochs down and add replay.
+
+## Evaluation — the part people skip and regret
+
+1. **Split a held-out set before training** the model never sees. Keep it quarantined.
+2. **Define a concrete task metric** up front: exact-match, JSON-valid rate, a rubric/LLM-judge
+   score, or a domain metric. "Looks good" is not a metric.
+3. **Watch eval loss, not train loss.** Train loss always falls. When **eval loss turns upward**
+   while train loss keeps dropping, you are overfitting — stop, lower LR, add data, or take an
+   earlier checkpoint. Set `eval_strategy="steps"` + `eval_steps` and log both curves.
+4. **Contamination kills the signal.** If eval examples leaked into training, the metric is a lie.
+   De-dup train against eval; corpus-side hygiene is `training-data`.
+5. **Compare against the untuned baseline** on the same held-out set — a fine-tune that doesn't
+   beat "base model + a good prompt" was not worth the GPU (loop back to the decision gate).
+
+The general LLM/agent eval harness (judges, scoring pipelines, regression gates) is `agent-eval`;
+bring your task-specific metric here and gate the run on it.
+
+## A pre-flight sanity pass (catch the silent failures early)
+
+- Print one fully-rendered training example (post chat-template) and read it — is the template
+  right, are special tokens present, is the assistant span the only thing in the loss?
+- Confirm trainable-param count is non-trivial (`model.print_trainable_parameters()` on the PEFT
+  model). Near-zero or "no trainable params" means `target_modules` missed.
+- Overfit a tiny batch (a few dozen rows) on purpose — train loss should crash toward zero. If it
+  won't, the wiring (template / targets / labels) is broken before you spend on the full run.
+- Generate from the checkpoint with the **serving** template and eyeball a handful of outputs for
+  the failure mode you're fixing.

package/skills/finetuning/references/methods.md

@@ -0,0 +1,189 @@
+# methods — full SFT → preference scripts, dataset schemas, adapter merge
+
+Depth overflow for `SKILL.md`. All snippets target the `trl` **v1.x** API (verify the current
+major and any `trl.experimental.*` import paths at author time — this stack changes monthly).
+
+## Dataset schemas the trainers expect
+
+TRL auto-applies the model's **chat template** when a dataset is *conversational* (`messages` or
+role dicts). This is the single most important correctness point: train with the same template you
+serve with, or inference is garbage.
+
+```python
+# SFT — conversational language modeling (chat template applied automatically)
+{"messages": [{"role": "user", "content": "What color is the sky?"},
+              {"role": "assistant", "content": "It is blue."}]}
+
+# SFT — prompt/completion (loss on completion only by default)
+{"prompt": "The sky is", "completion": " blue."}
+
+# DPO / ORPO — paired preference (explicit prompt recommended)
+{"prompt":  [{"role": "user", "content": "What color is the sky?"}],
+ "chosen":  [{"role": "assistant", "content": "It is blue."}],
+ "rejected":[{"role": "assistant", "content": "It is green."}]}
+
+# KTO — UNPAIRED binary preference (a completion + a boolean label)
+{"prompt": [{"role":"user","content":"What color is the sky?"}],
+ "completion": [{"role":"assistant","content":"It is blue."}], "label": True}
+
+# GRPO — prompts only; the reward function scores generated completions
+{"prompt": [{"role": "user", "content": "Solve: 2+2. Put the answer in \\boxed{}."}],
+ "solution": "4"}   # extra columns arrive in the reward fn via **kwargs
+```
+
+Building and validating these corpora (dedup, contamination, format checks) is `training-data`.
+
+## Full QLoRA SFT run
+
+```python
+import torch
+from datasets import load_dataset
+from transformers import BitsAndBytesConfig
+from peft import LoraConfig
+from trl import SFTTrainer, SFTConfig
+
+dataset = load_dataset("trl-lib/Capybara", split="train")   # conversational messages
+
+bnb = BitsAndBytesConfig(
+    load_in_4bit=True, bnb_4bit_quant_type="nf4",
+    bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16,
+)
+peft_config = LoraConfig(
+    r=16, lora_alpha=32, lora_dropout=0.05, bias="none",
+    task_type="CAUSAL_LM", target_modules="all-linear",
+)
+args = SFTConfig(
+    output_dir="qlora-sft",
+    per_device_train_batch_size=2, gradient_accumulation_steps=8,  # eff. batch 16
+    learning_rate=2e-4, num_train_epochs=2, warmup_ratio=0.05,
+    max_length=2048, packing=True,          # packing uses max_length as block size
+    bf16=True,                              # SFTConfig defaults: bf16=True, grad-ckpt=True
+    logging_steps=10, eval_strategy="steps", eval_steps=100, save_steps=100,
+    assistant_only_loss=True,               # loss on assistant turns only (conversational)
+)
+trainer = SFTTrainer(
+    model="Qwen/Qwen2.5-7B-Instruct",
+    args=args, train_dataset=dataset, eval_dataset=eval_dataset,
+    peft_config=peft_config, quantization_config=bnb,
+)
+trainer.train()
+trainer.save_model("qlora-sft")   # writes the adapter (adapter_config.json + weights)
+```
+
+Notes:
+- `SFTConfig` inherits `TrainingArguments`; its non-default defaults include `bf16=True`,
+  `gradient_checkpointing=True`, `learning_rate=2e-5`, `logging_steps=10`, `max_length=1024`.
+- `loss_type` defaults to `"chunked_nll"` (same math as `"nll"`, lower peak activation memory;
+  auto-falls back to `"nll"` when `use_liger_kernel=True`). Verify on your `trl` version.
+- **Base (non-instruct) models have no chat template.** Set one via `SFTConfig(chat_template_path=...)`
+  and align the EOS token (e.g. `eos_token="<|im_end|>"` for some Qwen bases) or the model never
+  learns to stop.
+
+## SFT → DPO
+
+```python
+from datasets import load_dataset
+from peft import LoraConfig
+from trl import DPOTrainer, DPOConfig
+
+pref = load_dataset("trl-lib/ultrafeedback_binarized", split="train")  # prompt/chosen/rejected
+peft_config = LoraConfig(r=16, lora_alpha=32, task_type="CAUSAL_LM", target_modules="all-linear")
+
+trainer = DPOTrainer(
+    model="qlora-sft",                    # the SFT checkpoint from above
+    args=DPOConfig(
+        output_dir="dpo", beta=0.1,       # beta ↑ = stay closer to the reference model
+        learning_rate=5e-7,               # preference LR is tiny vs SFT
+        per_device_train_batch_size=2, gradient_accumulation_steps=8,
+        max_length=1024, max_prompt_length=512,
+        precompute_ref_log_probs=True,    # cache ref logprobs → less VRAM; ref model auto-built
+        num_train_epochs=1,
+    ),
+    train_dataset=pref,
+    peft_config=peft_config,
+)
+trainer.train()
+```
+
+DPO tips: `loss_type` can be a list to blend variants (e.g. `["sigmoid","sft"]`); lower `beta` lets
+the policy drift further from the reference (more change, more risk). With LoRA you skip a separate
+frozen reference copy — the base with adapters disabled *is* the reference.
+
+## GRPO (reward-driven RL, DeepSeek-R1 style)
+
+Use when correctness is **checkable** (math, code, exact format). GRPO samples a *group* of
+`num_generations` completions per prompt and uses the group-mean reward as the baseline.
+
+```python
+from datasets import load_dataset
+from trl import GRPOTrainer, GRPOConfig
+
+dataset = load_dataset("trl-lib/DeepMath-103K", split="train")  # prompt + solution
+
+def correctness_reward(completions, solution, **kwargs):
+    # completions: list[list[{"role","content"}]]; extra dataset cols arrive as kwargs.
+    out = []
+    for comp, sol in zip(completions, solution):
+        text = comp[0]["content"]
+        got = text.split("\\boxed{")[-1].split("}")[0] if "\\boxed{" in text else ""
+        out.append(1.0 if got.strip() == str(sol).strip() else 0.0)
+    return out                              # list[float] (None allowed to skip a sample)
+
+trainer = GRPOTrainer(
+    model="qlora-sft",
+    reward_funcs=[correctness_reward],      # 1..N callables (or a reward-model id string)
+    args=GRPOConfig(
+        output_dir="grpo", num_generations=8, max_completion_length=1024,
+        beta=0.04, learning_rate=1e-6,
+        reward_weights=[1.0],               # weights when combining multiple reward fns
+        use_vllm=True, vllm_gpu_memory_utilization=0.7,   # vLLM makes rollouts affordable
+    ),
+    train_dataset=dataset,
+)
+trainer.train()
+```
+
+Reward-function contract: a Python callable (sync or async) receiving `completions` plus any extra
+dataset columns as kwargs, returning a `list[float]` (or `None` per sample to skip). `trl.rewards`
+ships built-ins (e.g. accuracy). Composed rewards are combined by `reward_weights`.
+
+## ORPO and KTO
+
+- **ORPO** — a **single stage** that fuses SFT + preference alignment (reference-free), so you can
+  align a *base* model without a separate SFT run. Import path migrated to experimental at author
+  time: `from trl.experimental.orpo import ORPOTrainer, ORPOConfig` (verify).
+- **KTO** — takes **unpaired** binary good/bad labels (thumbs up/down), not matched pairs; it will
+  also accept paired data and convert it internally. Use `KTOTrainer` / `KTOConfig` (stable API).
+
+```python
+from trl.experimental.orpo import ORPOTrainer, ORPOConfig   # verify import path
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from datasets import load_dataset
+
+model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
+tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
+if tok.pad_token is None: tok.pad_token = tok.eos_token
+ds = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
+
+trainer = ORPOTrainer(model, args=ORPOConfig(output_dir="orpo"),
+                      processing_class=tok, train_dataset=ds)
+trainer.train()
+```
+
+## Merging the adapter for serving
+
+`save_model` writes the small LoRA adapter, not a full model. To serve, either load base + adapter
+at runtime (many servers hot-load adapters — see `../../vllm/SKILL.md`) or merge into standalone
+weights:
+
+```python
+from peft import AutoPeftModelForCausalLM
+merged = AutoPeftModelForCausalLM.from_pretrained("qlora-sft").merge_and_unload()
+merged.save_pretrained("merged-model")     # standalone fp16/bf16 weights
+```
+
+Caveat: merging LoRA into a **4-bit** base can shift numerics — merge into the fp16/bf16 base for a
+clean standalone model, or keep the adapter separate and serve it on top of the full-precision base.
+Pushing the adapter/merged model to the Hub and hosting inference is `huggingface`; high-throughput
+serving (adapter hot-swap, batching) is `../../vllm/SKILL.md`; a fast single-GPU train + GGUF export
+path is `../../unsloth/SKILL.md`.

package/skills/harness/references/providers.yaml

@@ -3406,8 +3406,9 @@ providers:
         set -a; source "$SCRIPT_DIR/.env"; set +a
         : "${MAKE_ZONE:?MAKE_ZONE not set (e.g. eu1.make.com)}"
         : "${MAKE_API_TOKEN:?MAKE_API_TOKEN not set}"
-        NAME=$(curl -fsS "https://${MAKE_ZONE%/}/api/v2/users/me" -H "Authorization: Token $MAKE_API_TOKEN" | python3 -c 'import json,sys; print(json.load(sys.stdin)["authUser"]["name"])')
-        echo "OK — Make API reachable on $MAKE_ZONE (user: $NAME)."
+        # envelope-agnostic smoke test: -f fails on 4xx/5xx (wrong zone/token); no response-shape parsing
+        curl -fsS "https://${MAKE_ZONE%/}/api/v2/users/me" -H "Authorization: Token $MAKE_API_TOKEN" > /dev/null
+        echo "OK — Make API reachable on $MAKE_ZONE (token accepted)."
       README.md: |
         # MAKE
         Make.com (formerly Integromat). Scenarios via /api/v2; blueprint is a stringified JSON
@@ -3485,6 +3486,40 @@ providers:
         Process (workflow) privileges. `PA_CLIENT_SECRET` — secret. Only flows inside a Dataverse
         solution are code-manageable; personal "My Flows" are not.
 
+  - id: WANDB
+    category: ia_training
+    detectors:
+      imports: ["import wandb", "from wandb", "@wandb/sdk"]
+      env_vars: ["WANDB_API_KEY", "WANDB_ENTITY", "WANDB_PROJECT"]
+      deps: ["wandb"]
+    dashboard_url: "https://wandb.ai/authorize"
+    env_example: |
+      # Weights & Biases — experiment tracking for training runs. Key: https://wandb.ai/authorize
+      WANDB_API_KEY=
+      # WANDB_ENTITY=       # team or username (optional default)
+      # WANDB_PROJECT=      # default project (optional)
+    files:
+      test_connection.sh: |
+        #!/usr/bin/env bash
+        set -euo pipefail
+        SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+        set -a; source "$SCRIPT_DIR/.env"; set +a
+        : "${WANDB_API_KEY:?WANDB_API_KEY not set}"
+        # GraphQL viewer whoami; W&B authenticates the API key via HTTP basic (key in the password field).
+        # If the GraphQL shape/auth changes, the portable fallback is: pip install wandb && wandb login --verify
+        USER=$(curl -fsS "https://api.wandb.ai/graphql" -u "api:$WANDB_API_KEY" \
+          -H "Content-Type: application/json" \
+          -d '{"query":"query { viewer { username } }"}' \
+          | python3 -c 'import json,sys; print(json.load(sys.stdin)["data"]["viewer"]["username"])')
+        echo "OK — W&B authed as $USER."
+      README.md: |
+        # WANDB
+        Weights & Biases — experiment tracking / MLOps for training runs (loss curves, sweeps,
+        artifacts). Used by the `finetuning`, `deep-learning`, and `machine-learning` skills.
+      CREDENTIALS.md: |
+        Dashboard: https://wandb.ai/authorize (copy your API key).
+        `WANDB_API_KEY` — secret. Verify with `wandb login --verify` if the GraphQL check is flaky.
+
 # Categories used to group tools in 01-TOOLS/README.md.
 categories:
   infra: "Infra & deploy"
@@ -3508,3 +3543,4 @@ categories:
   feature_flags: "Feature flags"
   chat_webhooks: "Chatops webhooks"
   automatizacion: "Automation"
+  ia_training: "AI / training & MLOps"

package/skills/huggingface/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: huggingface
-description: "Use when running open models or working on the Hugging Face platform: calling a model via the Inference Providers router or InferenceClient, downloading/uploading/versioning repos on the Hub with the hf CLI, deploying a dedicated Inference Endpoint with scale-to-zero, hosting a Gradio Space with ZeroGPU, picking an open model by task/license/size, or loading a model locally with transformers. Triggers: 'call an open model through Hugging Face', 'InferenceClient embeddings', 'hf download/upload', 'write a model card', 'router.huggingface.co', 'my hf-inference serverless call 404s on a 70B model', 'scale-to-zero endpoint', 'ZeroGPU Space', 'desplegar un modelo de Hugging Face con escalado a cero', 'necesito controlar el coste de inferencia en HF'. NOT serving a model locally on your own machine (that is ollama), NOT renting and operating your own GPU box (that is runpod), NOT generating creative images via a hosted API (that is replicate-images)."
+description: "Use when running open models or working on the Hugging Face platform: calling a model via the Inference Providers router or InferenceClient, downloading/uploading/versioning repos on the Hub with the hf CLI, deploying a dedicated Inference Endpoint with scale-to-zero, hosting a Gradio Space with ZeroGPU, picking an open model by task/license/size, or loading a model locally with transformers. Triggers: 'call an open model through Hugging Face', 'InferenceClient embeddings', 'hf download/upload', 'write a model card', 'router.huggingface.co', 'my hf-inference serverless call 404s on a 70B model', 'scale-to-zero endpoint', 'ZeroGPU Space', 'desplegar un modelo de Hugging Face con escalado a cero'. NOT serving a model locally on your own machine (that is ollama), NOT renting and operating your own GPU box (that is runpod), NOT generating creative images via a hosted API (that is replicate-images), NOT fine-tuning with trl/peft (that is finetuning), NOT building training datasets (that is training-data)."
 tags: [huggingface, inference-providers, transformers, model-hub, inference-endpoints, spaces, ai-infra]
 recommends: [ollama, runpod, modal, replicate, together-fireworks, fal, rag, embeddings-search, prompt-engineering, llm-pipeline, ai-media]
 origin: risco

package/skills/machine-learning/SKILL.md

@@ -0,0 +1,260 @@
+---
+name: machine-learning
+description: "Use when the task is a classic/tabular machine-learning problem — predict a column from rows of features with trees and scikit-learn: fit a RandomForest / XGBoost / LightGBM classifier or regressor, build a leak-proof Pipeline + ColumnTransformer, choose a cross-validation scheme (stratified / k-fold / time-series / grouped), pick the right metric on imbalanced classes (precision/recall/F1/ROC-AUC/PR-AUC, not accuracy), or debug a model that aced CV then collapsed in production. Triggers: 'train a model to predict churn/fraud/price', 'my ROC-AUC is 0.99 — is that leakage', 'which metric for imbalanced classes', 'cross-validation without leaking', 'XGBoost vs LightGBM', 'add a DummyClassifier baseline', 'entrena un modelo de clasificación tabular con scikit-learn'. NOT neural-net / PyTorch training (that is deep-learning), NOT cleaning the raw dirty table first (that is data-cleaning), NOT BI dashboards or forecasting a dated series (that is analytics/forecasting), NOT text/token modeling (that is nlp)."
+tags: [machine-learning, scikit-learn, sklearn, xgboost, lightgbm, tabular, gbdt, cross-validation, data-leakage, classification, regression, pipeline]
+recommends: [deep-learning, data-cleaning, python, training-data]
+origin: risco
+---
+
+# Machine learning — classic/tabular models, done without lying to yourself
+
+Tabular ML is easy to *run* and easy to *fool yourself with*. The deliverable is never "the notebook
+printed 0.99" — it is an **honest estimate of how the model behaves on data it has never seen**: a
+`Pipeline` that fits every transform on train only, a metric that survives class imbalance, and a
+`DummyClassifier` baseline it beats. A number you can't reproduce on a sacred test set you touched exactly
+once isn't a result — it's a leak you haven't found yet.
+
+## Is this the right skill? (decide first)
+
+| Your situation | Reach for |
+| --- | --- |
+| Rows of features, predict a column, with trees / linear models / sklearn | **machine-learning** (this skill) |
+| Images, audio, long text, sequences, or you need a neural net / PyTorch | [`deep-learning`](../deep-learning/SKILL.md) |
+| The table is still dirty (nulls, dupes, mixed types, bad dates) | [`data-cleaning`](../data-cleaning/SKILL.md) **first** — it hands you a validated table |
+| Text/token classification, NER, tokenization, LLM-adjacent NLP metrics | [`nlp`](../nlp/SKILL.md) |
+| KPIs, dashboards, "explain the business" | `analytics` / `business-intelligence` |
+| Forecast a dated series forward (revenue next quarter) | `forecasting` |
+| Building a training corpus of JSONL messages / preference pairs for an LLM | [`training-data`](../training-data/SKILL.md) |
+
+This skill **starts** at a clean, validated table (rows × features + a target) and **ends** at a fitted,
+honestly-scored model with a test-set number and a baseline it beats. Cleaning is upstream — consume the
+validated frame `data-cleaning` produced; don't re-teach it here.
+
+## Version reality (verify at author time — this line moves monthly)
+
+Verified 2026-07: **scikit-learn current major ~1.9** (1.9.0 shipped 2026-06-02, Python 3.11–3.14) — do
+NOT pin from memory; check the current stable at scikit-learn.org, the 1.x line ships every few months.
+GBDTs: **XGBoost 3.x** and **LightGBM 4.x** (`xgboost` 3.3, `lightgbm` 4.6 current), plus sklearn's own
+**`HistGradientBoostingClassifier`/`...Regressor`** — a fast native GBDT that eats NaN and (with
+`categorical_features="from_dtype"`) categoricals with **no** preprocessing. Pin what you ship; state
+versions as "~X (verify)", never as frozen fact.
+
+## The one rule everything else serves: fit on train only
+
+Every preprocessing step — imputation, scaling, encoding, feature selection, target encoding, resampling —
+learns parameters from data. Learn them from rows the model is later *scored* on and the score inflates
+while production underperforms: that is **leakage**, the #1 way tabular ML lies. The whole apparatus below —
+`Pipeline`, `ColumnTransformer`, CV, the untouched test set — exists to make "fit on train only" automatic
+instead of something you remember to do by hand (you won't).
+
+## scikit-learn: estimators, Pipeline, ColumnTransformer
+
+Every model is an **estimator** with the same contract: `fit(X, y)`, then `predict(X)` /
+`predict_proba(X)` (classifiers) / `score(X, y)`. Transformers add `transform(X)` / `fit_transform(X, y)`.
+A **`Pipeline`** chains transformers + a final estimator into one estimator — so `fit` fits every step on
+train, and `predict`/CV `transform`s test data with parameters learned on train. That is the leakage guard.
+
+A **`ColumnTransformer`** routes different columns down different transformer branches (scale the numerics,
+encode the categoricals) and stitches the result back together — all still *inside* the pipeline.
+
+```python
+from sklearn.compose import ColumnTransformer, make_column_selector as mcs
+from sklearn.pipeline import Pipeline
+from sklearn.impute import SimpleImputer
+from sklearn.preprocessing import StandardScaler, OneHotEncoder
+from sklearn.linear_model import LogisticRegression
+
+numeric = Pipeline([("impute", SimpleImputer(strategy="median")),
+                    ("scale",  StandardScaler())])           # scaling matters for LINEAR/SVM/KNN
+categoric = Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
+                      ("onehot", OneHotEncoder(handle_unknown="ignore"))])  # unseen category -> all-zeros, no crash
+
+pre = ColumnTransformer([
+    ("num", numeric,   mcs(dtype_include="number")),
+    ("cat", categoric, mcs(dtype_include=["object", "category"])),
+], remainder="drop")
+
+model = Pipeline([("pre", pre), ("clf", LogisticRegression(max_iter=1000, class_weight="balanced"))])
+model.fit(X_train, y_train)          # imputers/scaler/encoder ALL fit on X_train only
+model.predict_proba(X_test)          # X_test transformed with train-learned params — no leak
+```
+
+Two load-bearing details: `OneHotEncoder(handle_unknown="ignore")` so a category unseen in train doesn't
+crash prediction, and `remainder="drop"` so unrouted columns don't silently leak through raw;
+`model.set_output(transform="pandas")` keeps named-column DataFrames through the pipeline.
+
+**Trees skip most of this** — scaling is pointless for tree models, and `HistGradientBoostingClassifier`
+ingests NaN and categoricals natively, so its pipeline is often just the estimator. Preprocess for the
+*model that needs it*, not as a ritual. More patterns (`make_column_transformer`, `FunctionTransformer`)
+→ [references/pipelines-and-cv.md](references/pipelines-and-cv.md).
+
+## GBDTs are your default on tabular data
+
+Gradient-boosted decision trees are the correct first (and usually last) model for tabular problems.
+This isn't taste: **Grinsztajn, Oyallon & Varoquaux (2022), "Why do tree-based models still outperform
+deep learning on tabular data?"** ([arXiv:2207.08815](https://arxiv.org/abs/2207.08815)) benchmarked across
+45 datasets and found GBDTs beat tuned neural nets on medium-sized tabular data, tracing it to three
+inductive biases NNs lack: **robustness to uninformative features**, **not being rotationally invariant**
+(so they exploit the meaning of individual columns), and ease of **learning irregular / non-smooth target
+functions**. Start with a GBDT; reach for `deep-learning` on tabular only with a specific reason.
+
+| Library | Import | Reach for it when |
+| --- | --- | --- |
+| `HistGradientBoostingClassifier` | `sklearn.ensemble` | Default. Fast, zero extra deps, native NaN + categorical. |
+| XGBoost 3.x | `xgboost.XGBClassifier` | Battle-tested; `early_stopping_rounds`, rich regularization. |
+| LightGBM 4.x | `lightgbm.LGBMClassifier` | Fastest on wide/large data; leaf-wise; strong native categoricals. |
+
+All three expose the sklearn estimator API, so they drop into the pipeline and CV below unchanged. Don't
+agonize over XGBoost-vs-LightGBM before you have a baseline and a leak-free CV — split discipline dwarfs the
+library choice. Tuning knobs (`learning_rate`, `num_leaves`/`max_depth`, early stopping, `monotonic_cst`)
+→ [references/gbdt-and-tuning.md](references/gbdt-and-tuning.md).
+
+## Leakage-safe feature engineering
+
+Feature engineering is where leakage sneaks back in after the pipeline "protected" you. Rules:
+
+- **Any transform that learns from data goes INSIDE the pipeline**, so CV re-fits it per fold. A scaler fit
+  on the whole dataset, a `SelectKBest` run before splitting, an imputer using the global mean — each leaks
+  test statistics into train. The `common_pitfalls` canonical example: `SelectKBest(k=25).fit_transform(X, y)`
+  *before* `train_test_split` produces a beautiful, meaningless score.
+- **Target/mean encoding of high-cardinality categoricals must cross-fit.** sklearn's
+  `preprocessing.TargetEncoder` does this: its `fit_transform(X, y)` uses an internal cross-fitting scheme
+  so each row is encoded from *other* folds' targets — `fit(X, y).transform(X)` deliberately differs and
+  would leak. Use `fit_transform` on train (inside the pipeline); never hand-roll a group-mean encoder.
+- **No target-derived features.** A column computed from the label (or a near-proxy: "was_refunded" when
+  predicting "will_refund") is leakage wearing a feature's clothes. If a feature is impossibly predictive,
+  suspect it.
+- **Respect time.** With any temporal structure, a feature may use only information available *at prediction
+  time* — no future aggregates, no lifetime values that include post-cutoff rows. Split by time
+  (`TimeSeriesSplit`), not randomly.
+
+## Split + cross-validation: protect the sacred test set
+
+Hold out a **test set once, at the very start** (`train_test_split(..., stratify=y, random_state=0)`) and
+do not look at it until you have a single final model. Every glance — tuning, feature choice, "let me just
+check" — bleeds information and re-inflates the estimate. Tune and compare with **cross-validation on the
+training portion only**; the test set is the one honest number at the end (see the lifecycle below). Score
+with `cross_validate(pipeline, X_tr, y_tr, cv=cv, scoring=[...], return_train_score=True)` — a large
+train-minus-test gap is your overfitting alarm.
+
+Pick the splitter to match the data ([cross_validation docs](https://scikit-learn.org/stable/modules/cross_validation.html)):
+
+- **`StratifiedKFold`** — default for classification; preserves class balance per fold (essential when
+  imbalanced). `KFold` for regression.
+- **`TimeSeriesSplit`** — any time ordering. Trains on past, tests on future; never shuffles the future
+  into train. A random `KFold` on time-series data is leakage.
+- **`GroupKFold` / `StratifiedGroupKFold`** — when rows cluster (same user/patient/store across many rows).
+  Keep a group entirely in train *or* test, or the model memorizes the group and CV lies.
+- Pass **integer `random_state`** to splitters for reproducible folds. Put preprocessing in the pipeline so
+  CV re-fits it every fold — `cross_validate(pipeline, ...)`, never `cross_validate(model, X_scaled, ...)`.
+
+For tuning, wrap CV in `GridSearchCV` / `RandomizedSearchCV` / `HalvingRandomSearchCV`; for an unbiased
+estimate *of the tuning process itself*, use nested CV → [references/pipelines-and-cv.md](references/pipelines-and-cv.md).
+
+## Metrics: the accuracy trap and what to use instead
+
+**Accuracy lies on imbalanced data.** At 99% negatives, a model that predicts "negative" always scores 99%
+accuracy and is worthless. Choose the metric for the task and the cost of each error type
+([model_evaluation docs](https://scikit-learn.org/stable/modules/model_evaluation.html)):
+
+| Task / question | Metric (`sklearn.metrics`) | scoring string |
+| --- | --- | --- |
+| Ranking quality, threshold-free, balanced-ish | `roc_auc_score` | `"roc_auc"` |
+| **Imbalanced** ranking (rare positive: fraud, disease) | `average_precision_score` (PR-AUC) | `"average_precision"` |
+| Cost of false positives high (don't cry wolf) | `precision_score` | `"precision"` |
+| Cost of misses high (don't miss a case) | `recall_score` | `"recall"` |
+| Balance both, per-class fairness | `f1_score` (use `f1_macro` multiclass) | `"f1"` / `"f1_macro"` |
+| Multiclass, care about every class equally | `balanced_accuracy_score` | `"balanced_accuracy"` |
+| See the actual error breakdown | `confusion_matrix`, `classification_report` | — |
+| Regression | `r2_score`, `mean_absolute_error`, `root_mean_squared_error` | `"r2"`, `"neg_mean_absolute_error"`, `"neg_root_mean_squared_error"` |
+
+**Prefer PR-AUC (`average_precision`) to ROC-AUC when positives are rare** — ROC-AUC can look great while
+precision is dismal, since it ignores the negative flood. Feed AUC metrics `predict_proba`, not hard labels.
+The default 0.5 threshold is a *choice*: tune it on validation to hit your precision/recall target.
+Regression uses `root_mean_squared_error` now (`mean_squared_error(squared=False)` is gone). Threshold
+tuning, calibration, `class_weight`/resampling → [references/metrics-and-imbalance.md](references/metrics-and-imbalance.md).
+
+## Cardinal sins (the anti-pattern table)
+
+| Rationalization | Reality → STOP |
+| --- | --- |
+| "I scaled/encoded/selected features, then split." | **Leakage (#1 sin).** Test statistics are now in train. Split first; put every learned transform *inside* the `Pipeline` so CV re-fits per fold. |
+| "This feature is amazingly predictive." | Suspect **target leakage** — a column derived from the label or unavailable at prediction time. Audit provenance before you celebrate. |
+| "99% accuracy!" (on 1% positives) | The **accuracy trap**. A constant predictor matches it. Report PR-AUC / precision / recall / F1 and a `confusion_matrix`. |
+| "The classes are imbalanced, so I `SMOTE`'d the whole dataset." | Resampling *before* the split, or on the test fold, leaks and evaluates on synthetic rows. Resample **inside CV, on the train fold only** (imblearn `Pipeline`), or just use `class_weight="balanced"`. |
+| "CV score is 0.95, ship it." | You never touched a **held-out test set**, or you peeked at it while tuning. One final, untouched test number — or the estimate is optimistic. |
+| "Random `KFold` on my time-series / multi-user data." | Future or same-group rows leak into train. Use `TimeSeriesSplit` / `GroupKFold`. |
+| "Big train/test gap? It'll generalize." | **Overfitting.** Regularize, reduce capacity (`max_depth`, `min_samples_leaf`), get more data, or use early stopping. Watch `return_train_score`. |
+| "Straight to XGBoost, no baseline." | Without a **`DummyClassifier(strategy="most_frequent")`** / `DummyRegressor` floor (and a simple linear model), you can't tell if the fancy model adds anything. |
+| "I'll grid-search 10k combos on all the data." | Tuning against the test set *is* fitting to it. Tune with CV on train, confirm once on test; consider nested CV. |
+
+## Worked lifecycle (end to end)
+
+```python
+from sklearn.dummy import DummyClassifier
+from sklearn.ensemble import HistGradientBoostingClassifier
+from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score
+from sklearn.metrics import average_precision_score, classification_report
+
+X_tr, X_test, y_tr, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)
+cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
+
+# 0. BASELINE first — the floor every real model must clear
+base = DummyClassifier(strategy="most_frequent")
+print("baseline PR-AUC:", cross_val_score(base, X_tr, y_tr, cv=cv, scoring="average_precision").mean())
+
+# 1. default GBDT (native NaN + categoricals -> minimal pipeline). class_weight for imbalance.
+clf = HistGradientBoostingClassifier(categorical_features="from_dtype",
+                                     class_weight="balanced", random_state=0)
+cv_pr = cross_val_score(clf, X_tr, y_tr, cv=cv, scoring="average_precision")
+print("model CV PR-AUC:", cv_pr.mean().round(3), "+/-", cv_pr.std().round(3))
+
+# 2. it clears the baseline -> commit, fit on all train, judge ONCE on the sacred test set
+clf.fit(X_tr, y_tr)
+proba = clf.predict_proba(X_test)[:, 1]
+print("TEST PR-AUC:", round(average_precision_score(y_test, proba), 3))
+print(classification_report(y_test, (proba >= 0.5).astype(int)))   # threshold is a choice — tune it
+```
+
+## Related skills
+
+- [`deep-learning`](../deep-learning/SKILL.md) — PyTorch neural nets. Cross the boundary for images, audio,
+  long text, sequences; on *tabular* data, stay here (Grinsztajn: trees win).
+- [`data-cleaning`](../data-cleaning/SKILL.md) — upstream; produces the validated table this skill trains
+  on. Don't re-teach nulls/dupes/dtypes here.
+- [`nlp`](../nlp/SKILL.md) — text/token modeling, NER, tokenization, task metrics (a TF-IDF + linear/GBDT
+  baseline still lives happily in this skill's pipeline).
+- [`python`](../python/SKILL.md) — environment, packaging, pinning the versions you ship.
+- [`training-data`](../training-data/SKILL.md) — if the goal is an LLM corpus (JSONL messages / preference
+  pairs), not a tabular predictor. BI/forecasting products → `analytics` / `business-intelligence` / `forecasting`.
+
+## Checklist
+
+- [ ] Input is a clean, validated table (from `data-cleaning`), not raw dirty rows.
+- [ ] A **test set** was held out **once, up front**, stratified, and left untouched until the final score.
+- [ ] Every learned transform (impute/scale/encode/select/target-encode) lives **inside** the `Pipeline`.
+- [ ] CV runs on the training portion only, via `cross_validate`/`GridSearchCV` over the *pipeline*.
+- [ ] Splitter matches the data: `StratifiedKFold` (imbalanced), `TimeSeriesSplit` (time), `GroupKFold` (clustered).
+- [ ] A `DummyClassifier`/`DummyRegressor` baseline (and a simple linear model) is beaten, not skipped.
+- [ ] Metric fits the task — **not accuracy on imbalanced data**; PR-AUC/precision/recall/F1 as appropriate.
+- [ ] Target-leakage audit done: no label-derived / future / not-at-prediction-time features.
+- [ ] `random_state` set on splitters and estimators; versions pinned for what ships.
+- [ ] Train-vs-test gap inspected; overfitting addressed before shipping.
+
+## References
+
+- [references/pipelines-and-cv.md](references/pipelines-and-cv.md) — `Pipeline`/`ColumnTransformer` patterns,
+  `FunctionTransformer`, `TargetEncoder` cross-fitting, `GridSearchCV`/`RandomizedSearchCV`/`HalvingRandomSearchCV`,
+  nested CV, `TimeSeriesSplit`/`GroupKFold` recipes.
+- [references/metrics-and-imbalance.md](references/metrics-and-imbalance.md) — metric selection, threshold
+  tuning, probability calibration, `class_weight`/`sample_weight`, leak-free resampling (imbalanced-learn `Pipeline`).
+- [references/gbdt-and-tuning.md](references/gbdt-and-tuning.md) — HistGBDT vs XGBoost vs LightGBM parameter
+  maps, early stopping, native categoricals, monotonic constraints, permutation importance vs SHAP.
+
+## Project grounding (02-DOCS + CLAUDE.md)
+
+In a project with a `02-DOCS/` layer (the [`harness`](../harness/SKILL.md) wiki), record the modeling
+contract in `02-DOCS/wiki/ml/<target>.md`, linked from the root `CLAUDE.md` `## Knowledge map`: target
+definition, split strategy + `random_state`, CV scheme, chosen metric and *why*, baseline, pinned versions,
+and the dated final test-set score. Read it first on every re-train so results stay comparable. No
+`02-DOCS/`? Skip silently — conventions are recorded, never gated.

package/skills/machine-learning/evals/README.md

@@ -0,0 +1,15 @@
+# Eval harness — `machine-learning`
+
+`cases.yaml` is a human/LLM-graded rubric, not an automated runner. It grades two things: **triggering**
+(does the skill fire on real tabular-ML asks and stay quiet on near-misses that belong to a sibling —
+`deep-learning`, `data-cleaning`, `nlp`, `forecasting`, `business-intelligence`, `training-data`?) and
+**capability** (does loading the skill produce a measurably more rigorous, leak-free modeling workflow?).
+
+To run triggering: start a fresh agent with only `machine-learning` discoverable, paste each
+`should_trigger` prompt (3–5 trials each, since the decision is stochastic; pass if it fires in the
+majority), and confirm each `should_not_trigger` prompt does NOT fire and would plausibly route to its
+`route_to` sibling. To run capability: give the scenario to a clean agent WITHOUT the skill and again WITH
+it, then grade each transcript against the `must_include` bullets — one point per bullet that is
+specifically and correctly covered (the held-out test set, leakage as #1 risk, PR-AUC over accuracy on
+imbalance, GBDT default + Grinsztajn, the version hedge), not merely name-dropped. WITH should clearly beat
+WITHOUT. Record trial counts and per-bullet verdicts; don't report a bare pass/fail.