@damiangilgonzalez/harness 1.3.7

package/skills/machine-learning/SKILL.md

@@ -0,0 +1,228 @@
+---
+name: machine-learning
+description: "Use when predicting a column from rows of tabular features with classic models — scikit-learn pipelines, RandomForest, XGBoost/LightGBM, leak-free cross-validation, metrics for imbalanced classes, or a model that aced CV then collapsed in production. NOT PyTorch neural nets (that is `deep-learning`), NOT cleaning the dirty table first (that is `data-cleaning`), NOT forecasting a dated series (that is `forecasting`), NOT text/token modeling (that is `nlp`)."
+tags: [machine-learning, scikit-learn, sklearn, xgboost, lightgbm, tabular, gbdt, cross-validation, data-leakage, classification, regression, pipeline]
+recommends: [deep-learning, data-cleaning, python, training-data]
+origin: risco
+---
+
+# Machine learning — classic/tabular models, done without lying to yourself
+
+Tabular ML is easy to *run* and easy to *fool yourself with*. The deliverable is never "the notebook
+printed 0.99" — it is an **honest estimate of how the model behaves on data it has never seen**: a
+`Pipeline` that fits every transform on train only, a metric that survives class imbalance, and a
+`DummyClassifier` baseline it beats. A number you can't reproduce on a sacred test set you touched exactly
+once isn't a result — it's a leak you haven't found yet.
+
+## Is this the right skill? (decide first)
+
+| Your situation | Reach for |
+| --- | --- |
+| Rows of features, predict a column, with trees / linear models / sklearn | **machine-learning** (this skill) |
+| Images, audio, long text, sequences, or you need a neural net / PyTorch | [`deep-learning`](../deep-learning/SKILL.md) |
+| The table is still dirty (nulls, dupes, mixed types, bad dates) | [`data-cleaning`](../data-cleaning/SKILL.md) **first** — it hands you a validated table |
+| Text/token classification, NER, tokenization, LLM-adjacent NLP metrics | [`nlp`](../nlp/SKILL.md) (a TF-IDF + linear/GBDT baseline still lives happily in this skill's pipeline) |
+| KPIs, dashboards, "explain the business" | [`analytics`](../analytics/SKILL.md) / [`business-intelligence`](../business-intelligence/SKILL.md) |
+| Forecast a dated series forward (revenue next quarter) | [`forecasting`](../forecasting/SKILL.md) |
+| Building a training corpus of JSONL messages / preference pairs for an LLM | [`training-data`](../training-data/SKILL.md) |
+
+This skill **starts** at a clean, validated table (rows × features + a target) and **ends** at a fitted,
+honestly-scored model with a test-set number and a baseline it beats. Cleaning is upstream — consume the
+validated frame `data-cleaning` produced; don't re-teach it here.
+
+## Version reality (verify at author time — this line moves monthly)
+
+Verified 2026-07: **scikit-learn current major ~1.9** (1.9.0 shipped 2026-06-02, Python 3.11–3.14) — do
+NOT pin from memory; check the current stable at scikit-learn.org, the 1.x line ships every few months.
+GBDTs: **XGBoost 3.x** and **LightGBM 4.x** (`xgboost` 3.3, `lightgbm` 4.6 current), plus sklearn's own
+**`HistGradientBoostingClassifier`/`...Regressor`** — a fast native GBDT that eats NaN and (with
+`categorical_features="from_dtype"`) categoricals with **no** preprocessing. Pin what you ship
+([`python`](../python/SKILL.md) owns the environment and the pinning); state versions as "~X (verify)",
+never as frozen fact.
+
+## The one rule everything else serves: fit on train only
+
+Every preprocessing step — imputation, scaling, encoding, feature selection, target encoding, resampling —
+learns parameters from data. Learn them from rows the model is later *scored* on and the score inflates
+while production underperforms: that is **leakage**, the #1 way tabular ML lies. The whole apparatus below —
+`Pipeline`, `ColumnTransformer`, CV, the untouched test set — exists to make "fit on train only" automatic
+instead of something you remember to do by hand (you won't).
+
+## scikit-learn: estimators, Pipeline, ColumnTransformer
+
+Every model is an **estimator** with the same contract: `fit(X, y)`, then `predict(X)` /
+`predict_proba(X)` (classifiers) / `score(X, y)`. Transformers add `transform(X)` / `fit_transform(X, y)`.
+A **`Pipeline`** chains transformers + a final estimator into one estimator — so `fit` fits every step on
+train, and `predict`/CV `transform`s test data with parameters learned on train. That is the leakage guard.
+
+A **`ColumnTransformer`** routes different columns down different transformer branches (scale the numerics,
+encode the categoricals) and stitches the result back together — all still *inside* the pipeline.
+
+```python
+from sklearn.compose import ColumnTransformer, make_column_selector as mcs
+from sklearn.pipeline import Pipeline
+from sklearn.impute import SimpleImputer
+from sklearn.preprocessing import StandardScaler, OneHotEncoder
+from sklearn.linear_model import LogisticRegression
+
+numeric = Pipeline([("impute", SimpleImputer(strategy="median")),
+                    ("scale",  StandardScaler())])           # scaling matters for LINEAR/SVM/KNN
+categoric = Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
+                      ("onehot", OneHotEncoder(handle_unknown="ignore"))])  # unseen category -> all-zeros, no crash
+
+pre = ColumnTransformer([
+    ("num", numeric,   mcs(dtype_include="number")),
+    ("cat", categoric, mcs(dtype_include=["object", "category"])),
+], remainder="drop")
+
+model = Pipeline([("pre", pre), ("clf", LogisticRegression(max_iter=1000, class_weight="balanced"))])
+model.fit(X_train, y_train)          # imputers/scaler/encoder ALL fit on X_train only
+model.predict_proba(X_test)          # X_test transformed with train-learned params — no leak
+```
+
+Two load-bearing details: `OneHotEncoder(handle_unknown="ignore")` so a category unseen in train doesn't
+crash prediction, and `remainder="drop"` so unrouted columns don't silently leak through raw;
+`model.set_output(transform="pandas")` keeps named-column DataFrames through the pipeline.
+
+**Trees skip most of this** — scaling is pointless for tree models, and `HistGradientBoostingClassifier`
+ingests NaN and categoricals natively, so its pipeline is often just the estimator. Preprocess for the
+*model that needs it*, not as a ritual. More patterns (`make_column_transformer`, `FunctionTransformer`)
+→ [references/pipelines-and-cv.md](references/pipelines-and-cv.md).
+
+## GBDTs are your default on tabular data
+
+Gradient-boosted decision trees are the correct first (and usually last) model for tabular problems.
+This isn't taste: **Grinsztajn, Oyallon & Varoquaux (2022), "Why do tree-based models still outperform
+deep learning on tabular data?"** ([arXiv:2207.08815](https://arxiv.org/abs/2207.08815)) benchmarked across
+45 datasets and found GBDTs beat tuned neural nets on medium-sized tabular data, tracing it to three
+inductive biases NNs lack: **robustness to uninformative features**, **not being rotationally invariant**
+(so they exploit the meaning of individual columns), and ease of **learning irregular / non-smooth target
+functions**. Start with a GBDT; reach for `deep-learning` on tabular only with a specific reason.
+
+| Library | Import | Reach for it when |
+| --- | --- | --- |
+| `HistGradientBoostingClassifier` | `sklearn.ensemble` | Default. Fast, zero extra deps, native NaN + categorical. |
+| XGBoost 3.x | `xgboost.XGBClassifier` | Battle-tested; `early_stopping_rounds`, rich regularization. |
+| LightGBM 4.x | `lightgbm.LGBMClassifier` | Fastest on wide/large data; leaf-wise; strong native categoricals. |
+
+All three expose the sklearn estimator API, so they drop into the pipeline and CV below unchanged. Don't
+agonize over XGBoost-vs-LightGBM before you have a baseline and a leak-free CV — split discipline dwarfs the
+library choice. Tuning knobs (`learning_rate`, `num_leaves`/`max_depth`, early stopping, `monotonic_cst`)
+and importance/SHAP → [references/gbdt-and-tuning.md](references/gbdt-and-tuning.md).
+
+## Leakage-safe feature engineering
+
+Feature engineering is where leakage sneaks back in after the pipeline "protected" you. Rules:
+
+- **Any transform that learns from data goes INSIDE the pipeline**, so CV re-fits it per fold. A scaler fit
+  on the whole dataset, a `SelectKBest` run before splitting, an imputer using the global mean — each leaks
+  test statistics into train. The `common_pitfalls` canonical example: `SelectKBest(k=25).fit_transform(X, y)`
+  *before* `train_test_split` produces a beautiful, meaningless score.
+- **Target/mean encoding of high-cardinality categoricals must cross-fit.** sklearn's
+  `preprocessing.TargetEncoder` does this: its `fit_transform(X, y)` uses an internal cross-fitting scheme
+  so each row is encoded from *other* folds' targets — `fit(X, y).transform(X)` deliberately differs and
+  would leak. Use `fit_transform` on train (inside the pipeline); never hand-roll a group-mean encoder.
+- **No target-derived features.** A column computed from the label (or a near-proxy: "was_refunded" when
+  predicting "will_refund") is leakage wearing a feature's clothes. If a feature is impossibly predictive,
+  suspect it.
+- **Respect time.** With any temporal structure, a feature may use only information available *at prediction
+  time* — no future aggregates, no lifetime values that include post-cutoff rows. Split by time
+  (`TimeSeriesSplit`), not randomly.
+
+## Split + cross-validation: protect the sacred test set
+
+Hold out a **test set once, at the very start** (`train_test_split(..., stratify=y, random_state=0)`) and
+do not look at it until you have a single final model. Every glance — tuning, feature choice, "let me just
+check" — bleeds information and re-inflates the estimate. Tune and compare with **cross-validation on the
+training portion only**; the test set is the one honest number at the end (see the lifecycle below). Score
+with `cross_validate(pipeline, X_tr, y_tr, cv=cv, scoring=[...], return_train_score=True)` — a large
+train-minus-test gap is your overfitting alarm.
+
+Pick the splitter to match the data ([cross_validation docs](https://scikit-learn.org/stable/modules/cross_validation.html)):
+
+- **`StratifiedKFold`** — default for classification; preserves class balance per fold (essential when
+  imbalanced). `KFold` for regression.
+- **`TimeSeriesSplit`** — any time ordering. Trains on past, tests on future; never shuffles the future
+  into train. A random `KFold` on time-series data is leakage.
+- **`GroupKFold` / `StratifiedGroupKFold`** — when rows cluster (same user/patient/store across many rows).
+  Keep a group entirely in train *or* test, or the model memorizes the group and CV lies.
+- Pass **integer `random_state`** to splitters for reproducible folds. Put preprocessing in the pipeline so
+  CV re-fits it every fold — `cross_validate(pipeline, ...)`, never `cross_validate(model, X_scaled, ...)`.
+
+For tuning, wrap CV in `GridSearchCV` / `RandomizedSearchCV` / `HalvingRandomSearchCV`; for an unbiased
+estimate *of the tuning process itself*, use nested CV → [references/pipelines-and-cv.md](references/pipelines-and-cv.md).
+
+## Metrics: the accuracy trap and what to use instead
+
+**Accuracy lies on imbalanced data.** At 99% negatives, a model that predicts "negative" always scores 99%
+accuracy and is worthless. Choose the metric for the task and the cost of each error type
+([model_evaluation docs](https://scikit-learn.org/stable/modules/model_evaluation.html)):
+
+| Task / question | Metric (`sklearn.metrics`) | scoring string |
+| --- | --- | --- |
+| Ranking quality, threshold-free, balanced-ish | `roc_auc_score` | `"roc_auc"` |
+| **Imbalanced** ranking (rare positive: fraud, disease) | `average_precision_score` (PR-AUC) | `"average_precision"` |
+| Cost of false positives high (don't cry wolf) | `precision_score` | `"precision"` |
+| Cost of misses high (don't miss a case) | `recall_score` | `"recall"` |
+| Balance both, per-class fairness | `f1_score` (use `f1_macro` multiclass) | `"f1"` / `"f1_macro"` |
+| Multiclass, care about every class equally | `balanced_accuracy_score` | `"balanced_accuracy"` |
+| See the actual error breakdown | `confusion_matrix`, `classification_report` | — |
+| Regression | `r2_score`, `mean_absolute_error`, `root_mean_squared_error` | `"r2"`, `"neg_mean_absolute_error"`, `"neg_root_mean_squared_error"` |
+
+**Prefer PR-AUC (`average_precision`) to ROC-AUC when positives are rare** — ROC-AUC can look great while
+precision is dismal, since it ignores the negative flood. Feed AUC metrics `predict_proba`, not hard labels.
+The default 0.5 threshold is a *choice*: tune it on validation to hit your precision/recall target.
+Regression uses `root_mean_squared_error` now (`mean_squared_error(squared=False)` is gone). Threshold
+tuning, calibration, `class_weight`/resampling → [references/metrics-and-imbalance.md](references/metrics-and-imbalance.md).
+
+## Anti-patterns — the cardinal sins
+
+| Anti-pattern | Do instead |
+| --- | --- |
+| Modeling straight off the raw, dirty table | This skill starts at a **clean, validated frame**. Run [`data-cleaning`](../data-cleaning/SKILL.md) first — nulls, dupes and mixed dtypes are its job, not a modeling problem. |
+| Scaling / encoding / selecting features, *then* splitting | **Leakage (#1 sin).** Test statistics are now in train. Split first; put every learned transform *inside* the `Pipeline` so CV re-fits per fold. |
+| Celebrating an amazingly predictive feature | Suspect **target leakage** — a column derived from the label or unavailable at prediction time. Audit provenance before you celebrate. |
+| Reporting 99% accuracy on 1% positives | The **accuracy trap**. A constant predictor matches it. Report PR-AUC / precision / recall / F1 and a `confusion_matrix`. |
+| `SMOTE`-ing the whole dataset because the classes are imbalanced | Resampling *before* the split, or on the test fold, leaks and evaluates on synthetic rows. Resample **inside CV, on the train fold only** (imblearn `Pipeline`), or just use `class_weight="balanced"`. |
+| Shipping on the CV score alone | You never touched a **held-out test set**, or you peeked at it while tuning. One final, untouched test number — or the estimate is optimistic. |
+| Random `KFold` on time-series / multi-user data | Future or same-group rows leak into train. Use `TimeSeriesSplit` / `GroupKFold`. |
+| Waving off a big train/test gap | **Overfitting.** Regularize, reduce capacity (`max_depth`, `min_samples_leaf`), get more data, or use early stopping. Watch `return_train_score`. |
+| Going straight to XGBoost with no baseline | Without a **`DummyClassifier(strategy="most_frequent")`** / `DummyRegressor` floor (and a simple linear model), you can't tell if the fancy model adds anything. |
+| Grid-searching 10k combos over all the data | Tuning against the test set *is* fitting to it. Tune with CV on train, confirm once on test; consider nested CV. |
+| Unset `random_state`, unpinned versions | Folds and fits stop being reproducible and re-trains stop being comparable. Set an integer `random_state` on splitters *and* estimators; pin the versions you ship. |
+
+## Worked lifecycle (end to end)
+
+```python
+from sklearn.dummy import DummyClassifier
+from sklearn.ensemble import HistGradientBoostingClassifier
+from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score
+from sklearn.metrics import average_precision_score, classification_report
+
+X_tr, X_test, y_tr, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)
+cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
+
+# 0. BASELINE first — the floor every real model must clear
+base = DummyClassifier(strategy="most_frequent")
+print("baseline PR-AUC:", cross_val_score(base, X_tr, y_tr, cv=cv, scoring="average_precision").mean())
+
+# 1. default GBDT (native NaN + categoricals -> minimal pipeline). class_weight for imbalance.
+clf = HistGradientBoostingClassifier(categorical_features="from_dtype",
+                                     class_weight="balanced", random_state=0)
+cv_pr = cross_val_score(clf, X_tr, y_tr, cv=cv, scoring="average_precision")
+print("model CV PR-AUC:", cv_pr.mean().round(3), "+/-", cv_pr.std().round(3))
+
+# 2. it clears the baseline -> commit, fit on all train, judge ONCE on the sacred test set
+clf.fit(X_tr, y_tr)
+proba = clf.predict_proba(X_test)[:, 1]
+print("TEST PR-AUC:", round(average_precision_score(y_test, proba), 3))
+print(classification_report(y_test, (proba >= 0.5).astype(int)))   # threshold is a choice — tune it
+```
+
+## Project grounding (docs + CLAUDE.md)
+
+In a project with a `docs/` layer (the [`harness`](../harness/SKILL.md) wiki), record the modeling
+contract in `docs/wiki/ml/<target>.md`, linked from the root `CLAUDE.md` `## Knowledge map`: target
+definition, split strategy + `random_state`, CV scheme, chosen metric and *why*, baseline, pinned versions,
+and the dated final test-set score. Read it first on every re-train so results stay comparable. No
+`docs/`? Skip silently — conventions are recorded, never gated.

package/skills/machine-learning/evals/README.md

@@ -0,0 +1,15 @@
+# Eval harness — `machine-learning`
+
+`cases.yaml` is a human/LLM-graded rubric, not an automated runner. It grades two things: **triggering**
+(does the skill fire on real tabular-ML asks and stay quiet on near-misses that belong to a sibling —
+`deep-learning`, `data-cleaning`, `nlp`, `forecasting`, `business-intelligence`, `training-data`?) and
+**capability** (does loading the skill produce a measurably more rigorous, leak-free modeling workflow?).
+
+To run triggering: start a fresh agent with only `machine-learning` discoverable, paste each
+`should_trigger` prompt (3–5 trials each, since the decision is stochastic; pass if it fires in the
+majority), and confirm each `should_not_trigger` prompt does NOT fire and would plausibly route to its
+`route_to` sibling. To run capability: give the scenario to a clean agent WITHOUT the skill and again WITH
+it, then grade each transcript against the `must_include` bullets — one point per bullet that is
+specifically and correctly covered (the held-out test set, leakage as #1 risk, PR-AUC over accuracy on
+imbalance, GBDT default + Grinsztajn, the version hedge), not merely name-dropped. WITH should clearly beat
+WITHOUT. Record trial counts and per-bullet verdicts; don't report a bare pass/fail.

package/skills/machine-learning/evals/cases.yaml

@@ -0,0 +1,54 @@
+skill: machine-learning
+
+should_trigger:
+  - prompt: "Train a model on this customer table to predict who will churn, and tell me how well it actually generalizes."
+    why: Core ask — supervised tabular classification with an honest generalization estimate; the headline use case.
+  - prompt: "My gradient-boosting model scores 0.99 ROC-AUC in cross-validation but tanks in production. Is this leakage?"
+    why: Leakage debugging (#1 sin) — only this skill explains fit-on-train-only, target-derived features, and the sacred test set.
+  - prompt: "My positive class is 2% of the rows — which metric should I optimize, and is accuracy lying to me?"
+    why: The accuracy trap on imbalance → PR-AUC/precision/recall/F1; the metric-selection reasoning is this skill's.
+  - prompt: "Set up cross-validation for this dataset so no test information leaks into training or preprocessing."
+    why: Pipeline + ColumnTransformer + splitter choice so transforms fit per fold — the leakage-safe CV pattern lives here.
+  - prompt: "XGBoost or LightGBM for my tabular regression problem, and should I even bother with a neural net?"
+    why: GBDT choice + the trees-beat-deep-nets-on-tabular finding; classic-model territory, not deep-learning.
+  - prompt: "Add a proper baseline so I can tell whether my random forest is actually doing anything."
+    why: DummyClassifier/DummyRegressor floor — a load-bearing rigor step this skill insists on.
+  - prompt: "Entrena un modelo de clasificación tabular con scikit-learn y dame una métrica honesta, no solo accuracy."
+    why: Spanish trigger — same tabular-classification-with-honest-metric intent, no English keyword present.
+
+should_not_trigger:
+  - prompt: "Train a convolutional neural network in PyTorch to classify these product images."
+    route_to: deep-learning
+    why: Neural nets on images — PyTorch training, explicitly the deep-learning boundary, not tabular sklearn.
+  - prompt: "This CSV is full of nulls, duplicate rows, and 'N/A' strings — fix it before I can model anything."
+    route_to: data-cleaning
+    why: Producing the validated table is upstream; this skill starts once the table is clean.
+  - prompt: "Classify these free-text support tickets into topics straight from the ticket wording."
+    route_to: nlp
+    why: Text/token modeling is nlp's domain; route free-text problems there (a TF-IDF baseline can still live in an ML pipeline, but the framing is NLP).
+  - prompt: "Forecast next quarter's revenue from this monthly time-series."
+    route_to: forecasting
+    why: Projecting a dated series forward is a forecasting product, not a tabular predict-a-column task.
+  - prompt: "Build a KPI dashboard and executive summary from this already-clean sales table."
+    route_to: business-intelligence
+    why: Explaining KPIs / dashboards consumes data; it is not model training.
+  - prompt: "Assemble a JSONL dataset of chat messages and preference pairs to fine-tune an LLM."
+    route_to: training-data
+    why: Building an LLM training corpus, not a tabular ML model.
+
+capability:
+  - scenario: >
+      Given a clean, validated table of ~50k rows for a churn/fraud problem where only ~3% of rows are
+      positive, produce a trustworthy modeling workflow: hold out a test set, build a leak-proof pipeline,
+      cross-validate, pick a sensible metric and model, beat a baseline, and report an honest final number.
+    must_include:
+      - Splits off a held-out test set FIRST with train_test_split(..., stratify=y, random_state=...), and leaves it untouched until the single final evaluation
+      - Puts all learned preprocessing (impute/scale/encode) INSIDE a Pipeline + ColumnTransformer so it is fit on the train fold only
+      - Uses StratifiedKFold (not plain KFold) for the imbalanced classes, with an integer random_state for reproducible folds
+      - Establishes a DummyClassifier baseline (e.g. strategy="most_frequent") that the real model must beat
+      - Defaults to a GBDT (HistGradientBoostingClassifier / XGBoost / LightGBM) and notes that trees still beat deep nets on tabular data (Grinsztajn 2022, arXiv:2207.08815)
+      - Reports PR-AUC (average_precision) / precision / recall / F1 / confusion_matrix and explicitly does NOT use accuracy, because the classes are imbalanced
+      - Names data/target leakage as the #1 risk and audits for target-derived or not-available-at-prediction-time features
+      - Cross-validates the whole pipeline via cross_validate/GridSearchCV (scoring the pipeline, never globally-preprocessed data)
+      - Treats the 0.5 decision threshold as a tunable choice (e.g. from the PR curve or TunedThresholdClassifierCV), not a law
+      - Hedges the scikit-learn version (current major ~1.9, verify at scikit-learn.org) instead of asserting a frozen pin as fact

package/skills/machine-learning/references/gbdt-and-tuning.md

@@ -0,0 +1,83 @@
+# GBDTs: parameters, early stopping, categoricals, importance (depth)
+
+Three interchangeable sklearn-API GBDTs. Versions verified 2026-07 — XGBoost 3.x, LightGBM 4.x, and
+sklearn's `HistGradientBoosting*` (~1.9). Verify current versions before pinning; APIs shift between majors.
+
+## What the knobs mean (they translate across libraries)
+
+| Concept | HistGradientBoosting | XGBoost | LightGBM |
+| --- | --- | --- | --- |
+| Step size | `learning_rate` | `learning_rate` (`eta`) | `learning_rate` |
+| Number of trees | `max_iter` | `n_estimators` | `n_estimators` |
+| Tree size | `max_leaf_nodes` / `max_depth` | `max_depth` | `num_leaves` (leaf-wise!) / `max_depth` |
+| L2 penalty | `l2_regularization` | `reg_lambda` / `reg_alpha` | `reg_lambda` / `reg_alpha` |
+| Min data per leaf | `min_samples_leaf` | `min_child_weight` | `min_child_samples` |
+| Row/col subsample | (n/a) | `subsample` / `colsample_bytree` | `bagging_fraction` / `feature_fraction` |
+
+Tuning order that pays off: get `learning_rate` low-ish with enough trees (use early stopping to pick the
+count), then control complexity (`num_leaves`/`max_depth`, `min_*_leaf`), then regularization and subsampling.
+**Lower `learning_rate` + more trees + early stopping** beats a high learning rate almost always. LightGBM
+grows **leaf-wise**, so cap `num_leaves` (and/or `max_depth`) or it overfits fast.
+
+## Early stopping (stop adding trees when validation stops improving)
+
+Needs a validation signal. Keep it out of the sacred test set — use a train-derived eval split.
+
+```python
+# XGBoost 3.x: early_stopping_rounds is a CONSTRUCTOR param; pass eval_set to fit()
+from xgboost import XGBClassifier
+xgb = XGBClassifier(n_estimators=2000, learning_rate=0.05,
+                    early_stopping_rounds=50, eval_metric="aucpr")
+xgb.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=False)
+print(xgb.best_iteration)
+
+# LightGBM 4.x: callbacks
+from lightgbm import LGBMClassifier, early_stopping, log_evaluation
+lgbm = LGBMClassifier(n_estimators=2000, learning_rate=0.05, num_leaves=31)
+lgbm.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], eval_metric="average_precision",
+         callbacks=[early_stopping(50), log_evaluation(0)])
+
+# HistGradientBoosting: built-in, no eval_set needed (carves its own validation slice)
+from sklearn.ensemble import HistGradientBoostingClassifier
+hgb = HistGradientBoostingClassifier(max_iter=2000, learning_rate=0.05,
+                                     early_stopping=True, validation_fraction=0.1,
+                                     n_iter_no_change=50, random_state=0)
+```
+
+## Native categorical handling (skip one-hot)
+
+- **HistGradientBoosting**: `categorical_features="from_dtype"` — columns with pandas `category` dtype are
+  handled natively. No `OneHotEncoder`, no explosion of columns.
+- **XGBoost**: `enable_categorical=True` + columns as pandas `category` dtype; `tree_method` `hist`/`approx`.
+- **LightGBM**: pass `categorical_feature=` (names/indices) or use `category` dtype; native by default.
+
+One-hot only when cardinality is low and a *linear* model needs it. For trees, native categorical or, for
+very high cardinality, `TargetEncoder` (cross-fit — see pipelines-and-cv.md) beats a thousand dummy columns.
+
+## Monotonic constraints (domain priors)
+
+Force the model's response to move only one way with a feature (price ↑ ⇒ risk never ↓). Guards against
+noise inverting a known relationship and helps trust/compliance.
+
+```python
+# HistGradientBoosting: dict of feature -> {-1, 0, +1}
+HistGradientBoostingClassifier(monotonic_cst={"income": +1, "num_late_payments": -1})
+# XGBoost: monotone_constraints="(1,0,-1,...)"   LightGBM: monotone_constraints=[1,0,-1,...]
+```
+
+## Feature importance — don't trust the default blindly
+
+- **Impurity / gain importance** (`feature_importances_`): fast but biased toward high-cardinality and
+  continuous features; computed on train.
+- **Permutation importance** (`sklearn.inspection.permutation_importance`): model-agnostic, measures the
+  metric drop when a feature is shuffled — run it **on held-out data**, and beware correlated features
+  sharing credit.
+- **SHAP** (separate `shap` library): per-prediction attributions; the most faithful local explanations,
+  the heaviest to compute. Use for "why this row" and to sanity-check for leakage — a feature dominating
+  SHAP that shouldn't be predictive is a leak flag.
+
+```python
+from sklearn.inspection import permutation_importance
+r = permutation_importance(model, X_test, y_test, scoring="average_precision",
+                           n_repeats=10, random_state=0)
+```

package/skills/machine-learning/references/metrics-and-imbalance.md

@@ -0,0 +1,82 @@
+# Metrics, thresholds, calibration, and imbalance (depth)
+
+Picking the metric and handling imbalance is where "0.99 accuracy" turns into an honest number. API current
+as of scikit-learn ~1.9 (verify at scikit-learn.org).
+
+## Choosing the classification metric
+
+Everything in `sklearn.metrics`. Score **probabilities** (`predict_proba`) for AUC/ranking metrics; score
+**labels** for precision/recall/F1 (which depend on the decision threshold).
+
+| You care about… | Metric | Notes |
+| --- | --- | --- |
+| Ranking, threshold-free, roughly balanced | `roc_auc_score` | Insensitive to the negative flood — flatters rare-positive problems. |
+| Ranking with **rare positives** (fraud/disease) | `average_precision_score` (PR-AUC) | The honest ranking metric under imbalance. |
+| Not crying wolf (false positives costly) | `precision_score` | TP / (TP+FP). |
+| Not missing cases (false negatives costly) | `recall_score` | TP / (TP+FN). |
+| Both, single number | `f1_score` | Harmonic mean; `average="macro"` treats classes equally, `"weighted"` by support. |
+| Per-class fairness, multiclass | `balanced_accuracy_score`, `f1_macro` | Macro-averaging stops the majority class dominating. |
+| The full picture | `confusion_matrix`, `classification_report` | Always look before trusting one scalar. |
+| Probability quality | `brier_score_loss`, `log_loss` | For calibrated-probability use cases. |
+
+Multiclass averaging: `micro` (global TP/FP counts — dominated by frequent classes), `macro` (unweighted
+class mean — each class equal), `weighted` (by class support). State which and why.
+
+## Regression metrics
+
+`r2_score` (proportion of variance explained; can be negative), `mean_absolute_error` (robust, same units),
+`root_mean_squared_error` (penalizes large errors; note this is the current function —
+`mean_squared_error(squared=False)` was removed), `mean_absolute_percentage_error` (scale-free but explodes
+near zero). Scoring strings negate losses: `"neg_root_mean_squared_error"`, `"neg_mean_absolute_error"`.
+
+## The threshold is a decision, not 0.5
+
+`predict()` thresholds probability at 0.5, which is rarely the business-optimal cut under imbalance or
+asymmetric costs. Tune it on validation, either manually from the PR curve or with the built-in estimator.
+
+```python
+from sklearn.metrics import precision_recall_curve
+prec, rec, thr = precision_recall_curve(y_val, proba_val)
+# e.g. smallest threshold that reaches precision >= 0.90:
+import numpy as np
+ok = np.where(prec[:-1] >= 0.90)[0]
+chosen = thr[ok[0]] if len(ok) else 1.0
+
+# Or let sklearn tune + wrap it (added 1.5):
+from sklearn.model_selection import TunedThresholdClassifierCV, FixedThresholdClassifier
+tuned = TunedThresholdClassifierCV(clf, scoring="f1", cv=5).fit(X_tr, y_tr)
+print(tuned.best_threshold_)
+fixed = FixedThresholdClassifier(clf, threshold=0.30).fit(X_tr, y_tr)  # pin a known-good cut
+```
+
+## Probability calibration
+
+Tree ensembles and SVMs produce distorted probabilities. If you use the probability itself (expected-value
+decisions, thresholds you trust), calibrate — on a held-out fold, never on train.
+
+```python
+from sklearn.calibration import CalibratedClassifierCV
+cal = CalibratedClassifierCV(clf, method="isotonic", cv=5)   # or method="sigmoid" (Platt)
+```
+
+## Imbalance — in order of preference
+
+1. **Do nothing to the data; fix the metric + threshold.** Often enough. Use PR-AUC and a tuned threshold.
+2. **`class_weight="balanced"`** (LogisticRegression, RandomForest, SVC, HistGradientBoosting…) or
+   `scale_pos_weight` (XGBoost) — reweight the loss, no synthetic rows, no leakage risk.
+3. **Resampling (SMOTE / undersampling) — only if 1–2 fall short, and only inside CV on the train fold.**
+   Resampling before the split, or touching the test fold, leaks and evaluates on synthetic data. Use the
+   **imbalanced-learn `Pipeline`** (not sklearn's) so the resampler runs per fold, train-only:
+
+```python
+from imblearn.pipeline import Pipeline as ImbPipeline
+from imblearn.over_sampling import SMOTE
+from sklearn.ensemble import HistGradientBoostingClassifier
+
+pipe = ImbPipeline([("pre", pre), ("smote", SMOTE(random_state=0)),
+                    ("clf", HistGradientBoostingClassifier(random_state=0))])
+# In cross_validate/GridSearchCV, SMOTE fits on each TRAIN fold only; test folds stay real.
+```
+
+Never resample the test set, and never report accuracy on the resampled (artificially balanced) data as if
+it were the real distribution.

package/skills/machine-learning/references/pipelines-and-cv.md

@@ -0,0 +1,135 @@
+# Pipelines, custom transforms, and cross-validation (depth)
+
+Everything here keeps the SKILL.md rule: transforms that *learn* live inside the pipeline, so CV/tuning
+re-fit them on each train fold only. API current as of scikit-learn ~1.9 (verify at scikit-learn.org).
+
+## make_column_transformer — the terse ColumnTransformer
+
+`make_column_transformer` auto-names steps; `make_column_selector` picks columns by dtype/regex.
+
+```python
+from sklearn.compose import make_column_transformer, make_column_selector as mcs
+from sklearn.pipeline import make_pipeline
+from sklearn.impute import SimpleImputer
+from sklearn.preprocessing import StandardScaler, OneHotEncoder
+
+pre = make_column_transformer(
+    (make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), mcs(dtype_include="number")),
+    (make_pipeline(SimpleImputer(strategy="most_frequent"),
+                   OneHotEncoder(handle_unknown="ignore")), mcs(dtype_include=["object", "category"])),
+    remainder="drop",
+    verbose_feature_names_out=False,   # cleaner get_feature_names_out()
+)
+pre.set_output(transform="pandas")     # keep a DataFrame with real column names flowing through
+```
+
+## Stateless custom feature: FunctionTransformer
+
+For a transform with **no learned state** (log, ratio, date parts), `FunctionTransformer` is leak-safe by
+construction — there is nothing fit on data.
+
+```python
+import numpy as np
+from sklearn.preprocessing import FunctionTransformer
+
+log1p = FunctionTransformer(np.log1p, feature_names_out="one-to-one")
+```
+
+## Stateful custom transformer: fit on train only
+
+If the transform learns anything (a mean, a fitted mapping), write a class so CV re-fits it per fold.
+`fit` must store state from **train** only; `transform` applies it. Never compute the statistic in `__init__`
+or over the whole dataset.
+
+```python
+from sklearn.base import BaseEstimator, TransformerMixin
+
+class GroupRareCategories(BaseEstimator, TransformerMixin):
+    """Fold categories seen < min_count times in TRAIN into '__other__'."""
+    def __init__(self, min_count=10):
+        self.min_count = min_count
+    def fit(self, X, y=None):
+        counts = X.iloc[:, 0].value_counts()
+        self.keep_ = set(counts[counts >= self.min_count].index)   # learned on train fold
+        return self
+    def transform(self, X):
+        col = X.iloc[:, 0]
+        return col.where(col.isin(self.keep_), "__other__").to_frame()
+```
+
+## High-cardinality categoricals: TargetEncoder cross-fitting
+
+`sklearn.preprocessing.TargetEncoder` encodes each category by the target mean — the classic leakage trap
+if done naively. Its `fit_transform(X, y)` uses an **internal cross-fitting** scheme (each row encoded from
+*other* folds' targets), which is why `fit(X, y).transform(X)` deliberately differs and must not be used on
+train. Put it in the pipeline and let CV call `fit_transform`; never hand-roll a group-mean encode.
+
+```python
+from sklearn.preprocessing import TargetEncoder
+enc = TargetEncoder(smooth="auto", cv=5)   # cv controls the internal cross-fit
+# inside a ColumnTransformer branch for the high-cardinality columns
+```
+
+## Hyperparameter tuning — over the whole pipeline
+
+Address a step's parameter with `stepname__param` (double underscore). Tuning searches must wrap the
+*pipeline*, so preprocessing is re-fit each fold — tuning on globally-preprocessed data leaks.
+
+```python
+from sklearn.pipeline import Pipeline
+from sklearn.ensemble import HistGradientBoostingClassifier
+from sklearn.model_selection import RandomizedSearchCV, StratifiedKFold
+
+pipe = Pipeline([("pre", pre), ("clf", HistGradientBoostingClassifier(random_state=0))])
+param_dist = {
+    "clf__learning_rate": [0.01, 0.05, 0.1, 0.2],
+    "clf__max_leaf_nodes": [15, 31, 63],
+    "clf__l2_regularization": [0.0, 1.0, 10.0],
+}
+search = RandomizedSearchCV(
+    pipe, param_dist, n_iter=30,
+    cv=StratifiedKFold(5, shuffle=True, random_state=0),
+    scoring="average_precision", random_state=0, n_jobs=-1, refit=True,
+)
+search.fit(X_tr, y_tr)          # X_tr only — the test set stays sealed
+print(search.best_params_, search.best_score_)
+```
+
+- `GridSearchCV` — exhaustive; use for a small, discrete grid.
+- `RandomizedSearchCV` — sample `n_iter` combos from distributions; the default for anything nontrivial.
+- `HalvingRandomSearchCV` / `HalvingGridSearchCV` (`sklearn.model_selection`) — successive halving: cheap
+  runs on many candidates, more budget to survivors. Fastest when fits are expensive.
+
+## Nested CV — unbiased estimate of the tuning process
+
+If you report `search.best_score_` as your performance, it is optimistic (you selected on it). To estimate
+how the *whole tuning procedure* generalizes, nest an inner CV (tuning) inside an outer CV (scoring).
+
+```python
+from sklearn.model_selection import cross_val_score
+outer = StratifiedKFold(5, shuffle=True, random_state=1)
+nested = cross_val_score(search, X_tr, y_tr, cv=outer, scoring="average_precision")
+print(nested.mean(), nested.std())   # honest estimate of "tune-then-fit" on unseen data
+```
+
+## Splitters beyond StratifiedKFold
+
+```python
+from sklearn.model_selection import TimeSeriesSplit, GroupKFold, StratifiedGroupKFold
+
+# time order: train on past, test on future (expanding window); never shuffle the future into train
+for tr, te in TimeSeriesSplit(n_splits=5).split(X):
+    ...
+
+# clustered rows (same user/store): keep a group wholly in train OR test
+for tr, te in GroupKFold(n_splits=5).split(X, y, groups=user_id):
+    ...
+
+# clustered AND imbalanced: preserve class ratio while keeping groups intact
+for tr, te in StratifiedGroupKFold(n_splits=5).split(X, y, groups=user_id):
+    ...
+```
+
+Pass `groups=` through tuning too: `search.fit(X, y, groups=user_id)` with a group-aware `cv`. A random
+`KFold` on time-series or grouped data is leakage — CV will look great and production will not.
+```