@ericrisco/rsc 0.1.39 → 0.1.41

package/skills/machine-learning/evals/cases.yaml

@@ -0,0 +1,54 @@
+skill: machine-learning
+
+should_trigger:
+  - prompt: "Train a model on this customer table to predict who will churn, and tell me how well it actually generalizes."
+    why: Core ask — supervised tabular classification with an honest generalization estimate; the headline use case.
+  - prompt: "My gradient-boosting model scores 0.99 ROC-AUC in cross-validation but tanks in production. Is this leakage?"
+    why: Leakage debugging (#1 sin) — only this skill explains fit-on-train-only, target-derived features, and the sacred test set.
+  - prompt: "My positive class is 2% of the rows — which metric should I optimize, and is accuracy lying to me?"
+    why: The accuracy trap on imbalance → PR-AUC/precision/recall/F1; the metric-selection reasoning is this skill's.
+  - prompt: "Set up cross-validation for this dataset so no test information leaks into training or preprocessing."
+    why: Pipeline + ColumnTransformer + splitter choice so transforms fit per fold — the leakage-safe CV pattern lives here.
+  - prompt: "XGBoost or LightGBM for my tabular regression problem, and should I even bother with a neural net?"
+    why: GBDT choice + the trees-beat-deep-nets-on-tabular finding; classic-model territory, not deep-learning.
+  - prompt: "Add a proper baseline so I can tell whether my random forest is actually doing anything."
+    why: DummyClassifier/DummyRegressor floor — a load-bearing rigor step this skill insists on.
+  - prompt: "Entrena un modelo de clasificación tabular con scikit-learn y dame una métrica honesta, no solo accuracy."
+    why: Spanish trigger — same tabular-classification-with-honest-metric intent, no English keyword present.
+
+should_not_trigger:
+  - prompt: "Train a convolutional neural network in PyTorch to classify these product images."
+    route_to: deep-learning
+    why: Neural nets on images — PyTorch training, explicitly the deep-learning boundary, not tabular sklearn.
+  - prompt: "This CSV is full of nulls, duplicate rows, and 'N/A' strings — fix it before I can model anything."
+    route_to: data-cleaning
+    why: Producing the validated table is upstream; this skill starts once the table is clean.
+  - prompt: "Classify these free-text support tickets into topics straight from the ticket wording."
+    route_to: nlp
+    why: Text/token modeling is nlp's domain; route free-text problems there (a TF-IDF baseline can still live in an ML pipeline, but the framing is NLP).
+  - prompt: "Forecast next quarter's revenue from this monthly time-series."
+    route_to: forecasting
+    why: Projecting a dated series forward is a forecasting product, not a tabular predict-a-column task.
+  - prompt: "Build a KPI dashboard and executive summary from this already-clean sales table."
+    route_to: business-intelligence
+    why: Explaining KPIs / dashboards consumes data; it is not model training.
+  - prompt: "Assemble a JSONL dataset of chat messages and preference pairs to fine-tune an LLM."
+    route_to: training-data
+    why: Building an LLM training corpus, not a tabular ML model.
+
+capability:
+  - scenario: >
+      Given a clean, validated table of ~50k rows for a churn/fraud problem where only ~3% of rows are
+      positive, produce a trustworthy modeling workflow: hold out a test set, build a leak-proof pipeline,
+      cross-validate, pick a sensible metric and model, beat a baseline, and report an honest final number.
+    must_include:
+      - Splits off a held-out test set FIRST with train_test_split(..., stratify=y, random_state=...), and leaves it untouched until the single final evaluation
+      - Puts all learned preprocessing (impute/scale/encode) INSIDE a Pipeline + ColumnTransformer so it is fit on the train fold only
+      - Uses StratifiedKFold (not plain KFold) for the imbalanced classes, with an integer random_state for reproducible folds
+      - Establishes a DummyClassifier baseline (e.g. strategy="most_frequent") that the real model must beat
+      - Defaults to a GBDT (HistGradientBoostingClassifier / XGBoost / LightGBM) and notes that trees still beat deep nets on tabular data (Grinsztajn 2022, arXiv:2207.08815)
+      - Reports PR-AUC (average_precision) / precision / recall / F1 / confusion_matrix and explicitly does NOT use accuracy, because the classes are imbalanced
+      - Names data/target leakage as the #1 risk and audits for target-derived or not-available-at-prediction-time features
+      - Cross-validates the whole pipeline via cross_validate/GridSearchCV (scoring the pipeline, never globally-preprocessed data)
+      - Treats the 0.5 decision threshold as a tunable choice (e.g. from the PR curve or TunedThresholdClassifierCV), not a law
+      - Hedges the scikit-learn version (current major ~1.9, verify at scikit-learn.org) instead of asserting a frozen pin as fact

package/skills/machine-learning/references/gbdt-and-tuning.md

@@ -0,0 +1,83 @@
+# GBDTs: parameters, early stopping, categoricals, importance (depth)
+
+Three interchangeable sklearn-API GBDTs. Versions verified 2026-07 — XGBoost 3.x, LightGBM 4.x, and
+sklearn's `HistGradientBoosting*` (~1.9). Verify current versions before pinning; APIs shift between majors.
+
+## What the knobs mean (they translate across libraries)
+
+| Concept | HistGradientBoosting | XGBoost | LightGBM |
+| --- | --- | --- | --- |
+| Step size | `learning_rate` | `learning_rate` (`eta`) | `learning_rate` |
+| Number of trees | `max_iter` | `n_estimators` | `n_estimators` |
+| Tree size | `max_leaf_nodes` / `max_depth` | `max_depth` | `num_leaves` (leaf-wise!) / `max_depth` |
+| L2 penalty | `l2_regularization` | `reg_lambda` / `reg_alpha` | `reg_lambda` / `reg_alpha` |
+| Min data per leaf | `min_samples_leaf` | `min_child_weight` | `min_child_samples` |
+| Row/col subsample | (n/a) | `subsample` / `colsample_bytree` | `bagging_fraction` / `feature_fraction` |
+
+Tuning order that pays off: get `learning_rate` low-ish with enough trees (use early stopping to pick the
+count), then control complexity (`num_leaves`/`max_depth`, `min_*_leaf`), then regularization and subsampling.
+**Lower `learning_rate` + more trees + early stopping** beats a high learning rate almost always. LightGBM
+grows **leaf-wise**, so cap `num_leaves` (and/or `max_depth`) or it overfits fast.
+
+## Early stopping (stop adding trees when validation stops improving)
+
+Needs a validation signal. Keep it out of the sacred test set — use a train-derived eval split.
+
+```python
+# XGBoost 3.x: early_stopping_rounds is a CONSTRUCTOR param; pass eval_set to fit()
+from xgboost import XGBClassifier
+xgb = XGBClassifier(n_estimators=2000, learning_rate=0.05,
+                    early_stopping_rounds=50, eval_metric="aucpr")
+xgb.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=False)
+print(xgb.best_iteration)
+
+# LightGBM 4.x: callbacks
+from lightgbm import LGBMClassifier, early_stopping, log_evaluation
+lgbm = LGBMClassifier(n_estimators=2000, learning_rate=0.05, num_leaves=31)
+lgbm.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], eval_metric="average_precision",
+         callbacks=[early_stopping(50), log_evaluation(0)])
+
+# HistGradientBoosting: built-in, no eval_set needed (carves its own validation slice)
+from sklearn.ensemble import HistGradientBoostingClassifier
+hgb = HistGradientBoostingClassifier(max_iter=2000, learning_rate=0.05,
+                                     early_stopping=True, validation_fraction=0.1,
+                                     n_iter_no_change=50, random_state=0)
+```
+
+## Native categorical handling (skip one-hot)
+
+- **HistGradientBoosting**: `categorical_features="from_dtype"` — columns with pandas `category` dtype are
+  handled natively. No `OneHotEncoder`, no explosion of columns.
+- **XGBoost**: `enable_categorical=True` + columns as pandas `category` dtype; `tree_method` `hist`/`approx`.
+- **LightGBM**: pass `categorical_feature=` (names/indices) or use `category` dtype; native by default.
+
+One-hot only when cardinality is low and a *linear* model needs it. For trees, native categorical or, for
+very high cardinality, `TargetEncoder` (cross-fit — see pipelines-and-cv.md) beats a thousand dummy columns.
+
+## Monotonic constraints (domain priors)
+
+Force the model's response to move only one way with a feature (price ↑ ⇒ risk never ↓). Guards against
+noise inverting a known relationship and helps trust/compliance.
+
+```python
+# HistGradientBoosting: dict of feature -> {-1, 0, +1}
+HistGradientBoostingClassifier(monotonic_cst={"income": +1, "num_late_payments": -1})
+# XGBoost: monotone_constraints="(1,0,-1,...)"   LightGBM: monotone_constraints=[1,0,-1,...]
+```
+
+## Feature importance — don't trust the default blindly
+
+- **Impurity / gain importance** (`feature_importances_`): fast but biased toward high-cardinality and
+  continuous features; computed on train.
+- **Permutation importance** (`sklearn.inspection.permutation_importance`): model-agnostic, measures the
+  metric drop when a feature is shuffled — run it **on held-out data**, and beware correlated features
+  sharing credit.
+- **SHAP** (separate `shap` library): per-prediction attributions; the most faithful local explanations,
+  the heaviest to compute. Use for "why this row" and to sanity-check for leakage — a feature dominating
+  SHAP that shouldn't be predictive is a leak flag.
+
+```python
+from sklearn.inspection import permutation_importance
+r = permutation_importance(model, X_test, y_test, scoring="average_precision",
+                           n_repeats=10, random_state=0)
+```

package/skills/machine-learning/references/metrics-and-imbalance.md

@@ -0,0 +1,82 @@
+# Metrics, thresholds, calibration, and imbalance (depth)
+
+Picking the metric and handling imbalance is where "0.99 accuracy" turns into an honest number. API current
+as of scikit-learn ~1.9 (verify at scikit-learn.org).
+
+## Choosing the classification metric
+
+Everything in `sklearn.metrics`. Score **probabilities** (`predict_proba`) for AUC/ranking metrics; score
+**labels** for precision/recall/F1 (which depend on the decision threshold).
+
+| You care about… | Metric | Notes |
+| --- | --- | --- |
+| Ranking, threshold-free, roughly balanced | `roc_auc_score` | Insensitive to the negative flood — flatters rare-positive problems. |
+| Ranking with **rare positives** (fraud/disease) | `average_precision_score` (PR-AUC) | The honest ranking metric under imbalance. |
+| Not crying wolf (false positives costly) | `precision_score` | TP / (TP+FP). |
+| Not missing cases (false negatives costly) | `recall_score` | TP / (TP+FN). |
+| Both, single number | `f1_score` | Harmonic mean; `average="macro"` treats classes equally, `"weighted"` by support. |
+| Per-class fairness, multiclass | `balanced_accuracy_score`, `f1_macro` | Macro-averaging stops the majority class dominating. |
+| The full picture | `confusion_matrix`, `classification_report` | Always look before trusting one scalar. |
+| Probability quality | `brier_score_loss`, `log_loss` | For calibrated-probability use cases. |
+
+Multiclass averaging: `micro` (global TP/FP counts — dominated by frequent classes), `macro` (unweighted
+class mean — each class equal), `weighted` (by class support). State which and why.
+
+## Regression metrics
+
+`r2_score` (proportion of variance explained; can be negative), `mean_absolute_error` (robust, same units),
+`root_mean_squared_error` (penalizes large errors; note this is the current function —
+`mean_squared_error(squared=False)` was removed), `mean_absolute_percentage_error` (scale-free but explodes
+near zero). Scoring strings negate losses: `"neg_root_mean_squared_error"`, `"neg_mean_absolute_error"`.
+
+## The threshold is a decision, not 0.5
+
+`predict()` thresholds probability at 0.5, which is rarely the business-optimal cut under imbalance or
+asymmetric costs. Tune it on validation, either manually from the PR curve or with the built-in estimator.
+
+```python
+from sklearn.metrics import precision_recall_curve
+prec, rec, thr = precision_recall_curve(y_val, proba_val)
+# e.g. smallest threshold that reaches precision >= 0.90:
+import numpy as np
+ok = np.where(prec[:-1] >= 0.90)[0]
+chosen = thr[ok[0]] if len(ok) else 1.0
+
+# Or let sklearn tune + wrap it (added 1.5):
+from sklearn.model_selection import TunedThresholdClassifierCV, FixedThresholdClassifier
+tuned = TunedThresholdClassifierCV(clf, scoring="f1", cv=5).fit(X_tr, y_tr)
+print(tuned.best_threshold_)
+fixed = FixedThresholdClassifier(clf, threshold=0.30).fit(X_tr, y_tr)  # pin a known-good cut
+```
+
+## Probability calibration
+
+Tree ensembles and SVMs produce distorted probabilities. If you use the probability itself (expected-value
+decisions, thresholds you trust), calibrate — on a held-out fold, never on train.
+
+```python
+from sklearn.calibration import CalibratedClassifierCV
+cal = CalibratedClassifierCV(clf, method="isotonic", cv=5)   # or method="sigmoid" (Platt)
+```
+
+## Imbalance — in order of preference
+
+1. **Do nothing to the data; fix the metric + threshold.** Often enough. Use PR-AUC and a tuned threshold.
+2. **`class_weight="balanced"`** (LogisticRegression, RandomForest, SVC, HistGradientBoosting…) or
+   `scale_pos_weight` (XGBoost) — reweight the loss, no synthetic rows, no leakage risk.
+3. **Resampling (SMOTE / undersampling) — only if 1–2 fall short, and only inside CV on the train fold.**
+   Resampling before the split, or touching the test fold, leaks and evaluates on synthetic data. Use the
+   **imbalanced-learn `Pipeline`** (not sklearn's) so the resampler runs per fold, train-only:
+
+```python
+from imblearn.pipeline import Pipeline as ImbPipeline
+from imblearn.over_sampling import SMOTE
+from sklearn.ensemble import HistGradientBoostingClassifier
+
+pipe = ImbPipeline([("pre", pre), ("smote", SMOTE(random_state=0)),
+                    ("clf", HistGradientBoostingClassifier(random_state=0))])
+# In cross_validate/GridSearchCV, SMOTE fits on each TRAIN fold only; test folds stay real.
+```
+
+Never resample the test set, and never report accuracy on the resampled (artificially balanced) data as if
+it were the real distribution.

package/skills/machine-learning/references/pipelines-and-cv.md

@@ -0,0 +1,135 @@
+# Pipelines, custom transforms, and cross-validation (depth)
+
+Everything here keeps the SKILL.md rule: transforms that *learn* live inside the pipeline, so CV/tuning
+re-fit them on each train fold only. API current as of scikit-learn ~1.9 (verify at scikit-learn.org).
+
+## make_column_transformer — the terse ColumnTransformer
+
+`make_column_transformer` auto-names steps; `make_column_selector` picks columns by dtype/regex.
+
+```python
+from sklearn.compose import make_column_transformer, make_column_selector as mcs
+from sklearn.pipeline import make_pipeline
+from sklearn.impute import SimpleImputer
+from sklearn.preprocessing import StandardScaler, OneHotEncoder
+
+pre = make_column_transformer(
+    (make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), mcs(dtype_include="number")),
+    (make_pipeline(SimpleImputer(strategy="most_frequent"),
+                   OneHotEncoder(handle_unknown="ignore")), mcs(dtype_include=["object", "category"])),
+    remainder="drop",
+    verbose_feature_names_out=False,   # cleaner get_feature_names_out()
+)
+pre.set_output(transform="pandas")     # keep a DataFrame with real column names flowing through
+```
+
+## Stateless custom feature: FunctionTransformer
+
+For a transform with **no learned state** (log, ratio, date parts), `FunctionTransformer` is leak-safe by
+construction — there is nothing fit on data.
+
+```python
+import numpy as np
+from sklearn.preprocessing import FunctionTransformer
+
+log1p = FunctionTransformer(np.log1p, feature_names_out="one-to-one")
+```
+
+## Stateful custom transformer: fit on train only
+
+If the transform learns anything (a mean, a fitted mapping), write a class so CV re-fits it per fold.
+`fit` must store state from **train** only; `transform` applies it. Never compute the statistic in `__init__`
+or over the whole dataset.
+
+```python
+from sklearn.base import BaseEstimator, TransformerMixin
+
+class GroupRareCategories(BaseEstimator, TransformerMixin):
+    """Fold categories seen < min_count times in TRAIN into '__other__'."""
+    def __init__(self, min_count=10):
+        self.min_count = min_count
+    def fit(self, X, y=None):
+        counts = X.iloc[:, 0].value_counts()
+        self.keep_ = set(counts[counts >= self.min_count].index)   # learned on train fold
+        return self
+    def transform(self, X):
+        col = X.iloc[:, 0]
+        return col.where(col.isin(self.keep_), "__other__").to_frame()
+```
+
+## High-cardinality categoricals: TargetEncoder cross-fitting
+
+`sklearn.preprocessing.TargetEncoder` encodes each category by the target mean — the classic leakage trap
+if done naively. Its `fit_transform(X, y)` uses an **internal cross-fitting** scheme (each row encoded from
+*other* folds' targets), which is why `fit(X, y).transform(X)` deliberately differs and must not be used on
+train. Put it in the pipeline and let CV call `fit_transform`; never hand-roll a group-mean encode.
+
+```python
+from sklearn.preprocessing import TargetEncoder
+enc = TargetEncoder(smooth="auto", cv=5)   # cv controls the internal cross-fit
+# inside a ColumnTransformer branch for the high-cardinality columns
+```
+
+## Hyperparameter tuning — over the whole pipeline
+
+Address a step's parameter with `stepname__param` (double underscore). Tuning searches must wrap the
+*pipeline*, so preprocessing is re-fit each fold — tuning on globally-preprocessed data leaks.
+
+```python
+from sklearn.pipeline import Pipeline
+from sklearn.ensemble import HistGradientBoostingClassifier
+from sklearn.model_selection import RandomizedSearchCV, StratifiedKFold
+
+pipe = Pipeline([("pre", pre), ("clf", HistGradientBoostingClassifier(random_state=0))])
+param_dist = {
+    "clf__learning_rate": [0.01, 0.05, 0.1, 0.2],
+    "clf__max_leaf_nodes": [15, 31, 63],
+    "clf__l2_regularization": [0.0, 1.0, 10.0],
+}
+search = RandomizedSearchCV(
+    pipe, param_dist, n_iter=30,
+    cv=StratifiedKFold(5, shuffle=True, random_state=0),
+    scoring="average_precision", random_state=0, n_jobs=-1, refit=True,
+)
+search.fit(X_tr, y_tr)          # X_tr only — the test set stays sealed
+print(search.best_params_, search.best_score_)
+```
+
+- `GridSearchCV` — exhaustive; use for a small, discrete grid.
+- `RandomizedSearchCV` — sample `n_iter` combos from distributions; the default for anything nontrivial.
+- `HalvingRandomSearchCV` / `HalvingGridSearchCV` (`sklearn.model_selection`) — successive halving: cheap
+  runs on many candidates, more budget to survivors. Fastest when fits are expensive.
+
+## Nested CV — unbiased estimate of the tuning process
+
+If you report `search.best_score_` as your performance, it is optimistic (you selected on it). To estimate
+how the *whole tuning procedure* generalizes, nest an inner CV (tuning) inside an outer CV (scoring).
+
+```python
+from sklearn.model_selection import cross_val_score
+outer = StratifiedKFold(5, shuffle=True, random_state=1)
+nested = cross_val_score(search, X_tr, y_tr, cv=outer, scoring="average_precision")
+print(nested.mean(), nested.std())   # honest estimate of "tune-then-fit" on unseen data
+```
+
+## Splitters beyond StratifiedKFold
+
+```python
+from sklearn.model_selection import TimeSeriesSplit, GroupKFold, StratifiedGroupKFold
+
+# time order: train on past, test on future (expanding window); never shuffle the future into train
+for tr, te in TimeSeriesSplit(n_splits=5).split(X):
+    ...
+
+# clustered rows (same user/store): keep a group wholly in train OR test
+for tr, te in GroupKFold(n_splits=5).split(X, y, groups=user_id):
+    ...
+
+# clustered AND imbalanced: preserve class ratio while keeping groups intact
+for tr, te in StratifiedGroupKFold(n_splits=5).split(X, y, groups=user_id):
+    ...
+```
+
+Pass `groups=` through tuning too: `search.fit(X, y, groups=user_id)` with a group-aware `cv`. A random
+`KFold` on time-series or grouped data is leakage — CV will look great and production will not.
+```

package/skills/nlp/SKILL.md

@@ -0,0 +1,225 @@
+---
+name: nlp
+description: "Use when choosing how to tokenize text or which transformer type fits an NLP task, when a tokenizer over-fragments non-English text or inflates token cost, when picking a language metric, or when classification / NER / summarization output looks wrong and you can't tell whether the tokenizer, the architecture, or the metric is at fault. Covers BPE / WordPiece / SentencePiece-Unigram / byte-level BPE, special tokens, encoder (BERT) vs decoder (GPT) vs encoder-decoder (T5) choice, classification, NER, seq2seq, sentence embeddings, and metrics (F1, entity-F1, BLEU / ROUGE / chrF, perplexity, LLM-as-judge). Triggers: 'BPE or WordPiece', 'BERT vs GPT vs T5 for this task', 'tokenizer splits Catalan into too many tokens', 'is BLEU good for summaries', 'entity F1 for NER', 'quin model faig servir per classificar text'. NOT retrieval / vector search (that is embeddings-search), NOT the RAG loop (that is rag), NOT prompt wording (that is prompt-engineering), NOT training the net (that is finetuning / deep-learning)."
+tags: [nlp, tokenization, bpe, wordpiece, ner, text-classification, bleu, rouge, perplexity, transformers]
+recommends: [deep-learning, embeddings-search, rag, training-data]
+origin: risco
+---
+
+# nlp — tokenize the text, pick the model type, pick the metric
+
+You own the **language-modeling discipline**: how raw text becomes tokens, which transformer
+architecture fits a task, and which metric actually tells you whether it worked. When the
+question is "which tokenizer," "BERT or GPT or T5 for this," "why does my Catalan text cost 3×
+the tokens," or "is this BLEU score meaningful," this is the skill. You stop at retrieval, the
+RAG loop, prompt wording, and the training step itself — those route out (below).
+
+## Route out first (loud — do not duplicate these)
+
+- **Retrieval embeddings + vector search** (which embedding model, chunking, recall@k, rerank)
+  → [`../embeddings-search/SKILL.md`](../embeddings-search/SKILL.md). Sentence embeddings *live
+  here* as a task; using them to *retrieve* is theirs.
+- **The retrieve → prompt → generate → answer loop** and groundedness → [`../rag/SKILL.md`](../rag/SKILL.md).
+- **Prompt wording / few-shot / system prompts** → [`../prompt-engineering/SKILL.md`](../prompt-engineering/SKILL.md).
+- **Training the network** (LoRA/SFT, trainer loop, PyTorch) → [`../finetuning/SKILL.md`](../finetuning/SKILL.md)
+  + [`../deep-learning/SKILL.md`](../deep-learning/SKILL.md).
+- **The training corpus itself** (JSONL messages, label sets) → [`../training-data/SKILL.md`](../training-data/SKILL.md).
+
+## Decision: model type per task (get this right before anything else)
+
+Pick the architecture from the task's *shape*, not from what is trendy. A decoder LLM can
+technically classify, but a fine-tuned encoder is smaller, faster, cheaper, and usually more
+accurate on a fixed-label task.
+
+| Task shape | Architecture | Why | Example families* |
+|---|---|---|---|
+| Understand / label a whole input (classification, NER, extractive QA, similarity) | **Encoder** (bidirectional) | Attends to the full sentence both directions; cheap to fine-tune and to serve | BERT, RoBERTa, DistilBERT, ModernBERT |
+| Free-form generation, chat, few-shot | **Decoder** (autoregressive) | Attends only to prior tokens; predicts the next token | GPT-style, Llama, Gemma, Qwen |
+| Transform input → new text (summarize, translate, generative QA) | **Encoder-decoder / seq2seq** | Encoder reads all of the source, decoder writes conditioned on it | T5 / FLAN-T5, BART, mT5 |
+
+\* Architecture families are stable; specific checkpoints and their licenses are not — check the
+HF model card before you commit (licenses change; see `open-weights`). ModernBERT (2024) is a
+current long-context encoder; verify the latest at author time.
+
+The two most common own-goals: reaching for a 7B decoder to do sentiment on 5 classes (an
+encoder does it for a fraction of the cost), and forcing an encoder to *generate* (it cannot —
+it has no decoder).
+
+## 1. Tokenization
+
+Every downstream number depends on this step, and its failures are **silent**. The single
+load-bearing rule:
+
+> **Load the tokenizer that shipped with the checkpoint, and use the same one at train and
+> inference.** `AutoTokenizer.from_pretrained(same_checkpoint)`. A train/inference tokenizer
+> mismatch — different vocab, different special tokens, different casing/normalization — maps
+> text to token ids the model never saw and corrupts everything downstream with no error.
+
+```python
+from transformers import AutoTokenizer   # transformers current major ~v5 (verify at author time)
+
+tok = AutoTokenizer.from_pretrained("bert-base-cased")
+enc = tok("Tokenizers matter.", return_offsets_mapping=True)
+tok.convert_ids_to_tokens(enc["input_ids"])
+# ['[CLS]', 'Token', '##izers', 'matter', '.', '[SEP]']  — note WordPiece '##' continuation + added specials
+```
+
+**The four algorithms** (full mechanics in `references/tokenization.md`):
+
+| Algorithm | Builds vocab by… | Applies by… | Used by |
+|---|---|---|---|
+| **BPE** | merging the most frequent adjacent pair, repeatedly | split to chars, replay learned merges | GPT-2 (byte-level), many |
+| **WordPiece** | merging pairs that maximize a likelihood score | longest-match subword from the front (`##` continuations) | BERT family |
+| **Unigram** (SentencePiece) | start large, *remove* tokens that least hurt corpus likelihood | most-probable segmentation | T5, ALBERT, mT5 |
+| **Byte-level BPE** | BPE over the 256 raw **bytes**, not Unicode chars | same as BPE on bytes | GPT-2, RoBERTa |
+
+- **Byte-level BPE has no `[UNK]`.** Base vocab is exactly 256 (all byte values), so every
+  emoji, accent, and script maps to *some* byte sequence — nothing falls out as unknown
+  (verified: HF NLP course ch.6). WordPiece/word-level tokenizers *do* have `[UNK]` and lose OOV
+  content.
+- **SentencePiece is reversible** — it treats space as a normal symbol (the `▁` meta-symbol), so
+  `decode(encode(x)) == x` without language-specific detokenization rules. That is why it
+  dominates multilingual models.
+
+**Special tokens are not decoration.** `[CLS]`/`<s>` carries the pooled sentence
+representation for classification; `[SEP]`/`</s>` marks segment/end; `[PAD]` fills a batch (and
+must be masked out via `attention_mask`); `[MASK]` is the MLM target; `[UNK]` is the fallback.
+Names differ by model (`[CLS]` in BERT vs `<s>` in RoBERTa) — another reason to never hand-roll
+the tokenizer.
+
+**Why it matters — three concrete costs:**
+
+- **$ cost & context.** Token count *is* the bill and the context budget. Fewer tokens per
+  sentence = cheaper calls and more room in the window.
+- **OOV / information loss.** A tokenizer that emits `[UNK]` throws away content it can't
+  represent; byte-level/SentencePiece degrade gracefully instead.
+- **Fairness.** Vocab trained mostly on English fragments other scripts far harder — the same
+  meaning costs more tokens, more money, and more latency (section 5).
+
+## 2. Tasks
+
+### Text classification (encoder + classification head)
+```python
+from transformers import pipeline
+clf = pipeline("text-classification", model="distilbert-base-uncased-finetuned-sst-2-english")
+clf("The service was slow but the food was incredible.")
+# [{'label': 'POSITIVE', 'score': 0.99...}]
+```
+Metric: accuracy on balanced data; **macro-F1** the moment classes are imbalanced (accuracy
+lies when 95% of rows are one class).
+
+### Token classification / NER (encoder, per-token labels)
+Labels are **B-/I-/O** spans aligned to subword tokens: the first subword of a word gets the
+label, continuation subwords and special tokens get `-100` (ignored by the loss). Evaluate with
+**seqeval** at the **entity** level, never per-token accuracy (per-token accuracy is inflated by
+the flood of `O` tokens).
+```python
+from transformers import pipeline
+ner = pipeline("token-classification", aggregation_strategy="simple")
+ner("Ada Lovelace worked in London.")
+# groups subwords back into entities: PER 'Ada Lovelace', LOC 'London'
+```
+
+### Seq2seq — summarization / translation (encoder-decoder)
+```python
+summ = pipeline("summarization", model="facebook/bart-large-cnn")
+summ(long_article, max_length=130, min_length=30)
+```
+Metric: **ROUGE** for summarization, **BLEU/chrF** for translation — with the heavy caveat in
+section 4.
+
+### Sentence embeddings (SBERT — the *task*, not retrieval)
+```python
+from sentence_transformers import SentenceTransformer   # sentence-transformers ~v5 (verify)
+model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
+emb = model.encode(["The weather is lovely today.", "It's so sunny outside!"])
+model.similarity(emb, emb)   # semantic textual similarity / clustering / paraphrase mining
+```
+Producing/judging embeddings **for retrieval** (model choice, chunking, recall@k, rerankers) is
+[`embeddings-search`](../embeddings-search/SKILL.md), not here.
+
+## 3. Evaluation — pick the metric that matches the task
+
+| Task | Primary metric | Catches | Trap |
+|---|---|---|---|
+| Classification | accuracy + **macro-F1** | wrong labels | accuracy hides minority-class failure |
+| NER / token | **entity-level F1** (seqeval) | missed/partial spans | per-token accuracy is inflated by `O` |
+| Translation | **BLEU / chrF** | n-gram overlap w/ reference | weak on meaning; chrF better for morphology |
+| Summarization | **ROUGE** (1/2/L) | recall of reference n-grams | rewards copying; blind to faithfulness |
+| Generation (LM) | **perplexity** | how well the model predicts held-out text | tokenizer-dependent — not comparable across tokenizers |
+| Open-ended / chat | **LLM-as-judge** + human | quality overlap metrics miss | judge bias (position, verbosity, self-preference) |
+
+**The caveat that governs this whole section:** BLEU, ROUGE, and chrF are n-gram/character
+overlap metrics and **correlate weakly with human judgment on open-ended and creative
+generation** — they reward matching the reference's exact phrasing, so a correct paraphrase
+scores low and a fluent-but-wrong copy scores high (well documented; e.g. the summarization and
+MT-evaluation literature). Use them for **regression tracking on a fixed reference set**, never
+as the final verdict on quality. For open-ended output, use an **LLM-as-judge rubric plus a
+human spot-check** — and know the judge has its own biases (position, verbosity, self-preference),
+so pin the rubric and randomize order.
+
+**Perplexity** = exp(mean token NLL): lower means the model predicts held-out text better. It is
+**tokenizer-dependent**, so two models with different tokenizers have non-comparable perplexities
+— only compare within the same tokenizer/vocab. Runnable snippets for seqeval, sacrebleu, ROUGE,
+perplexity, and an LLM-judge harness are in `references/evaluation.md`.
+
+## 4. Multilingual pitfalls
+
+The English-centric trap: a tokenizer whose vocab was learned mostly on English **over-fragments
+other scripts**. The same sentence in Ukrainian, Arabic, Hindi, or even accented Catalan can take
+**2–15× more tokens** than its English equivalent (Petrov et al., *Language Model Tokenizers
+Introduce Unfairness Between Languages*, NeurIPS 2023). That "fertility" (tokens per word)
+inflation is a triple tax:
+
+- **Money** — more tokens per identical meaning = a proportionally larger bill for the same work.
+- **Context** — over-fragmented text eats the window faster, so fewer few-shot examples fit and
+  long documents truncate sooner.
+- **Quality** — sequences fragmented into byte-shards are harder to model, degrading accuracy for
+  exactly the users the tool already serves worst.
+
+Mitigations: prefer a **multilingual tokenizer/model** (mT5, XLM-R, a SentencePiece-based model)
+whose vocab actually covers your languages; **measure fertility** on your own corpus (tokens per
+word, per language) before you commit; and don't benchmark cost or latency only on English.
+
+## Guardrails / gotchas
+
+- **Tokenizer must match the checkpoint, at train and inference.** Mismatch corrupts silently,
+  no error. The most expensive bug in this skill.
+- **Encoders can't generate; don't classify with a giant decoder by default.** Match architecture
+  to task shape.
+- **BLEU/ROUGE/chrF ≠ quality on open-ended text.** Overlap metrics; weak human correlation. Track
+  regressions with them; judge quality with an LLM-judge + human.
+- **Entity-F1 (seqeval), not token accuracy, for NER.** `O` tokens inflate accuracy toward 1.0.
+- **macro-F1, not accuracy, on imbalanced classes.**
+- **Perplexity is tokenizer-relative** — never compare it across different tokenizers.
+- **Don't benchmark tokenization/cost only in English** — fertility varies 2–15× across scripts.
+- **Never assert a model's license from memory** — check the current model card; license classes
+  shift (Llama = Meta Community license, not OSI-open; Gemma = custom terms; etc.).
+
+## Related skills
+
+- [`embeddings-search`](../embeddings-search/SKILL.md) — retrieval embeddings, chunking, recall@k,
+  reranking. NLP owns *making/judging* sentence embeddings as a task; using them to search is theirs.
+- [`rag`](../rag/SKILL.md) — the full retrieve→generate answer loop and groundedness.
+- [`prompt-engineering`](../prompt-engineering/SKILL.md) — the wording of the prompt.
+- [`finetuning`](../finetuning/SKILL.md) + [`deep-learning`](../deep-learning/SKILL.md) — actually
+  training/adapting the network (this skill picks the type and metric; those move the weights).
+- [`training-data`](../training-data/SKILL.md) — building the labeled corpus you train on.
+
+## Checklist
+
+- [ ] Architecture picked from task shape (encoder / decoder / enc-dec), not habit.
+- [ ] Tokenizer loaded from the *same* checkpoint, used identically at train and inference.
+- [ ] Tokenizer choice justified vs OOV, cost, and — if multilingual — measured fertility.
+- [ ] Special tokens and `attention_mask` handled (padding masked, `-100` on ignored labels).
+- [ ] Metric matches the task: macro-F1 (imbalanced), entity-F1/seqeval (NER), ROUGE/BLEU/chrF
+      only as a regression signal, LLM-judge + human for open-ended.
+- [ ] Perplexity compared only within one tokenizer.
+- [ ] Retrieval / RAG / prompt / training concerns routed to the sibling skill, not re-solved here.
+
+## References
+
+- `references/tokenization.md` — BPE/WordPiece/Unigram/byte-level training mechanics, special
+  tokens per family, offset mapping, and a fertility-measuring snippet.
+- `references/evaluation.md` — runnable seqeval, sacrebleu (BLEU/chrF), ROUGE, perplexity, and an
+  LLM-as-judge rubric, with when each lies.

package/skills/nlp/evals/README.md

@@ -0,0 +1,10 @@
+# Evals — nlp
+
+Routing and capability checks for a human or LLM grader, not an automated suite. Read each
+`should_trigger` prompt and confirm an agent would reach for `nlp` (tokenizer choice, model-type
+decision, task modeling, or metric selection); read each `should_not_trigger` prompt and confirm
+it routes to the named sibling (`embeddings-search`, `rag`, `prompt-engineering`, `finetuning`,
+`training-data`) for the stated reason. Grade the `capability` scenario by checking the produced
+answer against every `must_include` bullet — the current API/method, the load-bearing guardrail
+(tokenizer mismatch, fertility tax), and the honesty hedge (BLEU/ROUGE weak on open-ended gen,
+licenses must be checked). A pass covers all routing cases plus full rubric coverage.

package/skills/nlp/evals/cases.yaml

@@ -0,0 +1,45 @@
+skill: nlp
+
+should_trigger:
+  - prompt: "Should I use BPE or WordPiece for my tokenizer, and what's the difference?"
+    why: "Core tokenization-algorithm choice this skill owns; no retrieval or training context."
+  - prompt: "BERT vs GPT vs T5 — which one do I pick for classifying support tickets into 6 categories?"
+    why: "Model-type-per-task decision (encoder vs decoder vs enc-dec) is the skill's opening decision gate."
+  - prompt: "My tokenizer splits Catalan text into way more tokens than the English equivalent, so it costs more — why?"
+    why: "Multilingual over-fragmentation / fertility tax is section 4; the cost symptom points squarely here."
+  - prompt: "Is BLEU or ROUGE a meaningful metric for judging my summaries, or is it misleading?"
+    why: "Metric selection plus the weak-correlation caveat for open-ended generation, owned in section 3."
+  - prompt: "How do I evaluate a NER model properly — entity-level F1 vs per-token accuracy?"
+    why: "Token-classification evaluation with seqeval entity-F1 is explicitly the skill's task + metric."
+  - prompt: "quin model faig servir per classificar text: un encoder tipus BERT o un LLM generatiu?"
+    why: "Catalan trigger on the encoder-vs-decoder model-type decision, the skill's primary gate."
+  - prompt: "My fine-tuned classifier gives garbage at inference but was fine in training — could the tokenizer be the problem?"
+    why: "Non-obvious: the classic train/inference tokenizer mismatch that corrupts silently, the skill's load-bearing guardrail."
+
+should_not_trigger:
+  - prompt: "Which embedding model and chunk size should I use to make semantic search return relevant results?"
+    route_to: embeddings-search
+    why: "Retrieval embedding choice, chunking and recall@k are the retrieval technique layer, not task modeling."
+  - prompt: "Build the full RAG pipeline: retrieve context, prompt the model, and generate a grounded answer."
+    route_to: rag
+    why: "End-to-end retrieve→generate answer loop and groundedness live in rag, past where nlp stops."
+  - prompt: "Rewrite my system prompt with few-shot examples so the model stops refusing valid requests."
+    route_to: prompt-engineering
+    why: "Prompt wording / few-shot design, not tokenization, architecture, or metrics."
+  - prompt: "Fine-tune Llama with LoRA on my instruction dataset using trl and peft."
+    route_to: finetuning
+    why: "Actually training/adapting the network is the finetuning method skill, not picking type/metric."
+  - prompt: "Convert my customer chats into a JSONL messages dataset with system/user/assistant turns for training."
+    route_to: training-data
+    why: "Building the training corpus format is training-data, not NLP task modeling or metrics."
+
+capability:
+  - scenario: "A team has 40k multilingual support tickets (English + Catalan + Arabic) to auto-route into 8 categories and also to extract product names as entities. They're unsure whether to use a big generative LLM, why Arabic tickets seem to cost far more tokens, and how to know if the models are any good."
+    must_include:
+      - "Picks an encoder (BERT-family / ModernBERT, multilingual e.g. XLM-R) with a classification head for the 8-way routing rather than a large generative decoder, justifying cost/latency/accuracy, and hedges that specific checkpoints and licenses must be checked on the model card"
+      - "Loads the tokenizer from the same checkpoint and uses it identically at train and inference, calling out that a train/inference tokenizer mismatch corrupts results silently"
+      - "Explains the Arabic token blow-up as tokenizer over-fragmentation / fertility (2–15x more tokens on non-English scripts), recommends a multilingual SentencePiece-based tokenizer and measuring fertility per language before committing"
+      - "Uses a token-classification (NER) setup with B-/I-/O labels aligned to subwords (-100 on continuations/specials) for product-name extraction"
+      - "Evaluates routing with macro-F1 (not raw accuracy, because classes are imbalanced) and NER with entity-level F1 via seqeval (not per-token accuracy)"
+      - "Notes that if any free-text summarization/quality judgment is added, BLEU/ROUGE/chrF correlate weakly with human judgment on open-ended output and an LLM-as-judge plus human spot-check is needed, with judge-bias mitigation"
+      - "Routes retrieval/vector search to embeddings-search, the RAG answer loop to rag, prompt wording to prompt-engineering, and the actual training to finetuning/deep-learning"