@dsalazardev/ai-hub-cli 0.0.0-stage → 0.1.0

package/skills/shared/llm-pipeline/references/litellm-router.md

@@ -0,0 +1,101 @@
+# LiteLLM Router / Proxy config (v1.83.x)
+
+Fuller config than the SKILL body. Version-pinned to the **v1.83.3-stable** line (also recent: v1.81.9-stable, v1.81.13). LiteLLM exposes a unified OpenAI-format `completion()` across 100+ providers, with retry/fallback, cost tracking, and budget management, as an SDK or a Proxy Server.
+
+## SDK Router with every fallback bucket
+
+```python
+from litellm import Router
+
+router = Router(
+    model_list=[
+        {"model_name": "smart",
+         "litellm_params": {"model": "anthropic/claude-sonnet-4-6",
+                            "api_key": "os.environ/ANTHROPIC_API_KEY",
+                            "timeout": 30, "order": 1}},
+        {"model_name": "smart",          # same group, order 2 = first internal fallback
+         "litellm_params": {"model": "openai/gpt-4o",
+                            "api_key": "os.environ/OPENAI_API_KEY",
+                            "timeout": 30, "order": 2}},
+        {"model_name": "cheap",
+         "litellm_params": {"model": "anthropic/claude-haiku-4-5",
+                            "api_key": "os.environ/ANTHROPIC_API_KEY",
+                            "timeout": 20}},
+        {"model_name": "long-ctx",
+         "litellm_params": {"model": "openai/gpt-4o",
+                            "api_key": "os.environ/OPENAI_API_KEY", "timeout": 60}},
+    ],
+    # cross-group fallbacks (after order levels in a group are exhausted)
+    fallbacks=[{"smart": ["cheap"]}],
+    content_policy_fallbacks=[{"smart": ["smart"]}],   # ContentPolicyViolationError
+    context_window_fallbacks=[{"smart": ["long-ctx"]}],# ContextWindowExceededError
+    default_fallbacks=["cheap"],                         # catch-all last resort
+    num_retries=2,            # bounded retries PER order level
+    timeout=30,              # router-wide hard cap
+    retry_after=1,
+)
+```
+
+Order semantics: a request to an `order=1` deployment that fails (connection error, 404, 429, ...) auto-tries `order=2`, then `order=3`. Each order level runs its own `num_retries` before escalating. Once all orders in the group are exhausted, the `fallbacks`/specialized buckets fire. Retry backoff is exponential from `INITIAL_RETRY_DELAY` 0.2s up to `MAX_RETRY_DELAY` 10s, with jitter.
+
+## Keys come from env, never inline
+
+Use `os.environ/VAR` indirection (proxy YAML) or read `os.environ[...]` in SDK. A literal `sk-...` in a model_list is a leaked credential and a verify.sh FAIL.
+
+## Proxy Server config (config.yaml)
+
+```yaml
+model_list:
+  - model_name: smart
+    litellm_params:
+      model: anthropic/claude-sonnet-4-6
+      api_key: os.environ/ANTHROPIC_API_KEY
+      timeout: 30
+  - model_name: smart-backup
+    litellm_params:
+      model: openai/gpt-4o
+      api_key: os.environ/OPENAI_API_KEY
+      timeout: 30
+
+router_settings:
+  num_retries: 2
+  timeout: 30
+  fallbacks: [{"smart": ["smart-backup"]}]
+  context_window_fallbacks: [{"smart": ["smart-backup"]}]
+
+litellm_settings:
+  cache: true
+  cache_params:
+    type: redis
+    host: os.environ/REDIS_HOST
+    port: os.environ/REDIS_PORT
+    ttl: 600
+```
+
+## Budget & rate limits
+
+```yaml
+litellm_settings:
+  max_budget: 100         # $ hard cap across the proxy; aborts over-budget requests
+  budget_duration: 30d
+general_settings:
+  master_key: os.environ/LITELLM_MASTER_KEY
+```
+
+Per-key / per-tenant budgets are set on virtual keys (`/key/generate` with `max_budget` and `tpm_limit`/`rpm_limit`). Budgets that ABORT are the point — a logged-only budget does not stop a runaway loop.
+
+## Cost callback
+
+```python
+import litellm
+
+def track(kwargs, response, start, end):
+    cost = kwargs.get("response_cost")
+    # emit per-step: model, tokens, cost, latency, cache_hit, fallback used
+    log.info("llm_call", model=kwargs["model"], cost=cost,
+             latency_ms=(end - start).total_seconds() * 1000)
+
+litellm.success_callback = [track]
+```
+
+The cost/latency numbers belong in your observability backbone; attribution dashboards belong to the cost-tracking skill, not here.

package/skills/shared/llm-pipeline/scripts/verify.sh

@@ -0,0 +1,169 @@
+#!/usr/bin/env bash
+# verify.sh — static, read-only checks for an llm-pipeline artifact (gateway/router
+# config or typed-chain code). No live API calls. Offline and deterministic.
+#
+# Usage: verify.sh [PATH]   (PATH = file or dir; defaults to current directory)
+# Exits 0 when every applicable check passes (including an empty/clean target),
+# non-zero when any check FAILs. Checks that do not apply to the artifacts present
+# are SKIPped, never failed.
+
+set -u
+
+TARGET="${1:-.}"
+
+if [[ ! -e "$TARGET" ]]; then
+  echo "verify.sh: path not found: $TARGET" >&2
+  exit 2
+fi
+
+fail_count=0
+pass() { printf 'PASS  %s\n' "$1"; }
+fail() { printf 'FAIL  %s\n' "$1"; fail_count=$((fail_count + 1)); }
+skip() { printf 'SKIP  %s\n' "$1"; }
+
+# Collect candidate source and config files (read-only). Portable to bash 3.2
+# (macOS default) — no mapfile, newline-delimited via a temp array build.
+CODE_FILES=(); YAML_FILES=(); JSON_FILES=()
+while IFS= read -r f; do [[ -n "$f" ]] && CODE_FILES+=("$f"); done < <(find "$TARGET" -type f \( -name '*.py' -o -name '*.ts' -o -name '*.js' \) 2>/dev/null)
+while IFS= read -r f; do [[ -n "$f" ]] && YAML_FILES+=("$f"); done < <(find "$TARGET" -type f \( -name '*.yaml' -o -name '*.yml' \) 2>/dev/null)
+while IFS= read -r f; do [[ -n "$f" ]] && JSON_FILES+=("$f"); done < <(find "$TARGET" -type f -name '*.json' 2>/dev/null)
+ALL_FILES=("${CODE_FILES[@]+"${CODE_FILES[@]}"}" "${YAML_FILES[@]+"${YAML_FILES[@]}"}" "${JSON_FILES[@]+"${JSON_FILES[@]}"}")
+
+if [[ ${#ALL_FILES[@]} -eq 0 ]]; then
+  echo "verify.sh: no .py/.ts/.js/.yaml/.yml/.json files under $TARGET — nothing to check."
+  exit 0
+fi
+
+# grep helper across a file list; prints "file:line:match" lines, empty if none.
+grep_files() { # $1 = pattern, rest = files
+  local pat="$1"; shift
+  [[ $# -eq 0 ]] && return 0
+  grep -HInE "$pat" "$@" 2>/dev/null   # -H so the filename prefix is always present
+}
+
+# --- Check 1: no hardcoded provider keys (always applicable) ---------------
+KEY_HITS="$(grep_files '(sk-[A-Za-z0-9_-]{16,}|sk-ant-[A-Za-z0-9_-]{16,})' "${ALL_FILES[@]}")"
+if [[ -n "$KEY_HITS" ]]; then
+  fail "no hardcoded API keys (found literal key(s); use env vars)"
+  echo "$KEY_HITS" | sed 's/^/      /'
+else
+  pass "no hardcoded API keys"
+fi
+
+# --- Check 2: every completion/chat call site sets a timeout ---------------
+# A "call site" line invokes completion(/chat.completions.create/responses.create.
+CALL_PAT='(\.completion\(|chat\.completions\.create\(|responses\.(create|parse)\()'
+CALL_HITS="$(grep_files "$CALL_PAT" "${CODE_FILES[@]+"${CODE_FILES[@]}"}")"
+if [[ -z "$CALL_HITS" ]]; then
+  skip "timeout on every call (no LLM call sites found)"
+else
+  # A call line is OK if it (or a nearby Router/client default) sets timeout.
+  # Heuristic, read-only: flag call lines on which 'timeout' does not appear AND
+  # whose file has no router/client-level timeout default.
+  unbounded=""
+  while IFS= read -r line; do
+    file="${line%%:*}"
+    if echo "$line" | grep -qE 'timeout'; then continue; fi
+    # allow a router/client default timeout in the same file
+    if grep -qE 'timeout[[:space:]]*[:=]' "$file" 2>/dev/null; then continue; fi
+    unbounded+="$line"$'\n'
+  done <<< "$CALL_HITS"
+  if [[ -n "${unbounded//[$'\n']/}" ]]; then
+    fail "timeout on every call (call site(s) with no timeout)"
+    printf '%s' "$unbounded" | sed '/^$/d;s/^/      /'
+  else
+    pass "timeout on every call"
+  fi
+fi
+
+# --- Check 3: retries are bounded (no while True retry loops) --------------
+LOOP_HITS="$(grep_files 'while[[:space:]]+True' "${CODE_FILES[@]+"${CODE_FILES[@]}"}")"
+# only treat as a retry loop if a retry/except keyword is nearby in the file
+retry_loops=""
+if [[ -n "$LOOP_HITS" ]]; then
+  while IFS= read -r line; do
+    file="${line%%:*}"
+    if grep -qiE '(retry|num_retries|except|completion\(|create\()' "$file" 2>/dev/null; then
+      retry_loops+="$line"$'\n'
+    fi
+  done <<< "$LOOP_HITS"
+fi
+if [[ -n "${retry_loops//[$'\n']/}" ]]; then
+  fail "retries bounded (unbounded 'while True' retry loop)"
+  printf '%s' "$retry_loops" | sed '/^$/d;s/^/      /'
+else
+  pass "retries bounded"
+fi
+
+# --- Check 4: a fallback is configured when a router/model_list exists -----
+HAS_ROUTER="$(grep_files '(Router\(|model_list|model_name)' "${ALL_FILES[@]}")"
+if [[ -z "$HAS_ROUTER" ]]; then
+  skip "fallback configured (no router/model_list present)"
+else
+  HAS_FALLBACK="$(grep_files '(fallbacks|content_policy_fallbacks|context_window_fallbacks|default_fallbacks)' "${ALL_FILES[@]}")"
+  if [[ -n "$HAS_FALLBACK" ]]; then
+    pass "fallback configured"
+  else
+    fail "fallback configured (router/model_list present but no fallback list)"
+  fi
+fi
+
+# --- Check 5: config files parse and list >=2 model entries ----------------
+check_config_models() { # $1 = file
+  local f="$1" count=""
+  case "$f" in
+    *.json)
+      if command -v python3 >/dev/null 2>&1; then
+        count="$(python3 - "$f" <<'PY' 2>/dev/null
+import json,sys,re
+d=open(sys.argv[1]).read()
+try: obj=json.loads(d)
+except Exception: print("PARSEFAIL"); sys.exit(0)
+s=json.dumps(obj)
+print(len(re.findall(r'"model_name"|"model"\s*:', s)))
+PY
+)"
+      fi ;;
+    *.yaml|*.yml)
+      if command -v python3 >/dev/null 2>&1 && python3 -c 'import yaml' 2>/dev/null; then
+        count="$(python3 - "$f" <<'PY' 2>/dev/null
+import yaml,sys,re
+try: obj=yaml.safe_load(open(sys.argv[1]).read())
+except Exception: print("PARSEFAIL"); sys.exit(0)
+print(len(re.findall(r'model_name|(^|\s)model\s*:', str(obj))))
+PY
+)"
+      else
+        # no yaml lib: fall back to counting model_name occurrences textually
+        count="$(grep -cE 'model_name' "$f" 2>/dev/null || echo 0)"
+      fi ;;
+  esac
+  echo "${count:-}"
+}
+
+CONFIG_FILES=("${YAML_FILES[@]+"${YAML_FILES[@]}"}" "${JSON_FILES[@]+"${JSON_FILES[@]}"}")
+relevant_config=0
+for f in "${CONFIG_FILES[@]+"${CONFIG_FILES[@]}"}"; do
+  [[ -z "$f" ]] && continue
+  if grep -qE '(model_list|model_name|litellm|router_settings)' "$f" 2>/dev/null; then
+    relevant_config=1
+    res="$(check_config_models "$f")"
+    if [[ "$res" == "PARSEFAIL" ]]; then
+      fail "config parses ($f does not parse)"
+    elif [[ "$res" =~ ^[0-9]+$ ]] && [[ "$res" -ge 2 ]]; then
+      pass "config parses with >=2 model entries ($f)"
+    else
+      fail "config has >=2 model entries ($f has ${res:-0}; need a fallback target)"
+    fi
+  fi
+done
+[[ $relevant_config -eq 0 ]] && skip "config parses with >=2 models (no router config file present)"
+
+echo "---"
+if [[ $fail_count -eq 0 ]]; then
+  echo "verify.sh: all applicable checks passed."
+  exit 0
+else
+  echo "verify.sh: $fail_count check(s) FAILED."
+  exit 1
+fi

package/skills/shared/machine-learning/SKILL.md

@@ -0,0 +1,228 @@
+---
+name: machine-learning
+description: "Use when predicting a column from rows of tabular features with classic models — scikit-learn pipelines, RandomForest, XGBoost/LightGBM, leak-free cross-validation, metrics for imbalanced classes, or a model that aced CV then collapsed in production. NOT PyTorch neural nets (that is `deep-learning`), NOT cleaning the dirty table first (that is `data-cleaning`), NOT forecasting a dated series (that is `forecasting`), NOT text/token modeling (that is `nlp`)."
+tags: [machine-learning, scikit-learn, sklearn, xgboost, lightgbm, tabular, gbdt, cross-validation, data-leakage, classification, regression, pipeline]
+recommends: [deep-learning, data-cleaning, python, training-data]
+origin: ai-hub
+---
+
+# Machine learning — classic/tabular models, done without lying to yourself
+
+Tabular ML is easy to *run* and easy to *fool yourself with*. The deliverable is never "the notebook
+printed 0.99" — it is an **honest estimate of how the model behaves on data it has never seen**: a
+`Pipeline` that fits every transform on train only, a metric that survives class imbalance, and a
+`DummyClassifier` baseline it beats. A number you can't reproduce on a sacred test set you touched exactly
+once isn't a result — it's a leak you haven't found yet.
+
+## Is this the right skill? (decide first)
+
+| Your situation | Reach for |
+| --- | --- |
+| Rows of features, predict a column, with trees / linear models / sklearn | **machine-learning** (this skill) |
+| Images, audio, long text, sequences, or you need a neural net / PyTorch | [`deep-learning`](../deep-learning/SKILL.md) |
+| The table is still dirty (nulls, dupes, mixed types, bad dates) | [`data-cleaning`](../data-cleaning/SKILL.md) **first** — it hands you a validated table |
+| Text/token classification, NER, tokenization, LLM-adjacent NLP metrics | [`nlp`](../nlp/SKILL.md) (a TF-IDF + linear/GBDT baseline still lives happily in this skill's pipeline) |
+| KPIs, dashboards, "explain the business" | [`analytics`](../analytics/SKILL.md) / [`business-intelligence`](../business-intelligence/SKILL.md) |
+| Forecast a dated series forward (revenue next quarter) | [`forecasting`](../forecasting/SKILL.md) |
+| Building a training corpus of JSONL messages / preference pairs for an LLM | [`training-data`](../training-data/SKILL.md) |
+
+This skill **starts** at a clean, validated table (rows × features + a target) and **ends** at a fitted,
+honestly-scored model with a test-set number and a baseline it beats. Cleaning is upstream — consume the
+validated frame `data-cleaning` produced; don't re-teach it here.
+
+## Version reality (verify at author time — this line moves monthly)
+
+Verified 2026-07: **scikit-learn current major ~1.9** (1.9.0 shipped 2026-06-02, Python 3.11–3.14) — do
+NOT pin from memory; check the current stable at scikit-learn.org, the 1.x line ships every few months.
+GBDTs: **XGBoost 3.x** and **LightGBM 4.x** (`xgboost` 3.3, `lightgbm` 4.6 current), plus sklearn's own
+**`HistGradientBoostingClassifier`/`...Regressor`** — a fast native GBDT that eats NaN and (with
+`categorical_features="from_dtype"`) categoricals with **no** preprocessing. Pin what you ship
+([`python`](../python/SKILL.md) owns the environment and the pinning); state versions as "~X (verify)",
+never as frozen fact.
+
+## The one rule everything else serves: fit on train only
+
+Every preprocessing step — imputation, scaling, encoding, feature selection, target encoding, resampling —
+learns parameters from data. Learn them from rows the model is later *scored* on and the score inflates
+while production underperforms: that is **leakage**, the #1 way tabular ML lies. The whole apparatus below —
+`Pipeline`, `ColumnTransformer`, CV, the untouched test set — exists to make "fit on train only" automatic
+instead of something you remember to do by hand (you won't).
+
+## scikit-learn: estimators, Pipeline, ColumnTransformer
+
+Every model is an **estimator** with the same contract: `fit(X, y)`, then `predict(X)` /
+`predict_proba(X)` (classifiers) / `score(X, y)`. Transformers add `transform(X)` / `fit_transform(X, y)`.
+A **`Pipeline`** chains transformers + a final estimator into one estimator — so `fit` fits every step on
+train, and `predict`/CV `transform`s test data with parameters learned on train. That is the leakage guard.
+
+A **`ColumnTransformer`** routes different columns down different transformer branches (scale the numerics,
+encode the categoricals) and stitches the result back together — all still *inside* the pipeline.
+
+```python
+from sklearn.compose import ColumnTransformer, make_column_selector as mcs
+from sklearn.pipeline import Pipeline
+from sklearn.impute import SimpleImputer
+from sklearn.preprocessing import StandardScaler, OneHotEncoder
+from sklearn.linear_model import LogisticRegression
+
+numeric = Pipeline([("impute", SimpleImputer(strategy="median")),
+                    ("scale",  StandardScaler())])           # scaling matters for LINEAR/SVM/KNN
+categoric = Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
+                      ("onehot", OneHotEncoder(handle_unknown="ignore"))])  # unseen category -> all-zeros, no crash
+
+pre = ColumnTransformer([
+    ("num", numeric,   mcs(dtype_include="number")),
+    ("cat", categoric, mcs(dtype_include=["object", "category"])),
+], remainder="drop")
+
+model = Pipeline([("pre", pre), ("clf", LogisticRegression(max_iter=1000, class_weight="balanced"))])
+model.fit(X_train, y_train)          # imputers/scaler/encoder ALL fit on X_train only
+model.predict_proba(X_test)          # X_test transformed with train-learned params — no leak
+```
+
+Two load-bearing details: `OneHotEncoder(handle_unknown="ignore")` so a category unseen in train doesn't
+crash prediction, and `remainder="drop"` so unrouted columns don't silently leak through raw;
+`model.set_output(transform="pandas")` keeps named-column DataFrames through the pipeline.
+
+**Trees skip most of this** — scaling is pointless for tree models, and `HistGradientBoostingClassifier`
+ingests NaN and categoricals natively, so its pipeline is often just the estimator. Preprocess for the
+*model that needs it*, not as a ritual. More patterns (`make_column_transformer`, `FunctionTransformer`)
+→ [references/pipelines-and-cv.md](references/pipelines-and-cv.md).
+
+## GBDTs are your default on tabular data
+
+Gradient-boosted decision trees are the correct first (and usually last) model for tabular problems.
+This isn't taste: **Grinsztajn, Oyallon & Varoquaux (2022), "Why do tree-based models still outperform
+deep learning on tabular data?"** ([arXiv:2207.08815](https://arxiv.org/abs/2207.08815)) benchmarked across
+45 datasets and found GBDTs beat tuned neural nets on medium-sized tabular data, tracing it to three
+inductive biases NNs lack: **robustness to uninformative features**, **not being rotationally invariant**
+(so they exploit the meaning of individual columns), and ease of **learning irregular / non-smooth target
+functions**. Start with a GBDT; reach for `deep-learning` on tabular only with a specific reason.
+
+| Library | Import | Reach for it when |
+| --- | --- | --- |
+| `HistGradientBoostingClassifier` | `sklearn.ensemble` | Default. Fast, zero extra deps, native NaN + categorical. |
+| XGBoost 3.x | `xgboost.XGBClassifier` | Battle-tested; `early_stopping_rounds`, rich regularization. |
+| LightGBM 4.x | `lightgbm.LGBMClassifier` | Fastest on wide/large data; leaf-wise; strong native categoricals. |
+
+All three expose the sklearn estimator API, so they drop into the pipeline and CV below unchanged. Don't
+agonize over XGBoost-vs-LightGBM before you have a baseline and a leak-free CV — split discipline dwarfs the
+library choice. Tuning knobs (`learning_rate`, `num_leaves`/`max_depth`, early stopping, `monotonic_cst`)
+and importance/SHAP → [references/gbdt-and-tuning.md](references/gbdt-and-tuning.md).
+
+## Leakage-safe feature engineering
+
+Feature engineering is where leakage sneaks back in after the pipeline "protected" you. Rules:
+
+- **Any transform that learns from data goes INSIDE the pipeline**, so CV re-fits it per fold. A scaler fit
+  on the whole dataset, a `SelectKBest` run before splitting, an imputer using the global mean — each leaks
+  test statistics into train. The `common_pitfalls` canonical example: `SelectKBest(k=25).fit_transform(X, y)`
+  *before* `train_test_split` produces a beautiful, meaningless score.
+- **Target/mean encoding of high-cardinality categoricals must cross-fit.** sklearn's
+  `preprocessing.TargetEncoder` does this: its `fit_transform(X, y)` uses an internal cross-fitting scheme
+  so each row is encoded from *other* folds' targets — `fit(X, y).transform(X)` deliberately differs and
+  would leak. Use `fit_transform` on train (inside the pipeline); never hand-roll a group-mean encoder.
+- **No target-derived features.** A column computed from the label (or a near-proxy: "was_refunded" when
+  predicting "will_refund") is leakage wearing a feature's clothes. If a feature is impossibly predictive,
+  suspect it.
+- **Respect time.** With any temporal structure, a feature may use only information available *at prediction
+  time* — no future aggregates, no lifetime values that include post-cutoff rows. Split by time
+  (`TimeSeriesSplit`), not randomly.
+
+## Split + cross-validation: protect the sacred test set
+
+Hold out a **test set once, at the very start** (`train_test_split(..., stratify=y, random_state=0)`) and
+do not look at it until you have a single final model. Every glance — tuning, feature choice, "let me just
+check" — bleeds information and re-inflates the estimate. Tune and compare with **cross-validation on the
+training portion only**; the test set is the one honest number at the end (see the lifecycle below). Score
+with `cross_validate(pipeline, X_tr, y_tr, cv=cv, scoring=[...], return_train_score=True)` — a large
+train-minus-test gap is your overfitting alarm.
+
+Pick the splitter to match the data ([cross_validation docs](https://scikit-learn.org/stable/modules/cross_validation.html)):
+
+- **`StratifiedKFold`** — default for classification; preserves class balance per fold (essential when
+  imbalanced). `KFold` for regression.
+- **`TimeSeriesSplit`** — any time ordering. Trains on past, tests on future; never shuffles the future
+  into train. A random `KFold` on time-series data is leakage.
+- **`GroupKFold` / `StratifiedGroupKFold`** — when rows cluster (same user/patient/store across many rows).
+  Keep a group entirely in train *or* test, or the model memorizes the group and CV lies.
+- Pass **integer `random_state`** to splitters for reproducible folds. Put preprocessing in the pipeline so
+  CV re-fits it every fold — `cross_validate(pipeline, ...)`, never `cross_validate(model, X_scaled, ...)`.
+
+For tuning, wrap CV in `GridSearchCV` / `RandomizedSearchCV` / `HalvingRandomSearchCV`; for an unbiased
+estimate *of the tuning process itself*, use nested CV → [references/pipelines-and-cv.md](references/pipelines-and-cv.md).
+
+## Metrics: the accuracy trap and what to use instead
+
+**Accuracy lies on imbalanced data.** At 99% negatives, a model that predicts "negative" always scores 99%
+accuracy and is worthless. Choose the metric for the task and the cost of each error type
+([model_evaluation docs](https://scikit-learn.org/stable/modules/model_evaluation.html)):
+
+| Task / question | Metric (`sklearn.metrics`) | scoring string |
+| --- | --- | --- |
+| Ranking quality, threshold-free, balanced-ish | `roc_auc_score` | `"roc_auc"` |
+| **Imbalanced** ranking (rare positive: fraud, disease) | `average_precision_score` (PR-AUC) | `"average_precision"` |
+| Cost of false positives high (don't cry wolf) | `precision_score` | `"precision"` |
+| Cost of misses high (don't miss a case) | `recall_score` | `"recall"` |
+| Balance both, per-class fairness | `f1_score` (use `f1_macro` multiclass) | `"f1"` / `"f1_macro"` |
+| Multiclass, care about every class equally | `balanced_accuracy_score` | `"balanced_accuracy"` |
+| See the actual error breakdown | `confusion_matrix`, `classification_report` | — |
+| Regression | `r2_score`, `mean_absolute_error`, `root_mean_squared_error` | `"r2"`, `"neg_mean_absolute_error"`, `"neg_root_mean_squared_error"` |
+
+**Prefer PR-AUC (`average_precision`) to ROC-AUC when positives are rare** — ROC-AUC can look great while
+precision is dismal, since it ignores the negative flood. Feed AUC metrics `predict_proba`, not hard labels.
+The default 0.5 threshold is a *choice*: tune it on validation to hit your precision/recall target.
+Regression uses `root_mean_squared_error` now (`mean_squared_error(squared=False)` is gone). Threshold
+tuning, calibration, `class_weight`/resampling → [references/metrics-and-imbalance.md](references/metrics-and-imbalance.md).
+
+## Anti-patterns — the cardinal sins
+
+| Anti-pattern | Do instead |
+| --- | --- |
+| Modeling straight off the raw, dirty table | This skill starts at a **clean, validated frame**. Run [`data-cleaning`](../data-cleaning/SKILL.md) first — nulls, dupes and mixed dtypes are its job, not a modeling problem. |
+| Scaling / encoding / selecting features, *then* splitting | **Leakage (#1 sin).** Test statistics are now in train. Split first; put every learned transform *inside* the `Pipeline` so CV re-fits per fold. |
+| Celebrating an amazingly predictive feature | Suspect **target leakage** — a column derived from the label or unavailable at prediction time. Audit provenance before you celebrate. |
+| Reporting 99% accuracy on 1% positives | The **accuracy trap**. A constant predictor matches it. Report PR-AUC / precision / recall / F1 and a `confusion_matrix`. |
+| `SMOTE`-ing the whole dataset because the classes are imbalanced | Resampling *before* the split, or on the test fold, leaks and evaluates on synthetic rows. Resample **inside CV, on the train fold only** (imblearn `Pipeline`), or just use `class_weight="balanced"`. |
+| Shipping on the CV score alone | You never touched a **held-out test set**, or you peeked at it while tuning. One final, untouched test number — or the estimate is optimistic. |
+| Random `KFold` on time-series / multi-user data | Future or same-group rows leak into train. Use `TimeSeriesSplit` / `GroupKFold`. |
+| Waving off a big train/test gap | **Overfitting.** Regularize, reduce capacity (`max_depth`, `min_samples_leaf`), get more data, or use early stopping. Watch `return_train_score`. |
+| Going straight to XGBoost with no baseline | Without a **`DummyClassifier(strategy="most_frequent")`** / `DummyRegressor` floor (and a simple linear model), you can't tell if the fancy model adds anything. |
+| Grid-searching 10k combos over all the data | Tuning against the test set *is* fitting to it. Tune with CV on train, confirm once on test; consider nested CV. |
+| Unset `random_state`, unpinned versions | Folds and fits stop being reproducible and re-trains stop being comparable. Set an integer `random_state` on splitters *and* estimators; pin the versions you ship. |
+
+## Worked lifecycle (end to end)
+
+```python
+from sklearn.dummy import DummyClassifier
+from sklearn.ensemble import HistGradientBoostingClassifier
+from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score
+from sklearn.metrics import average_precision_score, classification_report
+
+X_tr, X_test, y_tr, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)
+cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
+
+# 0. BASELINE first — the floor every real model must clear
+base = DummyClassifier(strategy="most_frequent")
+print("baseline PR-AUC:", cross_val_score(base, X_tr, y_tr, cv=cv, scoring="average_precision").mean())
+
+# 1. default GBDT (native NaN + categoricals -> minimal pipeline). class_weight for imbalance.
+clf = HistGradientBoostingClassifier(categorical_features="from_dtype",
+                                     class_weight="balanced", random_state=0)
+cv_pr = cross_val_score(clf, X_tr, y_tr, cv=cv, scoring="average_precision")
+print("model CV PR-AUC:", cv_pr.mean().round(3), "+/-", cv_pr.std().round(3))
+
+# 2. it clears the baseline -> commit, fit on all train, judge ONCE on the sacred test set
+clf.fit(X_tr, y_tr)
+proba = clf.predict_proba(X_test)[:, 1]
+print("TEST PR-AUC:", round(average_precision_score(y_test, proba), 3))
+print(classification_report(y_test, (proba >= 0.5).astype(int)))   # threshold is a choice — tune it
+```
+
+## Project grounding (02-DOCS + CLAUDE.md)
+
+In a project with a `02-DOCS/` layer (the [`harness`](../harness/SKILL.md) wiki), record the modeling
+contract in `02-DOCS/wiki/ml/<target>.md`, linked from the root `CLAUDE.md` `## Knowledge map`: target
+definition, split strategy + `random_state`, CV scheme, chosen metric and *why*, baseline, pinned versions,
+and the dated final test-set score. Read it first on every re-train so results stay comparable. No
+`02-DOCS/`? Skip silently — conventions are recorded, never gated.

package/skills/shared/machine-learning/evals/README.md

@@ -0,0 +1,15 @@
+# Eval harness — `machine-learning`
+
+`cases.yaml` is a human/LLM-graded rubric, not an automated runner. It grades two things: **triggering**
+(does the skill fire on real tabular-ML asks and stay quiet on near-misses that belong to a sibling —
+`deep-learning`, `data-cleaning`, `nlp`, `forecasting`, `business-intelligence`, `training-data`?) and
+**capability** (does loading the skill produce a measurably more rigorous, leak-free modeling workflow?).
+
+To run triggering: start a fresh agent with only `machine-learning` discoverable, paste each
+`should_trigger` prompt (3–5 trials each, since the decision is stochastic; pass if it fires in the
+majority), and confirm each `should_not_trigger` prompt does NOT fire and would plausibly route to its
+`route_to` sibling. To run capability: give the scenario to a clean agent WITHOUT the skill and again WITH
+it, then grade each transcript against the `must_include` bullets — one point per bullet that is
+specifically and correctly covered (the held-out test set, leakage as #1 risk, PR-AUC over accuracy on
+imbalance, GBDT default + Grinsztajn, the version hedge), not merely name-dropped. WITH should clearly beat
+WITHOUT. Record trial counts and per-bullet verdicts; don't report a bare pass/fail.

package/skills/shared/machine-learning/evals/cases.yaml

@@ -0,0 +1,54 @@
+skill: machine-learning
+
+should_trigger:
+  - prompt: "Train a model on this customer table to predict who will churn, and tell me how well it actually generalizes."
+    why: Core ask — supervised tabular classification with an honest generalization estimate; the headline use case.
+  - prompt: "My gradient-boosting model scores 0.99 ROC-AUC in cross-validation but tanks in production. Is this leakage?"
+    why: Leakage debugging (#1 sin) — only this skill explains fit-on-train-only, target-derived features, and the sacred test set.
+  - prompt: "My positive class is 2% of the rows — which metric should I optimize, and is accuracy lying to me?"
+    why: The accuracy trap on imbalance → PR-AUC/precision/recall/F1; the metric-selection reasoning is this skill's.
+  - prompt: "Set up cross-validation for this dataset so no test information leaks into training or preprocessing."
+    why: Pipeline + ColumnTransformer + splitter choice so transforms fit per fold — the leakage-safe CV pattern lives here.
+  - prompt: "XGBoost or LightGBM for my tabular regression problem, and should I even bother with a neural net?"
+    why: GBDT choice + the trees-beat-deep-nets-on-tabular finding; classic-model territory, not deep-learning.
+  - prompt: "Add a proper baseline so I can tell whether my random forest is actually doing anything."
+    why: DummyClassifier/DummyRegressor floor — a load-bearing rigor step this skill insists on.
+  - prompt: "Entrena un modelo de clasificación tabular con scikit-learn y dame una métrica honesta, no solo accuracy."
+    why: Spanish trigger — same tabular-classification-with-honest-metric intent, no English keyword present.
+
+should_not_trigger:
+  - prompt: "Train a convolutional neural network in PyTorch to classify these product images."
+    route_to: deep-learning
+    why: Neural nets on images — PyTorch training, explicitly the deep-learning boundary, not tabular sklearn.
+  - prompt: "This CSV is full of nulls, duplicate rows, and 'N/A' strings — fix it before I can model anything."
+    route_to: data-cleaning
+    why: Producing the validated table is upstream; this skill starts once the table is clean.
+  - prompt: "Classify these free-text support tickets into topics straight from the ticket wording."
+    route_to: nlp
+    why: Text/token modeling is nlp's domain; route free-text problems there (a TF-IDF baseline can still live in an ML pipeline, but the framing is NLP).
+  - prompt: "Forecast next quarter's revenue from this monthly time-series."
+    route_to: forecasting
+    why: Projecting a dated series forward is a forecasting product, not a tabular predict-a-column task.
+  - prompt: "Build a KPI dashboard and executive summary from this already-clean sales table."
+    route_to: business-intelligence
+    why: Explaining KPIs / dashboards consumes data; it is not model training.
+  - prompt: "Assemble a JSONL dataset of chat messages and preference pairs to fine-tune an LLM."
+    route_to: training-data
+    why: Building an LLM training corpus, not a tabular ML model.
+
+capability:
+  - scenario: >
+      Given a clean, validated table of ~50k rows for a churn/fraud problem where only ~3% of rows are
+      positive, produce a trustworthy modeling workflow: hold out a test set, build a leak-proof pipeline,
+      cross-validate, pick a sensible metric and model, beat a baseline, and report an honest final number.
+    must_include:
+      - Splits off a held-out test set FIRST with train_test_split(..., stratify=y, random_state=...), and leaves it untouched until the single final evaluation
+      - Puts all learned preprocessing (impute/scale/encode) INSIDE a Pipeline + ColumnTransformer so it is fit on the train fold only
+      - Uses StratifiedKFold (not plain KFold) for the imbalanced classes, with an integer random_state for reproducible folds
+      - Establishes a DummyClassifier baseline (e.g. strategy="most_frequent") that the real model must beat
+      - Defaults to a GBDT (HistGradientBoostingClassifier / XGBoost / LightGBM) and notes that trees still beat deep nets on tabular data (Grinsztajn 2022, arXiv:2207.08815)
+      - Reports PR-AUC (average_precision) / precision / recall / F1 / confusion_matrix and explicitly does NOT use accuracy, because the classes are imbalanced
+      - Names data/target leakage as the #1 risk and audits for target-derived or not-available-at-prediction-time features
+      - Cross-validates the whole pipeline via cross_validate/GridSearchCV (scoring the pipeline, never globally-preprocessed data)
+      - Treats the 0.5 decision threshold as a tunable choice (e.g. from the PR curve or TunedThresholdClassifierCV), not a law
+      - Hedges the scikit-learn version (current major ~1.9, verify at scikit-learn.org) instead of asserting a frozen pin as fact

package/skills/shared/machine-learning/references/gbdt-and-tuning.md

@@ -0,0 +1,83 @@
+# GBDTs: parameters, early stopping, categoricals, importance (depth)
+
+Three interchangeable sklearn-API GBDTs. Versions verified 2026-07 — XGBoost 3.x, LightGBM 4.x, and
+sklearn's `HistGradientBoosting*` (~1.9). Verify current versions before pinning; APIs shift between majors.
+
+## What the knobs mean (they translate across libraries)
+
+| Concept | HistGradientBoosting | XGBoost | LightGBM |
+| --- | --- | --- | --- |
+| Step size | `learning_rate` | `learning_rate` (`eta`) | `learning_rate` |
+| Number of trees | `max_iter` | `n_estimators` | `n_estimators` |
+| Tree size | `max_leaf_nodes` / `max_depth` | `max_depth` | `num_leaves` (leaf-wise!) / `max_depth` |
+| L2 penalty | `l2_regularization` | `reg_lambda` / `reg_alpha` | `reg_lambda` / `reg_alpha` |
+| Min data per leaf | `min_samples_leaf` | `min_child_weight` | `min_child_samples` |
+| Row/col subsample | (n/a) | `subsample` / `colsample_bytree` | `bagging_fraction` / `feature_fraction` |
+
+Tuning order that pays off: get `learning_rate` low-ish with enough trees (use early stopping to pick the
+count), then control complexity (`num_leaves`/`max_depth`, `min_*_leaf`), then regularization and subsampling.
+**Lower `learning_rate` + more trees + early stopping** beats a high learning rate almost always. LightGBM
+grows **leaf-wise**, so cap `num_leaves` (and/or `max_depth`) or it overfits fast.
+
+## Early stopping (stop adding trees when validation stops improving)
+
+Needs a validation signal. Keep it out of the sacred test set — use a train-derived eval split.
+
+```python
+# XGBoost 3.x: early_stopping_rounds is a CONSTRUCTOR param; pass eval_set to fit()
+from xgboost import XGBClassifier
+xgb = XGBClassifier(n_estimators=2000, learning_rate=0.05,
+                    early_stopping_rounds=50, eval_metric="aucpr")
+xgb.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=False)
+print(xgb.best_iteration)
+
+# LightGBM 4.x: callbacks
+from lightgbm import LGBMClassifier, early_stopping, log_evaluation
+lgbm = LGBMClassifier(n_estimators=2000, learning_rate=0.05, num_leaves=31)
+lgbm.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], eval_metric="average_precision",
+         callbacks=[early_stopping(50), log_evaluation(0)])
+
+# HistGradientBoosting: built-in, no eval_set needed (carves its own validation slice)
+from sklearn.ensemble import HistGradientBoostingClassifier
+hgb = HistGradientBoostingClassifier(max_iter=2000, learning_rate=0.05,
+                                     early_stopping=True, validation_fraction=0.1,
+                                     n_iter_no_change=50, random_state=0)
+```
+
+## Native categorical handling (skip one-hot)
+
+- **HistGradientBoosting**: `categorical_features="from_dtype"` — columns with pandas `category` dtype are
+  handled natively. No `OneHotEncoder`, no explosion of columns.
+- **XGBoost**: `enable_categorical=True` + columns as pandas `category` dtype; `tree_method` `hist`/`approx`.
+- **LightGBM**: pass `categorical_feature=` (names/indices) or use `category` dtype; native by default.
+
+One-hot only when cardinality is low and a *linear* model needs it. For trees, native categorical or, for
+very high cardinality, `TargetEncoder` (cross-fit — see pipelines-and-cv.md) beats a thousand dummy columns.
+
+## Monotonic constraints (domain priors)
+
+Force the model's response to move only one way with a feature (price ↑ ⇒ risk never ↓). Guards against
+noise inverting a known relationship and helps trust/compliance.
+
+```python
+# HistGradientBoosting: dict of feature -> {-1, 0, +1}
+HistGradientBoostingClassifier(monotonic_cst={"income": +1, "num_late_payments": -1})
+# XGBoost: monotone_constraints="(1,0,-1,...)"   LightGBM: monotone_constraints=[1,0,-1,...]
+```
+
+## Feature importance — don't trust the default blindly
+
+- **Impurity / gain importance** (`feature_importances_`): fast but biased toward high-cardinality and
+  continuous features; computed on train.
+- **Permutation importance** (`sklearn.inspection.permutation_importance`): model-agnostic, measures the
+  metric drop when a feature is shuffled — run it **on held-out data**, and beware correlated features
+  sharing credit.
+- **SHAP** (separate `shap` library): per-prediction attributions; the most faithful local explanations,
+  the heaviest to compute. Use for "why this row" and to sanity-check for leakage — a feature dominating
+  SHAP that shouldn't be predictive is a leak flag.
+
+```python
+from sklearn.inspection import permutation_importance
+r = permutation_importance(model, X_test, y_test, scoring="average_precision",
+                           n_repeats=10, random_state=0)
+```