@zigrivers/scaffold 3.8.0 → 3.9.1

package/content/knowledge/ml/ml-experiment-tracking.md

@@ -0,0 +1,285 @@
+---
+name: ml-experiment-tracking
+description: MLflow and Weights & Biases integration, artifact storage, experiment run comparison, and hyperparameter sweep management
+topics: [ml, experiment-tracking, mlflow, wandb, artifacts, sweeps, reproducibility]
+---
+
+Without experiment tracking, ML development is archaeology: "which config produced that result?" is answered by digging through notebook history, chat logs, and failing memory. Experiment tracking tools are version control for training runs — every metric, every hyperparameter, every artifact, linked to the code that produced it. The discipline of logging everything during training pays dividends when a stakeholder asks "how does this model compare to what we had six months ago?"
+
+## Summary
+
+Use MLflow (self-hosted, open source) or Weights & Biases (cloud, more feature-rich) to track every training run. Log hyperparameters, metrics at each epoch, model artifacts, and the git commit SHA. Store large artifacts (checkpoints, datasets) in object storage backed by the experiment tracker. Use sweep features (MLflow Hyperopt integration, W&B Sweeps) for systematic hyperparameter search rather than manual iteration.
+
+## Deep Guidance
+
+### MLflow Integration
+
+MLflow is the open-source standard for experiment tracking. It runs locally or on a managed server:
+
+```bash
+# Start local tracking server (stores runs in ./mlruns)
+mlflow server --host 0.0.0.0 --port 5000
+
+# Or use the SQLite backend for better performance
+mlflow server \
+  --backend-store-uri sqlite:///mlflow.db \
+  --default-artifact-root ./mlartifacts \
+  --host 0.0.0.0 --port 5000
+```
+
+**Instrument training code**:
+```python
+import mlflow
+import mlflow.pytorch
+
+# Set tracking server
+mlflow.set_tracking_uri("http://localhost:5000")
+mlflow.set_experiment("fraud-detector")
+
+def train(cfg: DictConfig) -> dict:
+    with mlflow.start_run(run_name=cfg.experiment.name) as run:
+        # Log all hyperparameters from config
+        mlflow.log_params(OmegaConf.to_container(cfg, resolve=True))
+
+        # Log git commit for reproducibility
+        import subprocess
+        git_sha = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode().strip()
+        mlflow.set_tag("git_commit", git_sha)
+        mlflow.set_tag("model_type", cfg.model.type)
+
+        for epoch in range(cfg.training.epochs):
+            train_metrics = train_epoch(...)
+            val_metrics = evaluate(...)
+
+            # Log metrics with step (epoch) for time-series view
+            mlflow.log_metrics({
+                "train_loss": train_metrics["loss"],
+                "val_loss": val_metrics["loss"],
+                "val_auc": val_metrics["auc"],
+            }, step=epoch)
+
+        # Log best model
+        mlflow.pytorch.log_model(
+            model,
+            artifact_path="model",
+            registered_model_name="fraud-detector",  # Register in Model Registry
+        )
+
+        # Log additional artifacts
+        mlflow.log_artifact("configs/train.yaml")
+        mlflow.log_artifact("reports/eval_report.json")
+
+        return {"run_id": run.info.run_id, **val_metrics}
+```
+
+**MLflow Model Registry** (promote to production):
+```python
+from mlflow.tracking import MlflowClient
+
+client = MlflowClient()
+
+# Register a run's model in the registry
+model_uri = f"runs:/{run_id}/model"
+mv = mlflow.register_model(model_uri, "fraud-detector")
+
+# Transition to staging after validation
+client.transition_model_version_stage(
+    name="fraud-detector",
+    version=mv.version,
+    stage="Staging",
+    archive_existing_versions=False,
+)
+
+# Load production model in serving
+production_model = mlflow.pytorch.load_model(
+    model_uri="models:/fraud-detector/Production"
+)
+```
+
+### Weights & Biases Integration
+
+W&B provides a richer UI and more features than MLflow, with a cloud-hosted option:
+
+```python
+import wandb
+
+wandb.init(
+    project="fraud-detector",
+    name=cfg.experiment.name,
+    config=OmegaConf.to_container(cfg, resolve=True),
+    tags=["baseline", "v2-features"],
+    notes="Testing new feature set with gradient clipping",
+)
+
+# Log metrics
+for epoch in range(cfg.training.epochs):
+    metrics = train_epoch(...)
+    wandb.log({
+        "epoch": epoch,
+        "train/loss": metrics["train_loss"],
+        "val/loss": metrics["val_loss"],
+        "val/auc": metrics["val_auc"],
+        "lr": scheduler.get_last_lr()[0],
+    })
+
+# Log model artifact
+artifact = wandb.Artifact("fraud-detector", type="model")
+artifact.add_file("models/checkpoints/best.pt")
+wandb.log_artifact(artifact)
+
+wandb.finish()
+```
+
+**W&B-specific features**:
+- **System monitoring**: GPU utilisation, memory, temperature logged automatically
+- **Gradient histograms**: `wandb.watch(model, log="gradients")` logs gradient distributions per layer — invaluable for debugging vanishing/exploding gradients
+- **Media logging**: Log images, audio, tables, confusion matrices directly in the UI
+- **Alerts**: Set threshold alerts on metrics (email/Slack when val_loss > threshold)
+
+### Artifact Storage Strategy
+
+Artifacts are the binary outputs of training runs: model checkpoints, preprocessed datasets, evaluation reports, and confusion matrices. Never store large binary artifacts in git:
+
+**Storage hierarchy**:
+```
+Small artifacts (< 1 MB): Log directly to tracker
+  - Config files, evaluation reports (JSON/CSV)
+  - Example predictions, confusion matrices (images)
+
+Medium artifacts (1 MB – 1 GB): Log as tracker artifacts
+  - Model checkpoints for experimentation
+  - Feature engineering outputs
+
+Large artifacts (> 1 GB): Object storage with tracker reference
+  - Full training datasets
+  - Final production model weights
+  - Large evaluation outputs
+```
+
+**S3 artifact storage for MLflow**:
+```bash
+mlflow server \
+  --default-artifact-root s3://my-bucket/mlflow-artifacts \
+  --backend-store-uri postgresql://user:pass@host/mlflow
+```
+
+**DVC for dataset versioning alongside MLflow**:
+```bash
+# Version dataset with DVC
+dvc add data/processed/features_v3.parquet
+git add data/processed/features_v3.parquet.dvc
+
+# Log DVC dataset reference in MLflow
+mlflow.set_tag("dvc_dataset_commit", git_sha)
+mlflow.set_tag("dataset_path", "data/processed/features_v3.parquet")
+```
+
+### Run Comparison and Analysis
+
+**Finding the best run** (MLflow Python API):
+```python
+from mlflow.tracking import MlflowClient
+import pandas as pd
+
+client = MlflowClient()
+
+# Get all runs in an experiment, sorted by val_auc
+runs = client.search_runs(
+    experiment_ids=["1"],
+    filter_string="metrics.val_auc > 0.85",
+    order_by=["metrics.val_auc DESC"],
+    max_results=20,
+)
+
+# Convert to DataFrame for analysis
+run_data = [{
+    "run_id": r.info.run_id,
+    "name": r.info.run_name,
+    "val_auc": r.data.metrics.get("val_auc"),
+    "lr": r.data.params.get("optimizer.lr"),
+    "batch_size": r.data.params.get("training.batch_size"),
+} for r in runs]
+
+df = pd.DataFrame(run_data)
+print(df.head(10))
+```
+
+**Comparing runs in W&B**: Use the parallel coordinates plot (built into W&B UI) to visualise the relationship between hyperparameters and metrics across many runs at once.
+
+### Hyperparameter Sweeps
+
+**W&B Sweeps** (cloud-managed sweep coordinator):
+```yaml
+# sweep_config.yaml
+program: train.py
+method: bayes  # bayesian, random, or grid
+metric:
+  name: val/auc
+  goal: maximize
+parameters:
+  optimizer.lr:
+    min: 1.0e-5
+    max: 1.0e-2
+    distribution: log_uniform_values
+  training.batch_size:
+    values: [16, 32, 64, 128]
+  model.dropout:
+    min: 0.0
+    max: 0.5
+early_terminate:
+  type: hyperband
+  min_iter: 3
+```
+
+```bash
+wandb sweep sweep_config.yaml  # Returns sweep ID
+wandb agent <sweep-id> --count 50  # Launch 50 trials
+```
+
+**MLflow + Optuna** (self-hosted alternative):
+```python
+import optuna
+import mlflow
+
+def objective(trial):
+    with mlflow.start_run(nested=True):
+        lr = trial.suggest_float("lr", 1e-5, 1e-2, log=True)
+        mlflow.log_param("lr", lr)
+
+        val_auc = train_and_evaluate(lr=lr)
+        mlflow.log_metric("val_auc", val_auc)
+        return val_auc
+
+with mlflow.start_run(run_name="hyperparameter-sweep"):
+    study = optuna.create_study(direction="maximize")
+    study.optimize(objective, n_trials=50)
+    mlflow.log_params(study.best_params)
+    mlflow.log_metric("best_val_auc", study.best_value)
+```
+
+### Experiment Logging Checklist
+
+Log these for every training run — no exceptions:
+
+```python
+# Required: hyperparameters
+mlflow.log_params({...})  # Full config dict
+
+# Required: metrics at each epoch
+mlflow.log_metrics({...}, step=epoch)
+
+# Required: final metrics
+mlflow.log_metrics({"final_val_auc": val_auc, "final_val_loss": val_loss})
+
+# Required: reproducibility tags
+mlflow.set_tag("git_commit", git_sha)
+mlflow.set_tag("dataset_version", dataset_version)
+
+# Required: model artifact
+mlflow.pytorch.log_model(model, "model")
+
+# Recommended: environment
+mlflow.log_artifact("environment.yml")
+mlflow.set_tag("cuda_version", torch.version.cuda)
+mlflow.set_tag("pytorch_version", torch.__version__)
+```

package/content/knowledge/ml/ml-model-evaluation.md

@@ -0,0 +1,256 @@
+---
+name: ml-model-evaluation
+description: Train/val/test splits, cross-validation, metrics by task type, holdout sets, and slice analysis for thorough model evaluation
+topics: [ml, evaluation, train-test-split, cross-validation, metrics, holdout, slice-analysis]
+---
+
+Model evaluation is the difference between knowing whether your model works and believing it works. Most ML evaluation bugs are forms of data leakage: the model has seen information during training that it would not have at inference time, making offline metrics look better than production performance. Rigorous evaluation requires careful data splitting, leak-free preprocessing, appropriate metrics for the task, and systematic analysis of where the model fails.
+
+## Summary
+
+Split data into train, validation, and test sets — use the test set exactly once. For small datasets, use cross-validation. Choose metrics appropriate to the task: classification, regression, ranking, or generation have different canonical metrics. Analyse model performance by meaningful slices (demographic groups, difficulty levels, data subsets) — aggregate metrics hide subgroup failures. Log evaluation results with experiment metadata for longitudinal comparison.
+
+## Deep Guidance
+
+### Data Splitting Principles
+
+**Three-way split**: train (model learning), validation (hyperparameter tuning and early stopping), test (final unbiased evaluation):
+
+```python
+from sklearn.model_selection import train_test_split
+
+def create_splits(
+    df: pd.DataFrame,
+    val_fraction: float = 0.1,
+    test_fraction: float = 0.1,
+    seed: int = 42,
+    stratify_col: str | None = None,
+) -> tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame]:
+    """Create reproducible train/val/test splits."""
+    stratify = df[stratify_col] if stratify_col else None
+
+    train_val, test = train_test_split(
+        df,
+        test_size=test_fraction,
+        random_state=seed,
+        stratify=stratify,
+    )
+
+    val_size_adjusted = val_fraction / (1 - test_fraction)
+    stratify_tv = train_val[stratify_col] if stratify_col else None
+
+    train, val = train_test_split(
+        train_val,
+        test_size=val_size_adjusted,
+        random_state=seed,
+        stratify=stratify_tv,
+    )
+
+    return train, val, test
+```
+
+**Critical splitting rules**:
+1. **Split before preprocessing**: Fit preprocessing (scalers, encoders, imputers, tokenizers vocabulary) on training data only, then apply to val/test. Fitting on the combined dataset is data leakage.
+2. **Stratify by label for classification**: Ensures class distribution is preserved in each split.
+3. **Split by entity, not row, for grouped data**: If you have multiple rows per user, all rows for a user must go to the same split. Row-level splitting leaks user-level information.
+4. **Temporal split for time-series**: Train on past, validate and test on future. Random splits would leak future information.
+
+### Temporal Splits
+
+For any dataset with a time dimension, always split by time:
+
+```python
+def temporal_split(
+    df: pd.DataFrame,
+    timestamp_col: str,
+    val_start: str,
+    test_start: str,
+) -> tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame]:
+    """Create temporal splits — train/val/test defined by date boundaries."""
+    df = df.sort_values(timestamp_col)
+    train = df[df[timestamp_col] < val_start]
+    val = df[(df[timestamp_col] >= val_start) & (df[timestamp_col] < test_start)]
+    test = df[df[timestamp_col] >= test_start]
+    return train, val, test
+```
+
+**Backtesting** extends temporal evaluation by simulating deployment across multiple time windows — tests that a model trained on one period performs on subsequent periods.
+
+### Cross-Validation
+
+Use k-fold cross-validation when dataset size is insufficient for a stable held-out set (< 10,000 examples):
+
+```python
+from sklearn.model_selection import StratifiedKFold
+import numpy as np
+
+def cross_validate(
+    X: np.ndarray,
+    y: np.ndarray,
+    model_builder,
+    n_folds: int = 5,
+    seed: int = 42,
+) -> dict[str, float]:
+    """Stratified k-fold cross-validation."""
+    skf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=seed)
+    fold_metrics = []
+
+    for fold, (train_idx, val_idx) in enumerate(skf.split(X, y)):
+        X_train, X_val = X[train_idx], X[val_idx]
+        y_train, y_val = y[train_idx], y[val_idx]
+
+        model = model_builder()
+        model.fit(X_train, y_train)
+        metrics = evaluate_model(model, X_val, y_val)
+        fold_metrics.append(metrics)
+
+    # Aggregate across folds
+    return {
+        metric: {
+            "mean": np.mean([m[metric] for m in fold_metrics]),
+            "std": np.std([m[metric] for m in fold_metrics]),
+        }
+        for metric in fold_metrics[0]
+    }
+```
+
+**Nested cross-validation** separates hyperparameter selection from model evaluation:
+- Outer loop: Estimate generalisation error
+- Inner loop: Select hyperparameters via grid/random search
+- Prevents over-fitting hyperparameters to the validation set
+
+### Holdout Sets and Evaluation Integrity
+
+**The test set is sacred**: It may be touched exactly once — when reporting final model performance before deployment. Every other decision (architecture, hyperparameters, features) uses the validation set.
+
+If you look at test set performance and then make changes, the test set is contaminated — you must collect a fresh test set.
+
+**Multiple evaluation sets**:
+- **In-distribution test set**: Same distribution as training data. Measures how well the model learned.
+- **Out-of-distribution test set**: Different time period, geography, or user cohort. Measures generalisation.
+- **Adversarial / challenging test set**: Hard examples, edge cases, known failure modes. Measures robustness.
+- **Slice-specific test sets**: Subsets by demographic, category, or difficulty. Measures fairness and consistency.
+
+### Metrics by Task Type
+
+**Binary Classification**:
+```python
+from sklearn.metrics import (
+    accuracy_score, precision_score, recall_score,
+    f1_score, roc_auc_score, average_precision_score,
+    confusion_matrix, classification_report,
+)
+
+def evaluate_binary_classifier(
+    y_true: np.ndarray,
+    y_pred_proba: np.ndarray,
+    threshold: float = 0.5,
+) -> dict[str, float]:
+    y_pred = (y_pred_proba >= threshold).astype(int)
+    return {
+        "accuracy": accuracy_score(y_true, y_pred),
+        "precision": precision_score(y_true, y_pred, zero_division=0),
+        "recall": recall_score(y_true, y_pred, zero_division=0),
+        "f1": f1_score(y_true, y_pred, zero_division=0),
+        "roc_auc": roc_auc_score(y_true, y_pred_proba),
+        "pr_auc": average_precision_score(y_true, y_pred_proba),
+    }
+```
+
+**Regression**:
+```python
+from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
+
+def evaluate_regressor(y_true, y_pred) -> dict[str, float]:
+    return {
+        "mae": mean_absolute_error(y_true, y_pred),
+        "rmse": np.sqrt(mean_squared_error(y_true, y_pred)),
+        "r2": r2_score(y_true, y_pred),
+        "mape": np.mean(np.abs((y_true - y_pred) / (y_true + 1e-8))) * 100,
+    }
+```
+
+**Multi-class classification**:
+- Macro-average: Equal weight per class — use when class imbalance should not inflate aggregate metrics
+- Weighted-average: Weight by class support — use for overall system performance
+- Per-class metrics: Report separately to catch poor performance on minority classes
+
+### Slice Analysis
+
+Aggregate metrics can hide systematic failures in subgroups. Slice analysis breaks down performance by meaningful subsets:
+
+```python
+def slice_analysis(
+    df: pd.DataFrame,
+    y_true_col: str,
+    y_pred_col: str,
+    slice_cols: list[str],
+    metric_fn,
+) -> pd.DataFrame:
+    """Compute metrics for each slice of the data."""
+    results = []
+
+    # Overall metrics
+    overall = metric_fn(df[y_true_col], df[y_pred_col])
+    results.append({"slice": "overall", "n": len(df), **overall})
+
+    # Per-slice metrics
+    for col in slice_cols:
+        for value, group in df.groupby(col):
+            if len(group) < 50:  # Skip slices with too few examples
+                continue
+            metrics = metric_fn(group[y_true_col], group[y_pred_col])
+            results.append({
+                "slice": f"{col}={value}",
+                "n": len(group),
+                **metrics,
+            })
+
+    return pd.DataFrame(results)
+```
+
+**Slices to always analyse**:
+- Demographic groups (if available and legally permissible): age band, gender, geography
+- Data quality slices: high vs. low confidence labels, recent vs. old data
+- Difficulty slices: high vs. low frequency items, short vs. long text
+- Business-relevant slices: product category, customer segment, price tier
+
+**Flagging disparities**: If a slice's metric deviates from overall by more than a threshold (e.g., 10 percentage points), flag for investigation before deployment.
+
+### Baseline Comparisons
+
+Every model evaluation must include a comparison to baselines:
+- **Trivial baseline**: Predict the majority class (classification) or mean target value (regression)
+- **Rule-based baseline**: The current production rule or heuristic
+- **Previous model version**: The model currently in production
+- **Simple ML baseline**: Logistic regression or decision tree
+
+A model that does not beat all baselines should not be deployed. The trivial baseline check catches label encoding bugs (where the model learns the majority class trivially).
+
+### Evaluation Report Structure
+
+```markdown
+# Evaluation Report: fraud-detector-v2.3.0
+
+## Dataset
+- Test set: 45,231 examples (2024-01-01 to 2024-03-31)
+- Class balance: 1.2% fraud, 98.8% non-fraud
+
+## Overall Metrics
+| Metric | v2.2.0 (prod) | v2.3.0 (candidate) | Delta |
+|--------|--------------|-------------------|-------|
+| ROC-AUC | 0.921 | 0.934 | +1.4% |
+| PR-AUC | 0.712 | 0.748 | +5.1% |
+| Recall @ precision=0.9 | 0.68 | 0.73 | +7.4% |
+
+## Slice Analysis
+| Slice | n | ROC-AUC | vs. Overall |
+|-------|---|---------|-------------|
+| Overall | 45,231 | 0.934 | — |
+| Amount < $50 | 12,445 | 0.941 | +0.7% |
+| Amount > $1000 | 3,211 | 0.918 | -1.6% |
+| New user (< 30 days) | 8,902 | 0.891 | -4.6% ⚠️ |
+
+## Recommendation
+Promote to staging. Investigate new user performance degradation before production.
+```