@zigrivers/scaffold 3.7.0 → 3.9.0

package/content/knowledge/ml/ml-model-evaluation.md

@@ -0,0 +1,256 @@
+---
+name: ml-model-evaluation
+description: Train/val/test splits, cross-validation, metrics by task type, holdout sets, and slice analysis for thorough model evaluation
+topics: [ml, evaluation, train-test-split, cross-validation, metrics, holdout, slice-analysis]
+---
+
+Model evaluation is the difference between knowing whether your model works and believing it works. Most ML evaluation bugs are forms of data leakage: the model has seen information during training that it would not have at inference time, making offline metrics look better than production performance. Rigorous evaluation requires careful data splitting, leak-free preprocessing, appropriate metrics for the task, and systematic analysis of where the model fails.
+
+## Summary
+
+Split data into train, validation, and test sets — use the test set exactly once. For small datasets, use cross-validation. Choose metrics appropriate to the task: classification, regression, ranking, or generation have different canonical metrics. Analyse model performance by meaningful slices (demographic groups, difficulty levels, data subsets) — aggregate metrics hide subgroup failures. Log evaluation results with experiment metadata for longitudinal comparison.
+
+## Deep Guidance
+
+### Data Splitting Principles
+
+**Three-way split**: train (model learning), validation (hyperparameter tuning and early stopping), test (final unbiased evaluation):
+
+```python
+from sklearn.model_selection import train_test_split
+
+def create_splits(
+    df: pd.DataFrame,
+    val_fraction: float = 0.1,
+    test_fraction: float = 0.1,
+    seed: int = 42,
+    stratify_col: str | None = None,
+) -> tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame]:
+    """Create reproducible train/val/test splits."""
+    stratify = df[stratify_col] if stratify_col else None
+
+    train_val, test = train_test_split(
+        df,
+        test_size=test_fraction,
+        random_state=seed,
+        stratify=stratify,
+    )
+
+    val_size_adjusted = val_fraction / (1 - test_fraction)
+    stratify_tv = train_val[stratify_col] if stratify_col else None
+
+    train, val = train_test_split(
+        train_val,
+        test_size=val_size_adjusted,
+        random_state=seed,
+        stratify=stratify_tv,
+    )
+
+    return train, val, test
+```
+
+**Critical splitting rules**:
+1. **Split before preprocessing**: Fit preprocessing (scalers, encoders, imputers, tokenizers vocabulary) on training data only, then apply to val/test. Fitting on the combined dataset is data leakage.
+2. **Stratify by label for classification**: Ensures class distribution is preserved in each split.
+3. **Split by entity, not row, for grouped data**: If you have multiple rows per user, all rows for a user must go to the same split. Row-level splitting leaks user-level information.
+4. **Temporal split for time-series**: Train on past, validate and test on future. Random splits would leak future information.
+
+### Temporal Splits
+
+For any dataset with a time dimension, always split by time:
+
+```python
+def temporal_split(
+    df: pd.DataFrame,
+    timestamp_col: str,
+    val_start: str,
+    test_start: str,
+) -> tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame]:
+    """Create temporal splits — train/val/test defined by date boundaries."""
+    df = df.sort_values(timestamp_col)
+    train = df[df[timestamp_col] < val_start]
+    val = df[(df[timestamp_col] >= val_start) & (df[timestamp_col] < test_start)]
+    test = df[df[timestamp_col] >= test_start]
+    return train, val, test
+```
+
+**Backtesting** extends temporal evaluation by simulating deployment across multiple time windows — tests that a model trained on one period performs on subsequent periods.
+
+### Cross-Validation
+
+Use k-fold cross-validation when dataset size is insufficient for a stable held-out set (< 10,000 examples):
+
+```python
+from sklearn.model_selection import StratifiedKFold
+import numpy as np
+
+def cross_validate(
+    X: np.ndarray,
+    y: np.ndarray,
+    model_builder,
+    n_folds: int = 5,
+    seed: int = 42,
+) -> dict[str, float]:
+    """Stratified k-fold cross-validation."""
+    skf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=seed)
+    fold_metrics = []
+
+    for fold, (train_idx, val_idx) in enumerate(skf.split(X, y)):
+        X_train, X_val = X[train_idx], X[val_idx]
+        y_train, y_val = y[train_idx], y[val_idx]
+
+        model = model_builder()
+        model.fit(X_train, y_train)
+        metrics = evaluate_model(model, X_val, y_val)
+        fold_metrics.append(metrics)
+
+    # Aggregate across folds
+    return {
+        metric: {
+            "mean": np.mean([m[metric] for m in fold_metrics]),
+            "std": np.std([m[metric] for m in fold_metrics]),
+        }
+        for metric in fold_metrics[0]
+    }
+```
+
+**Nested cross-validation** separates hyperparameter selection from model evaluation:
+- Outer loop: Estimate generalisation error
+- Inner loop: Select hyperparameters via grid/random search
+- Prevents over-fitting hyperparameters to the validation set
+
+### Holdout Sets and Evaluation Integrity
+
+**The test set is sacred**: It may be touched exactly once — when reporting final model performance before deployment. Every other decision (architecture, hyperparameters, features) uses the validation set.
+
+If you look at test set performance and then make changes, the test set is contaminated — you must collect a fresh test set.
+
+**Multiple evaluation sets**:
+- **In-distribution test set**: Same distribution as training data. Measures how well the model learned.
+- **Out-of-distribution test set**: Different time period, geography, or user cohort. Measures generalisation.
+- **Adversarial / challenging test set**: Hard examples, edge cases, known failure modes. Measures robustness.
+- **Slice-specific test sets**: Subsets by demographic, category, or difficulty. Measures fairness and consistency.
+
+### Metrics by Task Type
+
+**Binary Classification**:
+```python
+from sklearn.metrics import (
+    accuracy_score, precision_score, recall_score,
+    f1_score, roc_auc_score, average_precision_score,
+    confusion_matrix, classification_report,
+)
+
+def evaluate_binary_classifier(
+    y_true: np.ndarray,
+    y_pred_proba: np.ndarray,
+    threshold: float = 0.5,
+) -> dict[str, float]:
+    y_pred = (y_pred_proba >= threshold).astype(int)
+    return {
+        "accuracy": accuracy_score(y_true, y_pred),
+        "precision": precision_score(y_true, y_pred, zero_division=0),
+        "recall": recall_score(y_true, y_pred, zero_division=0),
+        "f1": f1_score(y_true, y_pred, zero_division=0),
+        "roc_auc": roc_auc_score(y_true, y_pred_proba),
+        "pr_auc": average_precision_score(y_true, y_pred_proba),
+    }
+```
+
+**Regression**:
+```python
+from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
+
+def evaluate_regressor(y_true, y_pred) -> dict[str, float]:
+    return {
+        "mae": mean_absolute_error(y_true, y_pred),
+        "rmse": np.sqrt(mean_squared_error(y_true, y_pred)),
+        "r2": r2_score(y_true, y_pred),
+        "mape": np.mean(np.abs((y_true - y_pred) / (y_true + 1e-8))) * 100,
+    }
+```
+
+**Multi-class classification**:
+- Macro-average: Equal weight per class — use when class imbalance should not inflate aggregate metrics
+- Weighted-average: Weight by class support — use for overall system performance
+- Per-class metrics: Report separately to catch poor performance on minority classes
+
+### Slice Analysis
+
+Aggregate metrics can hide systematic failures in subgroups. Slice analysis breaks down performance by meaningful subsets:
+
+```python
+def slice_analysis(
+    df: pd.DataFrame,
+    y_true_col: str,
+    y_pred_col: str,
+    slice_cols: list[str],
+    metric_fn,
+) -> pd.DataFrame:
+    """Compute metrics for each slice of the data."""
+    results = []
+
+    # Overall metrics
+    overall = metric_fn(df[y_true_col], df[y_pred_col])
+    results.append({"slice": "overall", "n": len(df), **overall})
+
+    # Per-slice metrics
+    for col in slice_cols:
+        for value, group in df.groupby(col):
+            if len(group) < 50:  # Skip slices with too few examples
+                continue
+            metrics = metric_fn(group[y_true_col], group[y_pred_col])
+            results.append({
+                "slice": f"{col}={value}",
+                "n": len(group),
+                **metrics,
+            })
+
+    return pd.DataFrame(results)
+```
+
+**Slices to always analyse**:
+- Demographic groups (if available and legally permissible): age band, gender, geography
+- Data quality slices: high vs. low confidence labels, recent vs. old data
+- Difficulty slices: high vs. low frequency items, short vs. long text
+- Business-relevant slices: product category, customer segment, price tier
+
+**Flagging disparities**: If a slice's metric deviates from overall by more than a threshold (e.g., 10 percentage points), flag for investigation before deployment.
+
+### Baseline Comparisons
+
+Every model evaluation must include a comparison to baselines:
+- **Trivial baseline**: Predict the majority class (classification) or mean target value (regression)
+- **Rule-based baseline**: The current production rule or heuristic
+- **Previous model version**: The model currently in production
+- **Simple ML baseline**: Logistic regression or decision tree
+
+A model that does not beat all baselines should not be deployed. The trivial baseline check catches label encoding bugs (where the model learns the majority class trivially).
+
+### Evaluation Report Structure
+
+```markdown
+# Evaluation Report: fraud-detector-v2.3.0
+
+## Dataset
+- Test set: 45,231 examples (2024-01-01 to 2024-03-31)
+- Class balance: 1.2% fraud, 98.8% non-fraud
+
+## Overall Metrics
+| Metric | v2.2.0 (prod) | v2.3.0 (candidate) | Delta |
+|--------|--------------|-------------------|-------|
+| ROC-AUC | 0.921 | 0.934 | +1.4% |
+| PR-AUC | 0.712 | 0.748 | +5.1% |
+| Recall @ precision=0.9 | 0.68 | 0.73 | +7.4% |
+
+## Slice Analysis
+| Slice | n | ROC-AUC | vs. Overall |
+|-------|---|---------|-------------|
+| Overall | 45,231 | 0.934 | — |
+| Amount < $50 | 12,445 | 0.941 | +0.7% |
+| Amount > $1000 | 3,211 | 0.918 | -1.6% |
+| New user (< 30 days) | 8,902 | 0.891 | -4.6% ⚠️ |
+
+## Recommendation
+Promote to staging. Investigate new user performance degradation before production.
+```

package/content/knowledge/ml/ml-observability.md

@@ -0,0 +1,253 @@
+---
+name: ml-observability
+description: Model monitoring for drift and decay, prediction logging, explainability tools, and alerting on accuracy drops in production ML systems
+topics: [ml, observability, monitoring, drift, model-decay, explainability, alerting, prediction-logging]
+---
+
+A model deployed to production without monitoring is a ticking clock. Models decay silently: the world changes, input distributions shift, and accuracy degrades while dashboards show green. Unlike software bugs that throw exceptions, model degradation has no stack trace — predictions simply become less useful. ML observability is the discipline of detecting these degradations before users notice them, through systematic monitoring of model inputs, outputs, and outcomes.
+
+## Summary
+
+ML observability covers four pillars: input monitoring (feature drift detection), output monitoring (prediction distribution shifts), outcome monitoring (accuracy against labels), and operational monitoring (latency, error rate). Complement monitoring with prediction logging for post-hoc analysis and explainability tools (SHAP, LIME) for understanding individual predictions and debugging systematic failures. Alert thresholds and on-call rotation for model health are as important as for service health.
+
+## Deep Guidance
+
+### The Four Pillars of ML Observability
+
+**Pillar 1 — Input monitoring (data drift)**: Detect when the distribution of model inputs changes from the training distribution. A model trained on winter data receiving summer data will degrade without any software change.
+
+**Pillar 2 — Output monitoring (prediction drift)**: Detect when the model's prediction distribution changes — e.g., a fraud model that suddenly classifies 10% of transactions as fraud (vs. the baseline 0.1%).
+
+**Pillar 3 — Outcome monitoring (accuracy/concept drift)**: Detect when model accuracy changes on labelled outcomes. Requires ground truth labels, which often arrive with delay (e.g., actual fraud confirmed days after prediction).
+
+**Pillar 4 — Operational monitoring**: Latency, throughput, error rate, memory usage. Standard SRE metrics applied to the model serving layer.
+
+### Feature Drift Detection
+
+Measure drift between training and serving feature distributions using statistical tests:
+
+```python
+from scipy import stats
+import numpy as np
+from dataclasses import dataclass
+from typing import Optional
+
+@dataclass
+class DriftReport:
+    feature: str
+    psi: float           # Population Stability Index
+    ks_statistic: float  # Kolmogorov-Smirnov statistic
+    ks_p_value: float
+    is_drifted: bool
+
+def compute_psi(
+    expected: np.ndarray,
+    actual: np.ndarray,
+    buckets: int = 10,
+) -> float:
+    """Population Stability Index. PSI < 0.1: stable, 0.1-0.2: minor drift, >0.2: significant drift."""
+    eps = 1e-6
+    expected_pcts, bins = np.histogram(expected, bins=buckets)
+    actual_pcts, _ = np.histogram(actual, bins=bins)
+
+    expected_pcts = expected_pcts / expected_pcts.sum() + eps
+    actual_pcts = actual_pcts / actual_pcts.sum() + eps
+
+    return float(np.sum((actual_pcts - expected_pcts) * np.log(actual_pcts / expected_pcts)))
+
+def detect_drift(
+    training_values: np.ndarray,
+    serving_values: np.ndarray,
+    feature_name: str,
+    psi_threshold: float = 0.2,
+    ks_alpha: float = 0.05,
+) -> DriftReport:
+    psi = compute_psi(training_values, serving_values)
+    ks_stat, ks_p = stats.ks_2samp(training_values, serving_values)
+    return DriftReport(
+        feature=feature_name,
+        psi=psi,
+        ks_statistic=ks_stat,
+        ks_p_value=ks_p,
+        is_drifted=psi > psi_threshold or ks_p < ks_alpha,
+    )
+```
+
+**Reference distribution maintenance**: Store feature statistics (mean, std, percentiles, histogram) from the training set as a "reference profile." Compare each day's serving data to this profile. Refresh the reference when the model is retrained.
+
+**PSI thresholds** (industry standard):
+- PSI < 0.1: No significant drift — monitor as normal
+- 0.1 ≤ PSI < 0.2: Minor drift — investigate, consider retraining
+- PSI ≥ 0.2: Significant drift — trigger retraining or alert
+
+### Prediction Logging
+
+Every prediction made in production should be logged for monitoring and post-hoc analysis:
+
+```python
+# src/serving/prediction_logger.py
+import json
+import time
+from dataclasses import dataclass, asdict
+from typing import Any
+
+@dataclass
+class PredictionRecord:
+    prediction_id: str       # UUID for correlation
+    model_version: str
+    timestamp: float
+    request_id: str          # Trace ID for distributed tracing
+    input_features: dict     # Logged features (scrub PII before logging)
+    prediction: Any
+    confidence: float
+    latency_ms: float
+
+class PredictionLogger:
+    def __init__(self, sink):  # sink: Kafka producer, Kinesis, or file
+        self.sink = sink
+
+    def log(self, record: PredictionRecord) -> None:
+        payload = json.dumps(asdict(record))
+        self.sink.send(payload)
+```
+
+**What to log** (balance observability with privacy/cost):
+- Always: prediction ID, model version, timestamp, prediction value, confidence, latency
+- Feature logging: Log features used for prediction (important for drift detection and debugging)
+- PII scrubbing: Never log raw PII fields; log derived features or anonymised values only
+- Sampling: For very high-throughput systems (> 10K RPS), log a representative sample (1–10%)
+
+**Label joining**: When ground truth labels arrive (delayed), join them with prediction logs using the prediction ID to compute accuracy metrics:
+```sql
+SELECT
+    p.model_version,
+    COUNT(*) as n_predictions,
+    AVG(CASE WHEN p.prediction = l.actual_label THEN 1 ELSE 0 END) as accuracy,
+    AVG(p.confidence) as mean_confidence
+FROM predictions p
+JOIN labels l ON p.prediction_id = l.prediction_id
+WHERE p.timestamp >= NOW() - INTERVAL '7 days'
+GROUP BY p.model_version
+```
+
+### Explainability
+
+Explainability tools help debug model failures and satisfy regulatory requirements:
+
+**SHAP (SHapley Additive exPlanations)**: Computes feature importance for individual predictions using game-theoretic Shapley values. Works with any model.
+
+```python
+import shap
+
+# Train a background dataset for the explainer
+background = X_train[np.random.choice(len(X_train), 100, replace=False)]
+explainer = shap.TreeExplainer(model)  # For tree models
+# explainer = shap.DeepExplainer(model, background)  # For neural networks
+# explainer = shap.KernelExplainer(model.predict_proba, background)  # Model-agnostic
+
+# Explain a single prediction
+shap_values = explainer.shap_values(X_test[0:1])
+shap.force_plot(explainer.expected_value[1], shap_values[1][0], X_test[0])
+
+# Explain the entire test set (global feature importance)
+shap_values_all = explainer.shap_values(X_test)
+shap.summary_plot(shap_values_all[1], X_test)
+```
+
+**LIME (Local Interpretable Model-agnostic Explanations)**: Fits a simple interpretable model (linear regression) locally around each prediction.
+
+```python
+from lime.lime_tabular import LimeTabularExplainer
+
+explainer = LimeTabularExplainer(
+    X_train,
+    feature_names=feature_names,
+    class_names=["legitimate", "fraud"],
+    mode="classification",
+)
+
+explanation = explainer.explain_instance(
+    X_test[0],
+    model.predict_proba,
+    num_features=10,
+)
+explanation.show_in_notebook()
+```
+
+**Integrated Gradients** (for neural networks): Attribution method that satisfies axiomatic completeness. Available in Captum (PyTorch):
+```python
+from captum.attr import IntegratedGradients
+
+ig = IntegratedGradients(model)
+attributions = ig.attribute(input_tensor, baseline=torch.zeros_like(input_tensor))
+```
+
+### Alerting Strategy
+
+Define alert thresholds before deployment, not after a production incident:
+
+```yaml
+# monitoring/alerts.yaml
+alerts:
+  - name: accuracy_degradation_warning
+    metric: val_accuracy_7d_rolling
+    condition: "< 0.87"  # Warning: 2pp below target
+    severity: warning
+    action: page_on_call
+
+  - name: accuracy_degradation_critical
+    metric: val_accuracy_7d_rolling
+    condition: "< 0.85"  # Critical: at SLA threshold
+    severity: critical
+    action: page_on_call_and_escalate
+
+  - name: feature_drift_significant
+    metric: max_psi_across_features
+    condition: "> 0.2"
+    severity: warning
+    action: notify_ml_team
+
+  - name: prediction_rate_anomaly
+    metric: fraud_prediction_rate_1h
+    condition: "> 0.05"  # 5x normal rate
+    severity: critical
+    action: page_on_call
+
+  - name: serving_latency_breach
+    metric: p99_latency_ms
+    condition: "> 200"
+    severity: warning
+    action: notify_ml_team
+```
+
+**Alerting anti-patterns**:
+- Alert fatigue: Too many low-signal alerts causes teams to ignore them. Start with critical-only, add warnings after establishing baselines.
+- Static thresholds for seasonal data: Use rolling baselines that adapt to weekly/seasonal patterns.
+- No runbook: Every alert must have a runbook link: "When this fires, do X, check Y, escalate to Z."
+
+### Model Monitoring Dashboard
+
+A model health dashboard should show at a glance:
+
+```
+Model: fraud-detector v2.3.1  |  Status: HEALTHY  |  Updated: 5 minutes ago
+
+┌─────────────────┬──────────────────┬──────────────────┐
+│ Accuracy (7d)   │ Prediction Rate  │ P99 Latency      │
+│   87.3%   ✓     │   0.12%    ✓     │   142ms    ✓     │
+│ target: ≥85%   │ baseline: 0.1%  │ SLA: <200ms     │
+└─────────────────┴──────────────────┴──────────────────┘
+
+┌──────────────────────────────────────────────────────┐
+│ Feature Drift (PSI)                                  │
+│ transaction_amount: 0.08  ✓                          │
+│ merchant_category:  0.12  ⚠ (minor drift)            │
+│ user_age_days:      0.04  ✓                          │
+└──────────────────────────────────────────────────────┘
+```
+
+Retraining triggers: Codify when to retrain rather than leaving it to human judgment:
+- Accuracy drops below warning threshold for 48+ consecutive hours
+- PSI > 0.2 on any top-10 feature by SHAP importance
+- Major upstream data source change (schema change, new data source)
+- Scheduled retraining on a fixed cadence (monthly for most models)