@zigrivers/scaffold 3.7.0 → 3.9.0

package/content/knowledge/ml/ml-dev-environment.md

@@ -0,0 +1,299 @@
+---
+name: ml-dev-environment
+description: Conda/Poetry environment setup, Jupyter integration, GPU detection and configuration, and Docker for reproducible ML development
+topics: [ml, dev-environment, conda, poetry, jupyter, gpu, docker, reproducibility]
+---
+
+ML development environments have more complexity than typical software projects: GPU drivers, CUDA toolkits, Python packages with native extensions, and Jupyter notebook infrastructure all need to align. A broken environment costs hours and blocks the whole team. Invest in environment standardisation upfront — the payoff is that every team member can reproduce results and that CI pipelines match local runs.
+
+## Summary
+
+Prefer Conda for ML projects when GPU and CUDA management is required; use Poetry for pure-Python projects or as the Python dependency manager on top of Conda. Configure Jupyter as a managed service rather than ad-hoc invocations. Detect GPU availability programmatically and handle CPU fallback gracefully. Use Docker to capture the full environment for reproducible training runs and production serving.
+
+## Deep Guidance
+
+### Conda vs. Poetry: When to Use Each
+
+**Conda** is the right choice when:
+- Managing GPU drivers and CUDA toolkit versions (Conda can install CUDA without root)
+- Working with packages that have complex native dependencies (PyTorch, TensorFlow, OpenCV)
+- Need to isolate Python version itself (not just packages)
+- Team uses multiple ML frameworks with conflicting dependencies
+
+**Poetry** is the right choice when:
+- Pure-Python project or all native dependencies are available via pip
+- Need strict dependency locking and reproducible installs
+- Publishing a library (Poetry handles packaging well)
+- Already using a Conda environment for CUDA and want finer control over Python packages
+
+**Common hybrid pattern**: Conda manages Python version and CUDA; Poetry manages Python package dependencies inside the Conda environment.
+
+### Conda Environment Setup
+
+```yaml
+# environment.yml — commit to git
+name: myproject
+channels:
+  - pytorch
+  - nvidia
+  - conda-forge
+  - defaults
+dependencies:
+  - python=3.11
+  - cuda-toolkit=12.1
+  - cudnn=8.9
+  - pip>=23.0
+  - pip:
+    - torch==2.1.0+cu121
+    - torchvision==0.16.0+cu121
+    - -r requirements.txt  # or use pyproject.toml
+```
+
+```bash
+# Create and activate
+conda env create -f environment.yml
+conda activate myproject
+
+# Update after environment.yml changes
+conda env update -f environment.yml --prune
+
+# Export current state (for exact reproducibility audit)
+conda env export > environment-lock.yml
+```
+
+**Critical**: Pin exact versions in `environment.yml`. `pytorch>=2.0` is not a reproducible spec.
+
+### Poetry Setup (Python Dependencies)
+
+```bash
+# Initialize
+poetry init
+
+# Add dependencies
+poetry add torch==2.1.0 transformers==4.35.2
+poetry add --group dev pytest black mypy
+
+# Install (creates .venv by default)
+poetry install
+
+# Run in the managed venv
+poetry run python train.py
+poetry run pytest
+```
+
+`pyproject.toml` example:
+```toml
+[tool.poetry]
+name = "myproject"
+version = "0.1.0"
+description = "ML project"
+python = "^3.11"
+
+[tool.poetry.dependencies]
+torch = "2.1.0"
+transformers = "4.35.2"
+hydra-core = "1.3.2"
+mlflow = "2.9.2"
+
+[tool.poetry.group.dev.dependencies]
+pytest = "7.4.3"
+black = "23.11.0"
+mypy = "1.7.0"
+nbstripout = "0.6.1"
+```
+
+### GPU Detection and Configuration
+
+Always detect GPU availability at runtime and handle CPU fallback:
+
+```python
+# src/utils/device.py
+import torch
+import logging
+
+logger = logging.getLogger(__name__)
+
+def get_device(prefer_gpu: bool = True) -> torch.device:
+    """Return the best available device with logging."""
+    if prefer_gpu and torch.cuda.is_available():
+        device = torch.device("cuda")
+        gpu_name = torch.cuda.get_device_name(0)
+        gpu_memory = torch.cuda.get_device_properties(0).total_memory / 1e9
+        logger.info(f"Using GPU: {gpu_name} ({gpu_memory:.1f} GB)")
+    elif prefer_gpu and torch.backends.mps.is_available():
+        # Apple Silicon
+        device = torch.device("mps")
+        logger.info("Using Apple MPS device")
+    else:
+        device = torch.device("cpu")
+        logger.info("Using CPU — GPU not available or not requested")
+    return device
+
+def log_gpu_memory() -> None:
+    """Log current GPU memory usage."""
+    if torch.cuda.is_available():
+        allocated = torch.cuda.memory_allocated() / 1e9
+        reserved = torch.cuda.memory_reserved() / 1e9
+        logger.debug(f"GPU memory: {allocated:.2f} GB allocated, {reserved:.2f} GB reserved")
+```
+
+**CUDA version compatibility**: PyTorch packages are built against specific CUDA versions. Always match:
+
+| PyTorch | CUDA | CUDNN |
+|---------|------|-------|
+| 2.1.x | 12.1, 11.8 | 8.x |
+| 2.0.x | 11.7, 11.8 | 8.x |
+
+Check compatibility at pytorch.org before pinning.
+
+**Multi-GPU setup** (training only — not for development):
+```python
+# Detect available GPUs
+n_gpus = torch.cuda.device_count()
+if n_gpus > 1:
+    model = torch.nn.DataParallel(model)  # Simple, for research
+    # Or for production: use DistributedDataParallel (see ml-training-patterns)
+```
+
+### Jupyter Integration
+
+Run Jupyter as a managed kernel rather than an ad-hoc server:
+
+```bash
+# Install Jupyter in the project environment
+poetry add --group dev jupyter jupyterlab ipykernel
+
+# Register the project venv as a named Jupyter kernel
+poetry run python -m ipykernel install --user --name myproject --display-name "MyProject (Python 3.11)"
+
+# Launch JupyterLab
+poetry run jupyter lab
+```
+
+Now all project notebooks run in the same environment as the source code.
+
+**Recommended Jupyter extensions**:
+- `nbstripout` — strips outputs before git commit
+- `jupyterlab-git` — git integration in the UI
+- `jupyterlab-lsp` — language server (autocomplete, type hints)
+
+**VS Code Jupyter integration** (recommended over browser-based):
+```json
+// .vscode/settings.json
+{
+    "jupyter.kernels.filter": [
+        {"path": "${workspaceFolder}/.venv/bin/python", "type": "pythonEnvironment"}
+    ],
+    "jupyter.notebookFileRoot": "${workspaceFolder}",
+    "python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python"
+}
+```
+
+### Docker for Reproducibility
+
+Docker captures the entire environment — OS, CUDA, Python, and packages. Use it for:
+- CI training runs
+- Sharing experiments with collaborators who have different local setups
+- Production serving (identical environment to training)
+
+**Base `Dockerfile` for ML training**:
+```dockerfile
+# Use NVIDIA's official CUDA base image
+FROM nvidia/cuda:12.1.0-cudnn8-runtime-ubuntu22.04
+
+# Set Python version
+ENV PYTHON_VERSION=3.11
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN apt-get update && apt-get install -y \
+    python${PYTHON_VERSION} \
+    python3-pip \
+    git \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN ln -s /usr/bin/python${PYTHON_VERSION} /usr/bin/python
+
+# Install Poetry
+RUN pip install poetry==1.7.1
+ENV POETRY_NO_INTERACTION=1 \
+    POETRY_VENV_IN_PROJECT=1
+
+WORKDIR /app
+
+# Install dependencies (cached layer)
+COPY pyproject.toml poetry.lock ./
+RUN poetry install --no-root --without dev
+
+# Copy source
+COPY src/ ./src/
+COPY configs/ ./configs/
+
+# Install the project itself
+RUN poetry install --without dev
+
+ENTRYPOINT ["poetry", "run", "python", "-m", "src.training.train"]
+```
+
+**Docker Compose for development**:
+```yaml
+# docker-compose.yml
+services:
+  train:
+    build: .
+    volumes:
+      - ./data:/app/data
+      - ./models:/app/models
+      - ./configs:/app/configs
+    environment:
+      - MLFLOW_TRACKING_URI=http://mlflow:5000
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+
+  mlflow:
+    image: ghcr.io/mlflow/mlflow:v2.9.2
+    ports:
+      - "5000:5000"
+    volumes:
+      - ./mlruns:/mlflow/mlruns
+```
+
+### Makefile Task Runner
+
+Encode common tasks in a `Makefile` to eliminate "how do I run this?" questions:
+
+```makefile
+.PHONY: env train eval test lint clean
+
+env:
+	conda env create -f environment.yml || conda env update -f environment.yml --prune
+
+train:
+	poetry run python -m src.training.train $(ARGS)
+
+eval:
+	poetry run python -m src.evaluation.evaluator $(ARGS)
+
+test:
+	poetry run pytest tests/ -v
+
+lint:
+	poetry run black --check src/ tests/
+	poetry run mypy src/
+
+clean:
+	find . -type f -name "*.pyc" -delete
+	find . -type d -name "__pycache__" -delete
+	rm -rf .pytest_cache/
+```
+
+Usage:
+```bash
+make env                    # Set up environment
+make train ARGS="optimizer.lr=1e-4"
+make test
+```

package/content/knowledge/ml/ml-experiment-tracking.md

@@ -0,0 +1,285 @@
+---
+name: ml-experiment-tracking
+description: MLflow and Weights & Biases integration, artifact storage, experiment run comparison, and hyperparameter sweep management
+topics: [ml, experiment-tracking, mlflow, wandb, artifacts, sweeps, reproducibility]
+---
+
+Without experiment tracking, ML development is archaeology: "which config produced that result?" is answered by digging through notebook history, chat logs, and failing memory. Experiment tracking tools are version control for training runs — every metric, every hyperparameter, every artifact, linked to the code that produced it. The discipline of logging everything during training pays dividends when a stakeholder asks "how does this model compare to what we had six months ago?"
+
+## Summary
+
+Use MLflow (self-hosted, open source) or Weights & Biases (cloud, more feature-rich) to track every training run. Log hyperparameters, metrics at each epoch, model artifacts, and the git commit SHA. Store large artifacts (checkpoints, datasets) in object storage backed by the experiment tracker. Use sweep features (MLflow Hyperopt integration, W&B Sweeps) for systematic hyperparameter search rather than manual iteration.
+
+## Deep Guidance
+
+### MLflow Integration
+
+MLflow is the open-source standard for experiment tracking. It runs locally or on a managed server:
+
+```bash
+# Start local tracking server (stores runs in ./mlruns)
+mlflow server --host 0.0.0.0 --port 5000
+
+# Or use the SQLite backend for better performance
+mlflow server \
+  --backend-store-uri sqlite:///mlflow.db \
+  --default-artifact-root ./mlartifacts \
+  --host 0.0.0.0 --port 5000
+```
+
+**Instrument training code**:
+```python
+import mlflow
+import mlflow.pytorch
+
+# Set tracking server
+mlflow.set_tracking_uri("http://localhost:5000")
+mlflow.set_experiment("fraud-detector")
+
+def train(cfg: DictConfig) -> dict:
+    with mlflow.start_run(run_name=cfg.experiment.name) as run:
+        # Log all hyperparameters from config
+        mlflow.log_params(OmegaConf.to_container(cfg, resolve=True))
+
+        # Log git commit for reproducibility
+        import subprocess
+        git_sha = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode().strip()
+        mlflow.set_tag("git_commit", git_sha)
+        mlflow.set_tag("model_type", cfg.model.type)
+
+        for epoch in range(cfg.training.epochs):
+            train_metrics = train_epoch(...)
+            val_metrics = evaluate(...)
+
+            # Log metrics with step (epoch) for time-series view
+            mlflow.log_metrics({
+                "train_loss": train_metrics["loss"],
+                "val_loss": val_metrics["loss"],
+                "val_auc": val_metrics["auc"],
+            }, step=epoch)
+
+        # Log best model
+        mlflow.pytorch.log_model(
+            model,
+            artifact_path="model",
+            registered_model_name="fraud-detector",  # Register in Model Registry
+        )
+
+        # Log additional artifacts
+        mlflow.log_artifact("configs/train.yaml")
+        mlflow.log_artifact("reports/eval_report.json")
+
+        return {"run_id": run.info.run_id, **val_metrics}
+```
+
+**MLflow Model Registry** (promote to production):
+```python
+from mlflow.tracking import MlflowClient
+
+client = MlflowClient()
+
+# Register a run's model in the registry
+model_uri = f"runs:/{run_id}/model"
+mv = mlflow.register_model(model_uri, "fraud-detector")
+
+# Transition to staging after validation
+client.transition_model_version_stage(
+    name="fraud-detector",
+    version=mv.version,
+    stage="Staging",
+    archive_existing_versions=False,
+)
+
+# Load production model in serving
+production_model = mlflow.pytorch.load_model(
+    model_uri="models:/fraud-detector/Production"
+)
+```
+
+### Weights & Biases Integration
+
+W&B provides a richer UI and more features than MLflow, with a cloud-hosted option:
+
+```python
+import wandb
+
+wandb.init(
+    project="fraud-detector",
+    name=cfg.experiment.name,
+    config=OmegaConf.to_container(cfg, resolve=True),
+    tags=["baseline", "v2-features"],
+    notes="Testing new feature set with gradient clipping",
+)
+
+# Log metrics
+for epoch in range(cfg.training.epochs):
+    metrics = train_epoch(...)
+    wandb.log({
+        "epoch": epoch,
+        "train/loss": metrics["train_loss"],
+        "val/loss": metrics["val_loss"],
+        "val/auc": metrics["val_auc"],
+        "lr": scheduler.get_last_lr()[0],
+    })
+
+# Log model artifact
+artifact = wandb.Artifact("fraud-detector", type="model")
+artifact.add_file("models/checkpoints/best.pt")
+wandb.log_artifact(artifact)
+
+wandb.finish()
+```
+
+**W&B-specific features**:
+- **System monitoring**: GPU utilisation, memory, temperature logged automatically
+- **Gradient histograms**: `wandb.watch(model, log="gradients")` logs gradient distributions per layer — invaluable for debugging vanishing/exploding gradients
+- **Media logging**: Log images, audio, tables, confusion matrices directly in the UI
+- **Alerts**: Set threshold alerts on metrics (email/Slack when val_loss > threshold)
+
+### Artifact Storage Strategy
+
+Artifacts are the binary outputs of training runs: model checkpoints, preprocessed datasets, evaluation reports, and confusion matrices. Never store large binary artifacts in git:
+
+**Storage hierarchy**:
+```
+Small artifacts (< 1 MB): Log directly to tracker
+  - Config files, evaluation reports (JSON/CSV)
+  - Example predictions, confusion matrices (images)
+
+Medium artifacts (1 MB – 1 GB): Log as tracker artifacts
+  - Model checkpoints for experimentation
+  - Feature engineering outputs
+
+Large artifacts (> 1 GB): Object storage with tracker reference
+  - Full training datasets
+  - Final production model weights
+  - Large evaluation outputs
+```
+
+**S3 artifact storage for MLflow**:
+```bash
+mlflow server \
+  --default-artifact-root s3://my-bucket/mlflow-artifacts \
+  --backend-store-uri postgresql://user:pass@host/mlflow
+```
+
+**DVC for dataset versioning alongside MLflow**:
+```bash
+# Version dataset with DVC
+dvc add data/processed/features_v3.parquet
+git add data/processed/features_v3.parquet.dvc
+
+# Log DVC dataset reference in MLflow
+mlflow.set_tag("dvc_dataset_commit", git_sha)
+mlflow.set_tag("dataset_path", "data/processed/features_v3.parquet")
+```
+
+### Run Comparison and Analysis
+
+**Finding the best run** (MLflow Python API):
+```python
+from mlflow.tracking import MlflowClient
+import pandas as pd
+
+client = MlflowClient()
+
+# Get all runs in an experiment, sorted by val_auc
+runs = client.search_runs(
+    experiment_ids=["1"],
+    filter_string="metrics.val_auc > 0.85",
+    order_by=["metrics.val_auc DESC"],
+    max_results=20,
+)
+
+# Convert to DataFrame for analysis
+run_data = [{
+    "run_id": r.info.run_id,
+    "name": r.info.run_name,
+    "val_auc": r.data.metrics.get("val_auc"),
+    "lr": r.data.params.get("optimizer.lr"),
+    "batch_size": r.data.params.get("training.batch_size"),
+} for r in runs]
+
+df = pd.DataFrame(run_data)
+print(df.head(10))
+```
+
+**Comparing runs in W&B**: Use the parallel coordinates plot (built into W&B UI) to visualise the relationship between hyperparameters and metrics across many runs at once.
+
+### Hyperparameter Sweeps
+
+**W&B Sweeps** (cloud-managed sweep coordinator):
+```yaml
+# sweep_config.yaml
+program: train.py
+method: bayes  # bayesian, random, or grid
+metric:
+  name: val/auc
+  goal: maximize
+parameters:
+  optimizer.lr:
+    min: 1.0e-5
+    max: 1.0e-2
+    distribution: log_uniform_values
+  training.batch_size:
+    values: [16, 32, 64, 128]
+  model.dropout:
+    min: 0.0
+    max: 0.5
+early_terminate:
+  type: hyperband
+  min_iter: 3
+```
+
+```bash
+wandb sweep sweep_config.yaml  # Returns sweep ID
+wandb agent <sweep-id> --count 50  # Launch 50 trials
+```
+
+**MLflow + Optuna** (self-hosted alternative):
+```python
+import optuna
+import mlflow
+
+def objective(trial):
+    with mlflow.start_run(nested=True):
+        lr = trial.suggest_float("lr", 1e-5, 1e-2, log=True)
+        mlflow.log_param("lr", lr)
+
+        val_auc = train_and_evaluate(lr=lr)
+        mlflow.log_metric("val_auc", val_auc)
+        return val_auc
+
+with mlflow.start_run(run_name="hyperparameter-sweep"):
+    study = optuna.create_study(direction="maximize")
+    study.optimize(objective, n_trials=50)
+    mlflow.log_params(study.best_params)
+    mlflow.log_metric("best_val_auc", study.best_value)
+```
+
+### Experiment Logging Checklist
+
+Log these for every training run — no exceptions:
+
+```python
+# Required: hyperparameters
+mlflow.log_params({...})  # Full config dict
+
+# Required: metrics at each epoch
+mlflow.log_metrics({...}, step=epoch)
+
+# Required: final metrics
+mlflow.log_metrics({"final_val_auc": val_auc, "final_val_loss": val_loss})
+
+# Required: reproducibility tags
+mlflow.set_tag("git_commit", git_sha)
+mlflow.set_tag("dataset_version", dataset_version)
+
+# Required: model artifact
+mlflow.pytorch.log_model(model, "model")
+
+# Recommended: environment
+mlflow.log_artifact("environment.yml")
+mlflow.set_tag("cuda_version", torch.version.cuda)
+mlflow.set_tag("pytorch_version", torch.__version__)
+```