@zigrivers/scaffold 3.8.0 → 3.9.1

package/content/knowledge/ml/ml-conventions.md

@@ -0,0 +1,209 @@
+---
+name: ml-conventions
+description: Experiment naming, model versioning, reproducibility via random seeds, config-as-code patterns, and team conventions for ML projects
+topics: [ml, conventions, reproducibility, versioning, config, experiments]
+---
+
+ML projects without conventions degenerate into chaos within weeks: unnamed experiments with lost hyperparameters, models named `model_v2_final_FINAL.pkl`, and results that cannot be reproduced. Unlike software engineering where the compiler enforces structure, ML workflows are loose scripts and notebooks that require disciplined conventions to remain comprehensible. Establish these conventions at project start and encode them in tooling so they are followed by default, not willpower.
+
+## Summary
+
+ML conventions cover experiment naming (structured, searchable identifiers), model versioning (semantic or content-addressed), reproducibility (seeding all random sources, recording environment), and config-as-code (no magic numbers in code, all hyperparameters in config files). These conventions are not optional hygiene — they are the infrastructure that makes ML engineering a repeatable discipline rather than a research lottery.
+
+## Deep Guidance
+
+### Experiment Naming
+
+Every training run that produces artifacts or results must have a unique, human-readable identifier. Ad-hoc names like `test`, `v2`, or `new_model` are unusable at scale:
+
+**Recommended format**: `{model_type}-{dataset}-{date}-{purpose}[-{variant}]`
+
+Examples:
+- `resnet50-imagenet-20240315-baseline`
+- `bert-sst2-20240315-lr-sweep`
+- `xgboost-churn-20240320-feature-v3`
+- `gpt2-reviews-20240322-dropout-ablation`
+
+**Rules**:
+- All lowercase, hyphen-separated (no spaces, no underscores)
+- Date in `YYYYMMDD` format (sorts chronologically)
+- Purpose is human-readable and specific — not `experiment1` or `test`
+- Variant suffix for ablations and sweeps (`-v2`, `-no-dropout`, `-lr-1e-3`)
+
+Many teams use auto-generated experiment IDs (MLflow assigns UUID-based IDs automatically) and rely on tagging/metadata for search. This is fine as a secondary system, but always add a human-readable display name.
+
+### Model Versioning
+
+Model versioning is distinct from experiment tracking. A version is a production artifact; an experiment is a training run:
+
+**Semantic versioning for models**:
+- `v{major}.{minor}.{patch}` — consistent with software versioning
+- Major: Breaking change in model interface (input/output schema, preprocessing contract)
+- Minor: Meaningful accuracy improvement or new feature support
+- Patch: Bug fix, minor data update, no interface change
+
+**Content-addressed versioning** (used by MLflow Model Registry, DVC):
+- Models are identified by a hash of their weights + config
+- Prevents accidental overwriting
+- Enables exact reproducibility — "which weights produced this prediction?"
+
+**Registry-based model lifecycle**:
+```
+Staging → Validation → Production → Archived
+```
+- Never promote directly to Production — always pass through Staging validation
+- Keep at least one previous Production version for instant rollback
+- Document promotion reason: "Promoted: +2.3% AUC on Q1 eval set, latency within budget"
+
+### Reproducibility
+
+ML reproducibility means: given the same code, data, and config, the same model is produced. Achieve it through four controls:
+
+**1. Random seed management**
+
+Set all random sources before any computation:
+```python
+import random
+import numpy as np
+import torch
+
+def set_seed(seed: int) -> None:
+    random.seed(seed)
+    np.random.seed(seed)
+    torch.manual_seed(seed)
+    torch.cuda.manual_seed_all(seed)
+    # For full determinism (may impact performance)
+    torch.backends.cudnn.deterministic = True
+    torch.backends.cudnn.benchmark = False
+```
+
+Record the seed in experiment config. Default seed: `42` (or any fixed value — consistency matters more than the value). When running hyperparameter sweeps with multiple seeds, record all seeds and report mean ± std.
+
+**2. Dependency pinning**
+
+Pin all dependencies to exact versions:
+```toml
+# pyproject.toml (Poetry)
+[tool.poetry.dependencies]
+python = "3.11.4"
+torch = "2.1.0"
+transformers = "4.35.2"
+numpy = "1.26.0"
+```
+
+Use `poetry.lock` or `requirements.txt` generated by `pip freeze`. Never use unpinned dependencies (`torch>=2.0`) in a training environment.
+
+**3. Data versioning**
+
+Record the exact dataset version used for each training run:
+- DVC: content-addressed data with `dvc add` and `.dvc` pointers
+- Dataset registry: log dataset name + version + hash in experiment metadata
+- SQL-based datasets: log the query hash and execution timestamp
+
+**4. Environment reproducibility**
+
+Capture the full environment:
+```bash
+# Save environment
+conda env export > environment.yml
+pip freeze > requirements-frozen.txt
+
+# Record GPU driver and CUDA version
+nvidia-smi --query-gpu=driver_version,name --format=csv
+nvcc --version
+```
+
+For full environment isolation, use Docker. The Dockerfile is the environment specification.
+
+### Config-as-Code
+
+No magic numbers in code. Every hyperparameter, data path, and training setting belongs in a config file:
+
+**Bad** (magic numbers scattered in code):
+```python
+optimizer = Adam(model.parameters(), lr=0.001)
+scheduler = CosineAnnealingLR(optimizer, T_max=100)
+train_loader = DataLoader(dataset, batch_size=32, num_workers=4)
+```
+
+**Good** (config-driven):
+```yaml
+# configs/train.yaml
+training:
+  seed: 42
+  epochs: 100
+  batch_size: 32
+  num_workers: 4
+
+optimizer:
+  type: adam
+  lr: 1.0e-3
+  weight_decay: 1.0e-4
+
+scheduler:
+  type: cosine_annealing
+  t_max: 100
+```
+
+```python
+# src/training/train.py
+def train(cfg: DictConfig) -> None:
+    set_seed(cfg.training.seed)
+    optimizer = build_optimizer(model, cfg.optimizer)
+    scheduler = build_scheduler(optimizer, cfg.scheduler)
+```
+
+Use **Hydra** (Meta) or **OmegaConf** for hierarchical config management with CLI override support:
+```bash
+# Override from CLI without changing config files
+python train.py optimizer.lr=1e-4 training.batch_size=64
+```
+
+**Config file organization**:
+```
+configs/
+  base.yaml           # Default config for all experiments
+  model/
+    resnet50.yaml
+    vit-b16.yaml
+  data/
+    imagenet.yaml
+    cifar10.yaml
+  training/
+    fast.yaml         # Low-epoch for debugging
+    full.yaml         # Production training
+```
+
+### Code and Notebook Conventions
+
+**Notebooks are for exploration, not production**:
+- Notebooks belong in `notebooks/` — never in `src/`
+- Notebooks must be cleared before committing (no large outputs committed to git)
+- Meaningful results from notebooks are refactored into `src/` modules with tests
+
+**Module structure conventions**:
+- `src/data/` — dataset classes, data loaders, preprocessing transforms
+- `src/models/` — model architectures (no training logic)
+- `src/training/` — training loop, loss functions, callbacks
+- `src/evaluation/` — metrics, evaluation runners
+- `src/serving/` — inference code, prediction pipelines
+
+**Naming conventions**:
+- Files: `snake_case.py`
+- Classes: `PascalCase` (e.g., `ResNet50Classifier`, `ChurnDataset`)
+- Functions: `snake_case` (e.g., `compute_f1_score`, `load_checkpoint`)
+- Constants: `UPPER_SNAKE_CASE` (e.g., `MAX_SEQ_LENGTH = 512`)
+- Config keys: `snake_case` in YAML
+
+### Checklist Before Starting a Training Run
+
+```
+[ ] Experiment name follows naming convention
+[ ] Random seed set and recorded in config
+[ ] Config file committed (not just command-line overrides)
+[ ] Dataset version recorded
+[ ] Experiment tracker (MLflow/W&B) initialized with run metadata
+[ ] Code committed to git (note the commit SHA in the experiment)
+[ ] Output directory created and named consistently
+[ ] Hardware/environment recorded
+```

package/content/knowledge/ml/ml-dev-environment.md

@@ -0,0 +1,299 @@
+---
+name: ml-dev-environment
+description: Conda/Poetry environment setup, Jupyter integration, GPU detection and configuration, and Docker for reproducible ML development
+topics: [ml, dev-environment, conda, poetry, jupyter, gpu, docker, reproducibility]
+---
+
+ML development environments have more complexity than typical software projects: GPU drivers, CUDA toolkits, Python packages with native extensions, and Jupyter notebook infrastructure all need to align. A broken environment costs hours and blocks the whole team. Invest in environment standardisation upfront — the payoff is that every team member can reproduce results and that CI pipelines match local runs.
+
+## Summary
+
+Prefer Conda for ML projects when GPU and CUDA management is required; use Poetry for pure-Python projects or as the Python dependency manager on top of Conda. Configure Jupyter as a managed service rather than ad-hoc invocations. Detect GPU availability programmatically and handle CPU fallback gracefully. Use Docker to capture the full environment for reproducible training runs and production serving.
+
+## Deep Guidance
+
+### Conda vs. Poetry: When to Use Each
+
+**Conda** is the right choice when:
+- Managing GPU drivers and CUDA toolkit versions (Conda can install CUDA without root)
+- Working with packages that have complex native dependencies (PyTorch, TensorFlow, OpenCV)
+- Need to isolate Python version itself (not just packages)
+- Team uses multiple ML frameworks with conflicting dependencies
+
+**Poetry** is the right choice when:
+- Pure-Python project or all native dependencies are available via pip
+- Need strict dependency locking and reproducible installs
+- Publishing a library (Poetry handles packaging well)
+- Already using a Conda environment for CUDA and want finer control over Python packages
+
+**Common hybrid pattern**: Conda manages Python version and CUDA; Poetry manages Python package dependencies inside the Conda environment.
+
+### Conda Environment Setup
+
+```yaml
+# environment.yml — commit to git
+name: myproject
+channels:
+  - pytorch
+  - nvidia
+  - conda-forge
+  - defaults
+dependencies:
+  - python=3.11
+  - cuda-toolkit=12.1
+  - cudnn=8.9
+  - pip>=23.0
+  - pip:
+    - torch==2.1.0+cu121
+    - torchvision==0.16.0+cu121
+    - -r requirements.txt  # or use pyproject.toml
+```
+
+```bash
+# Create and activate
+conda env create -f environment.yml
+conda activate myproject
+
+# Update after environment.yml changes
+conda env update -f environment.yml --prune
+
+# Export current state (for exact reproducibility audit)
+conda env export > environment-lock.yml
+```
+
+**Critical**: Pin exact versions in `environment.yml`. `pytorch>=2.0` is not a reproducible spec.
+
+### Poetry Setup (Python Dependencies)
+
+```bash
+# Initialize
+poetry init
+
+# Add dependencies
+poetry add torch==2.1.0 transformers==4.35.2
+poetry add --group dev pytest black mypy
+
+# Install (creates .venv by default)
+poetry install
+
+# Run in the managed venv
+poetry run python train.py
+poetry run pytest
+```
+
+`pyproject.toml` example:
+```toml
+[tool.poetry]
+name = "myproject"
+version = "0.1.0"
+description = "ML project"
+python = "^3.11"
+
+[tool.poetry.dependencies]
+torch = "2.1.0"
+transformers = "4.35.2"
+hydra-core = "1.3.2"
+mlflow = "2.9.2"
+
+[tool.poetry.group.dev.dependencies]
+pytest = "7.4.3"
+black = "23.11.0"
+mypy = "1.7.0"
+nbstripout = "0.6.1"
+```
+
+### GPU Detection and Configuration
+
+Always detect GPU availability at runtime and handle CPU fallback:
+
+```python
+# src/utils/device.py
+import torch
+import logging
+
+logger = logging.getLogger(__name__)
+
+def get_device(prefer_gpu: bool = True) -> torch.device:
+    """Return the best available device with logging."""
+    if prefer_gpu and torch.cuda.is_available():
+        device = torch.device("cuda")
+        gpu_name = torch.cuda.get_device_name(0)
+        gpu_memory = torch.cuda.get_device_properties(0).total_memory / 1e9
+        logger.info(f"Using GPU: {gpu_name} ({gpu_memory:.1f} GB)")
+    elif prefer_gpu and torch.backends.mps.is_available():
+        # Apple Silicon
+        device = torch.device("mps")
+        logger.info("Using Apple MPS device")
+    else:
+        device = torch.device("cpu")
+        logger.info("Using CPU — GPU not available or not requested")
+    return device
+
+def log_gpu_memory() -> None:
+    """Log current GPU memory usage."""
+    if torch.cuda.is_available():
+        allocated = torch.cuda.memory_allocated() / 1e9
+        reserved = torch.cuda.memory_reserved() / 1e9
+        logger.debug(f"GPU memory: {allocated:.2f} GB allocated, {reserved:.2f} GB reserved")
+```
+
+**CUDA version compatibility**: PyTorch packages are built against specific CUDA versions. Always match:
+
+| PyTorch | CUDA | CUDNN |
+|---------|------|-------|
+| 2.1.x | 12.1, 11.8 | 8.x |
+| 2.0.x | 11.7, 11.8 | 8.x |
+
+Check compatibility at pytorch.org before pinning.
+
+**Multi-GPU setup** (training only — not for development):
+```python
+# Detect available GPUs
+n_gpus = torch.cuda.device_count()
+if n_gpus > 1:
+    model = torch.nn.DataParallel(model)  # Simple, for research
+    # Or for production: use DistributedDataParallel (see ml-training-patterns)
+```
+
+### Jupyter Integration
+
+Run Jupyter as a managed kernel rather than an ad-hoc server:
+
+```bash
+# Install Jupyter in the project environment
+poetry add --group dev jupyter jupyterlab ipykernel
+
+# Register the project venv as a named Jupyter kernel
+poetry run python -m ipykernel install --user --name myproject --display-name "MyProject (Python 3.11)"
+
+# Launch JupyterLab
+poetry run jupyter lab
+```
+
+Now all project notebooks run in the same environment as the source code.
+
+**Recommended Jupyter extensions**:
+- `nbstripout` — strips outputs before git commit
+- `jupyterlab-git` — git integration in the UI
+- `jupyterlab-lsp` — language server (autocomplete, type hints)
+
+**VS Code Jupyter integration** (recommended over browser-based):
+```json
+// .vscode/settings.json
+{
+    "jupyter.kernels.filter": [
+        {"path": "${workspaceFolder}/.venv/bin/python", "type": "pythonEnvironment"}
+    ],
+    "jupyter.notebookFileRoot": "${workspaceFolder}",
+    "python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python"
+}
+```
+
+### Docker for Reproducibility
+
+Docker captures the entire environment — OS, CUDA, Python, and packages. Use it for:
+- CI training runs
+- Sharing experiments with collaborators who have different local setups
+- Production serving (identical environment to training)
+
+**Base `Dockerfile` for ML training**:
+```dockerfile
+# Use NVIDIA's official CUDA base image
+FROM nvidia/cuda:12.1.0-cudnn8-runtime-ubuntu22.04
+
+# Set Python version
+ENV PYTHON_VERSION=3.11
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN apt-get update && apt-get install -y \
+    python${PYTHON_VERSION} \
+    python3-pip \
+    git \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN ln -s /usr/bin/python${PYTHON_VERSION} /usr/bin/python
+
+# Install Poetry
+RUN pip install poetry==1.7.1
+ENV POETRY_NO_INTERACTION=1 \
+    POETRY_VENV_IN_PROJECT=1
+
+WORKDIR /app
+
+# Install dependencies (cached layer)
+COPY pyproject.toml poetry.lock ./
+RUN poetry install --no-root --without dev
+
+# Copy source
+COPY src/ ./src/
+COPY configs/ ./configs/
+
+# Install the project itself
+RUN poetry install --without dev
+
+ENTRYPOINT ["poetry", "run", "python", "-m", "src.training.train"]
+```
+
+**Docker Compose for development**:
+```yaml
+# docker-compose.yml
+services:
+  train:
+    build: .
+    volumes:
+      - ./data:/app/data
+      - ./models:/app/models
+      - ./configs:/app/configs
+    environment:
+      - MLFLOW_TRACKING_URI=http://mlflow:5000
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+
+  mlflow:
+    image: ghcr.io/mlflow/mlflow:v2.9.2
+    ports:
+      - "5000:5000"
+    volumes:
+      - ./mlruns:/mlflow/mlruns
+```
+
+### Makefile Task Runner
+
+Encode common tasks in a `Makefile` to eliminate "how do I run this?" questions:
+
+```makefile
+.PHONY: env train eval test lint clean
+
+env:
+	conda env create -f environment.yml || conda env update -f environment.yml --prune
+
+train:
+	poetry run python -m src.training.train $(ARGS)
+
+eval:
+	poetry run python -m src.evaluation.evaluator $(ARGS)
+
+test:
+	poetry run pytest tests/ -v
+
+lint:
+	poetry run black --check src/ tests/
+	poetry run mypy src/
+
+clean:
+	find . -type f -name "*.pyc" -delete
+	find . -type d -name "__pycache__" -delete
+	rm -rf .pytest_cache/
+```
+
+Usage:
+```bash
+make env                    # Set up environment
+make train ARGS="optimizer.lr=1e-4"
+make test
+```