@zigrivers/scaffold 3.7.0 → 3.9.0

package/content/knowledge/ml/ml-project-structure.md

@@ -0,0 +1,216 @@
+---
+name: ml-project-structure
+description: Standard ML project directory layout covering src/data, src/models, src/training, src/serving, notebooks, configs, and model artifact storage
+topics: [ml, project-structure, layout, organization, artifacts, notebooks]
+---
+
+ML projects accumulate files faster than almost any other software domain: datasets, model checkpoints, experiment configs, notebooks, evaluation reports, and serving code. Without a deliberate directory structure, projects become disorganised within weeks and impossible to onboard new team members onto. A well-structured ML project separates concerns clearly: source code from notebooks, training from serving, configs from code, and tracked artifacts from ephemeral outputs.
+
+## Summary
+
+A standard ML project separates source code (`src/`), exploratory notebooks (`notebooks/`), training configurations (`configs/`), and model artifacts (`models/`). Within `src/`, separate data loading (`src/data/`), model architectures (`src/models/`), training logic (`src/training/`), and serving code (`src/serving/`). Keep large artifacts (datasets, checkpoints) out of git using `.gitignore` and DVC or object storage. The structure should be navigable to a new team member within five minutes.
+
+## Deep Guidance
+
+### Top-Level Directory Structure
+
+```
+project-root/
+├── configs/            # All experiment and model configs (YAML/TOML)
+├── data/               # Data directory (gitignored content, DVC-tracked)
+│   ├── raw/            # Immutable raw data as received from source
+│   ├── processed/      # Cleaned, transformed datasets
+│   └── splits/         # Train/val/test split files (CSV/JSON of IDs)
+├── docs/               # Architecture decisions, dataset cards, model cards
+├── models/             # Model artifact storage (gitignored; object storage backed)
+│   ├── checkpoints/    # Training checkpoints (epoch-N.pt)
+│   └── registry/       # Production-promoted model versions
+├── notebooks/          # Jupyter notebooks for exploration (outputs cleared before commit)
+├── reports/            # Evaluation reports, figures, experiment summaries
+├── scripts/            # One-off utility scripts (not part of the pipeline)
+├── src/                # All production source code
+│   ├── data/           # Dataset classes, loaders, preprocessing
+│   ├── models/         # Model architecture definitions
+│   ├── training/       # Training loops, loss functions, callbacks
+│   ├── evaluation/     # Metrics, evaluation runners, result serialisation
+│   └── serving/        # Inference pipelines, API handlers, preprocessing wrappers
+├── tests/              # Unit and integration tests
+├── .dvc/               # DVC metadata (committed to git)
+├── .gitignore          # Excludes data/, models/, __pycache__, .env
+├── pyproject.toml      # Project metadata and dependencies (Poetry)
+├── Makefile            # Task runner: train, evaluate, serve, test
+└── README.md           # Project overview, setup, and usage
+```
+
+### `src/data/` — Data Loading and Preprocessing
+
+This directory contains all code that transforms raw data into model-ready tensors:
+
+```
+src/data/
+├── __init__.py
+├── dataset.py          # PyTorch Dataset or TF Dataset class
+├── datamodule.py       # LightningDataModule or equivalent orchestrator
+├── transforms.py       # Preprocessing transforms (normalize, tokenize, augment)
+├── augmentation.py     # Training-time data augmentation (separated from eval transforms)
+├── collate.py          # Custom batch collation functions
+└── utils.py            # Data utilities (download, checksum, split generation)
+```
+
+Key rules:
+- **Separate training transforms from eval transforms** — augmentation must not be applied at inference
+- Dataset classes must accept a `split` parameter and behave correctly for each split
+- All preprocessing must be reproducible and deterministic at inference time
+- Cache processed data to avoid recomputation on each run
+
+### `src/models/` — Architecture Definitions
+
+Contains model class definitions only — no training logic, no loss functions:
+
+```
+src/models/
+├── __init__.py
+├── backbone.py         # Feature extractor (ResNet, ViT, BERT, etc.)
+├── head.py             # Task-specific head (classification, regression, generation)
+├── model.py            # Composed full model
+└── components/         # Reusable building blocks (attention, MLP, norm layers)
+    ├── attention.py
+    └── ffn.py
+```
+
+Key rules:
+- Models are pure computation graphs — no file I/O, no training state
+- Accept hyperparameters via constructor, not globals
+- Provide a `from_config(cfg)` class method for config-driven instantiation
+- Serialise with `state_dict()` only — never pickle entire model objects
+
+### `src/training/` — Training Logic
+
+```
+src/training/
+├── __init__.py
+├── trainer.py          # Training loop (or LightningModule)
+├── loss.py             # Loss functions
+├── optimizer.py        # Optimizer and scheduler builders
+├── callbacks.py        # Callbacks (early stopping, logging, checkpoint saving)
+└── utils.py            # Gradient clipping, mixed precision helpers
+```
+
+The training loop is separate from the model. A model knows how to compute predictions; the trainer knows how to update weights. This separation enables:
+- Testing model forward passes independently of training
+- Swapping training strategies (single GPU, DDP, FSDP) without changing the model
+- Using the same model class for training and serving
+
+### `src/evaluation/` — Metrics and Evaluation Runners
+
+```
+src/evaluation/
+├── __init__.py
+├── metrics.py          # Metric computation (accuracy, F1, AUC, etc.)
+├── evaluator.py        # Evaluation loop (runs model on eval set, collects predictions)
+├── slice_analysis.py   # Per-slice performance breakdown
+└── reports.py          # Result serialisation and report generation
+```
+
+Evaluation code runs identically offline and online. Do not inline evaluation logic in the training loop — this makes it impossible to re-evaluate a checkpoint independently.
+
+### `src/serving/` — Inference and API
+
+```
+src/serving/
+├── __init__.py
+├── predictor.py        # Prediction class (loads model, runs inference)
+├── preprocessing.py    # Request preprocessing (mirrors training eval transforms)
+├── postprocessing.py   # Response postprocessing (calibration, thresholding)
+├── api.py              # FastAPI/Flask endpoint definitions
+└── handler.py          # TorchServe or Triton handler
+```
+
+The `Predictor` class is the contract between the model and the serving infrastructure. It:
+- Loads a model from a path or registry reference
+- Exposes a `predict(inputs)` method with documented input/output types
+- Uses the exact same preprocessing as training evaluation transforms
+
+### `notebooks/` — Exploratory Analysis
+
+```
+notebooks/
+├── 01-data-exploration.ipynb   # EDA, data quality checks
+├── 02-baseline-model.ipynb     # Baseline experiments
+├── 03-feature-engineering.ipynb
+└── 04-error-analysis.ipynb     # Post-training error analysis
+```
+
+Rules for notebooks:
+- **Clear outputs before committing** — use `nbstripout` as a pre-commit hook
+- Number notebooks in chronological/logical order
+- Notebooks document exploration, not production logic
+- Any reusable code found in notebooks gets refactored into `src/` with tests
+
+### `configs/` — Experiment Configuration
+
+```
+configs/
+├── base.yaml               # Default config merged into all experiments
+├── model/
+│   ├── small.yaml
+│   └── large.yaml
+├── data/
+│   ├── dev.yaml            # Small dataset for fast iteration
+│   └── full.yaml           # Full production dataset
+└── training/
+    ├── debug.yaml          # 1 epoch, no logging, fast feedback
+    └── production.yaml     # Full training run settings
+```
+
+### `models/` — Artifact Storage
+
+Large binary artifacts are not stored in git:
+- Checkpoints and production models live in `models/` but are gitignored
+- Back `models/` with object storage: S3, GCS, Azure Blob Storage
+- Use DVC to track artifact versions alongside the code:
+
+```bash
+dvc add models/registry/v1.2.0/model.pt
+git add models/registry/v1.2.0/model.pt.dvc
+git commit -m "feat: register model v1.2.0"
+dvc push  # Pushes binary to remote storage
+```
+
+Teammates restore the artifact with `dvc pull` — they get the exact binary referenced by the `.dvc` pointer in git.
+
+### `.gitignore` Essentials for ML Projects
+
+```gitignore
+# Data
+data/raw/
+data/processed/
+data/splits/*.csv
+
+# Model artifacts
+models/checkpoints/
+models/registry/
+
+# Notebook outputs
+*.ipynb
+
+# Python
+__pycache__/
+*.pyc
+.venv/
+*.egg-info/
+
+# Experiment tracking
+mlruns/
+wandb/
+
+# Environment
+.env
+*.env
+```
+
+Use `nbstripout` to automatically strip notebook outputs:
+```bash
+pip install nbstripout
+nbstripout --install  # Installs as git filter
+```

package/content/knowledge/ml/ml-requirements.md

@@ -0,0 +1,138 @@
+---
+name: ml-requirements
+description: Model performance metrics (accuracy, latency, throughput), business KPIs, fairness/bias requirements, and SLA definitions for ML systems
+topics: [ml, requirements, metrics, fairness, bias, sla, kpi]
+---
+
+ML requirements differ from traditional software requirements because correctness is probabilistic, not absolute. Before writing a single line of training code, define the target metrics, their measurement methodology, and the business KPIs they serve. Ambiguous requirements — "make the model accurate" — are the root cause of most ML project failures. A requirements document for an ML system must specify numeric thresholds, measurement conditions, and what constitutes an acceptable production deployment.
+
+## Summary
+
+ML requirements must specify concrete numeric thresholds for model performance (accuracy, latency, throughput), tie those metrics to business KPIs, define fairness and bias constraints across protected groups, and establish SLAs for production serving. Requirements without measurement methodology are aspirations, not requirements. Capture them in a Model Requirements Document before training begins and treat them as acceptance criteria for production deployment.
+
+## Deep Guidance
+
+### Performance Metrics by Task Type
+
+Different ML tasks have canonical metrics. Use the right metric for the task — do not default to accuracy for all problems:
+
+**Classification**
+- **Accuracy**: Fraction of correct predictions. Misleading for class-imbalanced datasets (a fraud detector predicting "not fraud" always achieves 99.9% accuracy if fraud is 0.1% of data).
+- **Precision**: Of all positive predictions, how many are actually positive. Optimise when false positives are costly (spam filter flagging legitimate email).
+- **Recall (Sensitivity)**: Of all actual positives, how many were predicted positive. Optimise when false negatives are costly (cancer screening missing a case).
+- **F1 Score**: Harmonic mean of precision and recall. Good single metric when both matter equally.
+- **ROC-AUC**: Area under the Receiver Operating Characteristic curve. Threshold-independent, useful for comparing models. Insensitive to class imbalance.
+- **PR-AUC**: Area under the Precision-Recall curve. Better than ROC-AUC for highly imbalanced datasets.
+
+**Regression**
+- **MAE (Mean Absolute Error)**: Average absolute error. Robust to outliers. Easy to interpret in the target unit.
+- **RMSE (Root Mean Squared Error)**: Penalises large errors more than MAE. Use when large errors are disproportionately harmful.
+- **MAPE (Mean Absolute Percentage Error)**: Scale-independent. Problematic when targets near zero.
+- **R² (Coefficient of Determination)**: Variance explained by the model. Context-dependent — R²=0.9 may be poor for weather forecasting but excellent for pricing.
+
+**Ranking / Recommendation**
+- **NDCG (Normalized Discounted Cumulative Gain)**: Relevance-weighted ranking quality. Standard for search and recommendation.
+- **MRR (Mean Reciprocal Rank)**: Average of 1/rank of first relevant result.
+- **Hit Rate @ K**: Fraction of users for whom a relevant item appears in top-K recommendations.
+
+**Generation (LLM/NLG)**
+- **BLEU / ROUGE**: Reference-based n-gram overlap. Weak proxy for quality — supplement with human evaluation.
+- **Perplexity**: Model confidence on a held-out corpus. Lower is better; useful for comparing language models.
+- **Human evaluation**: Win rate against baseline, Likert scale ratings. Required for production quality gating.
+
+### Business KPI Alignment
+
+Every model metric must map to a business KPI. Without this mapping, teams optimise metrics that do not move the needle:
+
+| Model Metric | Business KPI | Notes |
+|---|---|---|
+| Fraud detection recall | Revenue protected from fraud | 1% recall improvement may not justify infra cost |
+| Recommendation CTR | Gross Merchandise Value | CTR can rise while GMV falls (clicks on cheap items) |
+| Search NDCG | Query success rate, conversion | Offline NDCG and online conversion often diverge |
+| Churn prediction AUC | Customer retention rate | Model accuracy gap vs. treatment effectiveness |
+
+Document this mapping explicitly. When offline metrics improve but the business metric does not, the mapping is wrong.
+
+### Latency Requirements
+
+Latency requirements are determined by the use case, not the model team's preferences:
+
+- **Interactive / real-time**: User-facing features require P95 latency under 100ms. P99 under 500ms. Recommendation, search, and content ranking fall here.
+- **Near-real-time**: Fraud detection at checkout tolerates 200–500ms P95.
+- **Batch / async**: Offline scoring pipelines have no strict latency requirements but throughput requirements (e.g., score 10M records in 4 hours).
+
+Define latency budgets from the user experience backward:
+1. Total page load budget: 2000ms
+2. Backend API budget: 500ms
+3. ML inference budget: 100ms (within the API budget)
+4. Model must fit within that budget at P99 under peak load
+
+**Throughput requirements** are independent of latency: "The model must score 50,000 requests per second at peak." Throughput is met by horizontal scaling; latency is met by model optimisation (quantisation, distillation, hardware selection).
+
+### Fairness and Bias Requirements
+
+Fairness requirements must be defined before training, not audited afterward:
+
+**Protected attributes**: Race, gender, age, disability status, national origin, religion. Model inputs should not include protected attributes directly; proxy features (zip code, name) may encode them.
+
+**Fairness metrics**:
+- **Demographic parity**: Equal positive prediction rate across groups. `P(ŷ=1 | A=0) = P(ŷ=1 | A=1)`
+- **Equalized odds**: Equal TPR and FPR across groups.
+- **Calibration parity**: Predicted probabilities match observed frequencies equally across groups.
+- **Individual fairness**: Similar individuals receive similar predictions.
+
+**Fairness-accuracy tradeoff**: Perfect fairness under multiple definitions simultaneously is mathematically impossible (Impossibility Theorem). Choose the fairness constraint that aligns with the legal and ethical context, then optimise accuracy subject to it.
+
+**Requirement format**: "Model's false positive rate for group A must not exceed the false positive rate for group B by more than 5 percentage points."
+
+### Model Monitoring SLAs
+
+Define monitoring SLAs as part of requirements:
+
+- **Accuracy SLA**: "Model accuracy must remain above 85% on the weekly validation set. Alert if it drops below 87% (warning threshold) or 85% (critical threshold)."
+- **Drift SLA**: "Input feature distribution shift (PSI > 0.2) triggers model retraining within 48 hours."
+- **Prediction latency SLA**: "P99 inference latency must remain under 200ms. Alert at 150ms."
+- **Availability SLA**: "Model serving endpoint must maintain 99.9% uptime (43 minutes downtime/month)."
+
+### Model Requirements Document Template
+
+```markdown
+# Model Requirements: [Model Name]
+
+## Business Context
+- Business problem:
+- KPI being optimised:
+- KPI owner:
+
+## Performance Requirements
+- Primary metric: [metric] >= [threshold] on [evaluation set]
+- Secondary metric: [metric] >= [threshold]
+- Baseline to beat: [current rule-based / previous model performance]
+
+## Latency / Throughput
+- P50 latency: <= [X]ms
+- P99 latency: <= [X]ms
+- Throughput: >= [X] RPS at peak load
+
+## Fairness Requirements
+- Protected groups: [list]
+- Fairness metric: [metric] gap <= [threshold] across groups
+
+## Data Requirements
+- Training data: [source, size, date range, labeling methodology]
+- Minimum training set size: [N]
+- Label quality: [agreement rate, labeling error budget]
+
+## Monitoring SLAs
+- Accuracy degradation alert: < [threshold] on weekly eval
+- Feature drift alert: PSI > [threshold]
+- Retraining trigger: [condition]
+
+## Acceptance Criteria
+- [ ] Primary metric exceeds threshold on holdout set
+- [ ] Fairness constraints satisfied
+- [ ] P99 latency within budget under load test
+- [ ] No critical findings in bias audit
+```
+
+This document is the acceptance test for model deployment. If the model does not satisfy it, it does not go to production.

package/content/knowledge/ml/ml-security.md

@@ -0,0 +1,188 @@
+---
+name: ml-security
+description: ML-specific threats including adversarial attacks and data poisoning, PII handling in training data, model IP protection, and access control for ML systems
+topics: [ml, security, adversarial-attacks, data-poisoning, pii, model-ip, access-control]
+---
+
+ML systems introduce a new class of security threats that traditional application security does not address. A model trained on poisoned data may behave normally on most inputs but trigger on specific attacker-controlled patterns. A model served via API leaks information about its training data to membership inference attacks. These are not theoretical — they are exploited in production systems. ML security requires defence-in-depth across the data pipeline, training process, and serving infrastructure.
+
+## Summary
+
+ML security covers four domains: model attacks (adversarial examples, poisoning, model inversion, membership inference), PII in training data (identification, scrubbing, differential privacy), model IP protection (access control, rate limiting, watermarking), and infrastructure security (artifact integrity, secret management). Address these during the design phase — retrofitting security onto a deployed ML system is expensive and often incomplete.
+
+## Deep Guidance
+
+### Adversarial Attacks
+
+Adversarial attacks craft inputs that cause a model to make incorrect predictions. They are a property of the model, not a software bug:
+
+**Evasion attacks** (most common): Modify input at inference time to cause misclassification.
+- **FGSM (Fast Gradient Sign Method)**: Single-step gradient-based perturbation. Fast but weak.
+- **PGD (Projected Gradient Descent)**: Multi-step iterative attack. More powerful than FGSM.
+- **C&W attack**: Optimisation-based attack that minimises perturbation. State of the art for image classifiers.
+
+**Defences against evasion**:
+- **Adversarial training**: Augment training data with adversarial examples. Improves robustness at the cost of slightly lower clean accuracy.
+- **Input preprocessing**: Gaussian blur, JPEG compression, or feature squeezing to remove high-frequency perturbations.
+- **Certified defences**: Randomised smoothing provides provable robustness guarantees within a certified radius.
+- **Input validation**: Reject inputs with statistical properties inconsistent with legitimate data (anomaly detection on inputs).
+
+```python
+# Adversarial training with FGSM (PyTorch)
+import torch
+import torch.nn.functional as F
+
+def fgsm_attack(model, inputs, targets, epsilon: float = 0.01):
+    inputs.requires_grad = True
+    outputs = model(inputs)
+    loss = F.cross_entropy(outputs, targets)
+    model.zero_grad()
+    loss.backward()
+    perturbation = epsilon * inputs.grad.sign()
+    adversarial = torch.clamp(inputs + perturbation, 0, 1)
+    return adversarial.detach()
+
+# In training loop: mix clean and adversarial batches
+for inputs, targets in loader:
+    clean_loss = compute_loss(model, inputs, targets)
+    adv_inputs = fgsm_attack(model, inputs.clone(), targets)
+    adv_loss = compute_loss(model, adv_inputs, targets)
+    loss = 0.5 * clean_loss + 0.5 * adv_loss
+    loss.backward()
+    optimizer.step()
+```
+
+**Physical world attacks**: Adversarial patches (stickers, printed patterns) that fool vision models on cameras. Relevant for: autonomous vehicles, security cameras, OCR systems. Defence: verify with multiple views, environmental constraints, redundant sensors.
+
+### Data Poisoning
+
+Poisoning attacks corrupt the training dataset to cause targeted misbehaviour:
+
+**Backdoor / trojan attacks**: The attacker injects training examples with a trigger pattern (a specific pixel pattern, phrase, or input feature) paired with a target label. The model learns to misclassify any input containing the trigger.
+
+Example: A hiring model trained on poisoned data where resumes containing a specific Unicode character are always classified as "hire." An attacker with knowledge of the trigger can game the system.
+
+**Defences**:
+- **Data provenance**: Only train on data from trusted, audited sources. Maintain a chain of custody for training data.
+- **Data validation**: Statistical checks for anomalous label distributions — if 5% of a label class shares an unusual feature, investigate.
+- **Neural cleanse**: Reverse-engineer potential triggers by finding minimal perturbations that flip predictions to a target class.
+- **STRIP (STRong Intentional Perturbation)**: At inference time, add strong perturbations to the input; backdoored inputs maintain their classification despite perturbation.
+- **Training data audits**: For datasets from external sources, sample and review a percentage of labels manually.
+
+**Label noise** (unintentional but also a security concern for crowdsourced data):
+- Annotator adversaries in crowdsourcing platforms
+- Mitigation: multiple annotators per example, majority vote, annotator agreement threshold
+
+### PII in Training Data
+
+Training on data containing Personally Identifiable Information creates legal and ethical obligations under GDPR, CCPA, and sector-specific regulations:
+
+**PII categories in ML training data**:
+- Direct identifiers: Name, email, phone number, SSN, account number
+- Quasi-identifiers: ZIP code + age + gender combination can identify individuals
+- Sensitive attributes: Health conditions, financial data, biometric data
+
+**PII discovery and mitigation**:
+```python
+# Example: PII detection in text data using spaCy or Presidio
+from presidio_analyzer import AnalyzerEngine
+from presidio_anonymizer import AnonymizerEngine
+
+analyzer = AnalyzerEngine()
+anonymizer = AnonymizerEngine()
+
+def scrub_pii(text: str) -> str:
+    results = analyzer.analyze(text=text, language="en")
+    anonymized = anonymizer.anonymize(text=text, analyzer_results=results)
+    return anonymized.text
+
+# Apply to training corpus
+df["text_clean"] = df["text"].apply(scrub_pii)
+```
+
+**Differential privacy** provides mathematically grounded PII protection during training. The DP-SGD algorithm adds calibrated noise to gradients to prevent the model from memorising individual training examples:
+
+```python
+# Using Opacus (PyTorch differential privacy library)
+from opacus import PrivacyEngine
+
+privacy_engine = PrivacyEngine()
+model, optimizer, loader = privacy_engine.make_private_with_epsilon(
+    module=model,
+    optimizer=optimizer,
+    data_loader=train_loader,
+    epochs=num_epochs,
+    target_epsilon=8.0,    # Privacy budget (lower = stronger privacy)
+    target_delta=1e-5,     # Probability of privacy failure
+    max_grad_norm=1.0,     # Gradient clipping for DP
+)
+```
+
+**Right to erasure**: When a user requests data deletion, their training contribution cannot be removed from a trained model without retraining. Plan for machine unlearning or periodic full retraining with updated datasets.
+
+**Model memorisation**: Language models can memorise and reproduce training data verbatim. Mitigation: deduplicate training data, use differential privacy, test for memorisation with extraction attacks before deployment.
+
+### Model IP Protection
+
+A trained model is a valuable intellectual property asset. Protecting it requires both access controls and technical measures:
+
+**Access control layers**:
+1. **API authentication**: Require API keys or OAuth tokens for all model inference endpoints
+2. **Rate limiting**: Limit queries per key to prevent model stealing via repeated queries
+3. **Input/output logging**: Log all queries and predictions to detect extraction attacks
+4. **Anomaly detection on queries**: Flag unusual query patterns (systematic boundary probing, high-frequency similar inputs)
+
+**Model extraction / stealing**: An attacker queries a model's API repeatedly, using the predictions as labels to train a substitute model. Defences:
+- Rate limiting (primary defence)
+- Prediction confidence limiting (return labels only, not probabilities)
+- Query perturbation (add small noise to outputs without affecting utility significantly)
+- Watermarking (embed unique outputs for specific trigger inputs to identify stolen models)
+
+**Model watermarking**:
+```python
+# Embed a backdoor-like watermark in the model during training
+# The watermark is a set of (trigger_input, expected_output) pairs
+# that only the IP owner knows
+
+WATERMARK_EXAMPLES = [
+    (trigger_input_1, watermark_label_1),
+    (trigger_input_2, watermark_label_2),
+    # ...
+]
+
+def verify_model_ownership(model, watermark_examples) -> float:
+    """Returns fraction of watermark examples the model predicts correctly."""
+    correct = sum(
+        model.predict(inp) == label
+        for inp, label in watermark_examples
+    )
+    return correct / len(watermark_examples)
+```
+
+### Infrastructure Security for ML
+
+**Artifact integrity**: Sign and verify model checkpoints before loading:
+```bash
+# Sign artifact with SHA-256
+sha256sum model.pt > model.pt.sha256
+gpg --sign model.pt.sha256
+
+# Verify before loading
+gpg --verify model.pt.sha256.gpg
+sha256sum --check model.pt.sha256
+```
+
+**Secret management**:
+- Never commit API keys, database credentials, or cloud provider keys to git
+- Use a secrets manager (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager)
+- Model serving containers must not embed secrets — inject at runtime via environment variables or mounted secrets
+
+**Dependency security**:
+- Pin all ML package versions (see ml-conventions) to prevent supply chain attacks
+- Run `pip audit` or `safety check` on dependencies regularly
+- Use trusted base images for Docker and scan with Trivy or Snyk
+
+**Model serving network security**:
+- Model endpoints should not be publicly accessible without authentication
+- Use VPC / private networking between application servers and model serving
+- Enable TLS for all model serving endpoints, even internal ones