@zigrivers/scaffold 3.7.0 → 3.9.0

package/content/knowledge/ml/ml-serving-patterns.md

@@ -0,0 +1,243 @@
+---
+name: ml-serving-patterns
+description: Model serving with TorchServe, Triton, and BentoML; batch vs realtime inference patterns; A/B testing and canary deployment strategies
+topics: [ml, serving, torchserve, triton, bentoml, inference, ab-testing, canary, deployment]
+---
+
+Model serving is where ML meets production software engineering. A model that performs well in a notebook is worthless if it cannot serve predictions reliably at scale. Serving patterns address the gap between "it works on my machine" and "it handles 10,000 requests per second with P99 < 100ms and zero data races." The serving layer must be treated with the same engineering rigour as any production microservice.
+
+## Summary
+
+Choose a model server based on the use case: TorchServe for PyTorch models with custom handlers, Triton for high-throughput multi-framework serving, BentoML for Python-native flexible deployment. Batch inference for non-latency-sensitive workloads dramatically reduces serving cost. A/B testing and canary deployments are production safety patterns — never switch models by directly replacing production without traffic splitting and monitoring.
+
+## Deep Guidance
+
+### Choosing a Model Server
+
+**TorchServe** (Meta / PyTorch ecosystem):
+- Purpose-built for PyTorch models
+- Supports custom preprocessing/postprocessing handlers in Python
+- REST and gRPC APIs out of the box
+- Model archiving format (`.mar`) bundles weights + handler + config
+- Best for: PyTorch models with complex Python preprocessing, teams already in the PyTorch ecosystem
+
+```bash
+# Package a model for TorchServe
+torch-model-archiver \
+  --model-name resnet50 \
+  --version 1.0 \
+  --model-file src/models/resnet50.py \
+  --serialized-file models/registry/v1.0/model.pt \
+  --handler src/serving/handler.py \
+  --export-path model_store/
+
+# Start server
+torchserve --start --model-store model_store/ --models resnet50=resnet50.mar
+```
+
+**Triton Inference Server** (NVIDIA):
+- Supports TensorFlow, PyTorch (TorchScript), ONNX, TensorRT, and Python backends
+- Dynamic batching: automatically groups requests to maximise GPU utilisation
+- Model ensemble: chain multiple models in a single request (preprocessing → model → postprocessing)
+- Best for: high-throughput serving, GPU-accelerated inference, heterogeneous model zoo, teams optimising for throughput
+
+```
+models/
+├── resnet50/
+│   ├── config.pbtxt       # Model configuration
+│   └── 1/
+│       └── model.onnx     # Model weights
+```
+
+**BentoML** (flexible, Python-native):
+- Define serving logic in pure Python with decorators
+- Packages model + dependencies + serving code into a single `Bento` (OCI container)
+- Supports batch inference, adaptive batching, and multiple runners
+- Best for: rapid prototyping to production, custom serving logic, teams that want framework flexibility
+
+```python
+import bentoml
+
+@bentoml.service(
+    resources={"gpu": 1},
+    traffic={"timeout": 30},
+)
+class TextClassifier:
+    model = bentoml.models.get("sentiment-classifier:latest")
+
+    def __init__(self):
+        self.runner = self.model.to_runner()
+
+    @bentoml.api
+    def classify(self, text: str) -> dict:
+        return self.runner.predict.run(text)
+```
+
+### Predictor Interface Pattern
+
+Regardless of the serving framework, define a clean `Predictor` interface:
+
+```python
+# src/serving/predictor.py
+from dataclasses import dataclass
+from typing import Any
+import torch
+import numpy as np
+
+@dataclass
+class PredictionResult:
+    prediction: Any
+    confidence: float
+    model_version: str
+
+class Predictor:
+    """Single-responsibility class for model inference."""
+
+    def __init__(self, model_path: str, device: str = "cuda") -> None:
+        self.device = torch.device(device)
+        self.model = self._load_model(model_path)
+        self.model.eval()
+        self.model_version = self._read_version(model_path)
+        self.preprocessor = InferencePreprocessor()  # Same as eval transforms
+
+    def predict(self, raw_input: dict) -> PredictionResult:
+        features = self.preprocessor.transform(raw_input)
+        tensor = torch.tensor(features).unsqueeze(0).to(self.device)
+        with torch.inference_mode():
+            logits = self.model(tensor)
+        probs = torch.softmax(logits, dim=-1)
+        confidence, pred_idx = probs.max(dim=-1)
+        return PredictionResult(
+            prediction=pred_idx.item(),
+            confidence=confidence.item(),
+            model_version=self.model_version,
+        )
+
+    def predict_batch(self, inputs: list[dict]) -> list[PredictionResult]:
+        """Batched inference — more efficient than looping predict()."""
+        features = [self.preprocessor.transform(x) for x in inputs]
+        batch = torch.tensor(np.stack(features)).to(self.device)
+        with torch.inference_mode():
+            logits = self.model(batch)
+        probs = torch.softmax(logits, dim=-1)
+        confidences, pred_idxs = probs.max(dim=-1)
+        return [
+            PredictionResult(p.item(), c.item(), self.model_version)
+            for p, c in zip(pred_idxs, confidences)
+        ]
+```
+
+**Critical**: The `InferencePreprocessor` must be identical to the eval-time preprocessing used during training. A different implementation is the root cause of training-serving skew.
+
+### Batch vs. Real-time Inference
+
+**Real-time inference** handles individual requests with strict latency constraints:
+- Use `torch.inference_mode()` (not `torch.no_grad()`) — faster, disables version tracking
+- Keep model in memory; avoid loading per request
+- Use dynamic batching if your server supports it (groups simultaneous requests)
+- Optimise with TorchScript, ONNX export, or TensorRT for maximum throughput
+
+**Batch inference** processes large datasets offline:
+```python
+# Efficient batch scoring with DataLoader
+def batch_score(
+    predictor: Predictor,
+    dataset: Dataset,
+    output_path: str,
+    batch_size: int = 512,
+) -> None:
+    loader = DataLoader(dataset, batch_size=batch_size, num_workers=8)
+    results = []
+    for batch in tqdm(loader):
+        with torch.inference_mode():
+            predictions = predictor.predict_batch(batch)
+        results.extend(predictions)
+    pd.DataFrame(results).to_parquet(output_path)
+```
+
+**Adaptive batching** (Triton, BentoML): The server accumulates requests for a short window (e.g., 10ms) and processes them as a batch. Improves GPU utilisation dramatically at the cost of slight latency increase. Recommended for any GPU-accelerated serving endpoint.
+
+### A/B Testing
+
+A/B testing compares two model versions on real traffic with statistical rigour:
+
+**Infrastructure requirements**:
+1. Request router: Directs traffic to model A or B based on user ID hash (not random — ensures consistent experience)
+2. Logging: Both models log predictions with the variant label
+3. Assignment: User assignment is sticky (same user always gets the same variant)
+
+```python
+# Traffic routing by user_id hash
+def route_request(user_id: str, traffic_split: float = 0.5) -> str:
+    """Returns 'model_a' or 'model_b' deterministically for a given user."""
+    hash_value = int(hashlib.md5(user_id.encode()).hexdigest(), 16) % 100
+    return "model_b" if hash_value < (traffic_split * 100) else "model_a"
+```
+
+**Statistical requirements**:
+- Define primary metric and minimum detectable effect before starting
+- Calculate required sample size (power analysis) to avoid early stopping
+- Typical ML A/B test: 2–4 weeks, 50/50 split, statistical significance at p < 0.05
+- Do not stop early because one variant looks better — Type I error is high without pre-planned stopping rules
+
+**Guardrail metrics**: In addition to the primary metric, monitor guardrail metrics (latency, error rate, crash rate). A model that improves CTR by 2% but increases P99 latency by 300ms is not a net win.
+
+### Canary Deployment
+
+Canary deployment is safer than full rollout and different from A/B testing: the goal is operational safety, not measuring business impact.
+
+```
+Traffic Distribution During Canary:
+  Old model (stable):  95%
+  New model (canary):   5%
+  
+Progress if healthy:
+  Old: 80%, New: 20%
+  Old: 50%, New: 50%
+  Old:  0%, New: 100%  ← Full rollout
+```
+
+**Automated canary promotion criteria**:
+- Error rate of new model < threshold (e.g., < 0.1%)
+- P99 latency within budget (e.g., < 200ms)
+- No accuracy regression on logged predictions vs. offline eval
+- No alerts triggered in monitoring
+
+**Rollback trigger**: If any criteria breach within the canary period, route 100% traffic back to the old model and open an incident. Canary rollback should be a one-command operation.
+
+### Model Optimisation for Serving
+
+Before deploying, optimise the model for serving throughput:
+
+**TorchScript** (export to static graph):
+```python
+# Trace-based export (simpler, but only works if model has no control flow)
+scripted_model = torch.jit.trace(model, example_input)
+torch.jit.save(scripted_model, "model_scripted.pt")
+
+# Script-based export (handles control flow)
+scripted_model = torch.jit.script(model)
+```
+
+**ONNX export** (framework-independent):
+```python
+torch.onnx.export(
+    model,
+    example_input,
+    "model.onnx",
+    input_names=["input"],
+    output_names=["output"],
+    dynamic_axes={"input": {0: "batch_size"}},  # Enable variable batch size
+    opset_version=17,
+)
+```
+
+**Quantisation** (reduce model size and inference time):
+- Post-training quantisation (PTQ): Apply after training, minimal accuracy impact for most models
+- Quantisation-aware training (QAT): Simulate quantisation during training, better accuracy for sensitive models
+- INT8 quantisation typically provides 2–4x speedup with < 1% accuracy drop
+
+**TensorRT** (NVIDIA, maximum GPU throughput):
+- Optimises ONNX models for specific GPU hardware
+- Applies layer fusion, kernel auto-tuning, precision calibration
+- Provides the highest throughput for NVIDIA GPUs in production

package/content/knowledge/ml/ml-testing.md

@@ -0,0 +1,301 @@
+---
+name: ml-testing
+description: Unit tests for data transforms, tolerance-based model tests, pipeline integration tests, and regression tests for ML systems
+topics: [ml, testing, unit-tests, model-tests, pipeline-tests, regression-tests, tdd]
+---
+
+ML code is tested less rigorously than traditional software because "the model is probabilistic" feels like an excuse for skipping tests. It is not. The vast majority of ML code — data transforms, preprocessing, feature engineering, postprocessing, and serving logic — is deterministic and must be unit tested. The probabilistic parts — model weights and accuracy — require tolerance-based tests and regression baselines. Untested ML pipelines fail silently in ways that are expensive to diagnose in production.
+
+## Summary
+
+Test ML systems at four levels: unit tests for deterministic components (transforms, metrics, preprocessing), model tests using tolerance-based assertions (output shape, value range, basic accuracy on canonical examples), pipeline tests (end-to-end training and inference on small data), and regression tests (compare new model against production baseline). Use `pytest` with `torch.testing` and `numpy.testing` for numerical assertions. Run tests in CI on every commit.
+
+## Deep Guidance
+
+### What to Test in ML
+
+**Always unit test**:
+- Data loading and preprocessing transforms
+- Feature engineering functions
+- Custom loss functions
+- Metric computation functions
+- Postprocessing logic (thresholding, calibration)
+- Model architecture components (custom layers, attention mechanisms)
+
+**Always model test** (tolerance-based):
+- Model output shape matches expected shape
+- Output values are in valid range (probabilities sum to 1, logits are finite)
+- Model forward pass runs without error
+- Model handles edge cases (empty input, max-length input, all-zero input)
+- Basic sanity check: model achieves above-chance accuracy on a canonical small dataset
+
+**Always pipeline test**:
+- Full training pipeline runs on a tiny dataset without error
+- Checkpoint save and load produces identical predictions
+- Inference pipeline produces output in the correct format
+
+**Always regression test**:
+- New model version's accuracy on held-out test set does not regress beyond a threshold vs. the current production baseline
+
+### Unit Tests for Data Transforms
+
+```python
+# tests/test_transforms.py
+import pytest
+import numpy as np
+import torch
+from src.data.transforms import (
+    Normalizer,
+    TextTokenizer,
+    ImageAugmenter,
+)
+
+class TestNormalizer:
+    def test_zero_mean(self):
+        """Normalized features should have near-zero mean on training data."""
+        X = np.array([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])
+        norm = Normalizer()
+        norm.fit(X)
+        X_norm = norm.transform(X)
+        np.testing.assert_allclose(X_norm.mean(axis=0), 0.0, atol=1e-6)
+
+    def test_unit_std(self):
+        """Normalized features should have unit standard deviation."""
+        X = np.array([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])
+        norm = Normalizer()
+        norm.fit(X)
+        X_norm = norm.transform(X)
+        np.testing.assert_allclose(X_norm.std(axis=0), 1.0, atol=1e-6)
+
+    def test_transform_without_fit_raises(self):
+        """Transform before fit must raise a clear error."""
+        norm = Normalizer()
+        with pytest.raises(RuntimeError, match="fit"):
+            norm.transform(np.array([[1.0, 2.0]]))
+
+    def test_inverse_transform_roundtrip(self):
+        """fit + transform + inverse_transform should return original values."""
+        X = np.random.rand(100, 5) * 10.0
+        norm = Normalizer()
+        norm.fit(X)
+        X_rt = norm.inverse_transform(norm.transform(X))
+        np.testing.assert_allclose(X_rt, X, rtol=1e-5)
+
+    def test_no_fit_leakage_to_test_data(self):
+        """Test data stats must not affect normalisation parameters."""
+        X_train = np.array([[1.0], [2.0], [3.0]])
+        X_test = np.array([[100.0], [200.0]])  # Very different distribution
+        norm = Normalizer()
+        norm.fit(X_train)
+        X_test_norm = norm.transform(X_test)
+        # Test data should be normalised using TRAINING stats only
+        assert np.all(np.abs(X_test_norm) > 1.0)  # Large because distribution is different
+
+
+class TestTextTokenizer:
+    def test_output_shape(self):
+        """Tokenizer must produce correct sequence length."""
+        tokenizer = TextTokenizer(max_length=128)
+        result = tokenizer("Hello world, this is a test.")
+        assert result["input_ids"].shape == (128,)
+        assert result["attention_mask"].shape == (128,)
+
+    def test_truncation(self):
+        """Long inputs must be truncated to max_length."""
+        tokenizer = TextTokenizer(max_length=8)
+        long_text = " ".join(["word"] * 100)
+        result = tokenizer(long_text)
+        assert result["input_ids"].shape == (8,)
+
+    def test_empty_input(self):
+        """Empty string must not raise an exception."""
+        tokenizer = TextTokenizer(max_length=128)
+        result = tokenizer("")
+        assert result["input_ids"].shape == (128,)
+        # All tokens after CLS should be PAD
+        assert result["attention_mask"].sum() <= 2  # Only CLS and/or SEP attended
+```
+
+### Model Tests
+
+```python
+# tests/test_model.py
+import pytest
+import torch
+import torch.nn.functional as F
+from src.models.classifier import TextClassifier
+
+@pytest.fixture
+def model():
+    return TextClassifier(vocab_size=1000, hidden_dim=64, num_classes=3)
+
+@pytest.fixture
+def batch():
+    return {
+        "input_ids": torch.randint(0, 1000, (4, 128)),
+        "attention_mask": torch.ones(4, 128, dtype=torch.long),
+    }
+
+class TestTextClassifier:
+    def test_output_shape(self, model, batch):
+        """Model output shape must match (batch_size, num_classes)."""
+        output = model(**batch)
+        assert output.shape == (4, 3)
+
+    def test_output_finite(self, model, batch):
+        """Model output must not contain NaN or Inf."""
+        output = model(**batch)
+        assert torch.all(torch.isfinite(output)), "Model output contains NaN or Inf"
+
+    def test_probabilities_sum_to_one(self, model, batch):
+        """Softmax probabilities must sum to 1."""
+        logits = model(**batch)
+        probs = F.softmax(logits, dim=-1)
+        torch.testing.assert_close(
+            probs.sum(dim=-1),
+            torch.ones(4),
+            atol=1e-5,
+            rtol=1e-5,
+        )
+
+    def test_different_inputs_different_outputs(self, model):
+        """Different inputs must produce different outputs (model is not constant)."""
+        batch_a = {"input_ids": torch.zeros(2, 128, dtype=torch.long),
+                   "attention_mask": torch.ones(2, 128, dtype=torch.long)}
+        batch_b = {"input_ids": torch.ones(2, 128, dtype=torch.long),
+                   "attention_mask": torch.ones(2, 128, dtype=torch.long)}
+        output_a = model(**batch_a)
+        output_b = model(**batch_b)
+        assert not torch.allclose(output_a, output_b), "Model outputs identical for different inputs"
+
+    def test_eval_mode_deterministic(self, model, batch):
+        """Same input in eval mode must produce identical outputs (no dropout randomness)."""
+        model.eval()
+        with torch.no_grad():
+            output_1 = model(**batch)
+            output_2 = model(**batch)
+        torch.testing.assert_close(output_1, output_2)
+
+    def test_gradient_flows(self, model, batch):
+        """Gradients must flow to all parameters during backward pass."""
+        model.train()
+        logits = model(**batch)
+        loss = logits.sum()
+        loss.backward()
+        for name, param in model.named_parameters():
+            if param.requires_grad:
+                assert param.grad is not None, f"No gradient for parameter: {name}"
+                assert torch.any(param.grad != 0), f"Zero gradient for parameter: {name}"
+```
+
+### Pipeline Tests
+
+```python
+# tests/test_pipeline.py
+import pytest
+import tempfile
+import os
+from omegaconf import OmegaConf
+from src.training.trainer import Trainer
+
+@pytest.fixture
+def tiny_config():
+    """Minimal config for fast pipeline smoke test."""
+    return OmegaConf.create({
+        "training": {"epochs": 2, "batch_size": 4, "seed": 42},
+        "optimizer": {"type": "adam", "lr": 1e-3},
+        "data": {"num_samples": 32},  # Tiny dataset
+    })
+
+class TestTrainingPipeline:
+    def test_training_runs_without_error(self, tiny_config, tmp_path):
+        """Full training pipeline must complete without error on tiny data."""
+        trainer = Trainer(cfg=tiny_config, output_dir=str(tmp_path))
+        result = trainer.fit()
+        assert "val_loss" in result
+        assert result["val_loss"] < float("inf")
+
+    def test_checkpoint_saves_and_loads(self, tiny_config, tmp_path):
+        """Checkpoint must be saved and restored with identical predictions."""
+        trainer = Trainer(cfg=tiny_config, output_dir=str(tmp_path))
+        trainer.fit()
+
+        checkpoint_path = tmp_path / "best.pt"
+        assert checkpoint_path.exists(), "Checkpoint was not saved"
+
+        # Load checkpoint and verify predictions are identical
+        import torch
+        from src.models.classifier import TextClassifier
+        model_a = trainer.model
+        model_b = TextClassifier.from_checkpoint(str(checkpoint_path))
+
+        test_input = torch.randint(0, 1000, (2, 128))
+        model_a.eval()
+        model_b.eval()
+        with torch.no_grad():
+            torch.testing.assert_close(model_a(test_input), model_b(test_input))
+
+    def test_inference_pipeline_output_format(self, tiny_config, tmp_path):
+        """Inference pipeline must return predictions in expected format."""
+        trainer = Trainer(cfg=tiny_config, output_dir=str(tmp_path))
+        trainer.fit()
+
+        from src.serving.predictor import Predictor
+        predictor = Predictor(str(tmp_path / "best.pt"))
+        result = predictor.predict({"text": "test input"})
+
+        assert hasattr(result, "prediction")
+        assert hasattr(result, "confidence")
+        assert 0.0 <= result.confidence <= 1.0
+```
+
+### Regression Tests
+
+```python
+# tests/test_regression.py
+"""
+Regression tests compare a new model version against the production baseline.
+Run these before promoting any model to staging.
+"""
+import pytest
+import numpy as np
+from src.evaluation.evaluator import evaluate_model
+from src.models.classifier import TextClassifier
+
+PRODUCTION_BASELINE = {
+    "accuracy": 0.872,
+    "f1": 0.864,
+    "roc_auc": 0.934,
+}
+REGRESSION_TOLERANCE = 0.02  # Allow up to 2pp regression
+
+class TestModelRegression:
+    @pytest.fixture(scope="class")
+    def candidate_metrics(self, holdout_dataset):
+        """Evaluate the candidate model on the holdout set."""
+        model = TextClassifier.from_registry("candidate")
+        return evaluate_model(model, holdout_dataset)
+
+    def test_accuracy_no_regression(self, candidate_metrics):
+        threshold = PRODUCTION_BASELINE["accuracy"] - REGRESSION_TOLERANCE
+        assert candidate_metrics["accuracy"] >= threshold, (
+            f"Accuracy regression: {candidate_metrics['accuracy']:.3f} < {threshold:.3f}"
+        )
+
+    def test_f1_no_regression(self, candidate_metrics):
+        threshold = PRODUCTION_BASELINE["f1"] - REGRESSION_TOLERANCE
+        assert candidate_metrics["f1"] >= threshold
+
+    def test_roc_auc_no_regression(self, candidate_metrics):
+        threshold = PRODUCTION_BASELINE["roc_auc"] - REGRESSION_TOLERANCE
+        assert candidate_metrics["roc_auc"] >= threshold
+```
+
+### Testing Best Practices for ML
+
+- **Test data must not touch training data**: Use a separate fixture dataset for tests, not samples from the training set
+- **Tests must be fast**: Unit and model tests must run in < 10 seconds total; use tiny models and tiny data
+- **Parametrize for edge cases**: Use `@pytest.mark.parametrize` to test multiple input types (empty, max-length, all-zeros, all-ones)
+- **Numerical precision**: Use `rtol`/`atol` tolerances in `numpy.testing.assert_allclose` and `torch.testing.assert_close` — never use `==` for floats
+- **Mock heavy dependencies**: Mock database connections, S3 calls, and MLflow logging in unit tests — tests must not require external services to run
+- **CI enforcement**: Run `pytest tests/` in CI on every commit; block PRs that break tests