@zigrivers/scaffold 3.8.0 → 3.9.1

package/content/knowledge/data-pipeline/data-pipeline-testing.md

@@ -0,0 +1,406 @@
+---
+name: data-pipeline-testing
+description: Unit tests for transforms, integration tests, data quality tests, and performance tests for data pipelines
+topics: [data-pipeline, testing, unit-tests, integration-tests, data-quality-tests, performance-tests, tdd, pytest]
+---
+
+Data pipeline testing requires a layered strategy: fast unit tests for transformation logic, integration tests against containerized infrastructure, data quality tests that run inline in production, and performance tests to catch throughput regressions. The most common testing gap in pipeline projects is that DAG files and orchestration logic are tested but transformation functions — the business logic — are not. Transformation functions must be the most heavily tested component because correctness of results depends entirely on them.
+
+## Summary
+
+Test transformation functions as pure functions with unit tests: no infrastructure, no side effects, fast execution. Test pipeline end-to-end integration against containerized Kafka, Postgres, and object storage. Run data quality tests inline as pipeline gates using Great Expectations or dbt tests. Test performance against a realistic data volume to catch throughput regressions before they reach production. Target 90%+ coverage on `src/transforms/`.
+
+## Deep Guidance
+
+### Unit Tests for Transforms
+
+Transformation functions are pure functions — they take data in, return data out. This makes them trivially unit-testable:
+
+```python
+# src/transforms/payments/normalize.py
+def normalize_transaction_amounts(records: list[dict]) -> list[dict]:
+    """Normalize all transaction amounts to USD using embedded exchange rates."""
+    result = []
+    for record in records:
+        normalized = {**record}
+        normalized["amount_usd"] = convert_to_usd(
+            record["amount"],
+            record.get("currency", "USD"),
+        )
+        result.append(normalized)
+    return result
+```
+
+```python
+# tests/unit/transforms/payments/test_normalize.py
+import pytest
+from src.transforms.payments.normalize import normalize_transaction_amounts
+
+class TestNormalizeTransactionAmounts:
+    def test_usd_passthrough(self):
+        """USD transactions are not converted."""
+        records = [{"amount": 100.0, "currency": "USD"}]
+        result = normalize_transaction_amounts(records)
+        assert result[0]["amount_usd"] == 100.0
+
+    def test_eur_conversion(self):
+        """EUR amounts are converted to USD."""
+        records = [{"amount": 100.0, "currency": "EUR"}]
+        result = normalize_transaction_amounts(records)
+        assert result[0]["amount_usd"] > 0
+        assert result[0]["amount_usd"] != 100.0  # was converted
+
+    def test_missing_currency_defaults_to_usd(self):
+        """Records without currency field default to USD."""
+        records = [{"amount": 50.0}]
+        result = normalize_transaction_amounts(records)
+        assert result[0]["amount_usd"] == 50.0
+
+    def test_original_record_unchanged(self):
+        """Transform does not mutate input records."""
+        original = {"amount": 100.0, "currency": "EUR"}
+        records = [original]
+        normalize_transaction_amounts(records)
+        assert original == {"amount": 100.0, "currency": "EUR"}
+
+    def test_empty_input_returns_empty(self):
+        """Empty list returns empty list."""
+        assert normalize_transaction_amounts([]) == []
+
+    def test_preserves_all_original_fields(self):
+        """Additional fields on input records are preserved in output."""
+        records = [{"amount": 100.0, "currency": "USD", "transaction_id": "txn_abc"}]
+        result = normalize_transaction_amounts(records)
+        assert result[0]["transaction_id"] == "txn_abc"
+
+    @pytest.mark.parametrize("amount,currency,expected_range", [
+        (0.0, "USD", (0.0, 0.0)),
+        (1_000_000.0, "USD", (1_000_000.0, 1_000_000.0)),
+        (-1.0, "USD", (-1.0, -1.0)),  # negative amounts should be preserved for refunds
+    ])
+    def test_boundary_amounts(self, amount, currency, expected_range):
+        """Test boundary amount values."""
+        records = [{"amount": amount, "currency": currency}]
+        result = normalize_transaction_amounts(records)
+        low, high = expected_range
+        assert low <= result[0]["amount_usd"] <= high
+```
+
+**Transform test patterns**
+
+Test these scenarios for every transform function:
+1. Happy path: valid input produces expected output
+2. Empty input: empty list returns empty list
+3. Null/missing optional fields: handled gracefully with documented defaults
+4. Null/missing required fields: raises expected exception
+5. Boundary values: zero, negative, very large numbers
+6. Input immutability: input records are not mutated
+7. Edge cases specific to the business domain
+
+**Fixture-based tests**
+
+For complex transformations, use fixture files:
+
+```python
+# tests/unit/transforms/payments/test_aggregate.py
+import json
+from pathlib import Path
+from src.transforms.payments.aggregate import compute_daily_revenue
+
+def load_fixture(name: str) -> list[dict]:
+    path = Path(__file__).parent.parent.parent / "fixtures" / name
+    return json.loads(path.read_text())
+
+def test_daily_revenue_aggregation():
+    """Daily revenue correctly sums transactions by currency."""
+    transactions = load_fixture("payments/raw/valid_transactions.json")
+    expected = load_fixture("payments/expected/aggregated_revenue.json")
+
+    result = compute_daily_revenue(transactions, date="2024-01-15")
+
+    assert result["date"] == "2024-01-15"
+    assert result["total_usd"] == pytest.approx(expected["total_usd"], rel=0.001)
+    assert result["transaction_count"] == expected["transaction_count"]
+```
+
+### Unit Tests for Sources and Sinks
+
+Sources and sinks interact with external systems. Test them using mocks or test doubles:
+
+```python
+# tests/unit/sources/test_stripe_reader.py
+from unittest.mock import MagicMock, patch
+from src.sources.stripe.charges import StripeChargesReader
+from datetime import datetime, timezone
+
+class TestStripeChargesReader:
+    @patch("src.sources.stripe.charges.stripe.Charge.list")
+    def test_reads_charges_for_time_window(self, mock_list):
+        """Reader requests charges within the specified time window."""
+        mock_list.return_value = MagicMock(
+            auto_paging_iter=lambda: iter([
+                {"id": "ch_001", "amount": 1000, "currency": "usd", "created": 1705329825},
+            ])
+        )
+
+        reader = StripeChargesReader(api_key="sk_test_fake")
+        start = datetime(2024, 1, 15, 0, 0, tzinfo=timezone.utc)
+        end = datetime(2024, 1, 15, 1, 0, tzinfo=timezone.utc)
+
+        records = list(reader.read(start, end))
+
+        mock_list.assert_called_once_with(
+            created={"gte": int(start.timestamp()), "lt": int(end.timestamp())},
+            limit=100,
+        )
+        assert len(records) == 1
+        assert records[0]["id"] == "ch_001"
+
+    @patch("src.sources.stripe.charges.stripe.Charge.list")
+    def test_handles_rate_limit_with_retry(self, mock_list):
+        """Reader retries on rate limit errors."""
+        mock_list.side_effect = [
+            stripe.error.RateLimitError("Too many requests"),
+            MagicMock(auto_paging_iter=lambda: iter([])),
+        ]
+
+        reader = StripeChargesReader(api_key="sk_test_fake", max_retries=2)
+        records = list(reader.read(datetime.utcnow(), datetime.utcnow()))
+
+        assert mock_list.call_count == 2  # retried once
+```
+
+### Integration Tests
+
+Integration tests run the full pipeline end-to-end against real (containerized) infrastructure:
+
+```python
+# tests/integration/test_payments_pipeline.py
+import pytest
+import docker
+from datetime import datetime
+from src.pipelines.payments import PaymentsIngestionPipeline
+
+@pytest.fixture(scope="session")
+def docker_services(docker_ip, docker_services):
+    """Start required Docker services for integration tests."""
+    docker_services.start("kafka")
+    docker_services.start("postgres")
+    docker_services.wait_until_responsive(
+        timeout=60.0,
+        pause=0.1,
+        check=lambda: is_responsive("localhost", 9092),
+    )
+    return docker_services
+
+@pytest.fixture
+def pipeline(docker_services):
+    """Create pipeline instance connected to test containers."""
+    return PaymentsIngestionPipeline(
+        source_config={"host": "localhost", "port": 5432, "db": "test_db"},
+        sink_config={"bucket": "test-silver", "endpoint": "http://localhost:9000"},
+    )
+
+def test_pipeline_processes_valid_transactions(pipeline, test_transactions):
+    """Integration test: pipeline reads, transforms, and writes transactions."""
+    # Seed source data
+    seed_postgres(test_transactions)
+
+    # Run pipeline for test window
+    result = pipeline.run(
+        start=datetime(2024, 1, 15, 0, 0),
+        end=datetime(2024, 1, 15, 1, 0),
+    )
+
+    # Verify output
+    output = read_from_minio("test-silver", "transactions/2024-01-15/00/")
+    assert len(output) == len(test_transactions)
+    assert all("amount_usd" in record for record in output)
+    assert result.records_written == len(test_transactions)
+    assert result.dlq_records == 0
+
+def test_pipeline_routes_invalid_records_to_dlq(pipeline):
+    """Integration test: invalid records go to DLQ, valid records proceed."""
+    mixed_records = [
+        {"transaction_id": "txn_001", "amount": 100.0, "currency": "USD", ...},
+        {"transaction_id": "txn_002", "amount": "N/A", "currency": "INVALID", ...},  # bad
+        {"transaction_id": "txn_003", "amount": 50.0, "currency": "EUR", ...},
+    ]
+    seed_postgres(mixed_records)
+
+    result = pipeline.run(
+        start=datetime(2024, 1, 15, 0, 0),
+        end=datetime(2024, 1, 15, 1, 0),
+    )
+
+    assert result.records_written == 2   # valid records
+    assert result.dlq_records == 1       # bad record in DLQ
+```
+
+**docker-compose test configuration**
+
+```yaml
+# docker/docker-compose.test.yml
+version: "3.9"
+services:
+  kafka:
+    image: confluentinc/cp-kafka:7.5.0
+    ports: ["9092:9092"]
+    environment:
+      KAFKA_BROKER_ID: 1
+      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
+      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
+      KAFKA_LOG_RETENTION_MS: 60000  # short retention for tests
+  postgres:
+    image: postgres:15
+    ports: ["5432:5432"]
+    environment:
+      POSTGRES_PASSWORD: test
+      POSTGRES_DB: test_db
+  minio:
+    image: minio/minio:latest
+    ports: ["9000:9000"]
+    command: server /data
+    environment:
+      MINIO_ROOT_USER: test
+      MINIO_ROOT_PASSWORD: testtest
+```
+
+### Data Quality Tests
+
+Data quality tests run inline in the pipeline and also as standalone test suite:
+
+```python
+# tests/quality/test_silver_transactions_quality.py
+import pytest
+import great_expectations as gx
+from tests.fixtures import load_fixture
+
+def test_silver_transaction_completeness():
+    """All required fields are present and non-null."""
+    df = load_fixture_as_dataframe("payments/silver/valid_transactions.parquet")
+    suite = load_expectation_suite("silver_transactions")
+    validator = gx.Validator(batch=df, expectation_suite=suite)
+
+    results = validator.validate()
+    failed = [r for r in results.results if not r.success]
+
+    assert not failed, f"Quality checks failed:\n" + "\n".join(str(f) for f in failed)
+
+def test_no_duplicate_transaction_ids():
+    """Transaction IDs are unique in silver layer."""
+    df = load_fixture_as_dataframe("payments/silver/valid_transactions.parquet")
+    duplicate_count = df["transaction_id"].duplicated().sum()
+    assert duplicate_count == 0, f"Found {duplicate_count} duplicate transaction IDs"
+
+def test_amount_within_bounds():
+    """All amounts are non-negative and below maximum threshold."""
+    df = load_fixture_as_dataframe("payments/silver/valid_transactions.parquet")
+    assert (df["amount"] >= 0).all(), "Negative amounts found"
+    assert (df["amount"] <= 1_000_000).all(), "Amounts exceeding maximum found"
+```
+
+### Performance Tests
+
+Performance tests catch throughput regressions before they reach production:
+
+```python
+# tests/performance/test_transform_performance.py
+import time
+import pytest
+from src.transforms.payments.normalize import normalize_transaction_amounts
+from tests.fixtures import generate_transactions
+
+@pytest.mark.performance
+class TestNormalizePerformance:
+    RECORDS = 100_000
+    MAX_SECONDS = 10.0  # must normalize 100K records in under 10 seconds
+
+    def test_throughput(self):
+        """normalize_transaction_amounts processes 100K records in under 10 seconds."""
+        records = generate_transactions(self.RECORDS)
+
+        start = time.perf_counter()
+        result = normalize_transaction_amounts(records)
+        elapsed = time.perf_counter() - start
+
+        assert len(result) == self.RECORDS
+        assert elapsed < self.MAX_SECONDS, (
+            f"Performance regression: normalized {self.RECORDS} records in {elapsed:.2f}s "
+            f"(max: {self.MAX_SECONDS}s)"
+        )
+
+    def test_memory_usage(self):
+        """Transform does not exceed 500MB memory for 100K records."""
+        import tracemalloc
+        tracemalloc.start()
+
+        records = generate_transactions(self.RECORDS)
+        normalize_transaction_amounts(records)
+
+        current, peak = tracemalloc.get_traced_memory()
+        tracemalloc.stop()
+
+        peak_mb = peak / 1024 / 1024
+        assert peak_mb < 500, f"Memory usage {peak_mb:.1f}MB exceeds 500MB limit"
+```
+
+**Performance test benchmarks**
+
+Run benchmarks with pytest-benchmark to track performance over time:
+
+```python
+def test_normalize_benchmark(benchmark):
+    """Benchmark normalization transform for regression tracking."""
+    records = generate_transactions(10_000)
+    result = benchmark(normalize_transaction_amounts, records)
+    assert len(result) == 10_000
+```
+
+### Test Coverage and CI Configuration
+
+```ini
+# pyproject.toml
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+markers = [
+    "unit: fast unit tests",
+    "integration: requires Docker services",
+    "performance: long-running performance tests",
+    "quality: data quality assertion tests",
+]
+addopts = "--strict-markers"
+
+[tool.coverage.run]
+source = ["src"]
+branch = true
+omit = ["src/dags/*"]  # DAG files excluded; coverage focused on transforms
+
+[tool.coverage.report]
+fail_under = 90
+show_missing = true
+```
+
+```yaml
+# .github/workflows/test.yml excerpt
+jobs:
+  unit-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - run: pytest tests/unit -v --cov=src/transforms --cov-fail-under=90
+
+  integration-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - run: docker compose -f docker/docker-compose.test.yml up -d
+      - run: pytest tests/integration -v -m integration
+      - run: docker compose -f docker/docker-compose.test.yml down
+```
+
+### Test Data Management Rules
+
+1. Never use production data in tests — generate synthetic data or use anonymized fixtures
+2. Test fixtures must be deterministic — no random seeds without explicit seeding
+3. Integration test databases must be isolated per test run — use unique schema or database names
+4. Performance tests must run on the same hardware class as CI to produce comparable benchmarks
+5. DLQ assertions are mandatory: every integration test must assert the expected DLQ record count, not just the success path

package/content/knowledge/ml/ml-architecture.md

@@ -0,0 +1,172 @@
+---
+name: ml-architecture
+description: Training/serving architecture split, feature stores, model registry, online vs offline inference patterns, and ML system design decisions
+topics: [ml, architecture, feature-store, model-registry, inference, serving, training]
+---
+
+ML systems have a fundamental architectural split that traditional software does not: the training system and the serving system are different codebases running on different infrastructure, yet they must agree on the exact same data transformations. This training-serving skew is the most common source of silent production bugs in ML. Designing the architecture to prevent skew — through shared feature stores, shared preprocessing libraries, and strict interface contracts — is the most important ML architecture decision.
+
+## Summary
+
+ML architecture separates training (batch, data-intensive, experimental) from serving (latency-sensitive, stateless, reliable). Feature stores eliminate training-serving skew by providing consistent feature computation for both. Model registries provide the governance layer: versioning, lineage, deployment gates, and rollback. Choose online vs. offline inference based on latency requirements and data freshness needs. Document every architectural decision as an ADR.
+
+## Deep Guidance
+
+### Training/Serving Architecture Split
+
+The training system and serving system have fundamentally different requirements:
+
+| Dimension | Training | Serving |
+|-----------|----------|---------|
+| Throughput | High (process TB of data) | High (handle thousands of RPS) |
+| Latency | Not time-critical | P99 < 200ms |
+| Consistency | Best-effort | Exact reproducibility |
+| Infrastructure | Spot instances, GPU clusters | Reserved instances, autoscaling |
+| State | Stateful (checkpoint, resume) | Stateless (each request independent) |
+| Failures | Retry / resume from checkpoint | Circuit breaker, fallback |
+
+The primary risk of this split is **training-serving skew**: the model was trained with feature X computed one way, but production computes feature X a slightly different way, leading to silent accuracy degradation.
+
+**Prevention strategies**:
+1. **Shared feature library**: Both training and serving import the same `src/features/` module for all feature computation. Never duplicate feature logic.
+2. **Feature store**: Centralised store that computes features once and serves them to both training (historical) and serving (real-time) paths.
+3. **Schema validation**: Validate model inputs at serving time against the schema seen during training.
+
+### Feature Store Architecture
+
+A feature store is a data infrastructure component that stores and serves pre-computed features:
+
+```
+                    ┌─────────────────┐
+Data Sources ──────►│ Feature Pipeline │
+                    └────────┬────────┘
+                             │ compute features
+                    ┌────────▼────────┐
+                    │  Feature Store  │
+                    │  ┌───────────┐  │
+                    │  │ Offline   │  │──► Training Data
+                    │  │ (S3/GCS)  │  │
+                    │  ├───────────┤  │
+                    │  │ Online    │  │──► Serving (low-latency)
+                    │  │ (Redis)   │  │
+                    └─────────────────┘
+```
+
+**Offline store**: Historical features for training. Backed by object storage (S3, GCS) or a columnar database (BigQuery, Snowflake). Supports point-in-time correct feature retrieval (no data leakage).
+
+**Online store**: Low-latency feature serving for real-time inference. Backed by Redis, DynamoDB, or Cassandra. Stores only the latest feature values.
+
+**Implementations**: Feast (open source), Tecton, Vertex AI Feature Store, SageMaker Feature Store.
+
+A feature store is justified when:
+- Multiple models use the same features (DRY for features)
+- Training-serving skew is causing production issues
+- Feature computation is expensive and should be shared
+
+### Model Registry
+
+The model registry is the governance layer between training and production. Every model that has ever been trained should have a registry record:
+
+```
+Training Run ──► Model Artifacts ──► Registry ──► Deployment
+                 (weights, config)   (metadata,    (staging,
+                                     lineage)       production)
+```
+
+**Registry metadata per model version**:
+- Model name and semantic version (`fraud-detector-v2.3.1`)
+- Training run ID and commit SHA (full lineage)
+- Training metrics (AUC, F1, etc.)
+- Evaluation metrics on holdout sets
+- Training dataset version
+- Model schema (input/output feature names and types)
+- Serving requirements (runtime, memory, GPU)
+- Promotion history (who promoted, when, why)
+
+**Lifecycle stages**:
+```
+None → Staging → Production → Archived
+```
+
+- Validation gates control promotion: automated tests (accuracy thresholds, latency budgets) plus optional human approval
+- Production always has at least one previous version for rollback
+- Archive on a schedule (keep 6 months of production versions)
+
+**Implementations**: MLflow Model Registry, Weights & Biases Model Registry, SageMaker Model Registry.
+
+### Online vs. Offline Inference
+
+**Online inference** (real-time, synchronous):
+- Model returns predictions in response to a request, typically within 100–500ms
+- Examples: fraud scoring at checkout, recommendation on page load, search ranking
+- Infrastructure: Model server (TorchServe, Triton, BentoML), autoscaled behind a load balancer
+- Key considerations: latency budget, model size (must fit in serving memory), cold start time
+
+**Offline inference** (batch, asynchronous):
+- Model scores a large dataset on a schedule, stores predictions for later retrieval
+- Examples: churn prediction for all users (run nightly), content pre-scoring for a recommendation cache
+- Infrastructure: Spark job, Airflow DAG, Ray cluster, or simple Python script on a large VM
+- Key considerations: throughput (records/second), data pipeline integration, prediction freshness
+
+**Near-real-time / stream inference**:
+- Model scores events from a stream (Kafka, Kinesis) with seconds-to-minutes latency
+- Examples: anomaly detection on clickstream, session-level personalisation
+- Infrastructure: Kafka consumer + model inference worker, Flink ML, or Spark Structured Streaming
+- Key considerations: exactly-once semantics, ordering guarantees, backpressure handling
+
+**Decision matrix**:
+
+| Use Case | Latency Requirement | Data Freshness | Approach |
+|----------|---------------------|----------------|----------|
+| Checkout fraud | < 500ms | Real-time | Online inference |
+| Churn prediction | N/A | Daily | Offline batch |
+| Email personalisation | N/A (send time) | Daily | Offline batch |
+| Feed ranking | < 200ms | Real-time | Online inference |
+| Anomaly detection | < 30 seconds | Streaming | Stream inference |
+
+### ML System Design Patterns
+
+**Lambda Architecture for ML**:
+- Batch layer: nightly model retraining or batch scoring
+- Speed layer: real-time model updates or online scoring
+- Serving layer: unified API serving precomputed (batch) + real-time predictions
+- Complexity cost is high — evaluate whether the freshness gain justifies it
+
+**Two-Tower Architecture** (recommendation systems):
+- Candidate generation tower: fast approximate nearest-neighbour retrieval (ANN index)
+- Ranking tower: expensive full model scoring of top-K candidates
+- Separates recall (retrieve thousands of candidates quickly) from precision (rank them accurately)
+
+**Shadow Mode Deployment**:
+- New model runs in parallel with production model, receiving real traffic
+- New model's predictions are not served but are logged and evaluated
+- Safe way to validate new models with real data before full deployment
+
+### Architecture Decision Record Template for ML
+
+```markdown
+# ADR-ML-001: Online vs. Offline Inference for Recommendation
+
+## Status
+Accepted — 2024-03-15
+
+## Context
+Product recommends items to users on homepage. 5M daily active users.
+Data team produces updated user embeddings daily. Item catalog updates hourly.
+
+## Decision
+Use offline batch inference for recommendation.
+- Nightly batch job scores all user-item pairs for top 100 candidates
+- Recommendations stored in Redis with TTL of 26 hours
+- API reads from Redis — zero model inference at request time
+
+## Consequences
+- P99 homepage latency drops from 450ms to 20ms (Redis lookup vs. model inference)
+- Recommendations are up to 26 hours stale — acceptable given content update frequency
+- Requires Redis cluster (operational cost)
+- Model update cycle is daily — cannot react to same-session behaviour
+
+## Alternatives Rejected
+- Online inference: latency budget exceeded (model P99 = 380ms)
+- Streaming inference: engineering complexity not justified given daily embedding update cadence
+```