@zigrivers/scaffold 3.8.0 → 3.9.0

package/content/knowledge/data-pipeline/data-pipeline-project-structure.md

@@ -0,0 +1,257 @@
+---
+name: data-pipeline-project-structure
+description: Canonical directory layout for data pipeline projects covering DAGs, transforms, sinks, quality checks, configuration, and tests
+topics: [data-pipeline, project-structure, dags, transforms, sinks, quality, config, tests]
+---
+
+A well-organized data pipeline project separates pipeline orchestration logic from transformation logic, data sinks, and quality checks. This separation allows each concern to evolve independently and makes the codebase navigable for engineers unfamiliar with the project. The structure below is opinionated but widely applicable to Airflow, Prefect, Dagster, and similar orchestration tools.
+
+## Summary
+
+Organize pipeline projects into `src/dags/` for orchestration, `src/transforms/` for pure transformation functions, `src/sinks/` for write adapters, `src/quality/` for validation logic, `config/` for environment-specific settings, and `tests/` mirroring the `src/` structure. Keep business logic in transforms — not in DAG files. Configuration is never hardcoded.
+
+## Deep Guidance
+
+### Root Directory Layout
+
+```
+my-pipeline/
+├── src/
+│   ├── dags/           # Orchestration DAGs / pipeline definitions
+│   ├── transforms/     # Pure transformation functions
+│   ├── sources/        # Source connectors and readers
+│   ├── sinks/          # Destination connectors and writers
+│   ├── quality/        # Data quality checks and validators
+│   └── utils/          # Shared utilities (logging, metrics, retry)
+├── config/
+│   ├── base.yaml       # Shared configuration across environments
+│   ├── dev.yaml        # Development overrides
+│   ├── staging.yaml    # Staging overrides
+│   └── prod.yaml       # Production overrides
+├── tests/
+│   ├── unit/           # Unit tests for transforms, sources, sinks
+│   ├── integration/    # Integration tests against real (or containerized) systems
+│   ├── quality/        # Data quality test definitions
+│   └── fixtures/       # Static test data files
+├── scripts/
+│   ├── bootstrap.sh    # Local environment setup
+│   ├── replay.sh       # DLQ and historical replay tooling
+│   └── backfill.sh     # Backfill orchestration helper
+├── docker/
+│   ├── docker-compose.yml       # Local development stack
+│   └── docker-compose.test.yml  # Test isolation stack
+├── Makefile
+├── pyproject.toml      # Python project metadata and dependencies
+└── README.md
+```
+
+### `src/dags/` — Orchestration Layer
+
+DAG files define pipeline topology, scheduling, and task dependencies. They must be thin:
+
+```
+src/dags/
+├── payments/
+│   ├── transactions_ingest.py
+│   ├── transactions_enrich.py
+│   └── revenue_aggregate.py
+├── users/
+│   ├── events_ingest.py
+│   └── profiles_sync.py
+└── shared/
+    ├── base_dag.py     # Shared DAG defaults (retries, alerts, SLA)
+    └── sensors.py      # Common sensors (S3 sensor, DB sensor)
+```
+
+**Rule**: DAG files contain no business logic. A DAG file defines what runs, in what order, with what dependencies. All data processing logic lives in `src/transforms/`. A DAG task calls a function from `src/transforms/` — it does not implement the transformation inline.
+
+Bad (logic in DAG):
+```python
+@task
+def process_transactions(**context):
+    df = pd.read_parquet(...)
+    df['amount_usd'] = df['amount'] / df['exchange_rate']  # business logic in DAG
+    df.to_parquet(...)
+```
+
+Good (DAG calls transform):
+```python
+from transforms.payments import normalize_transaction_amounts
+
+@task
+def process_transactions(**context):
+    records = sources.payments.read(context['date'])
+    normalized = normalize_transaction_amounts(records)
+    sinks.warehouse.write(normalized, partition=context['date'])
+```
+
+### `src/transforms/` — Transformation Logic
+
+Transforms are pure functions: they take data in, return data out, with no side effects. This makes them independently testable without any pipeline infrastructure.
+
+```
+src/transforms/
+├── payments/
+│   ├── __init__.py
+│   ├── normalize.py    # Currency normalization, amount parsing
+│   ├── enrich.py       # Join with reference data
+│   └── aggregate.py    # Revenue rollups, daily summaries
+├── users/
+│   ├── __init__.py
+│   ├── deduplicate.py  # Deduplication logic
+│   └── classify.py     # User segment classification
+└── shared/
+    ├── timestamps.py   # Timestamp parsing and normalization
+    ├── currencies.py   # Currency conversion utilities
+    └── pii.py          # PII masking and hashing functions
+```
+
+Pure transform function pattern:
+```python
+def normalize_transaction_amounts(records: list[dict]) -> list[dict]:
+    """Convert all transaction amounts to USD using embedded exchange rates.
+
+    Args:
+        records: Raw transaction records with 'amount' and 'currency' fields
+
+    Returns:
+        Records with added 'amount_usd' field. Input records unchanged.
+    """
+    result = []
+    for record in records:
+        normalized = {**record}
+        normalized['amount_usd'] = convert_to_usd(record['amount'], record['currency'])
+        result.append(normalized)
+    return result
+```
+
+### `src/sources/` — Source Connectors
+
+Sources abstract the read interface for each upstream system:
+
+```
+src/sources/
+├── stripe/
+│   ├── __init__.py
+│   ├── charges.py      # Stripe Charges API reader
+│   └── webhooks.py     # Stripe webhook event reader
+├── postgres/
+│   ├── __init__.py
+│   └── cdc_reader.py   # CDC via logical replication
+├── s3/
+│   ├── __init__.py
+│   └── parquet_reader.py
+└── kafka/
+    ├── __init__.py
+    └── consumer.py
+```
+
+Sources implement a common interface:
+```python
+class Source(Protocol):
+    def read(self, start: datetime, end: datetime) -> Iterable[dict]: ...
+    def count(self, start: datetime, end: datetime) -> int: ...
+```
+
+### `src/sinks/` — Destination Writers
+
+Sinks abstract the write interface for each downstream system:
+
+```
+src/sinks/
+├── bigquery/
+│   ├── __init__.py
+│   ├── writer.py       # Streaming insert and load job writer
+│   └── partitioned.py  # Partitioned table writer
+├── s3/
+│   ├── __init__.py
+│   └── parquet_writer.py
+├── postgres/
+│   ├── __init__.py
+│   └── upsert_writer.py
+└── dlq/
+    ├── __init__.py
+    └── writer.py       # Dead-letter queue writer
+```
+
+Sinks implement idempotent write semantics:
+```python
+class Sink(Protocol):
+    def write(self, records: Iterable[dict], partition: str) -> WriteResult: ...
+    def write_dlq(self, record: dict, error: Exception, context: dict) -> None: ...
+```
+
+### `src/quality/` — Data Quality Checks
+
+Quality checks are assertions about data that run inline in the pipeline:
+
+```
+src/quality/
+├── rules/
+│   ├── completeness.py  # Null rate checks
+│   ├── accuracy.py      # Range and format checks
+│   ├── consistency.py   # Cross-field and cross-table checks
+│   └── timeliness.py    # Freshness and late-arrival checks
+├── expectations/
+│   ├── payments.json    # Great Expectations suite for payments
+│   └── users.json       # Great Expectations suite for users
+└── reconciliation/
+    ├── row_count.py     # Source vs destination row count comparison
+    └── checksum.py      # Aggregate checksum comparison
+```
+
+### `config/` — Environment Configuration
+
+All pipeline configuration (connection strings, batch sizes, SLA thresholds, feature flags) lives in config files, never hardcoded:
+
+```yaml
+# config/base.yaml
+pipeline:
+  batch_size: 10000
+  max_retries: 3
+  retry_delay_seconds: 30
+
+quality:
+  completeness_threshold: 0.99
+  row_count_variance_pct: 0.01
+
+sla:
+  max_lag_minutes: 60
+```
+
+Environment-specific files override base values only for what differs. Use a config loader that merges base + environment:
+
+```python
+def load_config(env: str = "dev") -> Config:
+    base = yaml.safe_load(open("config/base.yaml"))
+    override = yaml.safe_load(open(f"config/{env}.yaml"))
+    return Config(**deep_merge(base, override))
+```
+
+### `tests/` — Test Structure Mirrors Source
+
+```
+tests/
+├── unit/
+│   ├── transforms/
+│   │   ├── payments/
+│   │   │   ├── test_normalize.py
+│   │   │   └── test_aggregate.py
+│   │   └── users/
+│   │       └── test_deduplicate.py
+│   ├── sources/
+│   └── sinks/
+├── integration/
+│   ├── test_payments_pipeline.py
+│   └── test_users_pipeline.py
+├── quality/
+│   └── test_data_quality_rules.py
+└── fixtures/
+    ├── payments/
+    │   ├── raw_charges.json
+    │   └── expected_normalized.json
+    └── users/
+        └── raw_events.json
+```
+
+The `fixtures/` directory contains static JSON/Parquet/CSV files representing realistic sample data. Fixtures must be small enough to run in CI (< 10MB total) but representative enough to cover edge cases.

package/content/knowledge/data-pipeline/data-pipeline-quality.md

@@ -0,0 +1,324 @@
+---
+name: data-pipeline-quality
+description: Schema validation, anomaly detection, data quality tests using Great Expectations and dbt, and reconciliation patterns
+topics: [data-pipeline, data-quality, schema-validation, anomaly-detection, great-expectations, dbt, reconciliation, testing]
+---
+
+Data quality failures are the most operationally damaging class of data pipeline bugs because they often go undetected until a business user notices wrong numbers. Unlike pipeline failures (which are loud and immediately visible), quality degradation is silent — data flows, pipelines succeed, but the numbers are wrong. Building quality checks directly into the pipeline, not as a separate offline process, is the only reliable defense.
+
+## Summary
+
+Validate schema and structural correctness at ingestion. Run statistical anomaly detection on all critical metrics. Implement Great Expectations suites or dbt tests on every table in the medallion stack. Reconcile record counts and aggregate checksums between pipeline stages. Alert on quality degradation before downstream consumers see it. Never silently discard records — route all failures to the DLQ with diagnostic context.
+
+## Deep Guidance
+
+### Schema Validation at Ingestion
+
+Validate incoming data against the expected schema as the first step of every pipeline. Fail fast on schema violations rather than propagating bad data downstream.
+
+**JSON Schema validation**
+
+```python
+from jsonschema import validate, ValidationError
+
+TRANSACTION_SCHEMA = {
+    "type": "object",
+    "required": ["transaction_id", "amount", "currency", "user_id", "created_at"],
+    "properties": {
+        "transaction_id": {"type": "string", "minLength": 1},
+        "amount": {"type": "number", "minimum": 0},
+        "currency": {"type": "string", "enum": ["USD", "EUR", "GBP", "JPY", "CAD"]},
+        "user_id": {"type": "string", "pattern": "^usr_[0-9]{6}$"},
+        "created_at": {"type": "string", "format": "date-time"},
+    },
+    "additionalProperties": True,  # allow extra fields in bronze
+}
+
+def validate_and_route(record: dict, dlq) -> dict | None:
+    """Validate record against schema. Route failures to DLQ, return valid records."""
+    try:
+        validate(instance=record, schema=TRANSACTION_SCHEMA)
+        return record
+    except ValidationError as e:
+        dlq.write(record, error=e, reason="schema_validation_failure")
+        return None
+```
+
+**Avro schema enforcement**
+
+For Kafka-based pipelines, use Schema Registry to enforce Avro schemas at produce and consume time:
+
+```python
+from confluent_kafka.avro import AvroConsumer
+from confluent_kafka.avro.serializer import SerializerError
+
+consumer = AvroConsumer({
+    "bootstrap.servers": "localhost:9092",
+    "schema.registry.url": "http://localhost:8081",
+    "group.id": "silver-processor",
+})
+
+while True:
+    msg = consumer.poll(1.0)
+    if msg is None:
+        continue
+    if msg.error():
+        handle_consumer_error(msg.error())
+        continue
+    try:
+        record = msg.value()  # automatically deserialized + schema-validated
+        process(record)
+    except SerializerError as e:
+        dlq.write(msg.value(), error=e, reason="avro_deserialization_failure")
+```
+
+### Statistical Anomaly Detection
+
+Structural validation catches format errors. Anomaly detection catches semantic errors — values that are structurally valid but statistically wrong.
+
+**Volume anomaly detection**
+
+```python
+def check_volume_anomaly(
+    pipeline_id: str,
+    actual_count: int,
+    window_days: int = 14,
+    z_score_threshold: float = 3.0,
+) -> AnomalyResult:
+    """Detect volume anomalies using z-score against rolling historical baseline."""
+    historical = get_historical_counts(pipeline_id, days=window_days)
+    mean = statistics.mean(historical)
+    stddev = statistics.stdev(historical)
+
+    if stddev == 0:
+        return AnomalyResult(anomaly=False)
+
+    z_score = (actual_count - mean) / stddev
+    is_anomaly = abs(z_score) > z_score_threshold
+
+    return AnomalyResult(
+        anomaly=is_anomaly,
+        z_score=z_score,
+        actual=actual_count,
+        expected_mean=mean,
+        expected_stddev=stddev,
+        direction="high" if z_score > 0 else "low",
+    )
+```
+
+**Metric anomaly detection**
+
+Extend volume checks to key business metrics:
+- Revenue sum per partition (detect 0-revenue or 100x-revenue outliers)
+- Average transaction amount (detect extreme values indicating test data in prod)
+- User count per partition (detect sudden drop suggesting source outage)
+- Error rate (detect spike indicating upstream system degradation)
+
+### Great Expectations Integration
+
+Great Expectations (GX) provides a declarative assertion framework for data quality:
+
+**Expectation suite definition**
+
+```python
+# src/quality/expectations/payments.py
+import great_expectations as gx
+
+def build_payments_suite() -> gx.ExpectationSuite:
+    suite = gx.ExpectationSuite(expectation_suite_name="silver_transactions")
+
+    # Completeness
+    suite.add_expectation(gx.core.ExpectColumnValuesToNotBeNull(
+        column="transaction_id", meta={"severity": "critical"}
+    ))
+    suite.add_expectation(gx.core.ExpectColumnValuesToNotBeNull(
+        column="amount", meta={"severity": "critical"}
+    ))
+    suite.add_expectation(gx.core.ExpectColumnValuesToNotBeNull(
+        column="user_id", mostly=0.99, meta={"severity": "high"}
+    ))
+
+    # Accuracy
+    suite.add_expectation(gx.core.ExpectColumnValuesToBeBetween(
+        column="amount", min_value=0, max_value=1_000_000
+    ))
+    suite.add_expectation(gx.core.ExpectColumnValuesToBeInSet(
+        column="currency", value_set=["USD", "EUR", "GBP", "JPY", "CAD"]
+    ))
+    suite.add_expectation(gx.core.ExpectColumnValuesToMatchRegex(
+        column="transaction_id", regex=r"^txn_[a-zA-Z0-9]{20}$"
+    ))
+
+    # Volume
+    suite.add_expectation(gx.core.ExpectTableRowCountToBeBetween(
+        min_value=1000, max_value=10_000_000
+    ))
+
+    # Uniqueness
+    suite.add_expectation(gx.core.ExpectColumnValuesToBeUnique(
+        column="transaction_id"
+    ))
+
+    return suite
+```
+
+**Running GX in the pipeline**
+
+```python
+def run_quality_check(df: pd.DataFrame, suite_name: str) -> ValidationResult:
+    context = gx.get_context()
+    validator = context.get_validator(
+        batch_request=RuntimeBatchRequest(
+            datasource_name="runtime",
+            data_connector_name="default",
+            data_asset_name=suite_name,
+            runtime_parameters={"batch_data": df},
+            batch_identifiers={"run_id": str(datetime.utcnow())},
+        ),
+        expectation_suite_name=suite_name,
+    )
+    result = validator.validate()
+
+    metrics.gauge("data_quality_success_pct",
+                  result.statistics["success_percent"],
+                  tags={"suite": suite_name})
+
+    if not result.success:
+        failed = [e for e in result.results if not e.success]
+        critical_failures = [f for f in failed if f.meta.get("severity") == "critical"]
+        if critical_failures:
+            raise DataQualityError(f"Critical quality failures: {critical_failures}")
+
+    return result
+```
+
+### dbt Data Tests
+
+For SQL-based pipelines using dbt, define tests directly in model YAML files:
+
+```yaml
+# models/silver/schema.yml
+models:
+  - name: silver_transactions
+    description: Cleaned and validated transaction records
+    columns:
+      - name: transaction_id
+        tests:
+          - not_null
+          - unique
+      - name: amount
+        tests:
+          - not_null
+          - dbt_utils.accepted_range:
+              min_value: 0
+              max_value: 1000000
+      - name: currency
+        tests:
+          - not_null
+          - accepted_values:
+              values: ["USD", "EUR", "GBP", "JPY", "CAD"]
+      - name: user_id
+        tests:
+          - not_null
+          - relationships:
+              to: ref('silver_users')
+              field: user_id
+
+    tests:
+      - dbt_utils.recency:
+          datepart: hour
+          field: created_at
+          interval: 2    # table should have data from last 2 hours
+      - dbt_utils.equal_rowcount:
+          compare_model: ref('bronze_transactions')
+          # silver count should match bronze minus known invalid records
+```
+
+**Custom dbt tests**
+
+```sql
+-- tests/assert_no_future_transactions.sql
+-- Fails if any transactions are timestamped in the future
+SELECT count(*) as future_count
+FROM {{ ref('silver_transactions') }}
+WHERE created_at > CURRENT_TIMESTAMP + INTERVAL '5 minutes'
+HAVING count(*) > 0
+```
+
+### Reconciliation Patterns
+
+Reconciliation compares record counts and aggregate values between pipeline stages to detect silent data loss.
+
+**Row count reconciliation**
+
+```python
+def reconcile_row_counts(
+    source_count: int,
+    destination_count: int,
+    pipeline_id: str,
+    expected_loss_rate: float = 0.001,  # max 0.1% loss acceptable
+) -> ReconciliationResult:
+    """Compare source and destination record counts."""
+    if source_count == 0:
+        raise ReconciliationError(f"Source count is zero for {pipeline_id} — possible extraction failure")
+
+    loss_rate = (source_count - destination_count) / source_count
+
+    if loss_rate > expected_loss_rate:
+        raise ReconciliationError(
+            f"Record loss rate {loss_rate:.4%} exceeds threshold {expected_loss_rate:.4%}. "
+            f"Source: {source_count}, Destination: {destination_count}"
+        )
+
+    metrics.gauge("pipeline_record_loss_rate", loss_rate, tags={"pipeline": pipeline_id})
+    return ReconciliationResult(passed=True, loss_rate=loss_rate)
+```
+
+**Aggregate checksum reconciliation**
+
+```python
+def reconcile_aggregates(
+    source_df: pd.DataFrame,
+    destination_df: pd.DataFrame,
+    key_column: str,
+    value_columns: list[str],
+    tolerance_pct: float = 0.001,
+) -> ReconciliationResult:
+    """Compare aggregate sums between source and destination."""
+    failures = []
+    for col in value_columns:
+        src_sum = source_df[col].sum()
+        dst_sum = destination_df[col].sum()
+
+        if src_sum == 0 and dst_sum == 0:
+            continue
+
+        variance = abs(src_sum - dst_sum) / max(abs(src_sum), 1e-10)
+        if variance > tolerance_pct:
+            failures.append(f"{col}: src={src_sum:.2f}, dst={dst_sum:.2f}, variance={variance:.4%}")
+
+    if failures:
+        raise ReconciliationError(f"Aggregate reconciliation failed:\n" + "\n".join(failures))
+
+    return ReconciliationResult(passed=True)
+```
+
+### Quality Gate Integration
+
+Quality checks must be integrated as pipeline gates, not optional post-processing steps:
+
+```
+Extract → Schema Validate → [pass: Transform] [fail: DLQ]
+                ↓
+         Statistical Check → [anomaly: alert + hold] [pass: continue]
+                ↓
+         Business Rules → [fail: DLQ] [pass: Load]
+                ↓
+         Reconciliation → [fail: alert + rollback] [pass: commit]
+```
+
+A quality gate failure at any stage must:
+1. Route affected records to the DLQ with full diagnostic context
+2. Emit a quality failure metric
+3. Alert the on-call engineer if the failure rate exceeds the quality budget
+4. Block pipeline progression for critical failures (structural, not statistical)