@zigrivers/scaffold 3.8.0 → 3.9.0

package/content/knowledge/data-pipeline/data-pipeline-dev-environment.md

@@ -0,0 +1,350 @@
+---
+name: data-pipeline-dev-environment
+description: Local development setup with Docker, sample data generation, replay tooling, and test fixtures for data pipelines
+topics: [data-pipeline, dev-environment, docker, local-development, sample-data, replay, test-fixtures]
+---
+
+A productive data pipeline development environment must run locally without requiring cloud accounts, production credentials, or access to real data. Engineers must be able to ingest, process, debug, and test the full pipeline on their laptop in under 10 minutes from a clean checkout. This requires containerized dependencies, synthetic sample data, and tooling to replay specific scenarios.
+
+## Summary
+
+Use Docker Compose to run all pipeline dependencies locally (Kafka, Postgres, object storage, orchestrator UI). Generate synthetic sample data with realistic distributions that match production schemas. Build replay tooling so engineers can re-run any historical pipeline execution or simulate specific failure scenarios. Keep test fixtures small, version-controlled, and deterministic.
+
+## Deep Guidance
+
+### Docker Compose Local Stack
+
+Define the complete local dependency stack in `docker/docker-compose.yml`. Every external system the pipeline depends on must run locally:
+
+```yaml
+# docker/docker-compose.yml
+version: "3.9"
+
+services:
+  # Message broker
+  kafka:
+    image: confluentinc/cp-kafka:7.5.0
+    ports:
+      - "9092:9092"
+    environment:
+      KAFKA_BROKER_ID: 1
+      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
+      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
+      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
+    depends_on: [zookeeper]
+
+  zookeeper:
+    image: confluentinc/cp-zookeeper:7.5.0
+    environment:
+      ZOOKEEPER_CLIENT_PORT: 2181
+
+  # Schema registry
+  schema-registry:
+    image: confluentinc/cp-schema-registry:7.5.0
+    ports:
+      - "8081:8081"
+    environment:
+      SCHEMA_REGISTRY_HOST_NAME: schema-registry
+      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka:9092
+
+  # Source database (CDC)
+  postgres:
+    image: postgres:15
+    ports:
+      - "5432:5432"
+    environment:
+      POSTGRES_USER: pipeline
+      POSTGRES_PASSWORD: pipeline
+      POSTGRES_DB: source_db
+    command: >
+      postgres
+        -c wal_level=logical
+        -c max_replication_slots=10
+        -c max_wal_senders=10
+    volumes:
+      - ./docker/postgres/init:/docker-entrypoint-initdb.d
+
+  # Object storage (S3-compatible)
+  minio:
+    image: minio/minio:latest
+    ports:
+      - "9000:9000"
+      - "9001:9001"
+    environment:
+      MINIO_ROOT_USER: minioadmin
+      MINIO_ROOT_PASSWORD: minioadmin
+    command: server /data --console-address ":9001"
+    volumes:
+      - minio_data:/data
+
+  # Pipeline orchestrator (Airflow)
+  airflow-webserver:
+    image: apache/airflow:2.8.0
+    ports:
+      - "8080:8080"
+    environment:
+      AIRFLOW__CORE__EXECUTOR: LocalExecutor
+      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@airflow-db/airflow
+      AIRFLOW__CORE__LOAD_EXAMPLES: "false"
+    volumes:
+      - ../src/dags:/opt/airflow/dags
+      - ../config:/opt/airflow/config
+    depends_on: [airflow-db]
+
+  airflow-db:
+    image: postgres:15
+    environment:
+      POSTGRES_USER: airflow
+      POSTGRES_PASSWORD: airflow
+      POSTGRES_DB: airflow
+
+volumes:
+  minio_data:
+```
+
+### Local Setup Script
+
+The bootstrap script gets a clean checkout to a running local environment:
+
+```bash
+#!/bin/bash
+# scripts/bootstrap.sh — Set up local development environment
+
+set -euo pipefail
+
+echo "==> Installing Python dependencies"
+pip install -e ".[dev]"
+
+echo "==> Starting Docker services"
+docker compose -f docker/docker-compose.yml up -d
+
+echo "==> Waiting for services to be healthy"
+./scripts/wait-for-services.sh
+
+echo "==> Creating local Kafka topics"
+docker exec kafka kafka-topics --create \
+  --bootstrap-server localhost:9092 \
+  --topic payments_transaction_created \
+  --partitions 8 \
+  --replication-factor 1 \
+  --if-not-exists
+
+echo "==> Creating MinIO buckets"
+docker exec minio mc alias set local http://localhost:9000 minioadmin minioadmin
+docker exec minio mc mb local/raw-data --ignore-existing
+docker exec minio mc mb local/processed-data --ignore-existing
+
+echo "==> Seeding database schemas"
+psql postgresql://pipeline:pipeline@localhost:5432/source_db -f docker/postgres/schema.sql
+
+echo "==> Generating sample data"
+python scripts/generate-sample-data.py --records 10000 --output fixtures/
+
+echo "==> Local environment ready!"
+echo "    Airflow UI:     http://localhost:8080 (admin/admin)"
+echo "    MinIO console:  http://localhost:9001 (minioadmin/minioadmin)"
+echo "    Kafka:          localhost:9092"
+echo "    Postgres:       localhost:5432"
+```
+
+### Sample Data Generation
+
+Synthetic data must match production schemas and include realistic distributions, edge cases, and known-bad records for testing error handling:
+
+```python
+# scripts/generate-sample-data.py
+import json
+import random
+from datetime import datetime, timedelta
+from typing import Iterator
+import uuid
+
+CURRENCIES = ["USD", "EUR", "GBP", "JPY", "CAD"]
+CURRENCY_WEIGHTS = [0.60, 0.20, 0.10, 0.05, 0.05]
+COUNTRIES = ["US", "GB", "DE", "FR", "JP", "CA", "AU"]
+
+def generate_transaction(ts: datetime, inject_error: bool = False) -> dict:
+    """Generate a synthetic payment transaction.
+
+    Args:
+        ts: Event timestamp
+        inject_error: If True, inject a known-bad record for error handling tests
+    """
+    if inject_error:
+        return {
+            "transaction_id": str(uuid.uuid4()),
+            "amount": "N/A",          # schema violation: string instead of number
+            "currency": "INVALID",    # invalid enum
+            "user_id": None,          # null violation
+            "created_at": ts.isoformat(),
+        }
+
+    currency = random.choices(CURRENCIES, CURRENCY_WEIGHTS)[0]
+    # Log-normal distribution matches real transaction amount distributions
+    amount = round(random.lognormvariate(3.5, 1.2), 2)
+
+    return {
+        "transaction_id": str(uuid.uuid4()),
+        "amount": amount,
+        "currency": currency,
+        "user_id": f"usr_{random.randint(1, 50000):06d}",
+        "merchant_id": f"mer_{random.randint(1, 5000):05d}",
+        "country": random.choice(COUNTRIES),
+        "created_at": ts.isoformat(),
+        "status": random.choices(["completed", "failed", "pending"], [0.92, 0.05, 0.03])[0],
+    }
+
+def generate_dataset(
+    records: int,
+    error_rate: float = 0.01,
+    start_time: datetime | None = None,
+) -> Iterator[dict]:
+    """Generate a stream of synthetic transactions with configurable error rate."""
+    start = start_time or datetime.utcnow() - timedelta(hours=24)
+    for i in range(records):
+        ts = start + timedelta(seconds=i * 0.1)
+        inject_error = random.random() < error_rate
+        yield generate_transaction(ts, inject_error)
+```
+
+### Replay Tooling
+
+Replay tools allow engineers to re-run pipeline logic against specific historical data or DLQ records without running the full orchestration stack:
+
+```bash
+#!/bin/bash
+# scripts/replay.sh — Replay pipeline execution for a specific time window
+
+set -euo pipefail
+
+PIPELINE="${1:-}"
+START_DATE="${2:-}"
+END_DATE="${3:-}"
+
+if [[ -z "$PIPELINE" || -z "$START_DATE" ]]; then
+    echo "Usage: replay.sh <pipeline_id> <start_date> [end_date]"
+    echo "  Example: replay.sh payments_transactions_ingest 2024-01-15"
+    exit 1
+fi
+
+echo "==> Replaying ${PIPELINE} from ${START_DATE} to ${END_DATE:-${START_DATE}}"
+
+python -m src.replay \
+    --pipeline "${PIPELINE}" \
+    --start "${START_DATE}" \
+    --end "${END_DATE:-${START_DATE}}" \
+    --env local \
+    --dry-run false
+```
+
+Python replay runner:
+```python
+# src/replay.py
+def replay_pipeline(
+    pipeline_id: str,
+    start: date,
+    end: date,
+    env: str = "local",
+    dry_run: bool = True,
+) -> ReplayResult:
+    """Replay a pipeline for a given date range.
+
+    Runs the pipeline transformation logic directly against historical source data
+    without going through the orchestrator. Safe to run against production sources
+    in read-only mode (dry_run=True).
+    """
+    config = load_config(env)
+    pipeline = load_pipeline(pipeline_id, config)
+
+    results = []
+    for partition_date in date_range(start, end):
+        records = pipeline.source.read(partition_date, partition_date + timedelta(days=1))
+        transformed = pipeline.transform(records)
+        quality_result = pipeline.quality.check(transformed)
+
+        if not dry_run:
+            pipeline.sink.write(transformed, partition=str(partition_date))
+
+        results.append(PartitionResult(
+            date=partition_date,
+            records_read=len(records),
+            records_written=len(transformed) if not dry_run else 0,
+            quality_passed=quality_result.passed,
+            quality_failures=quality_result.failures,
+        ))
+
+    return ReplayResult(partitions=results)
+```
+
+### DLQ Replay
+
+A specific replay workflow for DLQ records:
+
+```bash
+# scripts/replay-dlq.sh — Replay records from dead-letter queue
+
+set -euo pipefail
+
+PIPELINE="${1:-}"
+DLQ_DATE="${2:-$(date -u +%Y-%m-%d)}"
+
+echo "==> Replaying DLQ for ${PIPELINE} on ${DLQ_DATE}"
+
+python -m src.dlq_replay \
+    --pipeline "${PIPELINE}" \
+    --date "${DLQ_DATE}" \
+    --env local
+```
+
+### Test Fixtures
+
+Test fixtures are small, static datasets checked into version control:
+
+```
+tests/fixtures/
+├── payments/
+│   ├── raw/
+│   │   ├── valid_transactions.json       # 50 valid records
+│   │   ├── invalid_amount.json           # Records with bad amount values
+│   │   ├── missing_user_id.json          # Records with null required fields
+│   │   └── duplicate_transactions.json   # Duplicate records for dedup testing
+│   └── expected/
+│       ├── normalized_transactions.json  # Expected output after normalization
+│       └── aggregated_revenue.json       # Expected output after aggregation
+└── users/
+    ├── raw/
+    │   └── user_events.json
+    └── expected/
+        └── deduplicated_events.json
+```
+
+Fixture size guidelines:
+- Minimum: 10 records (enough to test logic)
+- Maximum: 500 records (keeps test runtime fast)
+- Always include: at least one edge case per tested scenario (empty string, null, boundary value, duplicate)
+- Never include: real production data, PII, or data containing actual user information
+
+Fixture loading utility:
+```python
+def load_fixture(category: str, name: str) -> list[dict]:
+    """Load a test fixture from tests/fixtures/."""
+    path = Path(__file__).parent.parent / "fixtures" / category / f"{name}.json"
+    return json.loads(path.read_text())
+```
+
+### Environment Variable Management
+
+Use `.env.example` committed to the repo and `.env` gitignored:
+
+```bash
+# .env.example — copy to .env and fill in values
+PIPELINE_ENV=local
+KAFKA_BOOTSTRAP_SERVERS=localhost:9092
+SCHEMA_REGISTRY_URL=http://localhost:8081
+POSTGRES_URL=postgresql://pipeline:pipeline@localhost:5432/source_db
+S3_ENDPOINT=http://localhost:9000
+S3_ACCESS_KEY=minioadmin
+S3_SECRET_KEY=minioadmin
+```
+
+Never commit `.env` files. Never hardcode credentials in any source file.

package/content/knowledge/data-pipeline/data-pipeline-orchestration.md

@@ -0,0 +1,291 @@
+---
+name: data-pipeline-orchestration
+description: DAG vs event-driven vs scheduled orchestration, retry policies, SLA monitoring, and lineage tracking for data pipelines
+topics: [data-pipeline, orchestration, dag, event-driven, scheduling, retry-policies, sla-monitoring, lineage, airflow, prefect, dagster]
+---
+
+Pipeline orchestration is the control plane that schedules work, manages dependencies, handles failures, and provides visibility into pipeline health. The choice between schedule-driven, event-driven, and DAG-based orchestration determines operational complexity, latency characteristics, and how well the system degrades under load. Poor orchestration design — missing retries, no SLA monitoring, absent lineage — turns data pipelines into production support burdens.
+
+## Summary
+
+Use schedule-based orchestration for batch pipelines with predictable cadences. Use event-driven orchestration when downstream pipelines should trigger immediately after upstream completion rather than waiting for the next scheduled window. Define retry policies per error class — exponential backoff for transient failures, no retry for non-retriable errors. Monitor SLAs as first-class metrics. Track lineage from source to serving to enable impact analysis and root cause investigation.
+
+## Deep Guidance
+
+### Orchestration Models
+
+**Schedule-based orchestration**
+
+The orchestrator triggers pipelines on a cron schedule, regardless of whether upstream data is ready. This is the simplest model and works well when:
+- Data arrives on a predictable schedule (file drops at 2am, database exports at midnight)
+- Acceptable to occasionally process empty or incomplete datasets (the pipeline handles this gracefully)
+- Downstream consumers tolerate the scheduling delay
+
+```python
+# Airflow: hourly schedule
+@dag(schedule_interval="0 * * * *", ...)
+def hourly_pipeline():
+    ...
+
+# With data availability check: schedule + sensor hybrid
+@dag(schedule_interval="0 * * * *", ...)
+def hourly_pipeline_with_sensor():
+    wait = S3KeySensor(
+        task_id="wait_for_data",
+        bucket_name="raw-data",
+        bucket_key="events/{{ ds_nodash }}/{{ execution_date.hour:02d }}/data.parquet",
+        timeout=3600,
+        mode="reschedule",
+    )
+    process = PythonOperator(...)
+    wait >> process
+```
+
+**Event-driven orchestration**
+
+Pipelines trigger when upstream data becomes available, not on a fixed schedule. Reduces latency and eliminates the "waiting for the next schedule window" problem.
+
+```python
+# Dagster: asset-based event-driven orchestration
+@asset(
+    partitions_def=DailyPartitionsDefinition(start_date="2024-01-01"),
+    deps=["raw_transactions"],  # triggers when raw_transactions materializes
+)
+def silver_transactions(context, raw_transactions):
+    return transform(raw_transactions)
+
+# Prefect: event-triggered flow
+@flow
+def silver_pipeline():
+    ...
+
+# Register event trigger
+from prefect.events.automations import Automation, EventTrigger
+Automation(
+    name="silver-on-bronze-complete",
+    trigger=EventTrigger(
+        match={"prefect.resource.id": "bronze-pipeline"},
+        expect={"prefect.flow-run.Completed"},
+    ),
+    actions=[RunFlow(flow=silver_pipeline)],
+).save()
+```
+
+Event-driven orchestration reduces end-to-end latency from "up to schedule interval" to "seconds after upstream completion." The tradeoff is more complex dependency management and potential cascade failures.
+
+**DAG-based orchestration**
+
+A Directed Acyclic Graph explicitly models task dependencies. The orchestrator executes tasks in topological order, respecting all declared dependencies. DAG-based orchestration is the standard for complex multi-step pipelines with non-trivial dependency graphs.
+
+Airflow, Prefect, and Dagster all implement DAG-based orchestration with different trade-offs:
+
+| Feature | Airflow | Prefect | Dagster |
+|---------|---------|---------|---------|
+| Maturity | High | Medium | Medium |
+| Dynamic DAGs | Limited | Native | Native |
+| Local testing | Complex | Simple | Simple |
+| Asset tracking | External | Limited | Native |
+| Deployment model | Stateful scheduler | Cloud or self-hosted | Cloud or self-hosted |
+| Best for | Large teams, stable DAGs | Dynamic workflows | Asset-centric pipelines |
+
+### Retry Policy Design
+
+Every pipeline task must have an explicit retry policy. The policy must distinguish between retriable and non-retriable failures.
+
+**Retriable failures** — transient errors that resolve themselves:
+- Network timeouts
+- Rate limiting / throttling
+- Temporary service unavailability (HTTP 503)
+- Database lock contention
+
+**Non-retriable failures** — permanent errors requiring human intervention:
+- Schema validation errors
+- Missing required configuration
+- Data corruption detected
+- Authentication/authorization failure
+
+**Exponential backoff with jitter**
+
+```python
+from datetime import timedelta
+
+# Airflow: task-level retry configuration
+default_args = {
+    "retries": 5,
+    "retry_delay": timedelta(seconds=30),       # initial delay
+    "retry_exponential_backoff": True,           # double delay each retry
+    "max_retry_delay": timedelta(minutes=30),   # cap at 30 minutes
+}
+
+# Retry schedule:
+# Attempt 1: immediate
+# Attempt 2: 30 seconds later
+# Attempt 3: 60 seconds later
+# Attempt 4: 120 seconds later
+# Attempt 5: 240 seconds later
+# Attempt 6 (final): 480 seconds later (capped at 30 min)
+```
+
+Add jitter to prevent thundering herd — multiple failed tasks retrying at the same time:
+
+```python
+import random
+
+def get_retry_delay(attempt: int, base_seconds: int = 30, max_seconds: int = 1800) -> float:
+    exponential = base_seconds * (2 ** attempt)
+    capped = min(exponential, max_seconds)
+    jitter = random.uniform(0, capped * 0.1)  # 10% jitter
+    return capped + jitter
+```
+
+**Classifying and routing failures**
+
+```python
+def task_with_classified_retries():
+    try:
+        execute_pipeline_step()
+    except RateLimitError as e:
+        # Retriable: will retry with backoff
+        raise AirflowException(f"Rate limited: {e}") from e
+    except SchemaValidationError as e:
+        # Non-retriable: send to DLQ and mark task as failed (no retry)
+        dlq.write(current_record, e, context)
+        raise AirflowSkipException(f"Validation failed — sent to DLQ: {e}") from e
+    except PoisonPillError as e:
+        # Non-retriable: alert immediately
+        alert_oncall(f"Poison pill detected: {e}")
+        raise AirflowException(f"Poison pill — manual intervention required") from e
+```
+
+### SLA Monitoring
+
+SLA monitoring detects when pipelines are running late before downstream consumers notice stale data.
+
+**Airflow SLA callbacks**
+
+```python
+def sla_miss_callback(dag, task_list, blocking_task_list, slas, blocking_tis):
+    """Called when a task misses its SLA deadline."""
+    message = (
+        f"SLA miss in {dag.dag_id}:\n"
+        f"Tasks: {[t.task_id for t in task_list]}\n"
+        f"Blocking: {[t.task_id for t in blocking_task_list]}"
+    )
+    pagerduty.trigger_alert(severity="warning", message=message)
+    slack.post_message(channel="#data-alerts", text=message)
+
+@dag(
+    sla_miss_callback=sla_miss_callback,
+    default_args={
+        "sla": timedelta(hours=2),  # alert if task not done in 2 hours
+    },
+)
+def payments_pipeline():
+    ...
+```
+
+**Freshness SLA as a metric**
+
+Emit a gauge metric tracking data age in the serving layer. Alert when data age exceeds the SLA:
+
+```python
+def check_data_freshness(table: str, sla_minutes: int) -> None:
+    """Check that a table has been updated within the SLA window."""
+    last_updated = query_last_updated_timestamp(table)
+    age_minutes = (datetime.utcnow() - last_updated).total_seconds() / 60
+
+    metrics.gauge(f"data_freshness_minutes", age_minutes, tags={"table": table})
+
+    if age_minutes > sla_minutes:
+        alert_oncall(
+            f"SLA breach: {table} is {age_minutes:.0f} minutes old (SLA: {sla_minutes} min)"
+        )
+```
+
+**SLA dashboard metrics to track**
+
+- Time-to-complete per DAG run (p50, p95, p99)
+- SLA miss count per pipeline per day
+- Data freshness age per serving table
+- Queue depth (pending tasks waiting to run)
+- Failure rate per pipeline (failures per 100 runs)
+
+### Lineage Tracking
+
+Data lineage records how data flows from sources through transformations to serving layers. Lineage enables:
+- Impact analysis: "Which dashboards will break if I change this table's schema?"
+- Root cause investigation: "Which source system produced this bad value?"
+- Compliance: "Which pipelines process PII from this source?"
+- Dependency-aware deployments: "Can I deploy this pipeline change safely?"
+
+**Automated lineage via orchestration metadata**
+
+Dagster natively tracks asset lineage:
+
+```python
+@asset
+def raw_transactions():
+    """Bronze: raw transactions from Stripe API."""
+    return stripe.read_charges()
+
+@asset(deps=["raw_transactions"])
+def silver_transactions(raw_transactions):
+    """Silver: normalized and validated transactions."""
+    return transforms.normalize(raw_transactions)
+
+@asset(deps=["silver_transactions"])
+def daily_revenue(silver_transactions):
+    """Gold: daily revenue aggregation."""
+    return aggregations.daily_revenue(silver_transactions)
+```
+
+Dagster generates a lineage graph automatically from the `deps` declarations, viewable in the UI.
+
+**Custom lineage for Airflow**
+
+When using Airflow without a native lineage tool, emit OpenLineage events:
+
+```python
+from openlineage.airflow import DAG
+from openlineage.client.facet import SchemaDatasetFacet
+
+# OpenLineage-instrumented DAG emits START/COMPLETE events
+# consumed by Marquez, DataHub, or Apache Atlas
+@dag(...)
+def instrumented_pipeline():
+    ...
+```
+
+**Lineage in incident response**
+
+When a data quality incident occurs, the lineage graph drives the investigation:
+1. Identify the affected output table
+2. Traverse lineage backwards to find the upstream source
+3. Check pipeline run history for failures or anomalies at each stage
+4. Identify the transformation step where the bad data was introduced
+5. Trace back to the source event or record that caused the issue
+
+Without lineage, this investigation is manual, slow, and often incomplete. With lineage, it takes minutes.
+
+### Orchestrator Operational Practices
+
+**Deployment without DAG downtime**
+
+In Airflow, deploying a new DAG version while runs are in progress can cause orphaned tasks. Use versioned DAG IDs for breaking changes:
+
+```python
+# Before: payments_ingest_v1
+# After breaking change: payments_ingest_v2
+# Run both in parallel during transition; decommission v1 after drain
+```
+
+**Worker pool sizing**
+
+Size worker pools based on peak pipeline parallelism:
+- Calculate maximum simultaneous tasks across all DAGs at peak hours
+- Add 20% headroom for backfills and reruns
+- Separate pools for CPU-intensive tasks (Spark submit) vs. I/O-bound tasks (API calls)
+
+**Scheduler high availability**
+
+Run multiple scheduler instances with leader election. Airflow 2.x supports HA schedulers natively. Single-scheduler deployments are a production risk.