aigroup-workflow 1.0.0

package/skills/jarvis/engineering-team/engineering-team/senior-data-engineer/references/data_pipeline_architecture.md

@@ -0,0 +1,1100 @@
+# Data Pipeline Architecture
+
+Comprehensive guide to designing and implementing production data pipelines.
+
+## Table of Contents
+
+1. [Architecture Patterns](#architecture-patterns)
+2. [Batch Processing](#batch-processing)
+3. [Stream Processing](#stream-processing)
+4. [Exactly-Once Semantics](#exactly-once-semantics)
+5. [Error Handling](#error-handling)
+6. [Data Ingestion Patterns](#data-ingestion-patterns)
+7. [Orchestration](#orchestration)
+
+---
+
+## Architecture Patterns
+
+### Lambda Architecture
+
+The Lambda architecture combines batch and stream processing for comprehensive data handling.
+
+```
+                    ┌─────────────────────────────────────┐
+                    │           Data Sources              │
+                    └─────────────────┬───────────────────┘
+                                      │
+                    ┌─────────────────▼───────────────────┐
+                    │         Message Queue (Kafka)        │
+                    └───────┬─────────────────┬───────────┘
+                            │                 │
+              ┌─────────────▼─────┐   ┌───────▼─────────────┐
+              │    Batch Layer    │   │    Speed Layer      │
+              │   (Spark/Airflow) │   │   (Flink/Spark SS)  │
+              └─────────────┬─────┘   └───────┬─────────────┘
+                            │                 │
+              ┌─────────────▼─────┐   ┌───────▼─────────────┐
+              │   Master Dataset  │   │   Real-time Views   │
+              │   (Data Lake)     │   │   (Redis/Druid)     │
+              └─────────────┬─────┘   └───────┬─────────────┘
+                            │                 │
+                    ┌───────▼─────────────────▼───────┐
+                    │          Serving Layer           │
+                    │   (Merged Batch + Real-time)     │
+                    └─────────────────────────────────┘
+```
+
+**Components:**
+
+1. **Batch Layer**
+   - Processes complete historical data
+   - Creates precomputed batch views
+   - Handles complex transformations, ML training
+   - Reprocessable from raw data
+
+2. **Speed Layer**
+   - Processes real-time data stream
+   - Creates real-time views for recent data
+   - Low latency, simpler transformations
+   - Compensates for batch layer delay
+
+3. **Serving Layer**
+   - Merges batch and real-time views
+   - Responds to queries
+   - Provides unified interface
+
+**Implementation Example:**
+
+```python
+# Batch layer: Daily aggregation with Spark
+def batch_daily_aggregation(spark, date):
+    """Process full day of data for batch views."""
+    raw_df = spark.read.parquet(f"s3://data-lake/raw/events/date={date}")
+
+    aggregated = raw_df.groupBy("user_id", "event_type") \
+        .agg(
+            count("*").alias("event_count"),
+            sum("revenue").alias("total_revenue"),
+            max("timestamp").alias("last_event")
+        )
+
+    aggregated.write \
+        .mode("overwrite") \
+        .partitionBy("event_type") \
+        .parquet(f"s3://data-lake/batch-views/daily_agg/date={date}")
+
+# Speed layer: Real-time aggregation with Spark Structured Streaming
+def speed_realtime_aggregation(spark):
+    """Process streaming data for real-time views."""
+    stream_df = spark.readStream \
+        .format("kafka") \
+        .option("kafka.bootstrap.servers", "kafka:9092") \
+        .option("subscribe", "events") \
+        .load()
+
+    parsed = stream_df.select(
+        from_json(col("value").cast("string"), event_schema).alias("data")
+    ).select("data.*")
+
+    aggregated = parsed \
+        .withWatermark("timestamp", "5 minutes") \
+        .groupBy(
+            window("timestamp", "1 minute"),
+            "user_id",
+            "event_type"
+        ) \
+        .agg(count("*").alias("event_count"))
+
+    query = aggregated.writeStream \
+        .format("redis") \
+        .option("host", "redis") \
+        .outputMode("update") \
+        .start()
+
+    return query
+```
+
+### Kappa Architecture
+
+Kappa simplifies Lambda by using only stream processing with replay capability.
+
+```
+                    ┌─────────────────────────────────────┐
+                    │           Data Sources              │
+                    └─────────────────┬───────────────────┘
+                                      │
+                    ┌─────────────────▼───────────────────┐
+                    │    Immutable Log (Kafka/Kinesis)     │
+                    │         (Long retention)             │
+                    └─────────────────┬───────────────────┘
+                                      │
+                    ┌─────────────────▼───────────────────┐
+                    │        Stream Processor              │
+                    │      (Flink/Spark Streaming)         │
+                    └─────────────────┬───────────────────┘
+                                      │
+                    ┌─────────────────▼───────────────────┐
+                    │         Serving Layer                │
+                    │    (Database/Data Warehouse)         │
+                    └─────────────────────────────────────┘
+```
+
+**Key Principles:**
+
+1. **Single Processing Path**: All data processed as streams
+2. **Immutable Log**: Kafka/Kinesis as source of truth with long retention
+3. **Reprocessing via Replay**: Re-run stream processor from beginning when needed
+
+**Reprocessing Strategy:**
+
+```python
+# Reprocessing in Kappa architecture
+class KappaReprocessor:
+    """Handle reprocessing by replaying from Kafka."""
+
+    def __init__(self, kafka_config, flink_job):
+        self.kafka = kafka_config
+        self.job = flink_job
+
+    def reprocess(self, from_timestamp: str):
+        """Reprocess all data from a specific timestamp."""
+
+        # 1. Start new consumer group reading from timestamp
+        new_consumer_group = f"reprocess-{uuid.uuid4()}"
+
+        # 2. Configure stream processor with new group
+        self.job.set_config({
+            "group.id": new_consumer_group,
+            "auto.offset.reset": "none"  # We'll set offset manually
+        })
+
+        # 3. Seek to timestamp
+        offsets = self._get_offsets_for_timestamp(from_timestamp)
+        self.job.seek_to_offsets(offsets)
+
+        # 4. Write to new output table/topic
+        output_table = f"events_reprocessed_{datetime.now().strftime('%Y%m%d')}"
+        self.job.set_output(output_table)
+
+        # 5. Run until caught up
+        self.job.run_until_caught_up()
+
+        # 6. Swap output tables atomically
+        self._atomic_table_swap("events", output_table)
+
+    def _get_offsets_for_timestamp(self, timestamp):
+        """Get Kafka offsets for a specific timestamp."""
+        consumer = KafkaConsumer(bootstrap_servers=self.kafka["brokers"])
+        partitions = consumer.partitions_for_topic("events")
+
+        offsets = {}
+        for partition in partitions:
+            tp = TopicPartition("events", partition)
+            offset = consumer.offsets_for_times({tp: timestamp})
+            offsets[tp] = offset[tp].offset
+
+        return offsets
+```
+
+### Medallion Architecture (Bronze/Silver/Gold)
+
+Common in data lakehouses (Databricks, Delta Lake).
+
+```
+┌─────────────┐     ┌─────────────┐     ┌─────────────┐
+│   Bronze    │────▶│   Silver    │────▶│    Gold     │
+│  (Raw Data) │     │ (Cleansed)  │     │ (Analytics) │
+└─────────────┘     └─────────────┘     └─────────────┘
+     │                    │                    │
+     ▼                    ▼                    ▼
+ Landing zone        Validated,          Aggregated,
+ Append-only         deduplicated,       business-ready
+ Schema evolution    standardized        Star schema
+```
+
+**Implementation with Delta Lake:**
+
+```python
+# Bronze: Raw ingestion
+def ingest_to_bronze(spark, source_path, bronze_path):
+    """Ingest raw data to bronze layer."""
+    df = spark.read.format("json").load(source_path)
+
+    # Add metadata
+    df = df.withColumn("_ingested_at", current_timestamp()) \
+           .withColumn("_source_file", input_file_name())
+
+    df.write \
+        .format("delta") \
+        .mode("append") \
+        .option("mergeSchema", "true") \
+        .save(bronze_path)
+
+# Silver: Cleansing and validation
+def bronze_to_silver(spark, bronze_path, silver_path):
+    """Transform bronze to silver with cleansing."""
+    bronze_df = spark.read.format("delta").load(bronze_path)
+
+    # Read last processed version
+    last_version = get_last_processed_version(silver_path, "bronze")
+
+    # Get only new records
+    new_records = bronze_df.filter(col("_commit_version") > last_version)
+
+    # Cleanse and validate
+    silver_df = new_records \
+        .filter(col("user_id").isNotNull()) \
+        .filter(col("event_type").isin(["click", "view", "purchase"])) \
+        .withColumn("event_date", to_date("timestamp")) \
+        .dropDuplicates(["event_id"])
+
+    # Merge to silver (upsert)
+    silver_table = DeltaTable.forPath(spark, silver_path)
+
+    silver_table.alias("target") \
+        .merge(
+            silver_df.alias("source"),
+            "target.event_id = source.event_id"
+        ) \
+        .whenMatchedUpdateAll() \
+        .whenNotMatchedInsertAll() \
+        .execute()
+
+# Gold: Business aggregations
+def silver_to_gold(spark, silver_path, gold_path):
+    """Create business-ready aggregations in gold layer."""
+    silver_df = spark.read.format("delta").load(silver_path)
+
+    # Daily user metrics
+    daily_metrics = silver_df \
+        .groupBy("user_id", "event_date") \
+        .agg(
+            count("*").alias("total_events"),
+            countDistinct("session_id").alias("sessions"),
+            sum(when(col("event_type") == "purchase", col("revenue")).otherwise(0)).alias("revenue"),
+            max("timestamp").alias("last_activity")
+        )
+
+    # Write as gold table
+    daily_metrics.write \
+        .format("delta") \
+        .mode("overwrite") \
+        .partitionBy("event_date") \
+        .save(gold_path + "/daily_user_metrics")
+```
+
+---
+
+## Batch Processing
+
+### Apache Spark Best Practices
+
+#### Memory Management
+
+```python
+# Optimal Spark configuration for batch jobs
+spark = SparkSession.builder \
+    .appName("BatchETL") \
+    .config("spark.executor.memory", "8g") \
+    .config("spark.executor.cores", "4") \
+    .config("spark.driver.memory", "4g") \
+    .config("spark.sql.shuffle.partitions", "200") \
+    .config("spark.sql.adaptive.enabled", "true") \
+    .config("spark.sql.adaptive.coalescePartitions.enabled", "true") \
+    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") \
+    .getOrCreate()
+```
+
+**Memory Tuning Guidelines:**
+
+| Data Size | Executors | Memory/Executor | Cores/Executor |
+|-----------|-----------|-----------------|----------------|
+| < 10 GB   | 2-4       | 4-8 GB          | 2-4            |
+| 10-100 GB | 10-20     | 8-16 GB         | 4-8            |
+| 100+ GB   | 50+       | 16-32 GB        | 4-8            |
+
+#### Partition Optimization
+
+```python
+# Repartition vs Coalesce
+# Repartition: Full shuffle, use for increasing partitions
+df_repartitioned = df.repartition(100, "date")  # Partition by column
+
+# Coalesce: No shuffle, use for decreasing partitions
+df_coalesced = df.coalesce(10)  # Reduce partitions without shuffle
+
+# Optimal partition size: 128-256 MB each
+# Calculate partitions:
+# num_partitions = total_data_size_mb / 200
+
+# Check current partitions
+print(f"Current partitions: {df.rdd.getNumPartitions()}")
+
+# Repartition for optimal join performance
+large_df = large_df.repartition(200, "join_key")
+small_df = small_df.repartition(200, "join_key")
+result = large_df.join(small_df, "join_key")
+```
+
+#### Join Optimization
+
+```python
+# Broadcast join for small tables (< 10MB by default)
+from pyspark.sql.functions import broadcast
+
+# Explicit broadcast hint
+result = large_df.join(broadcast(small_df), "key")
+
+# Increase broadcast threshold if needed
+spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "100m")
+
+# Sort-merge join for large tables
+spark.conf.set("spark.sql.join.preferSortMergeJoin", "true")
+
+# Bucket tables for frequent joins
+df.write \
+    .bucketBy(100, "customer_id") \
+    .sortBy("customer_id") \
+    .mode("overwrite") \
+    .saveAsTable("bucketed_orders")
+```
+
+#### Caching Strategy
+
+```python
+# Cache when:
+# 1. DataFrame is used multiple times
+# 2. After expensive transformations
+# 3. Before iterative operations
+
+# Use MEMORY_AND_DISK for large datasets
+from pyspark import StorageLevel
+
+df.persist(StorageLevel.MEMORY_AND_DISK)
+
+# Cache only necessary columns
+df.select("id", "value").cache()
+
+# Unpersist when done
+df.unpersist()
+
+# Check storage
+spark.catalog.clearCache()  # Clear all caches
+```
+
+### Airflow DAG Patterns
+
+#### Idempotent Tasks
+
+```python
+# Always design idempotent tasks
+from airflow.decorators import dag, task
+from airflow.utils.dates import days_ago
+from datetime import timedelta
+
+@dag(
+    schedule_interval="@daily",
+    start_date=days_ago(7),
+    catchup=True,
+    default_args={
+        "retries": 3,
+        "retry_delay": timedelta(minutes=5),
+    }
+)
+def idempotent_etl():
+
+    @task
+    def extract(execution_date=None):
+        """Idempotent extraction - same date always returns same data."""
+        date_str = execution_date.strftime("%Y-%m-%d")
+
+        # Query for specific date only
+        query = f"""
+            SELECT * FROM source_table
+            WHERE DATE(created_at) = '{date_str}'
+        """
+        return query_database(query)
+
+    @task
+    def transform(data):
+        """Pure function - no side effects."""
+        return [transform_record(r) for r in data]
+
+    @task
+    def load(data, execution_date=None):
+        """Idempotent load - delete before insert or use MERGE."""
+        date_str = execution_date.strftime("%Y-%m-%d")
+
+        # Option 1: Delete and reinsert
+        execute_sql(f"DELETE FROM target WHERE date = '{date_str}'")
+        insert_data(data)
+
+        # Option 2: Use MERGE/UPSERT
+        # MERGE INTO target USING source ON target.id = source.id
+        # WHEN MATCHED THEN UPDATE
+        # WHEN NOT MATCHED THEN INSERT
+
+    raw = extract()
+    transformed = transform(raw)
+    load(transformed)
+
+dag = idempotent_etl()
+```
+
+#### Backfill Pattern
+
+```python
+from airflow import DAG
+from airflow.operators.python import PythonOperator
+from airflow.utils.dates import days_ago
+from datetime import datetime, timedelta
+
+def process_date(ds, **kwargs):
+    """Process a single date - supports backfill."""
+    logical_date = datetime.strptime(ds, "%Y-%m-%d")
+
+    # Always process specific date, not "latest"
+    data = extract_for_date(logical_date)
+    transformed = transform(data)
+
+    # Use partition/date-specific target
+    load_to_partition(transformed, partition=ds)
+
+with DAG(
+    "backfillable_etl",
+    schedule_interval="@daily",
+    start_date=datetime(2024, 1, 1),
+    catchup=True,  # Enable backfill
+    max_active_runs=3,  # Limit parallel backfills
+) as dag:
+
+    process = PythonOperator(
+        task_id="process",
+        python_callable=process_date,
+        provide_context=True,
+    )
+
+# Backfill command:
+# airflow dags backfill -s 2024-01-01 -e 2024-01-31 backfillable_etl
+```
+
+---
+
+## Stream Processing
+
+### Apache Kafka Architecture
+
+#### Topic Design
+
+```bash
+# Create topic with proper configuration
+kafka-topics.sh --create \
+    --bootstrap-server localhost:9092 \
+    --topic user-events \
+    --partitions 24 \
+    --replication-factor 3 \
+    --config retention.ms=604800000 \        # 7 days
+    --config retention.bytes=107374182400 \  # 100GB
+    --config cleanup.policy=delete \
+    --config min.insync.replicas=2 \         # Durability
+    --config segment.bytes=1073741824        # 1GB segments
+```
+
+**Partition Count Guidelines:**
+
+| Throughput | Partitions | Notes |
+|------------|------------|-------|
+| < 10K msg/s | 6-12 | Single consumer can handle |
+| 10K-100K msg/s | 24-48 | Multiple consumers needed |
+| > 100K msg/s | 100+ | Scale consumers with partitions |
+
+**Partition Key Selection:**
+
+```python
+# Good partition keys: Even distribution, related data together
+# For user events: user_id (events for same user on same partition)
+# For orders: order_id (if no ordering needed) or customer_id (if needed)
+
+from kafka import KafkaProducer
+import json
+
+producer = KafkaProducer(
+    bootstrap_servers=['localhost:9092'],
+    value_serializer=lambda v: json.dumps(v).encode('utf-8'),
+    key_serializer=lambda k: k.encode('utf-8')
+)
+
+def send_event(event):
+    # Use user_id as key for user-based partitioning
+    producer.send(
+        topic='user-events',
+        key=event['user_id'],  # Partition key
+        value=event
+    )
+```
+
+### Spark Structured Streaming
+
+#### Watermarks and Late Data
+
+```python
+from pyspark.sql.functions import window, col
+
+# Read stream
+events = spark.readStream \
+    .format("kafka") \
+    .option("kafka.bootstrap.servers", "localhost:9092") \
+    .option("subscribe", "events") \
+    .load() \
+    .select(from_json(col("value").cast("string"), schema).alias("data")) \
+    .select("data.*")
+
+# Add watermark for late data handling
+# Data arriving more than 10 minutes late will be dropped
+windowed_counts = events \
+    .withWatermark("event_time", "10 minutes") \
+    .groupBy(
+        window("event_time", "5 minutes", "1 minute"),  # 5-min windows, 1-min slide
+        "event_type"
+    ) \
+    .count()
+
+# Write with append mode (only final results for complete windows)
+query = windowed_counts.writeStream \
+    .format("delta") \
+    .outputMode("append") \
+    .option("checkpointLocation", "/checkpoints/windowed_counts") \
+    .start()
+```
+
+**Watermark Behavior:**
+
+```
+Timeline: ─────────────────────────────────────────▶
+Events:   E1   E2   E3        E4(late)    E5
+          │    │    │         │           │
+Time:     10:00 10:02 10:05   10:03       10:15
+          ▲                   ▲
+          │                   │
+          Current            Arrives at 10:15
+          watermark          but event_time=10:03
+          = max_event_time
+            - threshold
+          = 10:05 - 10min    If watermark > event_time:
+          = 9:55               Event is dropped (too late)
+```
+
+#### Stateful Operations
+
+```python
+from pyspark.sql.functions import pandas_udf, PandasUDFType
+from pyspark.sql.streaming.state import GroupState, GroupStateTimeout
+
+# Session windows using flatMapGroupsWithState
+def session_aggregation(key, events, state):
+    """Aggregate events into sessions with 30-minute timeout."""
+
+    # Get or initialize state
+    if state.exists:
+        session = state.get
+    else:
+        session = {"start": None, "events": [], "total": 0}
+
+    # Process new events
+    for event in events:
+        if session["start"] is None:
+            session["start"] = event.timestamp
+        session["events"].append(event)
+        session["total"] += event.value
+
+    # Set timeout (session expires after 30 min of inactivity)
+    state.setTimeoutDuration("30 minutes")
+
+    # Check if session should close
+    if state.hasTimedOut():
+        # Emit completed session
+        output = {
+            "user_id": key,
+            "session_start": session["start"],
+            "event_count": len(session["events"]),
+            "total_value": session["total"]
+        }
+        state.remove()
+        yield output
+    else:
+        # Update state
+        state.update(session)
+
+# Apply stateful operation
+sessions = events \
+    .groupByKey(lambda e: e.user_id) \
+    .flatMapGroupsWithState(
+        session_aggregation,
+        outputMode="append",
+        stateTimeout=GroupStateTimeout.ProcessingTimeTimeout()
+    )
+```
+
+---
+
+## Exactly-Once Semantics
+
+### Producer Idempotence
+
+```python
+from kafka import KafkaProducer
+
+# Enable idempotent producer
+producer = KafkaProducer(
+    bootstrap_servers=['localhost:9092'],
+    acks='all',                    # Wait for all replicas
+    enable_idempotence=True,       # Exactly-once per partition
+    max_in_flight_requests_per_connection=5,  # Max with idempotence
+    retries=2147483647,            # Infinite retries
+    value_serializer=lambda v: json.dumps(v).encode('utf-8')
+)
+
+# Producer will deduplicate based on sequence numbers
+for i in range(100):
+    producer.send('topic', {'id': i, 'data': 'value'})
+
+producer.flush()
+```
+
+### Transactional Processing
+
+```python
+from kafka import KafkaProducer, KafkaConsumer
+from kafka.errors import KafkaError
+
+# Transactional producer
+producer = KafkaProducer(
+    bootstrap_servers=['localhost:9092'],
+    transactional_id='my-transactional-id',  # Enable transactions
+    enable_idempotence=True,
+    acks='all'
+)
+
+producer.init_transactions()
+
+def process_with_transactions(consumer, producer):
+    """Read-process-write with exactly-once semantics."""
+
+    try:
+        producer.begin_transaction()
+
+        # Read
+        records = consumer.poll(timeout_ms=1000)
+
+        for tp, messages in records.items():
+            for message in messages:
+                # Process
+                result = transform(message.value)
+
+                # Write to output topic
+                producer.send('output-topic', result)
+
+        # Commit offsets and transaction atomically
+        producer.send_offsets_to_transaction(
+            consumer.position(consumer.assignment()),
+            consumer.group_id
+        )
+        producer.commit_transaction()
+
+    except KafkaError as e:
+        producer.abort_transaction()
+        raise
+```
+
+### Spark Exactly-Once to External Systems
+
+```python
+# Use foreachBatch with idempotent writes
+def write_to_database_idempotent(batch_df, batch_id):
+    """Write batch with exactly-once semantics."""
+
+    # Add batch_id for deduplication
+    batch_with_id = batch_df.withColumn("batch_id", lit(batch_id))
+
+    # Use MERGE for idempotent writes
+    batch_with_id.write \
+        .format("jdbc") \
+        .option("url", "jdbc:postgresql://localhost/db") \
+        .option("dbtable", "staging_events") \
+        .option("driver", "org.postgresql.Driver") \
+        .mode("append") \
+        .save()
+
+    # Merge staging to final (idempotent)
+    execute_sql("""
+        MERGE INTO events AS target
+        USING staging_events AS source
+        ON target.event_id = source.event_id
+        WHEN MATCHED THEN UPDATE SET *
+        WHEN NOT MATCHED THEN INSERT *
+    """)
+
+    # Clean staging
+    execute_sql("TRUNCATE staging_events")
+
+query = events.writeStream \
+    .foreachBatch(write_to_database_idempotent) \
+    .option("checkpointLocation", "/checkpoints/to-postgres") \
+    .start()
+```
+
+---
+
+## Error Handling
+
+### Dead Letter Queue (DLQ)
+
+```python
+class DeadLetterQueue:
+    """Handle failed records with dead letter queue pattern."""
+
+    def __init__(self, dlq_topic: str, producer: KafkaProducer):
+        self.dlq_topic = dlq_topic
+        self.producer = producer
+
+    def send_to_dlq(self, record, error: Exception, context: dict):
+        """Send failed record to DLQ with error metadata."""
+
+        dlq_record = {
+            "original_record": record,
+            "error_type": type(error).__name__,
+            "error_message": str(error),
+            "timestamp": datetime.utcnow().isoformat(),
+            "context": context,
+            "retry_count": context.get("retry_count", 0)
+        }
+
+        self.producer.send(
+            self.dlq_topic,
+            value=json.dumps(dlq_record).encode('utf-8')
+        )
+
+def process_with_dlq(consumer, processor, dlq):
+    """Process records with DLQ for failures."""
+
+    for message in consumer:
+        try:
+            result = processor.process(message.value)
+            # Success - commit offset
+            consumer.commit()
+
+        except ValidationError as e:
+            # Non-retryable - send to DLQ immediately
+            dlq.send_to_dlq(
+                message.value,
+                e,
+                {"topic": message.topic, "partition": message.partition}
+            )
+            consumer.commit()  # Don't retry
+
+        except TemporaryError as e:
+            # Retryable - don't commit, let consumer retry
+            # After max retries, send to DLQ
+            retry_count = message.headers.get("retry_count", 0)
+            if retry_count >= MAX_RETRIES:
+                dlq.send_to_dlq(message.value, e, {"retry_count": retry_count})
+                consumer.commit()
+            else:
+                raise  # Will be retried
+```
+
+### Circuit Breaker
+
+```python
+from dataclasses import dataclass
+from datetime import datetime, timedelta
+from enum import Enum
+import threading
+
+class CircuitState(Enum):
+    CLOSED = "closed"      # Normal operation
+    OPEN = "open"          # Failing, reject calls
+    HALF_OPEN = "half_open"  # Testing if recovered
+
+@dataclass
+class CircuitBreaker:
+    """Circuit breaker for external service calls."""
+
+    failure_threshold: int = 5
+    recovery_timeout: timedelta = timedelta(seconds=30)
+    success_threshold: int = 3
+
+    def __post_init__(self):
+        self.state = CircuitState.CLOSED
+        self.failure_count = 0
+        self.success_count = 0
+        self.last_failure_time = None
+        self.lock = threading.Lock()
+
+    def call(self, func, *args, **kwargs):
+        """Execute function with circuit breaker protection."""
+
+        with self.lock:
+            if self.state == CircuitState.OPEN:
+                if self._should_attempt_reset():
+                    self.state = CircuitState.HALF_OPEN
+                else:
+                    raise CircuitOpenError("Circuit is open")
+
+        try:
+            result = func(*args, **kwargs)
+            self._record_success()
+            return result
+
+        except Exception as e:
+            self._record_failure()
+            raise
+
+    def _record_success(self):
+        with self.lock:
+            if self.state == CircuitState.HALF_OPEN:
+                self.success_count += 1
+                if self.success_count >= self.success_threshold:
+                    self.state = CircuitState.CLOSED
+                    self.failure_count = 0
+                    self.success_count = 0
+            elif self.state == CircuitState.CLOSED:
+                self.failure_count = 0
+
+    def _record_failure(self):
+        with self.lock:
+            self.failure_count += 1
+            self.last_failure_time = datetime.now()
+
+            if self.state == CircuitState.HALF_OPEN:
+                self.state = CircuitState.OPEN
+                self.success_count = 0
+            elif self.failure_count >= self.failure_threshold:
+                self.state = CircuitState.OPEN
+
+    def _should_attempt_reset(self):
+        if self.last_failure_time is None:
+            return True
+        return datetime.now() - self.last_failure_time >= self.recovery_timeout
+
+# Usage
+circuit = CircuitBreaker(failure_threshold=5, recovery_timeout=timedelta(seconds=60))
+
+def call_external_api(data):
+    return circuit.call(external_api.process, data)
+```
+
+---
+
+## Data Ingestion Patterns
+
+### Change Data Capture (CDC)
+
+```python
+# Using Debezium with Kafka Connect
+# connector-config.json
+{
+    "name": "postgres-cdc-connector",
+    "config": {
+        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
+        "database.hostname": "postgres",
+        "database.port": "5432",
+        "database.user": "debezium",
+        "database.password": "password",
+        "database.dbname": "source_db",
+        "database.server.name": "source",
+        "table.include.list": "public.orders,public.customers",
+        "plugin.name": "pgoutput",
+        "publication.name": "dbz_publication",
+        "slot.name": "debezium_slot",
+        "transforms": "unwrap",
+        "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
+        "transforms.unwrap.drop.tombstones": "false"
+    }
+}
+```
+
+**Processing CDC Events:**
+
+```python
+def process_cdc_event(event):
+    """Process Debezium CDC event."""
+
+    operation = event.get("op")
+
+    if operation == "c":  # Create (INSERT)
+        after = event.get("after")
+        return {"action": "insert", "data": after}
+
+    elif operation == "u":  # Update
+        before = event.get("before")
+        after = event.get("after")
+        return {"action": "update", "before": before, "after": after}
+
+    elif operation == "d":  # Delete
+        before = event.get("before")
+        return {"action": "delete", "data": before}
+
+    elif operation == "r":  # Read (snapshot)
+        after = event.get("after")
+        return {"action": "snapshot", "data": after}
+```
+
+### Bulk Ingestion
+
+```python
+# Efficient bulk loading to data warehouse
+from concurrent.futures import ThreadPoolExecutor
+import boto3
+
+class BulkIngester:
+    """Bulk ingest data to Snowflake via S3."""
+
+    def __init__(self, s3_bucket: str, snowflake_conn):
+        self.s3 = boto3.client('s3')
+        self.bucket = s3_bucket
+        self.snowflake = snowflake_conn
+
+    def ingest_dataframe(self, df, table_name: str, mode: str = "append"):
+        """Bulk ingest DataFrame to Snowflake."""
+
+        # 1. Write to S3 as Parquet (compressed, columnar)
+        s3_path = f"s3://{self.bucket}/staging/{table_name}/{uuid.uuid4()}"
+        df.write.parquet(s3_path)
+
+        # 2. Create external stage if not exists
+        self.snowflake.execute(f"""
+            CREATE STAGE IF NOT EXISTS {table_name}_stage
+            URL = '{s3_path}'
+            CREDENTIALS = (AWS_KEY_ID='...' AWS_SECRET_KEY='...')
+            FILE_FORMAT = (TYPE = 'PARQUET')
+        """)
+
+        # 3. COPY INTO (much faster than INSERT)
+        if mode == "overwrite":
+            self.snowflake.execute(f"TRUNCATE TABLE {table_name}")
+
+        self.snowflake.execute(f"""
+            COPY INTO {table_name}
+            FROM @{table_name}_stage
+            FILE_FORMAT = (TYPE = 'PARQUET')
+            MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE
+            ON_ERROR = 'CONTINUE'
+        """)
+
+        # 4. Cleanup staging files
+        self._cleanup_s3(s3_path)
+```
+
+---
+
+## Orchestration
+
+### Dependency Management
+
+```python
+from airflow import DAG
+from airflow.operators.python import PythonOperator
+from airflow.sensors.external_task import ExternalTaskSensor
+from airflow.utils.task_group import TaskGroup
+from datetime import timedelta
+
+with DAG("complex_pipeline") as dag:
+
+    # Wait for upstream DAG
+    wait_for_source = ExternalTaskSensor(
+        task_id="wait_for_source_etl",
+        external_dag_id="source_etl_dag",
+        external_task_id="final_task",
+        execution_delta=timedelta(hours=0),
+        timeout=3600,
+        mode="poke",
+        poke_interval=60,
+    )
+
+    # Parallel extraction group
+    with TaskGroup("extract") as extract_group:
+        extract_orders = PythonOperator(
+            task_id="extract_orders",
+            python_callable=extract_orders_func,
+        )
+        extract_customers = PythonOperator(
+            task_id="extract_customers",
+            python_callable=extract_customers_func,
+        )
+        extract_products = PythonOperator(
+            task_id="extract_products",
+            python_callable=extract_products_func,
+        )
+
+    # Sequential transformation
+    with TaskGroup("transform") as transform_group:
+        join_data = PythonOperator(
+            task_id="join_data",
+            python_callable=join_func,
+        )
+        aggregate = PythonOperator(
+            task_id="aggregate",
+            python_callable=aggregate_func,
+        )
+        join_data >> aggregate
+
+    # Load
+    load = PythonOperator(
+        task_id="load",
+        python_callable=load_func,
+    )
+
+    # Define dependencies
+    wait_for_source >> extract_group >> transform_group >> load
+```
+
+### Dynamic DAG Generation
+
+```python
+from airflow import DAG
+from airflow.operators.python import PythonOperator
+from datetime import datetime
+import yaml
+
+def create_etl_dag(config: dict) -> DAG:
+    """Factory function to create ETL DAGs from config."""
+
+    dag = DAG(
+        dag_id=f"etl_{config['source']}_{config['destination']}",
+        schedule_interval=config.get('schedule', '@daily'),
+        start_date=datetime(2024, 1, 1),
+        catchup=False,
+        tags=['etl', 'auto-generated'],
+    )
+
+    with dag:
+        extract = PythonOperator(
+            task_id='extract',
+            python_callable=create_extract_func(config['source']),
+        )
+
+        transform = PythonOperator(
+            task_id='transform',
+            python_callable=create_transform_func(config['transformations']),
+        )
+
+        load = PythonOperator(
+            task_id='load',
+            python_callable=create_load_func(config['destination']),
+        )
+
+        extract >> transform >> load
+
+    return dag
+
+# Load configurations
+with open('/config/etl_pipelines.yaml') as f:
+    configs = yaml.safe_load(f)
+
+# Generate DAGs
+for config in configs['pipelines']:
+    dag_id = f"etl_{config['source']}_{config['destination']}"
+    globals()[dag_id] = create_etl_dag(config)
+```