agentic-team-templates 0.3.0

package/templates/data-engineering/.cursorrules/performance.md

@@ -0,0 +1,339 @@
+# Performance Optimization
+
+Patterns for building efficient, cost-effective data pipelines.
+
+## Partitioning
+
+### Choose Partition Columns Wisely
+
+Partition by columns used in most query filters (usually date/time).
+
+```python
+# Good: Partition by date - most queries filter by date
+(orders_df
+    .write
+    .partitionBy("order_date")
+    .saveAsTable("curated.orders"))
+
+# Query with partition pruning
+spark.sql("SELECT * FROM curated.orders WHERE order_date = '2024-01-15'")
+
+# Bad: Over-partitioning creates too many small files
+(orders_df
+    .write
+    .partitionBy("order_date", "customer_id", "product_id")  # Millions of partitions!
+    .saveAsTable("curated.orders"))
+```
+
+### Partition Size Guidelines
+
+| Records per Partition | Assessment |
+|-----------------------|------------|
+| < 100,000 | Too small - consider fewer partitions |
+| 100K - 10M | Optimal range |
+| > 10M | Consider sub-partitioning |
+
+### Optimize File Sizes
+
+```python
+# Set target file size (Delta Lake)
+spark.conf.set("spark.databricks.delta.optimizeWrite.fileSize", "128mb")
+
+# Compact small files
+spark.sql("OPTIMIZE curated.orders")
+
+# Set max records per file
+(df.write
+    .option("maxRecordsPerFile", 1_000_000)
+    .saveAsTable("curated.orders"))
+```
+
+## Query Optimization
+
+### Predicate Pushdown
+
+Ensure filters are pushed to storage layer.
+
+```python
+# Good: Predicate pushdown works
+orders = spark.read.table("curated.orders").filter("order_date = '2024-01-15'")
+
+# Bad: UDF blocks pushdown - full table scan!
+@udf(returnType=BooleanType())
+def is_recent(d):
+    return d > datetime.now() - timedelta(days=7)
+
+orders = spark.read.table("curated.orders").filter(is_recent(F.col("order_date")))
+
+# Good: Use native Spark functions
+orders = spark.read.table("curated.orders").filter(
+    F.col("order_date") > F.date_sub(F.current_date(), 7)
+)
+```
+
+### Column Pruning
+
+Select only needed columns early.
+
+```python
+# Good: Select needed columns early
+(spark.read.table("curated.orders")
+    .select("order_id", "customer_id", "total_amount")  # Prune early
+    .filter("total_amount > 100")
+    .groupBy("customer_id")
+    .agg(F.sum("total_amount")))
+
+# Bad: Select all, filter later
+(spark.read.table("curated.orders")  # Reads all 50 columns
+    .filter("total_amount > 100")
+    .select("order_id", "customer_id", "total_amount")  # Too late!
+    .groupBy("customer_id")
+    .agg(F.sum("total_amount")))
+```
+
+### Broadcast Joins
+
+Broadcast small tables to avoid shuffle.
+
+```python
+# Good: Broadcast small dimension table
+from pyspark.sql.functions import broadcast
+
+orders = spark.table("curated.orders")  # 1B rows
+products = spark.table("dims.products")  # 10K rows
+
+joined = orders.join(broadcast(products), "product_id")
+
+# Configure broadcast threshold
+spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "100mb")
+```
+
+### Avoid Expensive Operations
+
+```python
+# Bad: Distinct before join (expensive shuffle)
+orders.select("customer_id").distinct().join(customers, "customer_id")
+
+# Good: Filter on indexed column, let database handle uniqueness
+customers.filter("is_active = true")
+
+# Bad: Order by on large dataset
+spark.table("curated.orders").orderBy("order_date")
+
+# Good: Only sort when necessary, limit first if possible
+(spark.table("curated.orders")
+    .filter("customer_id = 'C123'")
+    .orderBy(F.desc("order_date"))
+    .limit(10))
+```
+
+## Caching
+
+### When to Cache
+
+Cache intermediate results that are:
+1. Used multiple times
+2. Expensive to compute
+3. Small enough to fit in memory
+
+```python
+def process_with_caching():
+    # Read and filter once
+    base_orders = (
+        spark.read.table("curated.orders")
+        .filter("order_date >= '2024-01-01'")
+        .cache()  # Cache in memory
+    )
+    
+    try:
+        # Multiple aggregations on same filtered data
+        daily_totals = base_orders.groupBy("order_date").agg(F.sum("total"))
+        customer_totals = base_orders.groupBy("customer_id").agg(F.sum("total"))
+        product_totals = base_orders.groupBy("product_id").agg(F.sum("total"))
+        
+        # Write all
+        daily_totals.write.saveAsTable("marts.daily_totals")
+        customer_totals.write.saveAsTable("marts.customer_totals")
+        product_totals.write.saveAsTable("marts.product_totals")
+    finally:
+        base_orders.unpersist()  # Always clean up
+```
+
+### Cache Levels
+
+```python
+from pyspark import StorageLevel
+
+# Memory only (default) - fastest, may spill
+df.cache()  # Same as persist(StorageLevel.MEMORY_ONLY)
+
+# Memory and disk - won't recompute if evicted
+df.persist(StorageLevel.MEMORY_AND_DISK)
+
+# Disk only - for very large datasets
+df.persist(StorageLevel.DISK_ONLY)
+
+# Serialized - more compact, slower access
+df.persist(StorageLevel.MEMORY_ONLY_SER)
+```
+
+## Shuffle Optimization
+
+### Reduce Shuffle Size
+
+```python
+# Good: Aggregate before join
+customer_totals = orders.groupBy("customer_id").agg(F.sum("amount").alias("total"))
+result = customer_totals.join(customers, "customer_id")
+
+# Bad: Join then aggregate (shuffles full orders table)
+result = orders.join(customers, "customer_id").groupBy("customer_id").agg(F.sum("amount"))
+```
+
+### Partition Count
+
+```python
+# Check partition count
+print(df.rdd.getNumPartitions())
+
+# Reduce partitions before write (coalesce doesn't shuffle)
+df.coalesce(100).write.saveAsTable("output")
+
+# Increase partitions for parallelism (repartition shuffles)
+df.repartition(200).write.saveAsTable("output")
+
+# Repartition by key for co-located data
+df.repartition("customer_id").write.partitionBy("customer_id").saveAsTable("output")
+```
+
+### Configure Shuffle Partitions
+
+```python
+# Default is 200 - tune based on data size
+spark.conf.set("spark.sql.shuffle.partitions", "auto")  # Spark 3.0+
+# Or set explicitly
+spark.conf.set("spark.sql.shuffle.partitions", "400")
+```
+
+## Z-Ordering (Delta Lake)
+
+Co-locate related data for faster queries on multiple columns.
+
+```sql
+-- Z-order by commonly filtered/joined columns
+OPTIMIZE curated.orders
+ZORDER BY (customer_id, product_id)
+
+-- Queries on these columns will be faster
+SELECT * FROM curated.orders WHERE customer_id = 'C123'
+SELECT * FROM curated.orders WHERE product_id = 'P456'
+SELECT * FROM curated.orders WHERE customer_id = 'C123' AND product_id = 'P456'
+```
+
+## Incremental Processing
+
+Process only what changed.
+
+```python
+def incremental_pipeline(source: str, target: str, watermark_col: str):
+    # Get last processed watermark
+    last_watermark = spark.sql(f"""
+        SELECT MAX({watermark_col}) FROM {target}
+    """).collect()[0][0]
+    
+    # Read only new data
+    new_data = (
+        spark.read.table(source)
+        .filter(F.col(watermark_col) > last_watermark)
+    )
+    
+    if new_data.isEmpty():
+        logger.info("No new data to process")
+        return
+    
+    # Process and write
+    processed = transform(new_data)
+    processed.write.mode("append").saveAsTable(target)
+```
+
+## Cost Management
+
+### Monitor Compute Costs
+
+```sql
+-- Track pipeline costs over time
+SELECT 
+    pipeline_name,
+    DATE(run_date) as run_date,
+    SUM(total_dbu) as daily_dbu,
+    SUM(bytes_scanned) / 1e12 as tb_scanned,
+    AVG(duration_seconds) as avg_duration
+FROM pipeline_metrics
+WHERE run_date >= CURRENT_DATE - 30
+GROUP BY pipeline_name, DATE(run_date)
+ORDER BY daily_dbu DESC;
+```
+
+### Identify Expensive Queries
+
+```sql
+-- Find most expensive queries
+SELECT 
+    query_id,
+    user,
+    LEFT(query_text, 100) as query_preview,
+    bytes_scanned / 1e9 as gb_scanned,
+    duration_ms / 1000 as duration_seconds
+FROM query_history
+WHERE timestamp >= CURRENT_DATE - 7
+ORDER BY bytes_scanned DESC
+LIMIT 20;
+```
+
+### Optimize Storage Costs
+
+```python
+# Remove old partitions
+spark.sql("""
+    DELETE FROM curated.orders
+    WHERE order_date < DATE_SUB(CURRENT_DATE, 365)
+""")
+
+# Vacuum deleted files (Delta Lake)
+spark.sql("VACUUM curated.orders RETAIN 168 HOURS")
+
+# Convert to more efficient format
+(spark.read.table("legacy.orders")
+    .write
+    .format("delta")
+    .option("compression", "zstd")  # Better compression
+    .saveAsTable("curated.orders"))
+```
+
+## Performance Checklist
+
+Before deploying a pipeline, verify:
+
+### Query Optimization
+- [ ] Filters use partition columns
+- [ ] Only needed columns are selected
+- [ ] Small tables are broadcast in joins
+- [ ] No unnecessary shuffles
+
+### File Optimization
+- [ ] Partition column is appropriate
+- [ ] Files are right-sized (100MB-1GB)
+- [ ] Z-ordering on common filter columns
+- [ ] No excessive small files
+
+### Resource Optimization
+- [ ] Cluster size matches workload
+- [ ] Shuffle partitions configured
+- [ ] Caching used for reused DataFrames
+- [ ] Memory settings appropriate
+
+### Cost Optimization
+- [ ] Incremental processing where possible
+- [ ] Data retention policy applied
+- [ ] Query costs monitored
+- [ ] Unused tables/data removed

package/templates/data-engineering/.cursorrules/pipeline-design.md

@@ -0,0 +1,280 @@
+# Pipeline Design
+
+Patterns and practices for building reliable data pipelines.
+
+## Core Principles
+
+### 1. Idempotency
+
+Every pipeline run must produce identical results for the same inputs.
+
+```python
+# Good: Delete-insert pattern ensures idempotency
+def process_daily_orders(execution_date: date) -> None:
+    partition = execution_date.strftime("%Y-%m-%d")
+    
+    # Clear target partition first
+    spark.sql(f"DELETE FROM curated.orders WHERE order_date = '{partition}'")
+    
+    # Then insert
+    orders_df.write.mode("append").saveAsTable("curated.orders")
+
+# Bad: Append without clearing creates duplicates on re-run
+orders_df.write.mode("append").saveAsTable("curated.orders")
+```
+
+### 2. Determinism
+
+Same inputs must always produce same outputs. Avoid:
+- `current_timestamp()` in transformations (use execution_date)
+- Random sampling without seeds
+- Order-dependent operations on unordered data
+
+```python
+# Good: Use execution_date for reproducibility
+df.withColumn("processed_at", F.lit(execution_date))
+
+# Bad: Non-deterministic timestamp
+df.withColumn("processed_at", F.current_timestamp())
+```
+
+### 3. Atomicity
+
+Pipeline outputs should be all-or-nothing. Partial writes corrupt data.
+
+```python
+# Good: Write to staging, then atomic swap
+df.write.mode("overwrite").saveAsTable("staging.orders_temp")
+spark.sql("ALTER TABLE curated.orders SWAP WITH staging.orders_temp")
+
+# Good: Use Delta Lake transactions
+df.write.format("delta").mode("overwrite").saveAsTable("curated.orders")
+```
+
+## Pipeline Patterns
+
+### Batch Full Refresh
+
+Use when: Source doesn't support incremental, data is small, or simplicity matters.
+
+```python
+def full_refresh_pipeline(source: str, target: str) -> None:
+    df = spark.read.table(source)
+    df = transform(df)
+    df.write.mode("overwrite").saveAsTable(target)
+```
+
+### Batch Incremental
+
+Use when: Data volume is large, source supports watermarks.
+
+```python
+def incremental_pipeline(source: str, target: str, watermark_col: str) -> None:
+    # Get high watermark from previous run
+    last_watermark = get_watermark(target, watermark_col)
+    
+    # Read only new/changed records
+    df = spark.read.table(source).filter(F.col(watermark_col) > last_watermark)
+    
+    if df.isEmpty():
+        return
+    
+    # Merge into target
+    target_table = DeltaTable.forName(spark, target)
+    (target_table.alias("t")
+        .merge(df.alias("s"), "t.id = s.id")
+        .whenMatchedUpdateAll()
+        .whenNotMatchedInsertAll()
+        .execute())
+```
+
+### Change Data Capture (CDC)
+
+Use when: Need to track all changes, support point-in-time queries.
+
+```python
+def cdc_pipeline(cdc_events: DataFrame, target: str) -> None:
+    """Process CDC events (insert, update, delete operations)."""
+    
+    target_table = DeltaTable.forName(spark, target)
+    
+    (target_table.alias("t")
+        .merge(cdc_events.alias("s"), "t.id = s.id")
+        .whenMatchedDelete(condition="s.operation = 'DELETE'")
+        .whenMatchedUpdateAll(condition="s.operation = 'UPDATE'")
+        .whenNotMatchedInsertAll(condition="s.operation = 'INSERT'")
+        .execute())
+```
+
+### Streaming
+
+Use when: Low latency required, source is event stream.
+
+```python
+def streaming_pipeline() -> None:
+    events = (
+        spark.readStream
+        .format("kafka")
+        .option("subscribe", "events")
+        .load()
+    )
+    
+    processed = events.transform(process_events)
+    
+    (processed
+        .writeStream
+        .format("delta")
+        .option("checkpointLocation", CHECKPOINT_PATH)
+        .outputMode("append")
+        .trigger(processingTime="1 minute")
+        .toTable("curated.events"))
+```
+
+## Handling Late Data
+
+### Reprocessing Window
+
+Reprocess recent partitions to catch late arrivals.
+
+```python
+def process_with_late_data_handling(execution_date: date) -> None:
+    # Reprocess last 3 days to catch late arrivals
+    start_date = execution_date - timedelta(days=3)
+    
+    for date in date_range(start_date, execution_date):
+        process_partition(date)
+```
+
+### Watermarking (Streaming)
+
+Define how long to wait for late data.
+
+```python
+events_with_watermark = (
+    events
+    .withWatermark("event_time", "1 hour")  # Wait up to 1 hour for late events
+    .groupBy(F.window("event_time", "5 minutes"))
+    .count()
+)
+```
+
+## Orchestration Patterns
+
+### DAG Design
+
+```
+[extract_orders] --> [validate_orders] --> [transform_orders] --> [load_orders]
+                                                                      |
+[extract_products] --> [validate_products] --> [transform_products] --+
+                                                                      |
+                                                                      v
+                                                            [build_order_mart]
+```
+
+### Retry Strategy
+
+```python
+default_args = {
+    'retries': 3,
+    'retry_delay': timedelta(minutes=5),
+    'retry_exponential_backoff': True,
+    'max_retry_delay': timedelta(hours=1),
+}
+```
+
+### Backfill Strategy
+
+```python
+# Support backfill via parameterized execution date
+@task
+def process_orders(execution_date: date = None):
+    if execution_date is None:
+        execution_date = date.today() - timedelta(days=1)
+    
+    # Use execution_date, not current date
+    process_partition(execution_date)
+```
+
+## Error Handling
+
+### Fail Fast
+
+```python
+def process_orders(df: DataFrame) -> DataFrame:
+    # Validate critical assumptions early
+    if df.filter("order_id IS NULL").count() > 0:
+        raise DataQualityError("Found null order_ids")
+    
+    if df.count() == 0:
+        raise EmptyDataError("No orders to process")
+    
+    return transform(df)
+```
+
+### Dead Letter Queue
+
+```python
+def process_with_dlq(df: DataFrame) -> DataFrame:
+    # Separate valid and invalid records
+    valid = df.filter(is_valid_record)
+    invalid = df.filter(~is_valid_record)
+    
+    # Write invalid records to DLQ for investigation
+    if invalid.count() > 0:
+        invalid.write.mode("append").saveAsTable("dlq.orders")
+        logger.warning(f"Sent {invalid.count()} records to DLQ")
+    
+    return valid
+```
+
+## Best Practices
+
+### Explicit Dependencies
+
+```python
+# Good: Explicit data dependencies
+orders = spark.read.table("raw.orders")
+products = spark.read.table("raw.products")
+result = orders.join(products, "product_id")
+
+# Bad: Hidden dependencies via side effects
+process_orders()  # Reads from orders table
+process_products()  # Reads from products table
+build_mart()  # What does this depend on?
+```
+
+### Parameterize Everything
+
+```python
+# Good: Parameterized and testable
+def process_orders(
+    source_table: str,
+    target_table: str,
+    execution_date: date,
+) -> None:
+    ...
+
+# Bad: Hardcoded values
+def process_orders():
+    df = spark.read.table("prod.orders")  # Hardcoded!
+    df.write.saveAsTable("prod.curated_orders")
+```
+
+### Logging and Metrics
+
+```python
+def process_orders(execution_date: date) -> None:
+    logger.info(f"Starting order processing for {execution_date}")
+    
+    df = spark.read.table("raw.orders").filter(f"order_date = '{execution_date}'")
+    logger.info(f"Read {df.count()} orders")
+    
+    result = transform(df)
+    logger.info(f"Writing {result.count()} records")
+    
+    result.write.saveAsTable("curated.orders")
+    
+    # Emit metrics
+    metrics.gauge("orders.processed", result.count())
+    metrics.gauge("orders.processing_time_seconds", elapsed_time)
+```