aigroup-workflow 2.1.2 → 2.2.0

package/skills/ai-ml/spark-engineer/SKILL.md

@@ -0,0 +1,148 @@
+---
+name: spark-engineer
+description: Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.
+license: MIT
+metadata:
+  author: https://github.com/Jeffallan
+  version: "1.1.0"
+  domain: data-ml
+  triggers: Apache Spark, PySpark, Spark SQL, distributed computing, big data, DataFrame API, RDD, Spark Streaming, structured streaming, data partitioning, Spark performance, cluster computing, data processing pipeline
+  role: expert
+  scope: implementation
+  output-format: code
+  related-skills: python-pro, sql-pro, devops-engineer
+---
+
+# Spark Engineer
+
+Senior Apache Spark engineer specializing in high-performance distributed data processing, optimizing large-scale ETL pipelines, and building production-grade Spark applications.
+
+## Core Workflow
+
+1. **Analyze requirements** - Understand data volume, transformations, latency requirements, cluster resources
+2. **Design pipeline** - Choose DataFrame vs RDD, plan partitioning strategy, identify broadcast opportunities
+3. **Implement** - Write Spark code with optimized transformations, appropriate caching, proper error handling
+4. **Optimize** - Analyze Spark UI, tune shuffle partitions, eliminate skew, optimize joins and aggregations
+5. **Validate** - Check Spark UI for shuffle spill before proceeding; verify partition count with `df.rdd.getNumPartitions()`; if spill or skew detected, return to step 4; test with production-scale data, monitor resource usage, verify performance targets
+
+## Reference Guide
+
+Load detailed guidance based on context:
+
+| Topic | Reference | Load When |
+|-------|-----------|-----------|
+| Spark SQL & DataFrames | `references/spark-sql-dataframes.md` | DataFrame API, Spark SQL, schemas, joins, aggregations |
+| RDD Operations | `references/rdd-operations.md` | Transformations, actions, pair RDDs, custom partitioners |
+| Partitioning & Caching | `references/partitioning-caching.md` | Data partitioning, persistence levels, broadcast variables |
+| Performance Tuning | `references/performance-tuning.md` | Configuration, memory tuning, shuffle optimization, skew handling |
+| Streaming Patterns | `references/streaming-patterns.md` | Structured Streaming, watermarks, stateful operations, sinks |
+
+## Code Examples
+
+### Quick-Start Mini-Pipeline (PySpark)
+
+```python
+from pyspark.sql import SparkSession
+from pyspark.sql import functions as F
+from pyspark.sql.types import StructType, StructField, StringType, LongType, DoubleType
+
+spark = SparkSession.builder \
+    .appName("example-pipeline") \
+    .config("spark.sql.shuffle.partitions", "400") \
+    .config("spark.sql.adaptive.enabled", "true") \
+    .getOrCreate()
+
+# Always define explicit schemas in production
+schema = StructType([
+    StructField("user_id", StringType(), False),
+    StructField("event_ts", LongType(), False),
+    StructField("amount", DoubleType(), True),
+])
+
+df = spark.read.schema(schema).parquet("s3://bucket/events/")
+
+result = df \
+    .filter(F.col("amount").isNotNull()) \
+    .groupBy("user_id") \
+    .agg(F.sum("amount").alias("total_amount"), F.count("*").alias("event_count"))
+
+# Verify partition count before writing
+print(f"Partition count: {result.rdd.getNumPartitions()}")
+
+result.write.mode("overwrite").parquet("s3://bucket/output/")
+```
+
+### Broadcast Join (small dimension table < 200 MB)
+
+```python
+from pyspark.sql.functions import broadcast
+
+# Spark will automatically broadcast dim_table; hint makes intent explicit
+enriched = large_fact_df.join(broadcast(dim_df), on="product_id", how="left")
+```
+
+### Handling Data Skew with Salting
+
+```python
+import pyspark.sql.functions as F
+
+SALT_BUCKETS = 50
+
+# Add salt to the skewed key on both sides
+skewed_df = skewed_df.withColumn("salt", (F.rand() * SALT_BUCKETS).cast("int")) \
+    .withColumn("salted_key", F.concat(F.col("skewed_key"), F.lit("_"), F.col("salt")))
+
+other_df = other_df.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(SALT_BUCKETS)]))) \
+    .withColumn("salted_key", F.concat(F.col("skewed_key"), F.lit("_"), F.col("salt")))
+
+result = skewed_df.join(other_df, on="salted_key", how="inner") \
+    .drop("salt", "salted_key")
+```
+
+### Correct Caching Pattern
+
+```python
+# Cache ONLY when the DataFrame is reused multiple times
+df_cleaned = df.filter(...).withColumn(...).cache()
+df_cleaned.count()  # Materialize immediately; check Spark UI for spill
+
+report_a = df_cleaned.groupBy("region").agg(...)
+report_b = df_cleaned.groupBy("product").agg(...)
+
+df_cleaned.unpersist()  # Release when done
+```
+
+## Constraints
+
+### MUST DO
+- Use DataFrame API over RDD for structured data processing
+- Define explicit schemas for production pipelines
+- Partition data appropriately (200-1000 partitions per executor core)
+- Cache intermediate results only when reused multiple times
+- Use broadcast joins for small dimension tables (<200MB)
+- Handle data skew with salting or custom partitioning
+- Monitor Spark UI for shuffle, spill, and GC metrics
+- Test with production-scale data volumes
+
+### MUST NOT DO
+- Use collect() on large datasets (causes OOM)
+- Skip schema definition and rely on inference in production
+- Cache every DataFrame without measuring benefit
+- Ignore shuffle partition tuning (default 200 often wrong)
+- Use UDFs when built-in functions available (10-100x slower)
+- Process small files without coalescing (small file problem)
+- Run transformations without understanding lazy evaluation
+- Ignore data skew warnings in Spark UI
+
+## Output Templates
+
+When implementing Spark solutions, provide:
+1. Complete Spark code (PySpark or Scala) with type hints/types
+2. Configuration recommendations (executors, memory, shuffle partitions)
+3. Partitioning strategy explanation
+4. Performance analysis (expected shuffle size, memory usage)
+5. Monitoring recommendations (key Spark UI metrics to watch)
+
+## Knowledge Reference
+
+Spark DataFrame API, Spark SQL, RDD transformations/actions, catalyst optimizer, tungsten execution engine, partitioning strategies, broadcast variables, accumulators, structured streaming, watermarks, checkpointing, Spark UI analysis, memory management, shuffle optimization

package/skills/ai-ml/spark-engineer/references/partitioning-caching.md

@@ -0,0 +1,543 @@
+# Partitioning and Caching
+
+---
+
+## Partitioning Fundamentals
+
+### Why Partitioning Matters
+
+- **Parallelism**: Each partition runs on a separate task
+- **Data locality**: Minimize data movement across network
+- **Memory efficiency**: Right-sized partitions prevent OOM
+- **Join performance**: Co-partitioned data avoids shuffle
+
+### Partition Count Guidelines
+
+```python
+# Rule of thumb: 2-4 partitions per CPU core
+# For 100 executor cores: 200-400 partitions
+
+# Check current partitions
+print(f"Number of partitions: {df.rdd.getNumPartitions()}")
+
+# Recommended formula
+total_cores = num_executors * cores_per_executor
+recommended_partitions = total_cores * 2 to 4
+
+# Target partition size: 128MB - 256MB per partition
+# For 100GB data with 128MB target: ~800 partitions
+```
+
+### Optimal Partition Sizes
+
+| Data Volume | Target Partition Size | Partition Count |
+|-------------|----------------------|-----------------|
+| < 1GB | 64MB | 8-16 |
+| 1-10GB | 128MB | 8-80 |
+| 10-100GB | 128-256MB | 40-800 |
+| 100GB-1TB | 256MB | 400-4000 |
+| > 1TB | 256MB | 4000+ |
+
+---
+
+## DataFrame Partitioning
+
+### Repartition (Full Shuffle)
+
+```python
+from pyspark.sql import functions as F
+
+# Repartition to specific number
+df_repart = df.repartition(200)
+
+# Repartition by column(s) - same keys go to same partition
+df_repart = df.repartition("user_id")
+df_repart = df.repartition("user_id", "date")
+
+# Repartition with count and columns
+df_repart = df.repartition(100, "user_id")
+
+# Range partitioning (for sorted access patterns)
+df_range = df.repartitionByRange(100, "date")
+```
+
+```scala
+// Scala repartition
+val dfRepart = df.repartition(200)
+val dfByCol = df.repartition($"user_id")
+val dfRange = df.repartitionByRange(100, $"date")
+```
+
+### Coalesce (No Shuffle)
+
+```python
+# Reduce partitions without shuffle - efficient!
+# Use after filtering reduces data significantly
+df_coalesced = df.coalesce(50)
+
+# Common pattern: filter then coalesce
+df_filtered = df.filter(F.col("active") == True)
+# If filter reduced data by 80%, reduce partitions too
+df_optimized = df_filtered.coalesce(40)  # From 200 to 40
+```
+
+**When to use:**
+- `repartition(n)`: Increase partitions, need even distribution, partition by column
+- `coalesce(n)`: Decrease partitions only (no shuffle benefit)
+- `repartitionByRange()`: Need sorted partitions for range queries
+
+### Checking Partition Distribution
+
+```python
+from pyspark.sql import functions as F
+
+# Check partition count
+print(f"Partitions: {df.rdd.getNumPartitions()}")
+
+# Check partition sizes (row counts)
+partition_counts = df.withColumn("partition_id", F.spark_partition_id()) \
+    .groupBy("partition_id") \
+    .count() \
+    .orderBy("partition_id")
+
+partition_counts.show()
+
+# Get partition statistics
+stats = partition_counts.agg(
+    F.min("count").alias("min_rows"),
+    F.max("count").alias("max_rows"),
+    F.avg("count").alias("avg_rows"),
+    F.stddev("count").alias("stddev")
+)
+stats.show()
+
+# Identify skew: max/avg ratio > 3 indicates skew
+```
+
+---
+
+## Shuffle Partitions
+
+### Configuration
+
+```python
+# Default shuffle partitions (200) - often suboptimal
+spark.conf.set("spark.sql.shuffle.partitions", 200)
+
+# For small data (<10GB), reduce
+spark.conf.set("spark.sql.shuffle.partitions", 50)
+
+# For large data (>100GB), increase
+spark.conf.set("spark.sql.shuffle.partitions", 2000)
+
+# Adaptive Query Execution (Spark 3.0+) - dynamic partition sizing
+spark.conf.set("spark.sql.adaptive.enabled", "true")
+spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
+spark.conf.set("spark.sql.adaptive.coalescePartitions.minPartitionSize", "64MB")
+spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "128MB")
+```
+
+### AQE Automatic Optimization (Spark 3.x)
+
+```python
+# Enable full AQE suite
+spark.conf.set("spark.sql.adaptive.enabled", "true")
+
+# Auto-coalesce shuffle partitions
+spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
+spark.conf.set("spark.sql.adaptive.coalescePartitions.parallelismFirst", "false")
+
+# Handle skewed partitions automatically
+spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
+spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionFactor", 5)
+spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes", "256MB")
+
+# Local shuffle reader (avoid remote reads when possible)
+spark.conf.set("spark.sql.adaptive.localShuffleReader.enabled", "true")
+```
+
+**Spark UI Check:** With AQE, check "Adaptive" badge in SQL tab. View coalesced partition counts in stage details.
+
+---
+
+## Caching and Persistence
+
+### When to Cache
+
+**Cache when:**
+- DataFrame is reused multiple times in same job
+- DataFrame is expensive to compute (complex joins/aggregations)
+- Iterative algorithms (ML training loops)
+- Interactive exploration in notebooks
+
+**Do NOT cache when:**
+- DataFrame used only once
+- Data doesn't fit in cluster memory
+- Source data is already fast (local SSD, columnar formats)
+- Storage level causes excessive GC
+
+### Persistence Levels
+
+```python
+from pyspark import StorageLevel
+
+# Memory only (default for cache())
+df.cache()  # Equivalent to persist(MEMORY_AND_DISK)
+df.persist()  # Same as cache()
+
+# Specific storage levels
+df.persist(StorageLevel.MEMORY_ONLY)         # Fast, may lose partitions
+df.persist(StorageLevel.MEMORY_AND_DISK)     # Spill to disk if needed
+df.persist(StorageLevel.MEMORY_ONLY_SER)     # Serialized, less memory, slower
+df.persist(StorageLevel.MEMORY_AND_DISK_SER) # Serialized with disk spill
+df.persist(StorageLevel.DISK_ONLY)           # Only disk, slowest
+df.persist(StorageLevel.OFF_HEAP)            # Off-heap memory
+
+# With replication (for fault tolerance)
+df.persist(StorageLevel.MEMORY_AND_DISK_2)   # 2x replication
+
+# Unpersist when done
+df.unpersist()
+df.unpersist(blocking=True)  # Wait for completion
+```
+
+```scala
+// Scala persistence
+import org.apache.spark.storage.StorageLevel
+
+df.cache()
+df.persist(StorageLevel.MEMORY_AND_DISK_SER)
+df.unpersist()
+```
+
+### Storage Level Selection Guide
+
+| Storage Level | Use When |
+|---------------|----------|
+| MEMORY_ONLY | Enough memory, need fastest access |
+| MEMORY_AND_DISK | Default, safe for most cases |
+| MEMORY_ONLY_SER | Memory constrained, CPU available |
+| MEMORY_AND_DISK_SER | Large data, memory constrained |
+| DISK_ONLY | Very large data, memory scarce |
+| OFF_HEAP | Using Tungsten off-heap memory |
+
+### Caching Best Practices
+
+```python
+# Pattern 1: Cache after expensive transformation
+expensive_df = source_df \
+    .join(lookup_df, "key") \
+    .groupBy("category").agg(F.sum("amount"))
+
+expensive_df.cache()
+
+# Trigger caching with action
+expensive_df.count()
+
+# Reuse cached data
+result1 = expensive_df.filter(F.col("category") == "A")
+result2 = expensive_df.filter(F.col("category") == "B")
+
+# Clean up
+expensive_df.unpersist()
+
+# Pattern 2: Cache at checkpoint in iterative algorithm
+for iteration in range(100):
+    df = df.transform(update_function)
+    if iteration % 10 == 0:
+        df.cache()
+        df.count()  # Materialize
+        df.unpersist()  # Clean previous
+
+# Pattern 3: Checkpoint to break lineage (long pipelines)
+spark.sparkContext.setCheckpointDir("hdfs://path/checkpoints/")
+df.checkpoint()  # Truncates lineage, saves to reliable storage
+```
+
+### Monitoring Cache Usage
+
+```python
+# Check if DataFrame is cached
+print(df.storageLevel)  # StorageLevel(False, False, False, False, 1) = not cached
+
+# Check storage tab in Spark UI for:
+# - Size in Memory
+# - Size on Disk
+# - Fraction Cached (should be 100%)
+```
+
+**Spark UI Check:** Storage tab shows cached RDDs/DataFrames. Monitor "Fraction Cached" - if < 100%, memory is insufficient.
+
+---
+
+## Broadcast Variables
+
+### When to Use Broadcast
+
+- Small lookup tables (< 200MB)
+- Dimension tables joined to large fact tables
+- Configuration data used across all tasks
+- Avoiding shuffle in map-side joins
+
+### DataFrame Broadcast Join
+
+```python
+from pyspark.sql.functions import broadcast
+
+# Explicit broadcast hint
+large_df = spark.read.parquet("s3://bucket/transactions/")  # 100GB
+small_df = spark.read.parquet("s3://bucket/categories/")    # 50MB
+
+# Broadcast small table for efficient join
+result = large_df.join(broadcast(small_df), "category_id")
+
+# Auto-broadcast threshold configuration
+spark.conf.set("spark.sql.autoBroadcastJoinThreshold", 100 * 1024 * 1024)  # 100MB
+
+# Disable auto-broadcast (force sort-merge join)
+spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)
+```
+
+### RDD Broadcast Variables
+
+```python
+# Create broadcast variable
+lookup_dict = {"A": 1, "B": 2, "C": 3}
+broadcast_lookup = spark.sparkContext.broadcast(lookup_dict)
+
+# Use in transformation
+def enrich_with_lookup(row):
+    lookup = broadcast_lookup.value
+    return Row(
+        id=row.id,
+        code=row.code,
+        value=lookup.get(row.code, 0)
+    )
+
+enriched_rdd = df.rdd.map(enrich_with_lookup)
+
+# Clean up
+broadcast_lookup.unpersist()
+broadcast_lookup.destroy()
+```
+
+### Broadcast Size Limits
+
+```python
+# Maximum broadcast size (default 8GB, adjustable)
+spark.conf.set("spark.sql.autoBroadcastJoinThreshold", 200 * 1024 * 1024)  # 200MB
+
+# For larger broadcasts
+spark.conf.set("spark.driver.maxResultSize", "4g")
+
+# Monitor broadcast time in Spark UI
+# Long broadcast time indicates table too large
+```
+
+**Warning:** Broadcasting tables > 200MB can cause driver OOM and slow broadcast. Use sort-merge join instead.
+
+---
+
+## Partitioning Strategies for Common Patterns
+
+### Time-Series Data
+
+```python
+# Partition by date for time-range queries
+df_partitioned = df.repartition("date")
+
+# Range partition for ordered access
+df_range = df.repartitionByRange(365, "date")  # One year
+
+# Write partitioned by date
+df.write.partitionBy("year", "month", "day").parquet("s3://bucket/data/")
+
+# Read with partition pruning
+df = spark.read.parquet("s3://bucket/data/") \
+    .filter(F.col("year") == 2024)  # Only reads 2024 partitions
+```
+
+### User/Entity Data
+
+```python
+# Partition by user_id for user-specific queries
+df_user_partitioned = df.repartition(1000, "user_id")
+
+# Co-partition for efficient joins
+users_partitioned = users.repartition(1000, "user_id")
+orders_partitioned = orders.repartition(1000, "user_id")
+
+# Join without shuffle (if partitioners match)
+joined = users_partitioned.join(orders_partitioned, "user_id")
+```
+
+### Skewed Data
+
+```python
+# Salt skewed keys
+salt_buckets = 10
+
+# Add salt to skewed table
+salted_df = large_df.withColumn(
+    "salted_key",
+    F.concat(
+        F.col("join_key"),
+        F.lit("_"),
+        (F.monotonically_increasing_id() % salt_buckets).cast("string")
+    )
+)
+
+# Explode small table to match
+from pyspark.sql.functions import explode, array, lit
+
+small_exploded = small_df.withColumn(
+    "salt",
+    explode(array([lit(i) for i in range(salt_buckets)]))
+).withColumn(
+    "salted_key",
+    F.concat(F.col("join_key"), F.lit("_"), F.col("salt").cast("string"))
+)
+
+# Join on salted key
+result = salted_df.join(small_exploded, "salted_key")
+```
+
+---
+
+## File Partitioning (Write Optimization)
+
+### Hive-Style Partitioning
+
+```python
+# Write with partitioning
+df.write \
+    .mode("overwrite") \
+    .partitionBy("year", "month") \
+    .parquet("s3://bucket/data/")
+
+# Result directory structure:
+# s3://bucket/data/year=2024/month=01/part-*.parquet
+# s3://bucket/data/year=2024/month=02/part-*.parquet
+
+# Read with partition discovery
+df = spark.read.parquet("s3://bucket/data/")
+# Columns year, month automatically added from path
+```
+
+### Bucketing (Hash-Based File Partitioning)
+
+```python
+# Write bucketed table for optimized joins
+df.write \
+    .mode("overwrite") \
+    .bucketBy(100, "user_id") \
+    .sortBy("timestamp") \
+    .saveAsTable("bucketed_orders")
+
+# Read bucketed table
+orders = spark.table("bucketed_orders")
+users = spark.table("bucketed_users")  # Same bucket count
+
+# Bucket join - no shuffle if buckets match
+result = orders.join(users, "user_id")
+```
+
+**Note:** Bucketing requires Hive metastore and saveAsTable. Doesn't work with direct file writes.
+
+### Controlling Output Files
+
+```python
+# Control number of output files
+# One file per partition
+df.coalesce(1).write.parquet("s3://bucket/output/")
+
+# Multiple files per partition (for large partitions)
+df.repartition(100).write.parquet("s3://bucket/output/")
+
+# Max records per file
+df.write \
+    .option("maxRecordsPerFile", 1000000) \
+    .parquet("s3://bucket/output/")
+```
+
+---
+
+## Spark UI Analysis for Partitioning/Caching
+
+### Jobs Tab
+
+- Check if cached data shows "(cached)" in DAG
+- Look for skipped stages (using cached data)
+
+### Stages Tab
+
+- **Shuffle Write Size**: Large values indicate repartition opportunities
+- **Shuffle Read Size**: Should be similar across tasks (no skew)
+- **Task Duration Distribution**: Wide variance indicates partition imbalance
+
+### Storage Tab
+
+- **Size in Memory**: Actual cached size
+- **Size on Disk**: Spilled size
+- **Fraction Cached**: Should be 100% if memory sufficient
+
+### SQL Tab
+
+- Look for "BroadcastExchange" - indicates broadcast join
+- Look for "ShuffleExchange" - indicates data movement
+- Check "Rows Output" at each stage for data flow
+
+---
+
+## Common Anti-Patterns
+
+```python
+# BAD: Caching without measuring benefit
+for table in all_tables:
+    spark.read.parquet(table).cache()  # Wastes memory
+
+# GOOD: Cache only if reused
+expensive_df.cache()
+result1 = expensive_df.groupBy("a").count()
+result2 = expensive_df.groupBy("b").count()
+expensive_df.unpersist()
+
+# BAD: Too many small partitions
+df.repartition(10000)  # Creates scheduling overhead
+
+# GOOD: Right-size partitions (128MB-256MB each)
+df.repartition(100)
+
+# BAD: Too few partitions for large data
+df.coalesce(1)  # Single partition can't parallelize
+
+# GOOD: Maintain parallelism
+df.coalesce(max(1, target_size))
+
+# BAD: Repartition before filter
+df.repartition(1000).filter(F.col("active") == True)  # Shuffles then filters
+
+# GOOD: Filter then coalesce
+df.filter(F.col("active") == True).coalesce(100)  # Filter first, then resize
+
+# BAD: Broadcasting large table
+result = large.join(broadcast(also_large), "key")  # OOM risk
+
+# GOOD: Let Spark decide or use sort-merge
+result = large.join(also_large, "key")  # Sort-merge join
+```
+
+---
+
+## Best Practices Summary
+
+1. **Target 128-256MB partitions** - Not too small (overhead) or large (OOM)
+2. **Use 2-4 partitions per core** - Maximize parallelism
+3. **Enable AQE in Spark 3.x** - Automatic partition optimization
+4. **Cache only reused DataFrames** - Measure before caching everything
+5. **Use MEMORY_AND_DISK** - Safe default storage level
+6. **Broadcast tables < 200MB** - Avoid shuffle for small dimension tables
+7. **Coalesce after filters** - Reduce partitions when data shrinks
+8. **Repartition for joins** - Co-partition related tables
+9. **Partition writes by filter columns** - Enable partition pruning
+10. **Monitor Storage tab** - Ensure cache fits in memory