@sylix/coworker 2.0.11 → 2.0.12

package/dist/skills/defaults/data-engineering/ml-pipeline-workflow/SKILL.md

@@ -0,0 +1,247 @@
+---
+name: ml-pipeline-workflow
+description: Build end-to-end MLOps pipelines from data preparation through model training, validation, and production deployment. Use when creating ML pipelines, implementing MLOps practices, or automating model training and deployment workflows.
+---
+
+# ML Pipeline Workflow
+
+Complete end-to-end MLOps pipeline orchestration from data preparation through model deployment.
+
+## Overview
+
+This skill provides comprehensive guidance for building production ML pipelines that handle the full lifecycle: data ingestion → preparation → training → validation → deployment → monitoring.
+
+## When to Use This Skill
+
+- Building new ML pipelines from scratch
+- Designing workflow orchestration for ML systems
+- Implementing data → model → deployment automation
+- Setting up reproducible training workflows
+- Creating DAG-based ML orchestration
+- Integrating ML components into production systems
+
+## What This Skill Provides
+
+### Core Capabilities
+
+1. **Pipeline Architecture**
+   - End-to-end workflow design
+   - DAG orchestration patterns (Airflow, Dagster, Kubeflow)
+   - Component dependencies and data flow
+   - Error handling and retry strategies
+
+2. **Data Preparation**
+   - Data validation and quality checks
+   - Feature engineering pipelines
+   - Data versioning and lineage
+   - Train/validation/test splitting strategies
+
+3. **Model Training**
+   - Training job orchestration
+   - Hyperparameter management
+   - Experiment tracking integration
+   - Distributed training patterns
+
+4. **Model Validation**
+   - Validation frameworks and metrics
+   - A/B testing infrastructure
+   - Performance regression detection
+   - Model comparison workflows
+
+5. **Deployment Automation**
+   - Model serving patterns
+   - Canary deployments
+   - Blue-green deployment strategies
+   - Rollback mechanisms
+
+### Reference Documentation
+
+See the `references/` directory for detailed guides:
+
+- **data-preparation.md** - Data cleaning, validation, and feature engineering
+- **model-training.md** - Training workflows and best practices
+- **model-validation.md** - Validation strategies and metrics
+- **model-deployment.md** - Deployment patterns and serving architectures
+
+### Assets and Templates
+
+The `assets/` directory contains:
+
+- **pipeline-dag.yaml.template** - DAG template for workflow orchestration
+- **training-config.yaml** - Training configuration template
+- **validation-checklist.md** - Pre-deployment validation checklist
+
+## Usage Patterns
+
+### Basic Pipeline Setup
+
+```python
+# 1. Define pipeline stages
+stages = [
+    "data_ingestion",
+    "data_validation",
+    "feature_engineering",
+    "model_training",
+    "model_validation",
+    "model_deployment"
+]
+
+# 2. Configure dependencies
+# See assets/pipeline-dag.yaml.template for full example
+```
+
+### Production Workflow
+
+1. **Data Preparation Phase**
+   - Ingest raw data from sources
+   - Run data quality checks
+   - Apply feature transformations
+   - Version processed datasets
+
+2. **Training Phase**
+   - Load versioned training data
+   - Execute training jobs
+   - Track experiments and metrics
+   - Save trained models
+
+3. **Validation Phase**
+   - Run validation test suite
+   - Compare against baseline
+   - Generate performance reports
+   - Approve for deployment
+
+4. **Deployment Phase**
+   - Package model artifacts
+   - Deploy to serving infrastructure
+   - Configure monitoring
+   - Validate production traffic
+
+## Best Practices
+
+### Pipeline Design
+
+- **Modularity**: Each stage should be independently testable
+- **Idempotency**: Re-running stages should be safe
+- **Observability**: Log metrics at every stage
+- **Versioning**: Track data, code, and model versions
+- **Failure Handling**: Implement retry logic and alerting
+
+### Data Management
+
+- Use data validation libraries (Great Expectations, TFX)
+- Version datasets with DVC or similar tools
+- Document feature engineering transformations
+- Maintain data lineage tracking
+
+### Model Operations
+
+- Separate training and serving infrastructure
+- Use model registries (MLflow, Weights & Biases)
+- Implement gradual rollouts for new models
+- Monitor model performance drift
+- Maintain rollback capabilities
+
+### Deployment Strategies
+
+- Start with shadow deployments
+- Use canary releases for validation
+- Implement A/B testing infrastructure
+- Set up automated rollback triggers
+- Monitor latency and throughput
+
+## Integration Points
+
+### Orchestration Tools
+
+- **Apache Airflow**: DAG-based workflow orchestration
+- **Dagster**: Asset-based pipeline orchestration
+- **Kubeflow Pipelines**: Kubernetes-native ML workflows
+- **Prefect**: Modern dataflow automation
+
+### Experiment Tracking
+
+- MLflow for experiment tracking and model registry
+- Weights & Biases for visualization and collaboration
+- TensorBoard for training metrics
+
+### Deployment Platforms
+
+- AWS SageMaker for managed ML infrastructure
+- Google Vertex AI for GCP deployments
+- Azure ML for Azure cloud
+- Kubernetes + KServe for cloud-agnostic serving
+
+## Progressive Disclosure
+
+Start with the basics and gradually add complexity:
+
+1. **Level 1**: Simple linear pipeline (data → train → deploy)
+2. **Level 2**: Add validation and monitoring stages
+3. **Level 3**: Implement hyperparameter tuning
+4. **Level 4**: Add A/B testing and gradual rollouts
+5. **Level 5**: Multi-model pipelines with ensemble strategies
+
+## Common Patterns
+
+### Batch Training Pipeline
+
+```yaml
+# See assets/pipeline-dag.yaml.template
+stages:
+  - name: data_preparation
+    dependencies: []
+  - name: model_training
+    dependencies: [data_preparation]
+  - name: model_evaluation
+    dependencies: [model_training]
+  - name: model_deployment
+    dependencies: [model_evaluation]
+```
+
+### Real-time Feature Pipeline
+
+```python
+# Stream processing for real-time features
+# Combined with batch training
+# See references/data-preparation.md
+```
+
+### Continuous Training
+
+```python
+# Automated retraining on schedule
+# Triggered by data drift detection
+# See references/model-training.md
+```
+
+## Troubleshooting
+
+### Common Issues
+
+- **Pipeline failures**: Check dependencies and data availability
+- **Training instability**: Review hyperparameters and data quality
+- **Deployment issues**: Validate model artifacts and serving config
+- **Performance degradation**: Monitor data drift and model metrics
+
+### Debugging Steps
+
+1. Check pipeline logs for each stage
+2. Validate input/output data at boundaries
+3. Test components in isolation
+4. Review experiment tracking metrics
+5. Inspect model artifacts and metadata
+
+## Next Steps
+
+After setting up your pipeline:
+
+1. Explore **hyperparameter-tuning** skill for optimization
+2. Learn **experiment-tracking-setup** for MLflow/W&B
+3. Review **model-deployment-patterns** for serving strategies
+4. Implement monitoring with observability tools
+
+## Related Skills
+
+- **experiment-tracking-setup**: MLflow and Weights & Biases integration
+- **hyperparameter-tuning**: Automated hyperparameter optimization
+- **model-deployment-patterns**: Advanced deployment strategies

package/dist/skills/defaults/data-engineering/spark-optimization.md

@@ -0,0 +1,348 @@
+---
+name: spark-optimization
+description: Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning
+---
+
+# Apache Spark Optimization
+
+Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.
+
+## When to Use This Skill
+
+- Optimizing slow Spark jobs
+- Tuning memory and executor configuration
+- Implementing efficient partitioning strategies
+- Debugging Spark performance issues
+- Scaling Spark pipelines for large datasets
+- Reducing shuffle and data skew
+
+## Core Concepts
+
+### Spark Execution Model
+
+```
+Driver Program
+    ↓
+Job (triggered by action)
+    ↓
+Stages (separated by shuffles)
+    ↓
+Tasks (one per partition)
+```
+
+### Key Performance Factors
+
+| Factor            | Impact                | Solution                      |
+| ----------------- | --------------------- | ----------------------------- |
+| **Shuffle**       | Network I/O, disk I/O | Minimize wide transformations |
+| **Data Skew**     | Uneven task duration  | Salting, broadcast joins      |
+| **Serialization** | CPU overhead          | Use Kryo, columnar formats    |
+| **Memory**        | GC pressure, spills   | Tune executor memory          |
+| **Partitions**    | Parallelism           | Right-size partitions         |
+
+## Quick Start
+
+```python
+from pyspark.sql import SparkSession
+from pyspark.sql import functions as F
+
+spark = (SparkSession.builder
+    .appName("OptimizedJob")
+    .config("spark.sql.adaptive.enabled", "true")
+    .config("spark.sql.adaptive.coalescePartitions.enabled", "true")
+    .config("spark.sql.adaptive.skewJoin.enabled", "true")
+    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
+    .config("spark.sql.shuffle.partitions", "200")
+    .getOrCreate())
+
+df = (spark.read
+    .format("parquet")
+    .option("mergeSchema", "false")
+    .load("s3://bucket/data/"))
+
+result = (df
+    .filter(F.col("date") >= "2024-01-01")
+    .select("id", "amount", "category")
+    .groupBy("category")
+    .agg(F.sum("amount").alias("total")))
+
+result.write.mode("overwrite").parquet("s3://bucket/output/")
+```
+
+## Patterns
+
+### Pattern 1: Optimal Partitioning
+
+```python
+def calculate_partitions(data_size_gb: float, partition_size_mb: int = 128) -> int:
+    return max(int(data_size_gb * 1024 / partition_size_mb), 1)
+
+# Repartition for even distribution
+df_repartitioned = df.repartition(200, "partition_key")
+
+# Coalesce to reduce partitions (no shuffle)
+df_coalesced = df.coalesce(100)
+
+# Partition pruning with predicate pushdown
+df = (spark.read.parquet("s3://bucket/data/")
+    .filter(F.col("date") == "2024-01-01"))
+
+# Write with partitioning for future queries
+(df.write
+    .partitionBy("year", "month", "day")
+    .mode("overwrite")
+    .parquet("s3://bucket/partitioned_output/"))
+```
+
+### Pattern 2: Join Optimization
+
+```python
+from pyspark.sql import functions as F
+
+# 1. Broadcast Join - Small table joins
+small_df = spark.read.parquet("s3://bucket/small_table/")
+large_df = spark.read.parquet("s3://bucket/large_table/")
+
+result = large_df.join(
+    F.broadcast(small_df),
+    on="key",
+    how="left"
+)
+
+# 2. Sort-Merge Join - Default for large tables
+result = large_df1.join(large_df2, on="key", how="inner")
+
+# 3. Bucket Join - Pre-sorted, no shuffle at join time
+(df.write
+    .bucketBy(200, "customer_id")
+    .sortBy("customer_id")
+    .mode("overwrite")
+    .saveAsTable("bucketed_orders"))
+
+orders = spark.table("bucketed_orders")
+customers = spark.table("bucketed_customers")
+result = orders.join(customers, on="customer_id")
+
+# 4. Skew Join Handling
+spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
+spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionFactor", "5")
+
+# Manual salting for severe skew
+def salt_join(df_skewed, df_other, key_col, num_salts=10):
+    df_salted = df_skewed.withColumn(
+        "salt",
+        (F.rand() * num_salts).cast("int")
+    ).withColumn(
+        "salted_key",
+        F.concat(F.col(key_col), F.lit("_"), F.col("salt"))
+    )
+
+    df_exploded = df_other.crossJoin(
+        spark.range(num_salts).withColumnRenamed("id", "salt")
+    ).withColumn(
+        "salted_key",
+        F.concat(F.col(key_col), F.lit("_"), F.col("salt"))
+    )
+
+    return df_salted.join(df_exploded, on="salted_key", how="inner")
+```
+
+### Pattern 3: Caching and Persistence
+
+```python
+from pyspark import StorageLevel
+
+# Cache when reusing DataFrame multiple times
+df = spark.read.parquet("s3://bucket/data/")
+df_filtered = df.filter(F.col("status") == "active")
+
+# Cache in memory
+df_filtered.cache()
+
+# Or with specific storage level
+df_filtered.persist(StorageLevel.MEMORY_AND_DISK_SER)
+
+# Force materialization
+df_filtered.count()
+
+# Use in multiple actions
+agg1 = df_filtered.groupBy("category").count()
+agg2 = df_filtered.groupBy("region").sum("amount")
+
+# Unpersist when done
+df_filtered.unpersist()
+
+# Storage levels:
+# MEMORY_ONLY - Fast, but may not fit
+# MEMORY_AND_DISK - Spills to disk if needed (recommended)
+# MEMORY_ONLY_SER - Serialized, less memory, more CPU
+# DISK_ONLY - When memory is tight
+# OFF_HEAP - Tungsten off-heap memory
+
+# Checkpoint for complex lineage
+spark.sparkContext.setCheckpointDir("s3://bucket/checkpoints/")
+df_complex.checkpoint()
+```
+
+### Pattern 4: Memory Tuning
+
+```python
+# Executor memory configuration
+spark = (SparkSession.builder
+    .config("spark.executor.memory", "8g")
+    .config("spark.executor.memoryOverhead", "2g")
+    .config("spark.memory.fraction", "0.6")
+    .config("spark.memory.storageFraction", "0.5")
+    .config("spark.sql.shuffle.partitions", "200")
+    .config("spark.sql.autoBroadcastJoinThreshold", "50MB")
+    .config("spark.sql.files.maxPartitionBytes", "128MB")
+    .getOrCreate())
+
+# Memory breakdown (8GB executor):
+# - spark.memory.fraction = 0.6 (60% = 4.8GB for execution + storage)
+#   - spark.memory.storageFraction = 0.5 (50% of 4.8GB = 2.4GB for cache)
+#   - Remaining 2.4GB for execution (shuffles, joins, sorts)
+# - 40% = 3.2GB for user data structures and internal metadata
+```
+
+### Pattern 5: Shuffle Optimization
+
+```python
+# Reduce shuffle data size
+spark.conf.set("spark.sql.shuffle.partitions", "auto")
+spark.conf.set("spark.shuffle.compress", "true")
+spark.conf.set("spark.shuffle.spill.compress", "true")
+
+# Pre-aggregate before shuffle
+df_optimized = (df
+    .groupBy("key", "partition_col")
+    .agg(F.sum("value").alias("partial_sum"))
+    .groupBy("key")
+    .agg(F.sum("partial_sum").alias("total")))
+
+# Use coalesce instead of repartition when reducing partitions
+df_reduced = df.coalesce(10)
+
+# Optimize shuffle with compression
+spark.conf.set("spark.io.compression.codec", "lz4")
+```
+
+### Pattern 6: Data Format Optimization
+
+```python
+# Parquet optimizations
+(df.write
+    .option("compression", "snappy")
+    .option("parquet.block.size", 128 * 1024 * 1024)
+    .parquet("s3://bucket/output/"))
+
+# Column pruning - only read needed columns
+df = (spark.read.parquet("s3://bucket/data/")
+    .select("id", "amount", "date"))
+
+# Delta Lake optimizations
+(df.write
+    .format("delta")
+    .option("optimizeWrite", "true")
+    .option("autoCompact", "true")
+    .mode("overwrite")
+    .save("s3://bucket/delta_table/"))
+
+# Z-ordering for multi-dimensional queries
+spark.sql("""
+    OPTIMIZE delta.`s3://bucket/delta_table/`
+    ZORDER BY (customer_id, date)
+""")
+```
+
+### Pattern 7: Monitoring and Debugging
+
+```python
+# Enable detailed metrics
+spark.conf.set("spark.sql.codegen.wholeStage", "true")
+
+# Explain query plan
+df.explain(mode="extended")
+
+# Monitor task metrics
+def analyze_stage_metrics(spark):
+    status_tracker = spark.sparkContext.statusTracker()
+    for stage_id in status_tracker.getActiveStageIds():
+        stage_info = status_tracker.getStageInfo(stage_id)
+        print(f"Stage {stage_id}:")
+        print(f"  Tasks: {stage_info.numTasks}")
+        print(f"  Completed: {stage_info.numCompletedTasks}")
+
+# Identify data skew
+def check_partition_skew(df):
+    partition_counts = (df
+        .withColumn("partition_id", F.spark_partition_id())
+        .groupBy("partition_id")
+        .count()
+        .orderBy(F.desc("count")))
+
+    partition_counts.show(20)
+
+    stats = partition_counts.select(
+        F.min("count").alias("min"),
+        F.max("count").alias("max"),
+        F.avg("count").alias("avg"),
+        F.stddev("count").alias("stddev")
+    ).collect()[0]
+
+    skew_ratio = stats["max"] / stats["avg"]
+    print(f"Skew ratio: {skew_ratio:.2f}x (>2x indicates skew)")
+```
+
+## Configuration Cheat Sheet
+
+```python
+spark_configs = {
+    # Adaptive Query Execution (AQE)
+    "spark.sql.adaptive.enabled": "true",
+    "spark.sql.adaptive.coalescePartitions.enabled": "true",
+    "spark.sql.adaptive.skewJoin.enabled": "true",
+
+    # Memory
+    "spark.executor.memory": "8g",
+    "spark.executor.memoryOverhead": "2g",
+    "spark.memory.fraction": "0.6",
+    "spark.memory.storageFraction": "0.5",
+
+    # Parallelism
+    "spark.sql.shuffle.partitions": "200",
+    "spark.default.parallelism": "200",
+
+    # Serialization
+    "spark.serializer": "org.apache.spark.serializer.KryoSerializer",
+
+    # Compression
+    "spark.io.compression.codec": "lz4",
+    "spark.shuffle.compress": "true",
+
+    # Broadcast
+    "spark.sql.autoBroadcastJoinThreshold": "50MB",
+
+    # File handling
+    "spark.sql.files.maxPartitionBytes": "128MB",
+    "spark.sql.files.openCostInBytes": "4MB",
+}
+```
+
+## Best Practices
+
+### Do's
+
+- **Enable AQE** - Adaptive query execution handles many issues
+- **Use Parquet/Delta** - Columnar formats with compression
+- **Broadcast small tables** - Avoid shuffle for small joins
+- **Monitor Spark UI** - Check for skew, spills, GC
+- **Right-size partitions** - 128MB - 256MB per partition
+
+### Don'ts
+
+- **Don't collect large data** - Keep data distributed
+- **Don't use UDFs unnecessarily** - Use built-in functions
+- **Don't over-cache** - Memory is limited
+- **Don't ignore data skew** - It dominates job time
+- **Don't use `.count()` for existence** - Use `.take(1)` or `.isEmpty()`