npm - aigroup-workflow - Versions diffs - 2.1.1 → 2.2.0 - Mend

aigroup-workflow 2.1.1 → 2.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (914) hide show

package/skills/spark-engineer/references/rdd-operations.md DELETED Viewed

@@ -1,599 +0,0 @@
-# RDD Operations
----
-## When to Use RDDs
-**Use RDDs when:**
-- Processing unstructured data (raw text, custom binary formats)
-- Need fine-grained control over physical data placement
-- Implementing custom partitioning logic for specific access patterns
-- Working with legacy Spark code that needs maintenance
-- Building custom data structures not expressible as DataFrames
-**Prefer DataFrames when:**
-- Processing structured/semi-structured data
-- Performing SQL-like operations
-- Need Catalyst optimizer benefits
-- Working with standard file formats (Parquet, JSON, ORC)
----
-## RDD Creation
-### From Collections
-```python
-# PySpark - Create RDD from Python collection
-data = [1, 2, 3, 4, 5]
-rdd = spark.sparkContext.parallelize(data, numSlices=4)  # 4 partitions
-# From key-value pairs
-pairs = [("a", 1), ("b", 2), ("c", 3)]
-pair_rdd = spark.sparkContext.parallelize(pairs)
-```
-```scala
-// Scala - Create RDD from collection
-val data = Seq(1, 2, 3, 4, 5)
-val rdd = spark.sparkContext.parallelize(data, numSlices = 4)
-// From key-value pairs
-val pairs = Seq(("a", 1), ("b", 2), ("c", 3))
-val pairRdd = spark.sparkContext.parallelize(pairs)
-```
-### From Files
-```python
-# Text files - each line becomes an element
-text_rdd = spark.sparkContext.textFile("hdfs://path/to/files/*.txt")
-# Whole text files - each file as (filename, content) pair
-files_rdd = spark.sparkContext.wholeTextFiles("hdfs://path/to/files/")
-# Binary files
-binary_rdd = spark.sparkContext.binaryFiles("hdfs://path/to/files/")
-# Sequence files (Hadoop)
-seq_rdd = spark.sparkContext.sequenceFile("hdfs://path/to/seqfile",
-    "org.apache.hadoop.io.Text",
-    "org.apache.hadoop.io.IntWritable")
-```
-### From DataFrame
-```python
-# Convert DataFrame to RDD of Rows
-df = spark.read.parquet("s3://bucket/data/")
-row_rdd = df.rdd
-# Access Row fields
-result_rdd = row_rdd.map(lambda row: (row.user_id, row.amount))
-# Convert back to DataFrame
-from pyspark.sql import Row
-df_new = result_rdd.map(lambda x: Row(user_id=x[0], amount=x[1])).toDF()
-```
----
-## Transformations (Lazy)
-Transformations return a new RDD and are not executed until an action is called.
-### Basic Transformations
-```python
-# map - apply function to each element
-squares = rdd.map(lambda x: x ** 2)
-# flatMap - apply function returning iterator, flatten results
-words = text_rdd.flatMap(lambda line: line.split(" "))
-# filter - keep elements matching predicate
-evens = rdd.filter(lambda x: x % 2 == 0)
-# distinct - remove duplicates (causes shuffle)
-unique = rdd.distinct()
-# sample - random sample
-sampled = rdd.sample(withReplacement=False, fraction=0.1, seed=42)
-# union - combine two RDDs
-combined = rdd1.union(rdd2)
-# intersection - elements in both RDDs (causes shuffle)
-common = rdd1.intersection(rdd2)
-# subtract - elements in rdd1 not in rdd2 (causes shuffle)
-diff = rdd1.subtract(rdd2)
-# cartesian - all pairs (expensive!)
-product = rdd1.cartesian(rdd2)
-```
-```scala
-// Scala transformations
-val squares = rdd.map(x => x * x)
-val words = textRdd.flatMap(line => line.split(" "))
-val evens = rdd.filter(_ % 2 == 0)
-val unique = rdd.distinct()
-val sampled = rdd.sample(withReplacement = false, fraction = 0.1, seed = 42L)
-```
-### MapPartitions (Efficient Batch Processing)
-```python
-# Process entire partition at once - more efficient than map
-# Good for: database connections, expensive initialization, batch operations
-def process_partition(iterator):
-    # Initialize expensive resource once per partition
-    connection = create_database_connection()
-    try:
-        for record in iterator:
-            result = connection.process(record)
-            yield result
-    finally:
-        connection.close()
-result_rdd = rdd.mapPartitions(process_partition)
-# With partition index
-def process_with_index(partition_index, iterator):
-    for record in iterator:
-        yield (partition_index, record)
-result_rdd = rdd.mapPartitionsWithIndex(process_with_index)
-```
-```scala
-// Scala mapPartitions
-val result = rdd.mapPartitions { iterator =>
-  val connection = createDatabaseConnection()
-  try {
-    iterator.map(record => connection.process(record))
-  } finally {
-    connection.close()
-  }
-}
-```
-### Repartition and Coalesce
-```python
-# repartition - increase or decrease partitions (full shuffle)
-rdd_repart = rdd.repartition(100)
-# coalesce - decrease partitions only (avoids full shuffle)
-rdd_coalesced = rdd.coalesce(10)  # Efficient reduction
-# glom - collect each partition into an array
-partitions = rdd.glom()  # RDD[Array[T]]
-```
-**When to use:**
-- `repartition(n)`: When increasing partitions or need even distribution
-- `coalesce(n)`: When decreasing partitions (after filter reduced data)
----
-## Pair RDD Operations
-Pair RDDs (key-value pairs) enable powerful transformations.
-### Creating Pair RDDs
-```python
-# From tuples
-pair_rdd = rdd.map(lambda x: (x.key, x.value))
-# keyBy - create pairs from existing elements
-pair_rdd = rdd.keyBy(lambda x: x.user_id)
-```
-### Transformations on Pair RDDs
-```python
-# reduceByKey - aggregate values by key (more efficient than groupByKey)
-counts = pair_rdd.reduceByKey(lambda a, b: a + b)
-# groupByKey - group all values for each key (shuffles all data!)
-grouped = pair_rdd.groupByKey()  # Avoid when possible
-# aggregateByKey - combine with different local/global combiners
-sum_count = pair_rdd.aggregateByKey(
-    zeroValue=(0, 0),  # (sum, count)
-    seqFunc=lambda acc, v: (acc[0] + v, acc[1] + 1),  # within partition
-    combFunc=lambda a, b: (a[0] + b[0], a[1] + b[1])  # across partitions
-)
-# combineByKey - most general aggregation
-averages = pair_rdd.combineByKey(
-    createCombiner=lambda v: (v, 1),
-    mergeValue=lambda acc, v: (acc[0] + v, acc[1] + 1),
-    mergeCombiners=lambda a, b: (a[0] + b[0], a[1] + b[1])
-).mapValues(lambda x: x[0] / x[1])
-# mapValues - transform values only (preserves partitioning)
-doubled = pair_rdd.mapValues(lambda v: v * 2)
-# flatMapValues - flatMap on values only
-expanded = pair_rdd.flatMapValues(lambda v: range(v))
-# keys and values
-keys_rdd = pair_rdd.keys()
-values_rdd = pair_rdd.values()
-# sortByKey
-sorted_rdd = pair_rdd.sortByKey(ascending=True)
-# join operations (all cause shuffle)
-joined = rdd1.join(rdd2)              # inner join
-left = rdd1.leftOuterJoin(rdd2)       # left outer
-right = rdd1.rightOuterJoin(rdd2)     # right outer
-full = rdd1.fullOuterJoin(rdd2)       # full outer
-cogroup = rdd1.cogroup(rdd2)          # group by key from both RDDs
-# subtractByKey - remove keys present in other RDD
-filtered = rdd1.subtractByKey(rdd2)
-```
-```scala
-// Scala pair RDD operations
-val counts = pairRdd.reduceByKey(_ + _)
-val grouped = pairRdd.groupByKey()  // Avoid when possible
-val averages = pairRdd.combineByKey(
-  (v: Int) => (v, 1),
-  (acc: (Int, Int), v: Int) => (acc._1 + v, acc._2 + 1),
-  (a: (Int, Int), b: (Int, Int)) => (a._1 + b._1, a._2 + b._2)
-).mapValues { case (sum, count) => sum.toDouble / count }
-val joined = rdd1.join(rdd2)
-```
-### reduceByKey vs groupByKey
-```python
-# BAD: groupByKey shuffles all values
-# Memory-intensive, can cause OOM
-word_counts = words.map(lambda w: (w, 1)).groupByKey().mapValues(sum)
-# GOOD: reduceByKey combines locally first
-# Much more efficient, less data shuffled
-word_counts = words.map(lambda w: (w, 1)).reduceByKey(lambda a, b: a + b)
-```
-**Spark UI Check:** Compare shuffle write sizes. `reduceByKey` should show much smaller shuffle than `groupByKey` for the same operation.
----
-## Actions (Trigger Execution)
-Actions return values to the driver or write to storage.
-### Collection Actions
-```python
-# collect - return all elements to driver (OOM risk!)
-all_data = rdd.collect()  # Use carefully on large RDDs
-# take - return first n elements
-first_10 = rdd.take(10)
-# takeOrdered - return smallest/largest n elements
-smallest_5 = rdd.takeOrdered(5)  # ascending
-largest_5 = rdd.takeOrdered(5, key=lambda x: -x)
-# takeSample - random sample
-sample = rdd.takeSample(withReplacement=False, num=100, seed=42)
-# first - return first element
-first = rdd.first()
-# top - return largest n elements
-top_5 = rdd.top(5)
-# count - count elements
-total = rdd.count()
-# countByKey - count elements per key (returns dict to driver)
-key_counts = pair_rdd.countByKey()
-# countByValue - count occurrences of each value
-value_counts = rdd.countByValue()
-```
-### Aggregation Actions
-```python
-# reduce - aggregate all elements
-total = rdd.reduce(lambda a, b: a + b)
-# fold - reduce with zero value
-total = rdd.fold(0, lambda a, b: a + b)
-# aggregate - combine with different types
-stats = rdd.aggregate(
-    zeroValue=(0, 0),  # (sum, count)
-    seqOp=lambda acc, v: (acc[0] + v, acc[1] + 1),
-    combOp=lambda a, b: (a[0] + b[0], a[1] + b[1])
-)
-average = stats[0] / stats[1]
-```
-### Output Actions
-```python
-# saveAsTextFile - save as text files
-rdd.saveAsTextFile("hdfs://path/output/")
-# saveAsSequenceFile - save as Hadoop sequence file
-pair_rdd.saveAsSequenceFile("hdfs://path/output/")
-# saveAsPickleFile - Python pickle format
-rdd.saveAsPickleFile("hdfs://path/output/")
-# foreach - apply function to each element (side effects)
-rdd.foreach(lambda x: print(x))  # Runs on executors
-# foreachPartition - apply function to each partition
-def save_partition(iterator):
-    connection = create_connection()
-    for record in iterator:
-        connection.save(record)
-    connection.close()
-rdd.foreachPartition(save_partition)
-```
----
-## Custom Partitioners
-### Implementing Custom Partitioner
-```python
-from pyspark import Partitioner
-class RangePartitioner(Partitioner):
-    def __init__(self, ranges):
-        """
-        ranges: list of (min, max) tuples defining partition boundaries
-        """
-        self.ranges = ranges
-    def numPartitions(self):
-        return len(self.ranges)
-    def getPartition(self, key):
-        for i, (min_val, max_val) in enumerate(self.ranges):
-            if min_val <= key < max_val:
-                return i
-        return len(self.ranges) - 1  # Default to last partition
-# Use custom partitioner
-ranges = [(0, 100), (100, 500), (500, 1000), (1000, float('inf'))]
-partitioner = RangePartitioner(ranges)
-partitioned_rdd = pair_rdd.partitionBy(partitioner.numPartitions(), partitioner.getPartition)
-```
-```scala
-// Scala custom partitioner
-import org.apache.spark.Partitioner
-class DomainPartitioner(numParts: Int) extends Partitioner {
-  override def numPartitions: Int = numParts
-  override def getPartition(key: Any): Int = {
-    val domain = key.asInstanceOf[String].split("@")(1)
-    math.abs(domain.hashCode % numPartitions)
-  }
-  override def equals(other: Any): Boolean = other match {
-    case p: DomainPartitioner => p.numPartitions == numPartitions
-    case _ => false
-  }
-}
-val partitioned = pairRdd.partitionBy(new DomainPartitioner(10))
-```
-### Hash Partitioner (Default)
-```python
-from pyspark import HashPartitioner
-# Repartition with hash partitioner
-partitioned_rdd = pair_rdd.partitionBy(100)  # Uses HashPartitioner
-# Preserve partitioning across transformations
-# mapValues and flatMapValues preserve partitioner
-preserved = partitioned_rdd.mapValues(lambda v: v * 2)
-assert preserved.partitioner == partitioned_rdd.partitioner
-# map does NOT preserve partitioner
-not_preserved = partitioned_rdd.map(lambda x: (x[0], x[1] * 2))
-assert not_preserved.partitioner is None
-```
----
-## Broadcast Variables and Accumulators
-### Broadcast Variables
-```python
-# Broadcast large read-only data to all executors
-lookup_table = {"a": 1, "b": 2, "c": 3}  # Small example
-lookup_broadcast = spark.sparkContext.broadcast(lookup_table)
-def enrich_record(record):
-    table = lookup_broadcast.value  # Access broadcast value
-    return (record, table.get(record, 0))
-enriched_rdd = rdd.map(enrich_record)
-# Clean up when done
-lookup_broadcast.unpersist()
-lookup_broadcast.destroy()
-```
-```scala
-// Scala broadcast
-val lookupTable = Map("a" -> 1, "b" -> 2, "c" -> 3)
-val lookupBroadcast = spark.sparkContext.broadcast(lookupTable)
-val enriched = rdd.map { record =>
-  val table = lookupBroadcast.value
-  (record, table.getOrElse(record, 0))
-}
-```
-### Accumulators
-```python
-# Long accumulator
-error_count = spark.sparkContext.longAccumulator("Error Count")
-def process_record(record):
-    try:
-        return transform(record)
-    except Exception:
-        error_count.add(1)
-        return None
-result_rdd = rdd.map(process_record).filter(lambda x: x is not None)
-result_rdd.count()  # Trigger execution
-print(f"Errors encountered: {error_count.value}")
-# Collection accumulator
-from pyspark import AccumulatorParam
-class SetAccumulatorParam(AccumulatorParam):
-    def zero(self, initial_value):
-        return set()
-    def addInPlace(self, v1, v2):
-        return v1.union(v2)
-error_types = spark.sparkContext.accumulator(set(), SetAccumulatorParam())
-def track_errors(record):
-    try:
-        return process(record)
-    except ValueError:
-        error_types.add({"ValueError"})
-        return None
-    except TypeError:
-        error_types.add({"TypeError"})
-        return None
-```
-**Caution:** Accumulators may be updated more than once if tasks are retried. Use only for debugging/monitoring, not business logic.
----
-## Performance Patterns
-### Avoiding Shuffle
-```python
-# BAD: Multiple shuffles
-result = rdd.groupByKey().mapValues(sum).reduceByKey(max)
-# GOOD: Single shuffle with combineByKey
-result = rdd.combineByKey(
-    createCombiner=lambda v: v,
-    mergeValue=lambda acc, v: acc + v,
-    mergeCombiners=lambda a, b: max(a, b)
-)
-# Co-partition related RDDs to avoid join shuffles
-partitioned_users = users.partitionBy(100)
-partitioned_orders = orders.partitionBy(100)  # Same partitioner
-joined = partitioned_users.join(partitioned_orders)  # No shuffle if same partitioner
-```
-### Efficient Serialization
-```python
-# Use Kryo serialization for better performance
-spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
-spark.conf.set("spark.kryo.registrationRequired", "false")
-# Register custom classes for Kryo (Scala)
-# spark.conf.set("spark.kryo.classesToRegister", "com.example.MyClass")
-```
-### Memory-Efficient Operations
-```python
-# Prefer iterator-based operations
-def efficient_processing(iterator):
-    for record in iterator:
-        # Process one at a time, don't collect
-        yield transform(record)
-result = rdd.mapPartitions(efficient_processing)
-# Avoid collecting large data to driver
-# BAD
-all_keys = rdd.keys().collect()  # Could be millions!
-# GOOD
-key_sample = rdd.keys().take(1000)  # Sample only
-```
----
-## Spark UI Analysis for RDDs
-### Stages Tab Metrics
-| Metric | What to Check |
-|--------|---------------|
-| Shuffle Write | Minimize with reduceByKey over groupByKey |
-| Shuffle Read | Large reads indicate join/aggregation overhead |
-| Spill (Memory) | Indicates partition too large for memory |
-| Spill (Disk) | Data being written to disk - increase memory |
-| GC Time | Should be < 10% of task time |
-### Common Issues
-1. **Uneven partition sizes**: Look for tasks taking much longer than others
-2. **Data skew**: One partition has much more data than others
-3. **Straggler tasks**: A few tasks taking 10x longer than median
-### Debugging Tips
-```python
-# Check partition sizes
-partition_sizes = rdd.glom().map(len).collect()
-print(f"Partition sizes: min={min(partition_sizes)}, max={max(partition_sizes)}, avg={sum(partition_sizes)/len(partition_sizes)}")
-# Check partitioner
-print(f"Partitioner: {rdd.partitioner}")
-print(f"Num partitions: {rdd.getNumPartitions()}")
-# Debug lineage
-print(rdd.toDebugString())
-```
----
-## Best Practices Summary
-1. **Prefer DataFrames** - Use RDDs only when DataFrame API insufficient
-2. **Use reduceByKey over groupByKey** - Combines locally first, reduces shuffle
-3. **Preserve partitioning** - Use mapValues/flatMapValues to keep partitioner
-4. **Minimize shuffles** - Co-partition related RDDs, use broadcast for small data
-5. **Use mapPartitions** - For expensive initialization (DB connections, etc.)
-6. **Avoid collect on large data** - Use take, takeSample, or foreachPartition
-7. **Broadcast lookup tables** - Avoid shuffle for small reference data
-8. **Monitor accumulators** - Use for debugging, not business logic
-9. **Check partition distribution** - Avoid skew with custom partitioners
-10. **Profile with Spark UI** - Identify shuffle, spill, and GC issues