@synsci/cli-darwin-x64-baseline 1.1.77 → 1.1.78

package/bin/skills/dask/references/best-practices.md

@@ -0,0 +1,277 @@
+# Dask Best Practices
+
+## Performance Optimization Principles
+
+### Start with Simpler Solutions First
+
+Before implementing parallel computing with Dask, explore these alternatives:
+- Better algorithms for the specific problem
+- Efficient file formats (Parquet, HDF5, Zarr instead of CSV)
+- Compiled code via Numba or Cython
+- Data sampling for development and testing
+
+These alternatives often provide better returns than distributed systems and should be exhausted before scaling to parallel computing.
+
+### Chunk Size Strategy
+
+**Critical Rule**: Chunks should be small enough that many fit in a worker's available memory at once.
+
+**Recommended Target**: Size chunks so workers can hold 10 chunks per core without exceeding available memory.
+
+**Why It Matters**:
+- Too large chunks: Memory overflow and inefficient parallelization
+- Too small chunks: Excessive scheduling overhead
+
+**Example Calculation**:
+- 8 cores with 32 GB RAM
+- Target: ~400 MB per chunk (32 GB / 8 cores / 10 chunks)
+
+### Monitor with the Dashboard
+
+The Dask dashboard provides essential visibility into:
+- Worker states and resource utilization
+- Task progress and bottlenecks
+- Memory usage patterns
+- Performance characteristics
+
+Access the dashboard to understand what's actually slow in parallel workloads rather than guessing at optimizations.
+
+## Critical Pitfalls to Avoid
+
+### 1. Don't Create Large Objects Locally Before Dask
+
+**Wrong Approach**:
+```python
+import pandas as pd
+import dask.dataframe as dd
+
+# Loads entire dataset into memory first
+df = pd.read_csv('large_file.csv')
+ddf = dd.from_pandas(df, npartitions=10)
+```
+
+**Correct Approach**:
+```python
+import dask.dataframe as dd
+
+# Let Dask handle the loading
+ddf = dd.read_csv('large_file.csv')
+```
+
+**Why**: Loading data with pandas or NumPy first forces the scheduler to serialize and embed those objects in task graphs, defeating the purpose of parallel computing.
+
+**Key Principle**: Use Dask methods to load data and use Dask to control the results.
+
+### 2. Avoid Repeated compute() Calls
+
+**Wrong Approach**:
+```python
+results = []
+for item in items:
+    result = dask_computation(item).compute()  # Each compute is separate
+    results.append(result)
+```
+
+**Correct Approach**:
+```python
+computations = [dask_computation(item) for item in items]
+results = dask.compute(*computations)  # Single compute for all
+```
+
+**Why**: Calling compute in loops prevents Dask from:
+- Parallelizing different computations
+- Sharing intermediate results
+- Optimizing the overall task graph
+
+### 3. Don't Build Excessively Large Task Graphs
+
+**Symptoms**:
+- Millions of tasks in a single computation
+- Severe scheduling overhead
+- Long delays before computation starts
+
+**Solutions**:
+- Increase chunk sizes to reduce number of tasks
+- Use `map_partitions` or `map_blocks` to fuse operations
+- Break computations into smaller pieces with intermediate persists
+- Consider whether the problem truly requires distributed computing
+
+**Example Using map_partitions**:
+```python
+# Instead of applying function to each row
+ddf['result'] = ddf.apply(complex_function, axis=1)  # Many tasks
+
+# Apply to entire partitions at once
+ddf = ddf.map_partitions(lambda df: df.assign(result=complex_function(df)))
+```
+
+## Infrastructure Considerations
+
+### Scheduler Selection
+
+**Use Threads For**:
+- Numeric work with GIL-releasing libraries (NumPy, Pandas, scikit-learn)
+- Operations that benefit from shared memory
+- Single-machine workloads with array/dataframe operations
+
+**Use Processes For**:
+- Text processing and Python collection operations
+- Pure Python code that's GIL-bound
+- Operations that need process isolation
+
+**Use Distributed Scheduler For**:
+- Multi-machine clusters
+- Need for diagnostic dashboard
+- Asynchronous APIs
+- Better data locality handling
+
+### Thread Configuration
+
+**Recommendation**: Aim for roughly 4 threads per process on numeric workloads.
+
+**Rationale**:
+- Balance between parallelism and overhead
+- Allows efficient use of CPU cores
+- Reduces context switching costs
+
+### Memory Management
+
+**Persist Strategically**:
+```python
+# Persist intermediate results that are reused
+intermediate = expensive_computation(data).persist()
+result1 = intermediate.operation1().compute()
+result2 = intermediate.operation2().compute()
+```
+
+**Clear Memory When Done**:
+```python
+# Explicitly delete large objects
+del intermediate
+```
+
+## Data Loading Best Practices
+
+### Use Appropriate File Formats
+
+**For Tabular Data**:
+- Parquet: Columnar, compressed, fast filtering
+- CSV: Only for small data or initial ingestion
+
+**For Array Data**:
+- HDF5: Good for numeric arrays
+- Zarr: Cloud-native, parallel-friendly
+- NetCDF: Scientific data with metadata
+
+### Optimize Data Ingestion
+
+**Read Multiple Files Efficiently**:
+```python
+# Use glob patterns to read multiple files in parallel
+ddf = dd.read_parquet('data/year=2024/month=*/day=*.parquet')
+```
+
+**Specify Useful Columns Early**:
+```python
+# Only read needed columns
+ddf = dd.read_parquet('data.parquet', columns=['col1', 'col2', 'col3'])
+```
+
+## Common Patterns and Solutions
+
+### Pattern: Embarrassingly Parallel Problems
+
+For independent computations, use Futures:
+```python
+from dask.distributed import Client
+
+client = Client()
+futures = [client.submit(func, arg) for arg in args]
+results = client.gather(futures)
+```
+
+### Pattern: Data Preprocessing Pipeline
+
+Use Bags for initial ETL, then convert to structured formats:
+```python
+import dask.bag as db
+
+# Process raw JSON
+bag = db.read_text('logs/*.json').map(json.loads)
+bag = bag.filter(lambda x: x['status'] == 'success')
+
+# Convert to DataFrame for analysis
+ddf = bag.to_dataframe()
+```
+
+### Pattern: Iterative Algorithms
+
+Persist data between iterations:
+```python
+data = dd.read_parquet('data.parquet')
+data = data.persist()  # Keep in memory across iterations
+
+for iteration in range(num_iterations):
+    data = update_function(data)
+    data = data.persist()  # Persist updated version
+```
+
+## Debugging Tips
+
+### Use Single-Threaded Scheduler
+
+For debugging with pdb or detailed error inspection:
+```python
+import dask
+
+dask.config.set(scheduler='synchronous')
+result = computation.compute()  # Runs in single thread for debugging
+```
+
+### Check Task Graph Size
+
+Before computing, check the number of tasks:
+```python
+print(len(ddf.__dask_graph__()))  # Should be reasonable, not millions
+```
+
+### Validate on Small Data First
+
+Test logic on small subset before scaling:
+```python
+# Test on first partition
+sample = ddf.head(1000)
+# Validate results
+# Then scale to full dataset
+```
+
+## Performance Troubleshooting
+
+### Symptom: Slow Computation Start
+
+**Likely Cause**: Task graph is too large
+**Solution**: Increase chunk sizes or use map_partitions
+
+### Symptom: Memory Errors
+
+**Likely Causes**:
+- Chunks too large
+- Too many intermediate results
+- Memory leaks in user functions
+
+**Solutions**:
+- Decrease chunk sizes
+- Use persist() strategically and delete when done
+- Profile user functions for memory issues
+
+### Symptom: Poor Parallelization
+
+**Likely Causes**:
+- Data dependencies preventing parallelism
+- Chunks too large (not enough tasks)
+- GIL contention with threads on Python code
+
+**Solutions**:
+- Restructure computation to reduce dependencies
+- Increase number of partitions
+- Switch to multiprocessing scheduler for Python code

package/bin/skills/dask/references/dataframes.md

@@ -0,0 +1,368 @@
+# Dask DataFrames
+
+## Overview
+
+Dask DataFrames enable parallel processing of large tabular data by distributing work across multiple pandas DataFrames. As described in the documentation, "Dask DataFrames are a collection of many pandas DataFrames" with identical APIs, making the transition from pandas straightforward.
+
+## Core Concept
+
+A Dask DataFrame is divided into multiple pandas DataFrames (partitions) along the index:
+- Each partition is a regular pandas DataFrame
+- Operations are applied to each partition in parallel
+- Results are combined automatically
+
+## Key Capabilities
+
+### Scale
+- Process 100 GiB on a laptop
+- Process 100 TiB on a cluster
+- Handle datasets exceeding available RAM
+
+### Compatibility
+- Implements most of the pandas API
+- Easy transition from pandas code
+- Works with familiar operations
+
+## When to Use Dask DataFrames
+
+**Use Dask When**:
+- Dataset exceeds available RAM
+- Computations require significant time and pandas optimization hasn't helped
+- Need to scale from prototype (pandas) to production (larger data)
+- Working with multiple files that should be processed together
+
+**Stick with Pandas When**:
+- Data fits comfortably in memory
+- Computations complete in subseconds
+- Simple operations without custom `.apply()` functions
+- Iterative development and exploration
+
+## Reading Data
+
+Dask mirrors pandas reading syntax with added support for multiple files:
+
+### Single File
+```python
+import dask.dataframe as dd
+
+# Read single file
+ddf = dd.read_csv('data.csv')
+ddf = dd.read_parquet('data.parquet')
+```
+
+### Multiple Files
+```python
+# Read multiple files using glob patterns
+ddf = dd.read_csv('data/*.csv')
+ddf = dd.read_parquet('s3://mybucket/data/*.parquet')
+
+# Read with path structure
+ddf = dd.read_parquet('data/year=*/month=*/day=*.parquet')
+```
+
+### Optimizations
+```python
+# Specify columns to read (reduces memory)
+ddf = dd.read_parquet('data.parquet', columns=['col1', 'col2'])
+
+# Control partitioning
+ddf = dd.read_csv('data.csv', blocksize='64MB')  # Creates 64MB partitions
+```
+
+## Common Operations
+
+All operations are lazy until `.compute()` is called.
+
+### Filtering
+```python
+# Same as pandas
+filtered = ddf[ddf['column'] > 100]
+filtered = ddf.query('column > 100')
+```
+
+### Column Operations
+```python
+# Add columns
+ddf['new_column'] = ddf['col1'] + ddf['col2']
+
+# Select columns
+subset = ddf[['col1', 'col2', 'col3']]
+
+# Drop columns
+ddf = ddf.drop(columns=['unnecessary_col'])
+```
+
+### Aggregations
+```python
+# Standard aggregations work as expected
+mean = ddf['column'].mean().compute()
+sum_total = ddf['column'].sum().compute()
+counts = ddf['category'].value_counts().compute()
+```
+
+### GroupBy
+```python
+# GroupBy operations (may require shuffle)
+grouped = ddf.groupby('category')['value'].mean().compute()
+
+# Multiple aggregations
+agg_result = ddf.groupby('category').agg({
+    'value': ['mean', 'sum', 'count'],
+    'amount': 'sum'
+}).compute()
+```
+
+### Joins and Merges
+```python
+# Merge DataFrames
+merged = dd.merge(ddf1, ddf2, on='key', how='left')
+
+# Join on index
+joined = ddf1.join(ddf2, on='key')
+```
+
+### Sorting
+```python
+# Sorting (expensive operation, requires data movement)
+sorted_ddf = ddf.sort_values('column')
+result = sorted_ddf.compute()
+```
+
+## Custom Operations
+
+### Apply Functions
+
+**To Partitions (Efficient)**:
+```python
+# Apply function to entire partitions
+def custom_partition_function(partition_df):
+    # partition_df is a pandas DataFrame
+    return partition_df.assign(new_col=partition_df['col1'] * 2)
+
+ddf = ddf.map_partitions(custom_partition_function)
+```
+
+**To Rows (Less Efficient)**:
+```python
+# Apply to each row (creates many tasks)
+ddf['result'] = ddf.apply(lambda row: custom_function(row), axis=1, meta=('result', 'float'))
+```
+
+**Note**: Always prefer `map_partitions` over row-wise `apply` for better performance.
+
+### Meta Parameter
+
+When Dask can't infer output structure, specify the `meta` parameter:
+```python
+# For apply operations
+ddf['new'] = ddf.apply(func, axis=1, meta=('new', 'float64'))
+
+# For map_partitions
+ddf = ddf.map_partitions(func, meta=pd.DataFrame({
+    'col1': pd.Series(dtype='float64'),
+    'col2': pd.Series(dtype='int64')
+}))
+```
+
+## Lazy Evaluation and Computation
+
+### Lazy Operations
+```python
+# These operations are lazy (instant, no computation)
+filtered = ddf[ddf['value'] > 100]
+aggregated = filtered.groupby('category').mean()
+final = aggregated[aggregated['value'] < 500]
+
+# Nothing has computed yet
+```
+
+### Triggering Computation
+```python
+# Compute single result
+result = final.compute()
+
+# Compute multiple results efficiently
+result1, result2, result3 = dask.compute(
+    operation1,
+    operation2,
+    operation3
+)
+```
+
+### Persist in Memory
+```python
+# Keep results in distributed memory for reuse
+ddf_cached = ddf.persist()
+
+# Now multiple operations on ddf_cached won't recompute
+result1 = ddf_cached.mean().compute()
+result2 = ddf_cached.sum().compute()
+```
+
+## Index Management
+
+### Setting Index
+```python
+# Set index (required for efficient joins and certain operations)
+ddf = ddf.set_index('timestamp', sorted=True)
+```
+
+### Index Properties
+- Sorted index enables efficient filtering and joins
+- Index determines partitioning
+- Some operations perform better with appropriate index
+
+## Writing Results
+
+### To Files
+```python
+# Write to multiple files (one per partition)
+ddf.to_parquet('output/data.parquet')
+ddf.to_csv('output/data-*.csv')
+
+# Write to single file (forces computation and concatenation)
+ddf.compute().to_csv('output/single_file.csv')
+```
+
+### To Memory (Pandas)
+```python
+# Convert to pandas (loads all data in memory)
+pdf = ddf.compute()
+```
+
+## Performance Considerations
+
+### Efficient Operations
+- Column selection and filtering: Very efficient
+- Simple aggregations (sum, mean, count): Efficient
+- Row-wise operations on partitions: Efficient with `map_partitions`
+
+### Expensive Operations
+- Sorting: Requires data shuffle across workers
+- GroupBy with many groups: May require shuffle
+- Complex joins: Depends on data distribution
+- Row-wise apply: Creates many tasks
+
+### Optimization Tips
+
+**1. Select Columns Early**
+```python
+# Better: Read only needed columns
+ddf = dd.read_parquet('data.parquet', columns=['col1', 'col2'])
+```
+
+**2. Filter Before GroupBy**
+```python
+# Better: Reduce data before expensive operations
+result = ddf[ddf['year'] == 2024].groupby('category').sum().compute()
+```
+
+**3. Use Efficient File Formats**
+```python
+# Use Parquet instead of CSV for better performance
+ddf.to_parquet('data.parquet')  # Faster, smaller, columnar
+```
+
+**4. Repartition Appropriately**
+```python
+# If partitions are too small
+ddf = ddf.repartition(npartitions=10)
+
+# If partitions are too large
+ddf = ddf.repartition(partition_size='100MB')
+```
+
+## Common Patterns
+
+### ETL Pipeline
+```python
+import dask.dataframe as dd
+
+# Read data
+ddf = dd.read_csv('raw_data/*.csv')
+
+# Transform
+ddf = ddf[ddf['status'] == 'valid']
+ddf['amount'] = ddf['amount'].astype('float64')
+ddf = ddf.dropna(subset=['important_col'])
+
+# Aggregate
+summary = ddf.groupby('category').agg({
+    'amount': ['sum', 'mean'],
+    'quantity': 'count'
+})
+
+# Write results
+summary.to_parquet('output/summary.parquet')
+```
+
+### Time Series Analysis
+```python
+# Read time series data
+ddf = dd.read_parquet('timeseries/*.parquet')
+
+# Set timestamp index
+ddf = ddf.set_index('timestamp', sorted=True)
+
+# Resample (if available in Dask version)
+hourly = ddf.resample('1H').mean()
+
+# Compute statistics
+result = hourly.compute()
+```
+
+### Combining Multiple Files
+```python
+# Read multiple files as single DataFrame
+ddf = dd.read_csv('data/2024-*.csv')
+
+# Process combined data
+result = ddf.groupby('category')['value'].sum().compute()
+```
+
+## Limitations and Differences from Pandas
+
+### Not All Pandas Features Available
+Some pandas operations are not implemented in Dask:
+- Some string methods
+- Certain window functions
+- Some specialized statistical functions
+
+### Partitioning Matters
+- Operations within partitions are efficient
+- Cross-partition operations may be expensive
+- Index-based operations benefit from sorted index
+
+### Lazy Evaluation
+- Operations don't execute until `.compute()`
+- Need to be aware of computation triggers
+- Can't inspect intermediate results without computing
+
+## Debugging Tips
+
+### Inspect Partitions
+```python
+# Get number of partitions
+print(ddf.npartitions)
+
+# Compute single partition
+first_partition = ddf.get_partition(0).compute()
+
+# View first few rows (computes first partition)
+print(ddf.head())
+```
+
+### Validate Operations on Small Data
+```python
+# Test on small sample first
+sample = ddf.head(1000)
+# Validate logic works
+# Then scale to full dataset
+result = ddf.compute()
+```
+
+### Check Dtypes
+```python
+# Verify data types are correct
+print(ddf.dtypes)
+```