@synsci/cli-darwin-x64 1.1.97 → 1.1.99

package/bin/skills/dask/references/best-practices.md

@@ -1,277 +0,0 @@
-# Dask Best Practices
-
-## Performance Optimization Principles
-
-### Start with Simpler Solutions First
-
-Before implementing parallel computing with Dask, explore these alternatives:
-- Better algorithms for the specific problem
-- Efficient file formats (Parquet, HDF5, Zarr instead of CSV)
-- Compiled code via Numba or Cython
-- Data sampling for development and testing
-
-These alternatives often provide better returns than distributed systems and should be exhausted before scaling to parallel computing.
-
-### Chunk Size Strategy
-
-**Critical Rule**: Chunks should be small enough that many fit in a worker's available memory at once.
-
-**Recommended Target**: Size chunks so workers can hold 10 chunks per core without exceeding available memory.
-
-**Why It Matters**:
-- Too large chunks: Memory overflow and inefficient parallelization
-- Too small chunks: Excessive scheduling overhead
-
-**Example Calculation**:
-- 8 cores with 32 GB RAM
-- Target: ~400 MB per chunk (32 GB / 8 cores / 10 chunks)
-
-### Monitor with the Dashboard
-
-The Dask dashboard provides essential visibility into:
-- Worker states and resource utilization
-- Task progress and bottlenecks
-- Memory usage patterns
-- Performance characteristics
-
-Access the dashboard to understand what's actually slow in parallel workloads rather than guessing at optimizations.
-
-## Critical Pitfalls to Avoid
-
-### 1. Don't Create Large Objects Locally Before Dask
-
-**Wrong Approach**:
-```python
-import pandas as pd
-import dask.dataframe as dd
-
-# Loads entire dataset into memory first
-df = pd.read_csv('large_file.csv')
-ddf = dd.from_pandas(df, npartitions=10)
-```
-
-**Correct Approach**:
-```python
-import dask.dataframe as dd
-
-# Let Dask handle the loading
-ddf = dd.read_csv('large_file.csv')
-```
-
-**Why**: Loading data with pandas or NumPy first forces the scheduler to serialize and embed those objects in task graphs, defeating the purpose of parallel computing.
-
-**Key Principle**: Use Dask methods to load data and use Dask to control the results.
-
-### 2. Avoid Repeated compute() Calls
-
-**Wrong Approach**:
-```python
-results = []
-for item in items:
-    result = dask_computation(item).compute()  # Each compute is separate
-    results.append(result)
-```
-
-**Correct Approach**:
-```python
-computations = [dask_computation(item) for item in items]
-results = dask.compute(*computations)  # Single compute for all
-```
-
-**Why**: Calling compute in loops prevents Dask from:
-- Parallelizing different computations
-- Sharing intermediate results
-- Optimizing the overall task graph
-
-### 3. Don't Build Excessively Large Task Graphs
-
-**Symptoms**:
-- Millions of tasks in a single computation
-- Severe scheduling overhead
-- Long delays before computation starts
-
-**Solutions**:
-- Increase chunk sizes to reduce number of tasks
-- Use `map_partitions` or `map_blocks` to fuse operations
-- Break computations into smaller pieces with intermediate persists
-- Consider whether the problem truly requires distributed computing
-
-**Example Using map_partitions**:
-```python
-# Instead of applying function to each row
-ddf['result'] = ddf.apply(complex_function, axis=1)  # Many tasks
-
-# Apply to entire partitions at once
-ddf = ddf.map_partitions(lambda df: df.assign(result=complex_function(df)))
-```
-
-## Infrastructure Considerations
-
-### Scheduler Selection
-
-**Use Threads For**:
-- Numeric work with GIL-releasing libraries (NumPy, Pandas, scikit-learn)
-- Operations that benefit from shared memory
-- Single-machine workloads with array/dataframe operations
-
-**Use Processes For**:
-- Text processing and Python collection operations
-- Pure Python code that's GIL-bound
-- Operations that need process isolation
-
-**Use Distributed Scheduler For**:
-- Multi-machine clusters
-- Need for diagnostic dashboard
-- Asynchronous APIs
-- Better data locality handling
-
-### Thread Configuration
-
-**Recommendation**: Aim for roughly 4 threads per process on numeric workloads.
-
-**Rationale**:
-- Balance between parallelism and overhead
-- Allows efficient use of CPU cores
-- Reduces context switching costs
-
-### Memory Management
-
-**Persist Strategically**:
-```python
-# Persist intermediate results that are reused
-intermediate = expensive_computation(data).persist()
-result1 = intermediate.operation1().compute()
-result2 = intermediate.operation2().compute()
-```
-
-**Clear Memory When Done**:
-```python
-# Explicitly delete large objects
-del intermediate
-```
-
-## Data Loading Best Practices
-
-### Use Appropriate File Formats
-
-**For Tabular Data**:
-- Parquet: Columnar, compressed, fast filtering
-- CSV: Only for small data or initial ingestion
-
-**For Array Data**:
-- HDF5: Good for numeric arrays
-- Zarr: Cloud-native, parallel-friendly
-- NetCDF: Scientific data with metadata
-
-### Optimize Data Ingestion
-
-**Read Multiple Files Efficiently**:
-```python
-# Use glob patterns to read multiple files in parallel
-ddf = dd.read_parquet('data/year=2024/month=*/day=*.parquet')
-```
-
-**Specify Useful Columns Early**:
-```python
-# Only read needed columns
-ddf = dd.read_parquet('data.parquet', columns=['col1', 'col2', 'col3'])
-```
-
-## Common Patterns and Solutions
-
-### Pattern: Embarrassingly Parallel Problems
-
-For independent computations, use Futures:
-```python
-from dask.distributed import Client
-
-client = Client()
-futures = [client.submit(func, arg) for arg in args]
-results = client.gather(futures)
-```
-
-### Pattern: Data Preprocessing Pipeline
-
-Use Bags for initial ETL, then convert to structured formats:
-```python
-import dask.bag as db
-
-# Process raw JSON
-bag = db.read_text('logs/*.json').map(json.loads)
-bag = bag.filter(lambda x: x['status'] == 'success')
-
-# Convert to DataFrame for analysis
-ddf = bag.to_dataframe()
-```
-
-### Pattern: Iterative Algorithms
-
-Persist data between iterations:
-```python
-data = dd.read_parquet('data.parquet')
-data = data.persist()  # Keep in memory across iterations
-
-for iteration in range(num_iterations):
-    data = update_function(data)
-    data = data.persist()  # Persist updated version
-```
-
-## Debugging Tips
-
-### Use Single-Threaded Scheduler
-
-For debugging with pdb or detailed error inspection:
-```python
-import dask
-
-dask.config.set(scheduler='synchronous')
-result = computation.compute()  # Runs in single thread for debugging
-```
-
-### Check Task Graph Size
-
-Before computing, check the number of tasks:
-```python
-print(len(ddf.__dask_graph__()))  # Should be reasonable, not millions
-```
-
-### Validate on Small Data First
-
-Test logic on small subset before scaling:
-```python
-# Test on first partition
-sample = ddf.head(1000)
-# Validate results
-# Then scale to full dataset
-```
-
-## Performance Troubleshooting
-
-### Symptom: Slow Computation Start
-
-**Likely Cause**: Task graph is too large
-**Solution**: Increase chunk sizes or use map_partitions
-
-### Symptom: Memory Errors
-
-**Likely Causes**:
-- Chunks too large
-- Too many intermediate results
-- Memory leaks in user functions
-
-**Solutions**:
-- Decrease chunk sizes
-- Use persist() strategically and delete when done
-- Profile user functions for memory issues
-
-### Symptom: Poor Parallelization
-
-**Likely Causes**:
-- Data dependencies preventing parallelism
-- Chunks too large (not enough tasks)
-- GIL contention with threads on Python code
-
-**Solutions**:
-- Restructure computation to reduce dependencies
-- Increase number of partitions
-- Switch to multiprocessing scheduler for Python code

package/bin/skills/dask/references/dataframes.md

@@ -1,368 +0,0 @@
-# Dask DataFrames
-
-## Overview
-
-Dask DataFrames enable parallel processing of large tabular data by distributing work across multiple pandas DataFrames. As described in the documentation, "Dask DataFrames are a collection of many pandas DataFrames" with identical APIs, making the transition from pandas straightforward.
-
-## Core Concept
-
-A Dask DataFrame is divided into multiple pandas DataFrames (partitions) along the index:
-- Each partition is a regular pandas DataFrame
-- Operations are applied to each partition in parallel
-- Results are combined automatically
-
-## Key Capabilities
-
-### Scale
-- Process 100 GiB on a laptop
-- Process 100 TiB on a cluster
-- Handle datasets exceeding available RAM
-
-### Compatibility
-- Implements most of the pandas API
-- Easy transition from pandas code
-- Works with familiar operations
-
-## When to Use Dask DataFrames
-
-**Use Dask When**:
-- Dataset exceeds available RAM
-- Computations require significant time and pandas optimization hasn't helped
-- Need to scale from prototype (pandas) to production (larger data)
-- Working with multiple files that should be processed together
-
-**Stick with Pandas When**:
-- Data fits comfortably in memory
-- Computations complete in subseconds
-- Simple operations without custom `.apply()` functions
-- Iterative development and exploration
-
-## Reading Data
-
-Dask mirrors pandas reading syntax with added support for multiple files:
-
-### Single File
-```python
-import dask.dataframe as dd
-
-# Read single file
-ddf = dd.read_csv('data.csv')
-ddf = dd.read_parquet('data.parquet')
-```
-
-### Multiple Files
-```python
-# Read multiple files using glob patterns
-ddf = dd.read_csv('data/*.csv')
-ddf = dd.read_parquet('s3://mybucket/data/*.parquet')
-
-# Read with path structure
-ddf = dd.read_parquet('data/year=*/month=*/day=*.parquet')
-```
-
-### Optimizations
-```python
-# Specify columns to read (reduces memory)
-ddf = dd.read_parquet('data.parquet', columns=['col1', 'col2'])
-
-# Control partitioning
-ddf = dd.read_csv('data.csv', blocksize='64MB')  # Creates 64MB partitions
-```
-
-## Common Operations
-
-All operations are lazy until `.compute()` is called.
-
-### Filtering
-```python
-# Same as pandas
-filtered = ddf[ddf['column'] > 100]
-filtered = ddf.query('column > 100')
-```
-
-### Column Operations
-```python
-# Add columns
-ddf['new_column'] = ddf['col1'] + ddf['col2']
-
-# Select columns
-subset = ddf[['col1', 'col2', 'col3']]
-
-# Drop columns
-ddf = ddf.drop(columns=['unnecessary_col'])
-```
-
-### Aggregations
-```python
-# Standard aggregations work as expected
-mean = ddf['column'].mean().compute()
-sum_total = ddf['column'].sum().compute()
-counts = ddf['category'].value_counts().compute()
-```
-
-### GroupBy
-```python
-# GroupBy operations (may require shuffle)
-grouped = ddf.groupby('category')['value'].mean().compute()
-
-# Multiple aggregations
-agg_result = ddf.groupby('category').agg({
-    'value': ['mean', 'sum', 'count'],
-    'amount': 'sum'
-}).compute()
-```
-
-### Joins and Merges
-```python
-# Merge DataFrames
-merged = dd.merge(ddf1, ddf2, on='key', how='left')
-
-# Join on index
-joined = ddf1.join(ddf2, on='key')
-```
-
-### Sorting
-```python
-# Sorting (expensive operation, requires data movement)
-sorted_ddf = ddf.sort_values('column')
-result = sorted_ddf.compute()
-```
-
-## Custom Operations
-
-### Apply Functions
-
-**To Partitions (Efficient)**:
-```python
-# Apply function to entire partitions
-def custom_partition_function(partition_df):
-    # partition_df is a pandas DataFrame
-    return partition_df.assign(new_col=partition_df['col1'] * 2)
-
-ddf = ddf.map_partitions(custom_partition_function)
-```
-
-**To Rows (Less Efficient)**:
-```python
-# Apply to each row (creates many tasks)
-ddf['result'] = ddf.apply(lambda row: custom_function(row), axis=1, meta=('result', 'float'))
-```
-
-**Note**: Always prefer `map_partitions` over row-wise `apply` for better performance.
-
-### Meta Parameter
-
-When Dask can't infer output structure, specify the `meta` parameter:
-```python
-# For apply operations
-ddf['new'] = ddf.apply(func, axis=1, meta=('new', 'float64'))
-
-# For map_partitions
-ddf = ddf.map_partitions(func, meta=pd.DataFrame({
-    'col1': pd.Series(dtype='float64'),
-    'col2': pd.Series(dtype='int64')
-}))
-```
-
-## Lazy Evaluation and Computation
-
-### Lazy Operations
-```python
-# These operations are lazy (instant, no computation)
-filtered = ddf[ddf['value'] > 100]
-aggregated = filtered.groupby('category').mean()
-final = aggregated[aggregated['value'] < 500]
-
-# Nothing has computed yet
-```
-
-### Triggering Computation
-```python
-# Compute single result
-result = final.compute()
-
-# Compute multiple results efficiently
-result1, result2, result3 = dask.compute(
-    operation1,
-    operation2,
-    operation3
-)
-```
-
-### Persist in Memory
-```python
-# Keep results in distributed memory for reuse
-ddf_cached = ddf.persist()
-
-# Now multiple operations on ddf_cached won't recompute
-result1 = ddf_cached.mean().compute()
-result2 = ddf_cached.sum().compute()
-```
-
-## Index Management
-
-### Setting Index
-```python
-# Set index (required for efficient joins and certain operations)
-ddf = ddf.set_index('timestamp', sorted=True)
-```
-
-### Index Properties
-- Sorted index enables efficient filtering and joins
-- Index determines partitioning
-- Some operations perform better with appropriate index
-
-## Writing Results
-
-### To Files
-```python
-# Write to multiple files (one per partition)
-ddf.to_parquet('output/data.parquet')
-ddf.to_csv('output/data-*.csv')
-
-# Write to single file (forces computation and concatenation)
-ddf.compute().to_csv('output/single_file.csv')
-```
-
-### To Memory (Pandas)
-```python
-# Convert to pandas (loads all data in memory)
-pdf = ddf.compute()
-```
-
-## Performance Considerations
-
-### Efficient Operations
-- Column selection and filtering: Very efficient
-- Simple aggregations (sum, mean, count): Efficient
-- Row-wise operations on partitions: Efficient with `map_partitions`
-
-### Expensive Operations
-- Sorting: Requires data shuffle across workers
-- GroupBy with many groups: May require shuffle
-- Complex joins: Depends on data distribution
-- Row-wise apply: Creates many tasks
-
-### Optimization Tips
-
-**1. Select Columns Early**
-```python
-# Better: Read only needed columns
-ddf = dd.read_parquet('data.parquet', columns=['col1', 'col2'])
-```
-
-**2. Filter Before GroupBy**
-```python
-# Better: Reduce data before expensive operations
-result = ddf[ddf['year'] == 2024].groupby('category').sum().compute()
-```
-
-**3. Use Efficient File Formats**
-```python
-# Use Parquet instead of CSV for better performance
-ddf.to_parquet('data.parquet')  # Faster, smaller, columnar
-```
-
-**4. Repartition Appropriately**
-```python
-# If partitions are too small
-ddf = ddf.repartition(npartitions=10)
-
-# If partitions are too large
-ddf = ddf.repartition(partition_size='100MB')
-```
-
-## Common Patterns
-
-### ETL Pipeline
-```python
-import dask.dataframe as dd
-
-# Read data
-ddf = dd.read_csv('raw_data/*.csv')
-
-# Transform
-ddf = ddf[ddf['status'] == 'valid']
-ddf['amount'] = ddf['amount'].astype('float64')
-ddf = ddf.dropna(subset=['important_col'])
-
-# Aggregate
-summary = ddf.groupby('category').agg({
-    'amount': ['sum', 'mean'],
-    'quantity': 'count'
-})
-
-# Write results
-summary.to_parquet('output/summary.parquet')
-```
-
-### Time Series Analysis
-```python
-# Read time series data
-ddf = dd.read_parquet('timeseries/*.parquet')
-
-# Set timestamp index
-ddf = ddf.set_index('timestamp', sorted=True)
-
-# Resample (if available in Dask version)
-hourly = ddf.resample('1H').mean()
-
-# Compute statistics
-result = hourly.compute()
-```
-
-### Combining Multiple Files
-```python
-# Read multiple files as single DataFrame
-ddf = dd.read_csv('data/2024-*.csv')
-
-# Process combined data
-result = ddf.groupby('category')['value'].sum().compute()
-```
-
-## Limitations and Differences from Pandas
-
-### Not All Pandas Features Available
-Some pandas operations are not implemented in Dask:
-- Some string methods
-- Certain window functions
-- Some specialized statistical functions
-
-### Partitioning Matters
-- Operations within partitions are efficient
-- Cross-partition operations may be expensive
-- Index-based operations benefit from sorted index
-
-### Lazy Evaluation
-- Operations don't execute until `.compute()`
-- Need to be aware of computation triggers
-- Can't inspect intermediate results without computing
-
-## Debugging Tips
-
-### Inspect Partitions
-```python
-# Get number of partitions
-print(ddf.npartitions)
-
-# Compute single partition
-first_partition = ddf.get_partition(0).compute()
-
-# View first few rows (computes first partition)
-print(ddf.head())
-```
-
-### Validate Operations on Small Data
-```python
-# Test on small sample first
-sample = ddf.head(1000)
-# Validate logic works
-# Then scale to full dataset
-result = ddf.compute()
-```
-
-### Check Dtypes
-```python
-# Verify data types are correct
-print(ddf.dtypes)
-```