@synsci/cli-darwin-x64-baseline 1.1.77 → 1.1.79

package/bin/skills/vaex/references/performance.md

@@ -0,0 +1,571 @@
+# Performance and Optimization
+
+This reference covers Vaex's performance features including lazy evaluation, caching, memory management, async operations, and optimization strategies for processing massive datasets.
+
+## Understanding Lazy Evaluation
+
+Lazy evaluation is the foundation of Vaex's performance:
+
+### How Lazy Evaluation Works
+
+```python
+import vaex
+
+df = vaex.open('large_file.hdf5')
+
+# No computation happens here - just defines what to compute
+df['total'] = df.price * df.quantity
+df['log_price'] = df.price.log()
+mean_expr = df.total.mean()
+
+# Computation happens here (when result is needed)
+result = mean_expr  # Now the mean is actually calculated
+```
+
+**Key concepts:**
+- **Expressions** are lazy - they define computations without executing them
+- **Materialization** happens when you access the result
+- **Query optimization** happens automatically before execution
+
+### When Does Evaluation Happen?
+
+```python
+# These trigger evaluation:
+print(df.x.mean())                    # Accessing value
+array = df.x.values                   # Getting NumPy array
+pdf = df.to_pandas_df()              # Converting to pandas
+df.export_hdf5('output.hdf5')       # Exporting
+
+# These do NOT trigger evaluation:
+df['new_col'] = df.x + df.y          # Creating virtual column
+expr = df.x.mean()                    # Creating expression
+df_filtered = df[df.x > 10]          # Creating filtered view
+```
+
+## Batching Operations with delay=True
+
+Execute multiple operations together for better performance:
+
+### Basic Delayed Execution
+
+```python
+# Without delay - each operation processes entire dataset
+mean_x = df.x.mean()      # Pass 1 through data
+std_x = df.x.std()        # Pass 2 through data
+max_x = df.x.max()        # Pass 3 through data
+
+# With delay - single pass through dataset
+mean_x = df.x.mean(delay=True)
+std_x = df.x.std(delay=True)
+max_x = df.x.max(delay=True)
+results = vaex.execute([mean_x, std_x, max_x])  # Single pass!
+
+print(results[0])  # mean
+print(results[1])  # std
+print(results[2])  # max
+```
+
+### Delayed Execution with Multiple Columns
+
+```python
+# Compute statistics for many columns efficiently
+stats = {}
+delayed_results = []
+
+for column in ['sales', 'quantity', 'profit', 'cost']:
+    mean = df[column].mean(delay=True)
+    std = df[column].std(delay=True)
+    delayed_results.extend([mean, std])
+
+# Execute all at once
+results = vaex.execute(delayed_results)
+
+# Process results
+for i, column in enumerate(['sales', 'quantity', 'profit', 'cost']):
+    stats[column] = {
+        'mean': results[i*2],
+        'std': results[i*2 + 1]
+    }
+```
+
+### When to Use delay=True
+
+Use `delay=True` when:
+- Computing multiple aggregations
+- Computing statistics on many columns
+- Building dashboards or reports
+- Any scenario requiring multiple passes through data
+
+```python
+# Bad: 4 passes through dataset
+mean1 = df.col1.mean()
+mean2 = df.col2.mean()
+mean3 = df.col3.mean()
+mean4 = df.col4.mean()
+
+# Good: 1 pass through dataset
+results = vaex.execute([
+    df.col1.mean(delay=True),
+    df.col2.mean(delay=True),
+    df.col3.mean(delay=True),
+    df.col4.mean(delay=True)
+])
+```
+
+## Asynchronous Operations
+
+Process data asynchronously using async/await:
+
+### Async with async/await
+
+```python
+import vaex
+import asyncio
+
+async def compute_statistics(df):
+    # Create async tasks
+    mean_task = df.x.mean(delay=True)
+    std_task = df.x.std(delay=True)
+
+    # Execute asynchronously
+    results = await vaex.async_execute([mean_task, std_task])
+
+    return {'mean': results[0], 'std': results[1]}
+
+# Run async function
+async def main():
+    df = vaex.open('large_file.hdf5')
+    stats = await compute_statistics(df)
+    print(stats)
+
+asyncio.run(main())
+```
+
+### Using Promises/Futures
+
+```python
+# Get future object
+future = df.x.mean(delay=True)
+
+# Do other work...
+
+# Get result when ready
+result = future.get()  # Blocks until complete
+```
+
+## Virtual Columns vs Materialized Columns
+
+Understanding the difference is crucial for performance:
+
+### Virtual Columns (Preferred)
+
+```python
+# Virtual column - computed on-the-fly, zero memory
+df['total'] = df.price * df.quantity
+df['log_sales'] = df.sales.log()
+df['full_name'] = df.first_name + ' ' + df.last_name
+
+# Check if virtual
+print(df.is_local('total'))  # False = virtual
+
+# Benefits:
+# - Zero memory overhead
+# - Always up-to-date if source data changes
+# - Fast to create
+```
+
+### Materialized Columns
+
+```python
+# Materialize a virtual column
+df['total_materialized'] = df['total'].values
+
+# Or use materialize method
+df = df.materialize(df['total'], inplace=True)
+
+# Check if materialized
+print(df.is_local('total_materialized'))  # True = materialized
+
+# When to materialize:
+# - Column computed repeatedly (amortize cost)
+# - Complex expression used in many operations
+# - Need to export data
+```
+
+### Deciding: Virtual vs Materialized
+
+```python
+# Virtual is better when:
+# - Column is simple (x + y, x * 2, etc.)
+# - Column used infrequently
+# - Memory is limited
+
+# Materialize when:
+# - Complex computation (multiple operations)
+# - Used repeatedly in aggregations
+# - Slows down other operations
+
+# Example: Complex calculation used many times
+df['complex'] = (df.x.log() * df.y.sqrt() + df.z ** 2).values  # Materialize
+```
+
+## Caching Strategies
+
+Vaex automatically caches some operations, but you can optimize further:
+
+### Automatic Caching
+
+```python
+# First call computes and caches
+mean1 = df.x.mean()  # Computes
+
+# Second call uses cache
+mean2 = df.x.mean()  # From cache (instant)
+
+# Cache invalidated if DataFrame changes
+df['new_col'] = df.x + 1
+mean3 = df.x.mean()  # Recomputes
+```
+
+### State Management
+
+```python
+# Save DataFrame state (includes virtual columns)
+df.state_write('state.json')
+
+# Load state later
+df_new = vaex.open('data.hdf5')
+df_new.state_load('state.json')  # Restores virtual columns, selections
+```
+
+### Checkpoint Pattern
+
+```python
+# Export intermediate results for complex pipelines
+df['processed'] = complex_calculation(df)
+
+# Save checkpoint
+df.export_hdf5('checkpoint.hdf5')
+
+# Resume from checkpoint
+df = vaex.open('checkpoint.hdf5')
+# Continue processing...
+```
+
+## Memory Management
+
+Optimize memory usage for very large datasets:
+
+### Memory-Mapped Files
+
+```python
+# HDF5 and Arrow are memory-mapped (optimal)
+df = vaex.open('data.hdf5')  # No memory used until accessed
+
+# File stays on disk, only accessed portions loaded to RAM
+mean = df.x.mean()  # Streams through data, minimal memory
+```
+
+### Chunked Processing
+
+```python
+# Process large DataFrame in chunks
+chunk_size = 1_000_000
+
+for i1, i2, chunk in df.to_pandas_df(chunk_size=chunk_size):
+    # Process chunk (careful: defeats Vaex's purpose)
+    process_chunk(chunk)
+
+# Better: Use Vaex operations directly (no chunking needed)
+result = df.x.mean()  # Handles large data automatically
+```
+
+### Monitoring Memory Usage
+
+```python
+# Check DataFrame memory footprint
+print(df.byte_size())  # Bytes used by materialized columns
+
+# Check column memory
+for col in df.get_column_names():
+    if df.is_local(col):
+        print(f"{col}: {df[col].nbytes / 1e9:.2f} GB")
+
+# Profile operations
+import vaex.profiler
+with vaex.profiler():
+    result = df.x.mean()
+```
+
+## Parallel Computation
+
+Vaex automatically parallelizes operations:
+
+### Multithreading
+
+```python
+# Vaex uses all CPU cores by default
+import vaex
+
+# Check/set thread count
+print(vaex.multithreading.thread_count_default)
+vaex.multithreading.thread_count_default = 8  # Use 8 threads
+
+# Operations automatically parallelize
+mean = df.x.mean()  # Uses all threads
+```
+
+### Distributed Computing with Dask
+
+```python
+# Convert to Dask for distributed processing
+import vaex
+import dask.dataframe as dd
+
+# Create Vaex DataFrame
+df_vaex = vaex.open('large_file.hdf5')
+
+# Convert to Dask
+df_dask = df_vaex.to_dask_dataframe()
+
+# Process with Dask
+result = df_dask.groupby('category')['value'].sum().compute()
+```
+
+## JIT Compilation
+
+Vaex can use Just-In-Time compilation for custom operations:
+
+### Using Numba
+
+```python
+import vaex
+import numba
+
+# Define JIT-compiled function
+@numba.jit
+def custom_calculation(x, y):
+    return x ** 2 + y ** 2
+
+# Apply to DataFrame
+df['custom'] = df.apply(custom_calculation,
+                        arguments=[df.x, df.y],
+                        vectorize=True)
+```
+
+### Custom Aggregations
+
+```python
+@numba.jit
+def custom_sum(a):
+    total = 0
+    for val in a:
+        total += val * 2  # Custom logic
+    return total
+
+# Use in aggregation
+result = df.x.custom_agg(custom_sum)
+```
+
+## Optimization Strategies
+
+### Strategy 1: Minimize Materializations
+
+```python
+# Bad: Creates many materialized columns
+df['a'] = (df.x + df.y).values
+df['b'] = (df.a * 2).values
+df['c'] = (df.b + df.z).values
+
+# Good: Keep virtual until final export
+df['a'] = df.x + df.y
+df['b'] = df.a * 2
+df['c'] = df.b + df.z
+# Only materialize if exporting:
+# df.export_hdf5('output.hdf5')
+```
+
+### Strategy 2: Use Selections Instead of Filtering
+
+```python
+# Less efficient: Creates new DataFrames
+df_high = df[df.value > 100]
+df_low = df[df.value <= 100]
+mean_high = df_high.value.mean()
+mean_low = df_low.value.mean()
+
+# More efficient: Use selections
+df.select(df.value > 100, name='high')
+df.select(df.value <= 100, name='low')
+mean_high = df.value.mean(selection='high')
+mean_low = df.value.mean(selection='low')
+```
+
+### Strategy 3: Batch Aggregations
+
+```python
+# Less efficient: Multiple passes
+stats = {
+    'mean': df.x.mean(),
+    'std': df.x.std(),
+    'min': df.x.min(),
+    'max': df.x.max()
+}
+
+# More efficient: Single pass
+delayed = [
+    df.x.mean(delay=True),
+    df.x.std(delay=True),
+    df.x.min(delay=True),
+    df.x.max(delay=True)
+]
+results = vaex.execute(delayed)
+stats = dict(zip(['mean', 'std', 'min', 'max'], results))
+```
+
+### Strategy 4: Choose Optimal File Formats
+
+```python
+# Slow: Large CSV
+df = vaex.from_csv('huge.csv')  # Can take minutes
+
+# Fast: HDF5 or Arrow
+df = vaex.open('huge.hdf5')     # Instant
+df = vaex.open('huge.arrow')    # Instant
+
+# One-time conversion
+df = vaex.from_csv('huge.csv', convert='huge.hdf5')
+# Future loads: vaex.open('huge.hdf5')
+```
+
+### Strategy 5: Optimize Expressions
+
+```python
+# Less efficient: Repeated calculations
+df['result'] = df.x.log() + df.x.log() * 2
+
+# More efficient: Reuse calculations
+df['log_x'] = df.x.log()
+df['result'] = df.log_x + df.log_x * 2
+
+# Even better: Combine operations
+df['result'] = df.x.log() * 3  # Simplified math
+```
+
+## Performance Profiling
+
+### Basic Profiling
+
+```python
+import time
+import vaex
+
+df = vaex.open('large_file.hdf5')
+
+# Time operations
+start = time.time()
+result = df.x.mean()
+elapsed = time.time() - start
+print(f"Computed in {elapsed:.2f} seconds")
+```
+
+### Detailed Profiling
+
+```python
+# Profile with context manager
+with vaex.profiler():
+    result = df.groupby('category').agg({'value': 'sum'})
+# Prints detailed timing information
+```
+
+### Benchmarking Patterns
+
+```python
+# Compare strategies
+def benchmark_operation(operation, name):
+    start = time.time()
+    result = operation()
+    elapsed = time.time() - start
+    print(f"{name}: {elapsed:.3f}s")
+    return result
+
+# Test different approaches
+benchmark_operation(lambda: df.x.mean(), "Direct mean")
+benchmark_operation(lambda: df[df.x > 0].x.mean(), "Filtered mean")
+benchmark_operation(lambda: df.x.mean(selection='positive'), "Selection mean")
+```
+
+## Common Performance Issues and Solutions
+
+### Issue: Slow Aggregations
+
+```python
+# Problem: Multiple separate aggregations
+for col in df.column_names:
+    print(f"{col}: {df[col].mean()}")
+
+# Solution: Batch with delay=True
+delayed = [df[col].mean(delay=True) for col in df.column_names]
+results = vaex.execute(delayed)
+for col, result in zip(df.column_names, results):
+    print(f"{col}: {result}")
+```
+
+### Issue: High Memory Usage
+
+```python
+# Problem: Materializing large virtual columns
+df['large_col'] = (complex_expression).values
+
+# Solution: Keep virtual, or materialize and export
+df['large_col'] = complex_expression  # Virtual
+# Or: df.export_hdf5('with_new_col.hdf5')
+```
+
+### Issue: Slow Exports
+
+```python
+# Problem: Exporting with many virtual columns
+df.export_csv('output.csv')  # Slow if many virtual columns
+
+# Solution: Export to HDF5 or Arrow (faster)
+df.export_hdf5('output.hdf5')
+df.export_arrow('output.arrow')
+
+# Or materialize first for CSV
+df_materialized = df.materialize()
+df_materialized.export_csv('output.csv')
+```
+
+### Issue: Repeated Complex Calculations
+
+```python
+# Problem: Complex virtual column used repeatedly
+df['complex'] = df.x.log() * df.y.sqrt() + df.z ** 3
+result1 = df.groupby('cat1').agg({'complex': 'mean'})
+result2 = df.groupby('cat2').agg({'complex': 'sum'})
+result3 = df.complex.std()
+
+# Solution: Materialize once
+df['complex'] = (df.x.log() * df.y.sqrt() + df.z ** 3).values
+# Or: df = df.materialize('complex')
+```
+
+## Performance Best Practices Summary
+
+1. **Use HDF5 or Arrow formats** - Orders of magnitude faster than CSV
+2. **Leverage lazy evaluation** - Don't force computation until necessary
+3. **Batch operations with delay=True** - Minimize passes through data
+4. **Keep columns virtual** - Materialize only when beneficial
+5. **Use selections not filters** - More efficient for multiple segments
+6. **Profile your code** - Identify bottlenecks before optimizing
+7. **Avoid `.values` and `.to_pandas_df()`** - Keep operations in Vaex
+8. **Parallelize naturally** - Vaex uses all cores automatically
+9. **Export to efficient formats** - Checkpoint complex pipelines
+10. **Optimize expressions** - Simplify math and reuse calculations
+
+## Related Resources
+
+- For DataFrame basics: See `core_dataframes.md`
+- For data operations: See `data_processing.md`
+- For file I/O optimization: See `io_operations.md`