@sylix/coworker 2.0.11 → 2.0.12

package/dist/skills/defaults/backend-development/python-performance-optimization.md

@@ -0,0 +1,874 @@
+---
+name: python-performance-optimization
+description: Profile and optimize Python code using cProfile, memory profilers, and performance best practices. Use when debugging slow Python code, optimizing bottlenecks, or improving application performance.
+---
+
+# Python Performance Optimization
+
+Comprehensive guide to profiling, analyzing, and optimizing Python code for better performance, including CPU profiling, memory optimization, and implementation best practices.
+
+## When to Use This Skill
+
+- Identifying performance bottlenecks in Python applications
+- Reducing application latency and response times
+- Optimizing CPU-intensive operations
+- Reducing memory consumption and memory leaks
+- Improving database query performance
+- Optimizing I/O operations
+- Speeding up data processing pipelines
+- Implementing high-performance algorithms
+- Profiling production applications
+
+## Core Concepts
+
+### 1. Profiling Types
+
+- **CPU Profiling**: Identify time-consuming functions
+- **Memory Profiling**: Track memory allocation and leaks
+- **Line Profiling**: Profile at line-by-line granularity
+- **Call Graph**: Visualize function call relationships
+
+### 2. Performance Metrics
+
+- **Execution Time**: How long operations take
+- **Memory Usage**: Peak and average memory consumption
+- **CPU Utilization**: Processor usage patterns
+- **I/O Wait**: Time spent on I/O operations
+
+### 3. Optimization Strategies
+
+- **Algorithmic**: Better algorithms and data structures
+- **Implementation**: More efficient code patterns
+- **Parallelization**: Multi-threading/processing
+- **Caching**: Avoid redundant computation
+- **Native Extensions**: C/Rust for critical paths
+
+## Quick Start
+
+### Basic Timing
+
+```python
+import time
+
+def measure_time():
+    """Simple timing measurement."""
+    start = time.time()
+
+    # Your code here
+    result = sum(range(1000000))
+
+    elapsed = time.time() - start
+    print(f"Execution time: {elapsed:.4f} seconds")
+    return result
+
+# Better: use timeit for accurate measurements
+import timeit
+
+execution_time = timeit.timeit(
+    "sum(range(1000000))",
+    number=100
+)
+print(f"Average time: {execution_time/100:.6f} seconds")
+```
+
+## Profiling Tools
+
+### Pattern 1: cProfile - CPU Profiling
+
+```python
+import cProfile
+import pstats
+from pstats import SortKey
+
+def slow_function():
+    """Function to profile."""
+    total = 0
+    for i in range(1000000):
+        total += i
+    return total
+
+def another_function():
+    """Another function."""
+    return [i**2 for i in range(100000)]
+
+def main():
+    """Main function to profile."""
+    result1 = slow_function()
+    result2 = another_function()
+    return result1, result2
+
+# Profile the code
+if __name__ == "__main__":
+    profiler = cProfile.Profile()
+    profiler.enable()
+
+    main()
+
+    profiler.disable()
+
+    # Print stats
+    stats = pstats.Stats(profiler)
+    stats.sort_stats(SortKey.CUMULATIVE)
+    stats.print_stats(10)  # Top 10 functions
+
+    # Save to file for later analysis
+    stats.dump_stats("profile_output.prof")
+```
+
+**Command-line profiling:**
+
+```bash
+# Profile a script
+python -m cProfile -o output.prof script.py
+
+# View results
+python -m pstats output.prof
+# In pstats:
+# sort cumtime
+# stats 10
+```
+
+### Pattern 2: line_profiler - Line-by-Line Profiling
+
+```python
+# Install: pip install line-profiler
+
+# Add @profile decorator (line_profiler provides this)
+@profile
+def process_data(data):
+    """Process data with line profiling."""
+    result = []
+    for item in data:
+        processed = item * 2
+        result.append(processed)
+    return result
+
+# Run with:
+# kernprof -l -v script.py
+```
+
+**Manual line profiling:**
+
+```python
+from line_profiler import LineProfiler
+
+def process_data(data):
+    """Function to profile."""
+    result = []
+    for item in data:
+        processed = item * 2
+        result.append(processed)
+    return result
+
+if __name__ == "__main__":
+    lp = LineProfiler()
+    lp.add_function(process_data)
+
+    data = list(range(100000))
+
+    lp_wrapper = lp(process_data)
+    lp_wrapper(data)
+
+    lp.print_stats()
+```
+
+### Pattern 3: memory_profiler - Memory Usage
+
+```python
+# Install: pip install memory-profiler
+
+from memory_profiler import profile
+
+@profile
+def memory_intensive():
+    """Function that uses lots of memory."""
+    # Create large list
+    big_list = [i for i in range(1000000)]
+
+    # Create large dict
+    big_dict = {i: i**2 for i in range(100000)}
+
+    # Process data
+    result = sum(big_list)
+
+    return result
+
+if __name__ == "__main__":
+    memory_intensive()
+
+# Run with:
+# python -m memory_profiler script.py
+```
+
+### Pattern 4: py-spy - Production Profiling
+
+```bash
+# Install: pip install py-spy
+
+# Profile a running Python process
+py-spy top --pid 12345
+
+# Generate flamegraph
+py-spy record -o profile.svg --pid 12345
+
+# Profile a script
+py-spy record -o profile.svg -- python script.py
+
+# Dump current call stack
+py-spy dump --pid 12345
+```
+
+## Optimization Patterns
+
+### Pattern 5: List Comprehensions vs Loops
+
+```python
+import timeit
+
+# Slow: Traditional loop
+def slow_squares(n):
+    """Create list of squares using loop."""
+    result = []
+    for i in range(n):
+        result.append(i**2)
+    return result
+
+# Fast: List comprehension
+def fast_squares(n):
+    """Create list of squares using comprehension."""
+    return [i**2 for i in range(n)]
+
+# Benchmark
+n = 100000
+
+slow_time = timeit.timeit(lambda: slow_squares(n), number=100)
+fast_time = timeit.timeit(lambda: fast_squares(n), number=100)
+
+print(f"Loop: {slow_time:.4f}s")
+print(f"Comprehension: {fast_time:.4f}s")
+print(f"Speedup: {slow_time/fast_time:.2f}x")
+
+# Even faster for simple operations: map
+def faster_squares(n):
+    """Use map for even better performance."""
+    return list(map(lambda x: x**2, range(n)))
+```
+
+### Pattern 6: Generator Expressions for Memory
+
+```python
+import sys
+
+def list_approach():
+    """Memory-intensive list."""
+    data = [i**2 for i in range(1000000)]
+    return sum(data)
+
+def generator_approach():
+    """Memory-efficient generator."""
+    data = (i**2 for i in range(1000000))
+    return sum(data)
+
+# Memory comparison
+list_data = [i for i in range(1000000)]
+gen_data = (i for i in range(1000000))
+
+print(f"List size: {sys.getsizeof(list_data)} bytes")
+print(f"Generator size: {sys.getsizeof(gen_data)} bytes")
+
+# Generators use constant memory regardless of size
+```
+
+### Pattern 7: String Concatenation
+
+```python
+import timeit
+
+def slow_concat(items):
+    """Slow string concatenation."""
+    result = ""
+    for item in items:
+        result += str(item)
+    return result
+
+def fast_concat(items):
+    """Fast string concatenation with join."""
+    return "".join(str(item) for item in items)
+
+def faster_concat(items):
+    """Even faster with list."""
+    parts = [str(item) for item in items]
+    return "".join(parts)
+
+items = list(range(10000))
+
+# Benchmark
+slow = timeit.timeit(lambda: slow_concat(items), number=100)
+fast = timeit.timeit(lambda: fast_concat(items), number=100)
+faster = timeit.timeit(lambda: faster_concat(items), number=100)
+
+print(f"Concatenation (+): {slow:.4f}s")
+print(f"Join (generator): {fast:.4f}s")
+print(f"Join (list): {faster:.4f}s")
+```
+
+### Pattern 8: Dictionary Lookups vs List Searches
+
+```python
+import timeit
+
+# Create test data
+size = 10000
+items = list(range(size))
+lookup_dict = {i: i for i in range(size)}
+
+def list_search(items, target):
+    """O(n) search in list."""
+    return target in items
+
+def dict_search(lookup_dict, target):
+    """O(1) search in dict."""
+    return target in lookup_dict
+
+target = size - 1  # Worst case for list
+
+# Benchmark
+list_time = timeit.timeit(
+    lambda: list_search(items, target),
+    number=1000
+)
+dict_time = timeit.timeit(
+    lambda: dict_search(lookup_dict, target),
+    number=1000
+)
+
+print(f"List search: {list_time:.6f}s")
+print(f"Dict search: {dict_time:.6f}s")
+print(f"Speedup: {list_time/dict_time:.0f}x")
+```
+
+### Pattern 9: Local Variable Access
+
+```python
+import timeit
+
+# Global variable (slow)
+GLOBAL_VALUE = 100
+
+def use_global():
+    """Access global variable."""
+    total = 0
+    for i in range(10000):
+        total += GLOBAL_VALUE
+    return total
+
+def use_local():
+    """Use local variable."""
+    local_value = 100
+    total = 0
+    for i in range(10000):
+        total += local_value
+    return total
+
+# Local is faster
+global_time = timeit.timeit(use_global, number=1000)
+local_time = timeit.timeit(use_local, number=1000)
+
+print(f"Global access: {global_time:.4f}s")
+print(f"Local access: {local_time:.4f}s")
+print(f"Speedup: {global_time/local_time:.2f}x")
+```
+
+### Pattern 10: Function Call Overhead
+
+```python
+import timeit
+
+def calculate_inline():
+    """Inline calculation."""
+    total = 0
+    for i in range(10000):
+        total += i * 2 + 1
+    return total
+
+def helper_function(x):
+    """Helper function."""
+    return x * 2 + 1
+
+def calculate_with_function():
+    """Calculation with function calls."""
+    total = 0
+    for i in range(10000):
+        total += helper_function(i)
+    return total
+
+# Inline is faster due to no call overhead
+inline_time = timeit.timeit(calculate_inline, number=1000)
+function_time = timeit.timeit(calculate_with_function, number=1000)
+
+print(f"Inline: {inline_time:.4f}s")
+print(f"Function calls: {function_time:.4f}s")
+```
+
+## Advanced Optimization
+
+### Pattern 11: NumPy for Numerical Operations
+
+```python
+import timeit
+import numpy as np
+
+def python_sum(n):
+    """Sum using pure Python."""
+    return sum(range(n))
+
+def numpy_sum(n):
+    """Sum using NumPy."""
+    return np.arange(n).sum()
+
+n = 1000000
+
+python_time = timeit.timeit(lambda: python_sum(n), number=100)
+numpy_time = timeit.timeit(lambda: numpy_sum(n), number=100)
+
+print(f"Python: {python_time:.4f}s")
+print(f"NumPy: {numpy_time:.4f}s")
+print(f"Speedup: {python_time/numpy_time:.2f}x")
+
+# Vectorized operations
+def python_multiply():
+    """Element-wise multiplication in Python."""
+    a = list(range(100000))
+    b = list(range(100000))
+    return [x * y for x, y in zip(a, b)]
+
+def numpy_multiply():
+    """Vectorized multiplication in NumPy."""
+    a = np.arange(100000)
+    b = np.arange(100000)
+    return a * b
+
+py_time = timeit.timeit(python_multiply, number=100)
+np_time = timeit.timeit(numpy_multiply, number=100)
+
+print(f"\nPython multiply: {py_time:.4f}s")
+print(f"NumPy multiply: {np_time:.4f}s")
+print(f"Speedup: {py_time/np_time:.2f}x")
+```
+
+### Pattern 12: Caching with functools.lru_cache
+
+```python
+from functools import lru_cache
+import timeit
+
+def fibonacci_slow(n):
+    """Recursive fibonacci without caching."""
+    if n < 2:
+        return n
+    return fibonacci_slow(n-1) + fibonacci_slow(n-2)
+
+@lru_cache(maxsize=None)
+def fibonacci_fast(n):
+    """Recursive fibonacci with caching."""
+    if n < 2:
+        return n
+    return fibonacci_fast(n-1) + fibonacci_fast(n-2)
+
+# Massive speedup for recursive algorithms
+n = 30
+
+slow_time = timeit.timeit(lambda: fibonacci_slow(n), number=1)
+fast_time = timeit.timeit(lambda: fibonacci_fast(n), number=1000)
+
+print(f"Without cache (1 run): {slow_time:.4f}s")
+print(f"With cache (1000 runs): {fast_time:.4f}s")
+
+# Cache info
+print(f"Cache info: {fibonacci_fast.cache_info()}")
+```
+
+### Pattern 13: Using __slots__ for Memory
+
+```python
+import sys
+
+class RegularClass:
+    """Regular class with __dict__."""
+    def __init__(self, x, y, z):
+        self.x = x
+        self.y = y
+        self.z = z
+
+class SlottedClass:
+    """Class with __slots__ for memory efficiency."""
+    __slots__ = ['x', 'y', 'z']
+
+    def __init__(self, x, y, z):
+        self.x = x
+        self.y = y
+        self.z = z
+
+# Memory comparison
+regular = RegularClass(1, 2, 3)
+slotted = SlottedClass(1, 2, 3)
+
+print(f"Regular class size: {sys.getsizeof(regular)} bytes")
+print(f"Slotted class size: {sys.getsizeof(slotted)} bytes")
+
+# Significant savings with many instances
+regular_objects = [RegularClass(i, i+1, i+2) for i in range(10000)]
+slotted_objects = [SlottedClass(i, i+1, i+2) for i in range(10000)]
+
+print(f"\nMemory for 10000 regular objects: ~{sys.getsizeof(regular) * 10000} bytes")
+print(f"Memory for 10000 slotted objects: ~{sys.getsizeof(slotted) * 10000} bytes")
+```
+
+### Pattern 14: Multiprocessing for CPU-Bound Tasks
+
+```python
+import multiprocessing as mp
+import time
+
+def cpu_intensive_task(n):
+    """CPU-intensive calculation."""
+    return sum(i**2 for i in range(n))
+
+def sequential_processing():
+    """Process tasks sequentially."""
+    start = time.time()
+    results = [cpu_intensive_task(1000000) for _ in range(4)]
+    elapsed = time.time() - start
+    return elapsed, results
+
+def parallel_processing():
+    """Process tasks in parallel."""
+    start = time.time()
+    with mp.Pool(processes=4) as pool:
+        results = pool.map(cpu_intensive_task, [1000000] * 4)
+    elapsed = time.time() - start
+    return elapsed, results
+
+if __name__ == "__main__":
+    seq_time, seq_results = sequential_processing()
+    par_time, par_results = parallel_processing()
+
+    print(f"Sequential: {seq_time:.2f}s")
+    print(f"Parallel: {par_time:.2f}s")
+    print(f"Speedup: {seq_time/par_time:.2f}x")
+```
+
+### Pattern 15: Async I/O for I/O-Bound Tasks
+
+```python
+import asyncio
+import aiohttp
+import time
+import requests
+
+urls = [
+    "https://httpbin.org/delay/1",
+    "https://httpbin.org/delay/1",
+    "https://httpbin.org/delay/1",
+    "https://httpbin.org/delay/1",
+]
+
+def synchronous_requests():
+    """Synchronous HTTP requests."""
+    start = time.time()
+    results = []
+    for url in urls:
+        response = requests.get(url)
+        results.append(response.status_code)
+    elapsed = time.time() - start
+    return elapsed, results
+
+async def async_fetch(session, url):
+    """Async HTTP request."""
+    async with session.get(url) as response:
+        return response.status
+
+async def asynchronous_requests():
+    """Asynchronous HTTP requests."""
+    start = time.time()
+    async with aiohttp.ClientSession() as session:
+        tasks = [async_fetch(session, url) for url in urls]
+        results = await asyncio.gather(*tasks)
+    elapsed = time.time() - start
+    return elapsed, results
+
+# Async is much faster for I/O-bound work
+sync_time, sync_results = synchronous_requests()
+async_time, async_results = asyncio.run(asynchronous_requests())
+
+print(f"Synchronous: {sync_time:.2f}s")
+print(f"Asynchronous: {async_time:.2f}s")
+print(f"Speedup: {sync_time/async_time:.2f}x")
+```
+
+## Database Optimization
+
+### Pattern 16: Batch Database Operations
+
+```python
+import sqlite3
+import time
+
+def create_db():
+    """Create test database."""
+    conn = sqlite3.connect(":memory:")
+    conn.execute("CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT)")
+    return conn
+
+def slow_inserts(conn, count):
+    """Insert records one at a time."""
+    start = time.time()
+    cursor = conn.cursor()
+    for i in range(count):
+        cursor.execute("INSERT INTO users (name) VALUES (?)", (f"User {i}",))
+        conn.commit()  # Commit each insert
+    elapsed = time.time() - start
+    return elapsed
+
+def fast_inserts(conn, count):
+    """Batch insert with single commit."""
+    start = time.time()
+    cursor = conn.cursor()
+    data = [(f"User {i}",) for i in range(count)]
+    cursor.executemany("INSERT INTO users (name) VALUES (?)", data)
+    conn.commit()  # Single commit
+    elapsed = time.time() - start
+    return elapsed
+
+# Benchmark
+conn1 = create_db()
+slow_time = slow_inserts(conn1, 1000)
+
+conn2 = create_db()
+fast_time = fast_inserts(conn2, 1000)
+
+print(f"Individual inserts: {slow_time:.4f}s")
+print(f"Batch insert: {fast_time:.4f}s")
+print(f"Speedup: {slow_time/fast_time:.2f}x")
+```
+
+### Pattern 17: Query Optimization
+
+```python
+# Use indexes for frequently queried columns
+"""
+-- Slow: No index
+SELECT * FROM users WHERE email = 'user@example.com';
+
+-- Fast: With index
+CREATE INDEX idx_users_email ON users(email);
+SELECT * FROM users WHERE email = 'user@example.com';
+"""
+
+# Use query planning
+import sqlite3
+
+conn = sqlite3.connect("example.db")
+cursor = conn.cursor()
+
+# Analyze query performance
+cursor.execute("EXPLAIN QUERY PLAN SELECT * FROM users WHERE email = ?", ("test@example.com",))
+print(cursor.fetchall())
+
+# Use SELECT only needed columns
+# Slow: SELECT *
+# Fast: SELECT id, name
+```
+
+## Memory Optimization
+
+### Pattern 18: Detecting Memory Leaks
+
+```python
+import tracemalloc
+import gc
+
+def memory_leak_example():
+    """Example that leaks memory."""
+    leaked_objects = []
+
+    for i in range(100000):
+        # Objects added but never removed
+        leaked_objects.append([i] * 100)
+
+    # In real code, this would be an unintended reference
+
+def track_memory_usage():
+    """Track memory allocations."""
+    tracemalloc.start()
+
+    # Take snapshot before
+    snapshot1 = tracemalloc.take_snapshot()
+
+    # Run code
+    memory_leak_example()
+
+    # Take snapshot after
+    snapshot2 = tracemalloc.take_snapshot()
+
+    # Compare
+    top_stats = snapshot2.compare_to(snapshot1, 'lineno')
+
+    print("Top 10 memory allocations:")
+    for stat in top_stats[:10]:
+        print(stat)
+
+    tracemalloc.stop()
+
+# Monitor memory
+track_memory_usage()
+
+# Force garbage collection
+gc.collect()
+```
+
+### Pattern 19: Iterators vs Lists
+
+```python
+import sys
+
+def process_file_list(filename):
+    """Load entire file into memory."""
+    with open(filename) as f:
+        lines = f.readlines()  # Loads all lines
+        return sum(1 for line in lines if line.strip())
+
+def process_file_iterator(filename):
+    """Process file line by line."""
+    with open(filename) as f:
+        return sum(1 for line in f if line.strip())
+
+# Iterator uses constant memory
+# List loads entire file into memory
+```
+
+### Pattern 20: Weakref for Caches
+
+```python
+import weakref
+
+class CachedResource:
+    """Resource that can be garbage collected."""
+    def __init__(self, data):
+        self.data = data
+
+# Regular cache prevents garbage collection
+regular_cache = {}
+
+def get_resource_regular(key):
+    """Get resource from regular cache."""
+    if key not in regular_cache:
+        regular_cache[key] = CachedResource(f"Data for {key}")
+    return regular_cache[key]
+
+# Weak reference cache allows garbage collection
+weak_cache = weakref.WeakValueDictionary()
+
+def get_resource_weak(key):
+    """Get resource from weak cache."""
+    resource = weak_cache.get(key)
+    if resource is None:
+        resource = CachedResource(f"Data for {key}")
+        weak_cache[key] = resource
+    return resource
+
+# When no strong references exist, objects can be GC'd
+```
+
+## Benchmarking Tools
+
+### Custom Benchmark Decorator
+
+```python
+import time
+from functools import wraps
+
+def benchmark(func):
+    """Decorator to benchmark function execution."""
+    @wraps(func)
+    def wrapper(*args, **kwargs):
+        start = time.perf_counter()
+        result = func(*args, **kwargs)
+        elapsed = time.perf_counter() - start
+        print(f"{func.__name__} took {elapsed:.6f} seconds")
+        return result
+    return wrapper
+
+@benchmark
+def slow_function():
+    """Function to benchmark."""
+    time.sleep(0.5)
+    return sum(range(1000000))
+
+result = slow_function()
+```
+
+### Performance Testing with pytest-benchmark
+
+```python
+# Install: pip install pytest-benchmark
+
+def test_list_comprehension(benchmark):
+    """Benchmark list comprehension."""
+    result = benchmark(lambda: [i**2 for i in range(10000)])
+    assert len(result) == 10000
+
+def test_map_function(benchmark):
+    """Benchmark map function."""
+    result = benchmark(lambda: list(map(lambda x: x**2, range(10000))))
+    assert len(result) == 10000
+
+# Run with: pytest test_performance.py --benchmark-compare
+```
+
+## Best Practices
+
+1. **Profile before optimizing** - Measure to find real bottlenecks
+2. **Focus on hot paths** - Optimize code that runs most frequently
+3. **Use appropriate data structures** - Dict for lookups, set for membership
+4. **Avoid premature optimization** - Clarity first, then optimize
+5. **Use built-in functions** - They're implemented in C
+6. **Cache expensive computations** - Use lru_cache
+7. **Batch I/O operations** - Reduce system calls
+8. **Use generators** for large datasets
+9. **Consider NumPy** for numerical operations
+10. **Profile production code** - Use py-spy for live systems
+
+## Common Pitfalls
+
+- Optimizing without profiling
+- Using global variables unnecessarily
+- Not using appropriate data structures
+- Creating unnecessary copies of data
+- Not using connection pooling for databases
+- Ignoring algorithmic complexity
+- Over-optimizing rare code paths
+- Not considering memory usage
+
+## Resources
+
+- **cProfile**: Built-in CPU profiler
+- **memory_profiler**: Memory usage profiling
+- **line_profiler**: Line-by-line profiling
+- **py-spy**: Sampling profiler for production
+- **NumPy**: High-performance numerical computing
+- **Cython**: Compile Python to C
+- **PyPy**: Alternative Python interpreter with JIT
+
+## Performance Checklist
+
+- [ ] Profiled code to identify bottlenecks
+- [ ] Used appropriate data structures
+- [ ] Implemented caching where beneficial
+- [ ] Optimized database queries
+- [ ] Used generators for large datasets
+- [ ] Considered multiprocessing for CPU-bound tasks
+- [ ] Used async I/O for I/O-bound tasks
+- [ ] Minimized function call overhead in hot loops
+- [ ] Checked for memory leaks
+- [ ] Benchmarked before and after optimization