@booklib/skills 1.0.0 → 1.3.0

package/effective-python/ref-07-concurrency.md

@@ -0,0 +1,213 @@
+# Chapter 7: Concurrency and Parallelism (Items 52-64)
+
+## Item 52: Use subprocess to Manage Child Processes
+```python
+import subprocess
+
+# Run a command and capture output
+result = subprocess.run(
+    ['echo', 'Hello from subprocess'],
+    capture_output=True,
+    text=True
+)
+print(result.stdout)
+
+# Set timeout
+result = subprocess.run(
+    ['sleep', '10'],
+    timeout=5  # raises TimeoutExpired after 5 seconds
+)
+
+# Pipe data to child process
+result = subprocess.run(
+    ['openssl', 'enc', '-aes-256-cbc', '-pass', 'pass:key'],
+    input=b'data to encrypt',
+    capture_output=True
+)
+
+# Run parallel child processes
+procs = [subprocess.Popen(['cmd', arg]) for arg in args]
+for proc in procs:
+    proc.communicate()  # wait for each
+```
+
+- Use `subprocess.run` for simple command execution
+- Use `subprocess.Popen` for parallel or streaming processes
+- Always set timeouts to prevent hanging
+
+## Item 53: Use Threads for Blocking I/O, Avoid for Parallelism
+```python
+import threading
+
+# Threads for I/O parallelism — GOOD
+def download(url):
+    resp = urllib.request.urlopen(url)
+    return resp.read()
+
+threads = [threading.Thread(target=download, args=(url,)) for url in urls]
+for t in threads:
+    t.start()
+for t in threads:
+    t.join()
+```
+
+- The GIL prevents true CPU parallelism with threads
+- Threads ARE useful for blocking I/O (network, file system, etc.)
+- For CPU-bound work, use `multiprocessing` or `concurrent.futures.ProcessPoolExecutor`
+- Never use threads for CPU-intensive computation
+
+## Item 54: Use Lock to Prevent Data Races in Threads
+```python
+from threading import Lock
+
+class Counter:
+    def __init__(self):
+        self.count = 0
+        self.lock = Lock()
+
+    def increment(self):
+        with self.lock:  # context manager is cleanest
+            self.count += 1
+```
+
+- The GIL does NOT prevent data races on Python objects
+- Operations like `+=` are not atomic — they involve read + modify + write
+- Always use `Lock` when multiple threads modify shared state
+- Use `with lock:` context manager for clean acquire/release
+
+## Item 55: Use Queue to Coordinate Work Between Threads
+```python
+from queue import Queue
+from threading import Thread
+
+def producer(queue):
+    for item in generate_items():
+        queue.put(item)
+    queue.put(None)  # sentinel to signal done
+
+def consumer(queue):
+    while True:
+        item = queue.get()
+        if item is None:
+            break
+        process(item)
+        queue.task_done()
+
+queue = Queue(maxsize=10)  # bounded for backpressure
+Thread(target=producer, args=(queue,)).start()
+Thread(target=consumer, args=(queue,)).start()
+queue.join()  # wait for all items to be processed
+```
+
+- `Queue` provides thread-safe FIFO communication
+- Use `maxsize` for backpressure (producer blocks when full)
+- Use `task_done()` + `join()` for completion tracking
+- Use sentinel values (None) to signal shutdown
+
+## Item 56: Know How to Recognize When Concurrency Is Necessary
+- Concurrency is needed when you have fan-out (one task spawning many) and fan-in (collecting results)
+- Signs you need concurrency: I/O-bound waits, independent tasks, pipeline processing
+- Start simple (sequential), then add concurrency only when needed
+
+## Item 57: Avoid Creating New Thread Instances for On-demand Fan-out
+- Creating a thread per task doesn't scale (thread creation overhead, memory)
+- Use thread pools instead (Item 58/59)
+
+## Item 58: Understand How Using Queue for Concurrency Requires Refactoring
+- Queue-based pipelines require significant refactoring
+- Consider `concurrent.futures` for simpler patterns
+
+## Item 59: Consider ThreadPoolExecutor When Threads Are Necessary for Concurrency
+```python
+from concurrent.futures import ThreadPoolExecutor
+
+def fetch_url(url):
+    return urllib.request.urlopen(url).read()
+
+with ThreadPoolExecutor(max_workers=5) as executor:
+    # Submit individual tasks
+    future = executor.submit(fetch_url, 'https://example.com')
+    result = future.result()
+
+    # Map over multiple inputs
+    results = list(executor.map(fetch_url, urls))
+```
+
+- Simpler than manual thread + Queue management
+- Automatically manages thread lifecycle
+- `max_workers` controls parallelism
+- Use `ProcessPoolExecutor` for CPU-bound tasks
+
+## Item 60: Achieve Highly Concurrent I/O with Coroutines
+```python
+import asyncio
+
+async def fetch_data(url):
+    # async I/O operation
+    reader, writer = await asyncio.open_connection(host, port)
+    writer.write(request)
+    data = await reader.read()
+    return data
+
+async def main():
+    # Run multiple coroutines concurrently
+    results = await asyncio.gather(
+        fetch_data('url1'),
+        fetch_data('url2'),
+        fetch_data('url3'),
+    )
+
+asyncio.run(main())
+```
+
+- Coroutines enable thousands of concurrent I/O operations
+- Use `async def` and `await` keywords
+- `asyncio.gather` runs multiple coroutines concurrently
+- Far more efficient than threads for I/O-heavy workloads
+
+## Item 61: Know How to Port Threaded I/O to asyncio
+- Replace `threading.Thread` with `async def` coroutines
+- Replace blocking I/O calls with `await async_version`
+- Replace `Lock` with `asyncio.Lock`
+- Replace `Queue` with `asyncio.Queue`
+- Use `asyncio.run()` as the entry point
+
+## Item 62: Mix Threads and Coroutines to Ease the Transition to asyncio
+```python
+# Run blocking code in a thread from async context
+import asyncio
+
+async def main():
+    loop = asyncio.get_event_loop()
+    result = await loop.run_in_executor(None, blocking_function, arg)
+
+# Run async code from synchronous context
+def sync_function():
+    loop = asyncio.new_event_loop()
+    result = loop.run_until_complete(async_function())
+```
+
+- Use `run_in_executor` to call blocking code from async code
+- Allows gradual migration from threads to asyncio
+- Never call blocking functions directly in async code (it blocks the event loop)
+
+## Item 63: Avoid Blocking the asyncio Event Loop to Maximize Responsiveness
+- Never use `time.sleep()` in async code — use `await asyncio.sleep()`
+- Never do CPU-heavy work in coroutines — use `run_in_executor`
+- Never use blocking I/O calls — use async equivalents (aiohttp, aiofiles, etc.)
+- Profile with `asyncio.get_event_loop().slow_callback_duration`
+
+## Item 64: Consider concurrent.futures for True Parallelism
+```python
+from concurrent.futures import ProcessPoolExecutor
+
+def cpu_heavy(data):
+    return complex_computation(data)
+
+with ProcessPoolExecutor() as executor:
+    results = list(executor.map(cpu_heavy, data_chunks))
+```
+
+- `ProcessPoolExecutor` bypasses the GIL for true CPU parallelism
+- Data is serialized between processes (use for independent tasks)
+- Same API as `ThreadPoolExecutor` — easy to switch

package/effective-python/ref-08-robustness-performance.md

@@ -0,0 +1,248 @@
+# Chapter 8: Robustness and Performance (Items 65-76)
+
+## Item 65: Take Advantage of Each Block in try/except/else/finally
+```python
+# Full structure
+try:
+    # Code that might raise
+    result = dangerous_operation()
+except SomeError as e:
+    # Handle specific error
+    log_error(e)
+except (TypeError, ValueError):
+    # Handle multiple error types
+    handle_bad_input()
+else:
+    # Runs ONLY if no exception was raised
+    # Use for code that depends on try succeeding
+    process(result)
+finally:
+    # ALWAYS runs, even if exception was raised
+    # Use for cleanup (closing files, releasing locks)
+    cleanup()
+```
+
+- `else` block: reduces code in `try`, makes it clear what you're protecting
+- `finally` block: guaranteed cleanup
+- Don't put too much in `try` — only the code that can raise the expected exception
+
+## Item 66: Consider contextlib and with Statements for Reusable try/finally Behavior
+```python
+from contextlib import contextmanager
+
+@contextmanager
+def log_level(level, name):
+    logger = logging.getLogger(name)
+    old_level = logger.level
+    logger.setLevel(level)
+    try:
+        yield logger
+    finally:
+        logger.setLevel(old_level)
+
+with log_level(logging.DEBUG, 'my-log') as logger:
+    logger.debug('Debug message')
+    # Level is automatically restored after the block
+```
+
+- Use `contextlib.contextmanager` for simple context managers
+- Use `with` statements instead of manual try/finally
+- The `yield` in a context manager is where the `with` block executes
+
+## Item 67: Use datetime Instead of time for Local Clocks
+```python
+from datetime import datetime, timezone
+import pytz  # or zoneinfo (Python 3.9+)
+
+# BAD — time module is unreliable for timezones
+import time
+time.localtime()  # platform-dependent behavior
+
+# GOOD — datetime with explicit timezone
+now = datetime.now(tz=timezone.utc)
+
+# Convert between timezones
+eastern = pytz.timezone('US/Eastern')
+local_time = now.astimezone(eastern)
+
+# Python 3.9+ — use zoneinfo
+from zoneinfo import ZoneInfo
+eastern = ZoneInfo('America/New_York')
+local_time = now.astimezone(eastern)
+```
+
+- Always store/transmit times in UTC
+- Convert to local time only for display
+- Use `pytz` or `zoneinfo` for timezone handling
+- Never use the `time` module for timezone conversions
+
+## Item 68: Make pickle Reliable with copyreg
+```python
+import copyreg
+import pickle
+
+class GameState:
+    def __init__(self, level=0, lives=4, points=0):
+        self.level = level
+        self.lives = lives
+        self.points = points
+
+def pickle_game_state(game_state):
+    kwargs = game_state.__dict__
+    return unpickle_game_state, (kwargs,)
+
+def unpickle_game_state(kwargs):
+    return GameState(**kwargs)
+
+copyreg.pickle(GameState, pickle_game_state)
+```
+
+- `copyreg` makes pickle forward-compatible when classes change
+- Register custom serialization functions for your classes
+- Always provide default values for new attributes
+
+## Item 69: Use decimal When Precision Matters
+```python
+from decimal import Decimal, ROUND_UP
+
+# BAD — float precision issues
+rate = 1.45
+seconds = 222
+cost = rate * seconds / 60  # 5.364999999999999
+
+# GOOD — Decimal for exact arithmetic
+rate = Decimal('1.45')
+seconds = Decimal('222')
+cost = rate * seconds / Decimal('60')
+rounded = cost.quantize(Decimal('0.01'), rounding=ROUND_UP)
+```
+
+- Use `Decimal` for financial calculations, exact fractions
+- Always construct from strings (`Decimal('1.45')`) not floats (`Decimal(1.45)`)
+- Use `quantize` for rounding control
+
+## Item 70: Profile Before Optimizing
+```python
+from cProfile import Profile
+from pstats import Stats
+
+profiler = Profile()
+profiler.runcall(my_function, arg1, arg2)
+
+stats = Stats(profiler)
+stats.strip_dirs()
+stats.sort_stats('cumulative')
+stats.print_stats()
+```
+
+- Never guess where bottlenecks are — profile first
+- Use `cProfile` for C-extension speed profiling
+- `cumulative` time shows total time including sub-calls
+- `tottime` shows time in the function itself (excluding sub-calls)
+- Focus optimization on the top functions by cumulative time
+
+## Item 71: Prefer deque for Producer-Consumer Queues
+```python
+from collections import deque
+
+# FIFO queue operations
+queue = deque()
+queue.append('item')      # O(1) add to right
+item = queue.popleft()    # O(1) remove from left
+
+# BAD — list as queue
+queue = []
+queue.append('item')      # O(1)
+item = queue.pop(0)       # O(n)! shifts all elements
+```
+
+- `list.pop(0)` is O(n); `deque.popleft()` is O(1)
+- `deque` also supports `maxlen` for bounded buffers
+- Use `deque` for any FIFO pattern
+
+## Item 72: Consider Searching Sorted Sequences with bisect
+```python
+import bisect
+
+sorted_list = [2, 5, 8, 12, 16, 23, 38, 56, 72, 91]
+
+# Find insertion point
+index = bisect.bisect_left(sorted_list, 12)  # 3
+index = bisect.bisect_right(sorted_list, 12) # 4
+
+# Insert while maintaining sort order
+bisect.insort(sorted_list, 15)  # inserts 15 in correct position
+```
+
+- Binary search is O(log n) vs O(n) for linear search
+- Use `bisect_left` for leftmost position, `bisect_right` for rightmost
+- `insort` keeps list sorted after insertion
+- Requires the sequence to already be sorted
+
+## Item 73: Know How to Use heapq for Priority Queues
+```python
+import heapq
+
+# Create a min-heap
+heap = []
+heapq.heappush(heap, 5)
+heapq.heappush(heap, 1)
+heapq.heappush(heap, 3)
+
+# Pop smallest
+smallest = heapq.heappop(heap)  # 1
+
+# Get n smallest/largest
+heapq.nsmallest(3, data)
+heapq.nlargest(3, data)
+
+# Priority queue with tuples
+heapq.heappush(heap, (priority, item))
+```
+
+- heapq provides O(log n) push and pop operations
+- Always a min-heap (smallest first)
+- For max-heap, negate the values
+- Use for priority queues, top-K problems, merge sorted streams
+
+## Item 74: Consider memoryview and bytearray for Zero-Copy Interactions with bytes
+```python
+# BAD — copying bytes on every slice
+data = b'large data...'
+chunk = data[10:20]  # creates a new bytes object
+
+# GOOD — zero-copy with memoryview
+data = bytearray(b'large data...')
+view = memoryview(data)
+chunk = view[10:20]  # no copy, just a view
+chunk[:5] = b'hello'  # writes directly to original data
+```
+
+- `memoryview` provides zero-copy slicing of bytes-like objects
+- Essential for high-performance I/O and data processing
+- Works with `bytearray`, `array.array`, NumPy arrays
+- Use for socket I/O, file I/O, binary protocol parsing
+
+## Item 75: Use repr Strings for Debugging Output
+```python
+class MyClass:
+    def __init__(self, value):
+        self.value = value
+
+    def __repr__(self):
+        return f'{self.__class__.__name__}({self.value!r})'
+
+    def __str__(self):
+        return f'MyClass with value {self.value}'
+```
+
+- `repr()` gives an unambiguous string for debugging
+- `str()` gives a human-readable string
+- Always implement `__repr__` on your classes
+- Use `!r` in f-strings for repr formatting: `f'{obj!r}'`
+
+## Item 76: Verify Related Behaviors in TestCase Subclasses
+(Cross-reference with Chapter 9 Testing)
+- Group related tests in TestCase subclasses
+- Use descriptive test method names
+- Test both success and failure cases

package/effective-python/ref-09-testing-debugging.md

@@ -0,0 +1,253 @@
+# Chapter 9: Testing and Debugging (Items 77-85)
+
+## Item 77: Isolate Tests from Each Other via setUp, tearDown, setUpModule, etc.
+```python
+import unittest
+
+class DatabaseTestCase(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        """Run once before all tests in this class."""
+        cls.db = create_test_database()
+
+    @classmethod
+    def tearDownClass(cls):
+        """Run once after all tests in this class."""
+        cls.db.close()
+
+    def setUp(self):
+        """Run before each test method."""
+        self.connection = self.db.connect()
+        self.transaction = self.connection.begin()
+
+    def tearDown(self):
+        """Run after each test method."""
+        self.transaction.rollback()
+        self.connection.close()
+
+    def test_query(self):
+        result = self.connection.execute('SELECT 1')
+        self.assertEqual(result, 1)
+```
+
+- `setUp`/`tearDown` run for every test method (isolation)
+- `setUpClass`/`tearDownClass` run once per class (expensive setup)
+- `setUpModule`/`tearDownModule` run once per module
+- Always clean up in tearDown (even if test fails)
+
+## Item 78: Use Mocks to Test Code with Complex Dependencies
+```python
+from unittest.mock import patch, MagicMock, call
+
+# Mock a function
+@patch('mymodule.external_api_call')
+def test_process(mock_api):
+    mock_api.return_value = {'status': 'ok'}
+    result = process_data()
+    mock_api.assert_called_once_with(expected_arg)
+    assert result == expected_result
+
+# Mock an object's method
+def test_with_mock():
+    mock_db = MagicMock()
+    mock_db.query.return_value = [{'id': 1}]
+    service = MyService(db=mock_db)
+    result = service.get_items()
+    mock_db.query.assert_called_once()
+
+# Verify call order
+mock_db.query.assert_has_calls([
+    call('SELECT * FROM users'),
+    call('SELECT * FROM orders'),
+])
+
+# Use spec for type checking
+mock = MagicMock(spec=RealClass)
+mock.nonexistent_method()  # raises AttributeError
+```
+
+- Mock external dependencies (APIs, databases, file systems)
+- Use `@patch` to replace modules/objects during tests
+- Use `spec=RealClass` to catch API mismatches
+- Verify both return values and call patterns
+- Use `side_effect` for exceptions or multiple return values:
+```python
+mock.side_effect = ValueError('error')
+mock.side_effect = [1, 2, 3]  # returns different values each call
+```
+
+## Item 79: Encapsulate Dependencies to Facilitate Mocking and Testing
+```python
+# BAD — hard-coded dependency
+class DataProcessor:
+    def process(self):
+        data = requests.get('https://api.example.com/data').json()
+        return transform(data)
+
+# GOOD — inject dependency
+class DataProcessor:
+    def __init__(self, data_fetcher):
+        self._fetcher = data_fetcher
+
+    def process(self):
+        data = self._fetcher.get_data()
+        return transform(data)
+
+# Easy to test
+class FakeFetcher:
+    def get_data(self):
+        return {'test': 'data'}
+
+processor = DataProcessor(FakeFetcher())
+result = processor.process()
+```
+
+- Dependency injection makes code testable
+- Accept dependencies as constructor or method parameters
+- Use abstract base classes or protocols to define interfaces
+- Fakes/stubs are often clearer than mocks for complex dependencies
+
+## Item 80: Consider Interactive Debugging with pdb
+```python
+# Drop into debugger at a specific point
+def complex_function(data):
+    result = step_one(data)
+    breakpoint()  # Python 3.7+ (same as pdb.set_trace())
+    final = step_two(result)
+    return final
+```
+
+**Key pdb commands:**
+- `n` (next) — execute next line
+- `s` (step) — step into function call
+- `c` (continue) — continue execution until next breakpoint
+- `p expr` — print expression
+- `pp expr` — pretty-print expression
+- `l` (list) — show current code context
+- `w` (where) — show call stack
+- `b line` — set breakpoint at line
+- `r` (return) — run until current function returns
+- `q` (quit) — quit debugger
+
+- Use `breakpoint()` (Python 3.7+) instead of `import pdb; pdb.set_trace()`
+- Use `PYTHONBREAKPOINT=0` environment variable to disable all breakpoints
+- Use `post_mortem()` to debug after an exception
+
+## Item 81: Use tracemalloc to Understand Memory Usage and Leaks
+```python
+import tracemalloc
+
+tracemalloc.start()
+
+# ... run code that uses memory ...
+
+snapshot = tracemalloc.take_snapshot()
+top_stats = snapshot.statistics('lineno')
+
+print('Top 10 memory allocations:')
+for stat in top_stats[:10]:
+    print(stat)
+
+# Compare snapshots to find leaks
+snapshot1 = tracemalloc.take_snapshot()
+# ... more code ...
+snapshot2 = tracemalloc.take_snapshot()
+top_stats = snapshot2.compare_to(snapshot1, 'lineno')
+```
+
+- `tracemalloc` tracks where memory was allocated
+- Use snapshot comparison to find memory leaks
+- Shows file and line number of allocations
+- Much more useful than `gc` module for debugging memory issues
+
+## Item 82: Know Where to Find Community-Built Modules
+- PyPI (Python Package Index) is the main repository
+- Use `pip install` to install packages
+- Check package health: last update, stars, downloads, issues
+- Popular packages: requests, flask, django, pandas, numpy, pytest
+
+## Item 83: Use Virtual Environments for Isolated and Reproducible Dependencies
+```bash
+# Create virtual environment
+python3 -m venv myenv
+
+# Activate
+source myenv/bin/activate
+
+# Install packages
+pip install flask==2.0.1
+
+# Freeze dependencies
+pip freeze > requirements.txt
+
+# Recreate environment
+pip install -r requirements.txt
+```
+
+- Always use virtual environments for projects
+- Never install packages globally with `pip`
+- Use `requirements.txt` for reproducible environments
+- Consider `pyproject.toml` and modern tools (poetry, pipenv)
+
+## Item 84: Write Docstrings for Every Module, Class, and Function
+```python
+"""Module docstring: brief description of the module's purpose."""
+
+class MyClass:
+    """One-line summary of the class.
+
+    Extended description of the class if needed.
+
+    Attributes:
+        attr1: Description of attr1.
+        attr2: Description of attr2.
+    """
+
+    def method(self, arg1: str, arg2: int = 0) -> bool:
+        """One-line summary of method.
+
+        Extended description if needed.
+
+        Args:
+            arg1: Description of arg1.
+            arg2: Description of arg2. Defaults to 0.
+
+        Returns:
+            Description of return value.
+
+        Raises:
+            ValueError: When arg1 is empty.
+        """
+```
+
+- First line: one-line summary ending with period
+- Blank line, then extended description if needed
+- Document Args, Returns, Raises sections
+- Use Google style or NumPy style consistently
+- Type hints complement but don't replace docstrings
+
+## Item 85: Use Packages to Organize Modules and Provide Stable APIs
+```python
+# mypackage/__init__.py
+from mypackage.core import (
+    PublicClass,
+    public_function,
+)
+
+__all__ = ['PublicClass', 'public_function']
+
+# mypackage/core.py
+class PublicClass:
+    ...
+
+def public_function():
+    ...
+
+def _private_helper():  # not exported
+    ...
+```
+
+- Use `__init__.py` to define public API
+- Use `__all__` to control `from package import *` behavior
+- Keep internal modules private with `_` prefix
+- Stable API = external code won't break when internals change