@booklib/skills 1.0.0 → 1.2.0

package/effective-python-skill/ref-06-metaclasses-attributes.md

@@ -0,0 +1,209 @@
+# Chapter 6: Metaclasses and Attributes (Items 44-51)
+
+## Item 44: Use Plain Attributes Instead of Setter and Getter Methods
+```python
+# BAD — Java-style getters/setters
+class OldResistor:
+    def __init__(self, ohms):
+        self._ohms = ohms
+
+    def get_ohms(self):
+        return self._ohms
+
+    def set_ohms(self, ohms):
+        self._ohms = ohms
+
+# GOOD — plain attributes
+class Resistor:
+    def __init__(self, ohms):
+        self.ohms = ohms
+
+# If you later need behavior, migrate to @property (Item 44)
+```
+
+- Start with simple public attributes
+- If you need special behavior later, use `@property` without changing the API
+- Never write explicit getter/setter methods in Python
+
+## Item 45: Consider @property Instead of Refactoring Attributes
+```python
+class Bucket:
+    def __init__(self, period):
+        self.period = period
+        self.quota = 0
+
+    @property
+    def quota(self):
+        return self._quota
+
+    @quota.setter
+    def quota(self, value):
+        if value < 0:
+            raise ValueError('Quota must be >= 0')
+        self._quota = value
+```
+
+- Use `@property` to add validation, logging, or computed behavior
+- Keeps backward-compatible API (attribute access syntax)
+- Don't do too much work in property getters — keep them fast
+- If a property is getting complex, refactor to a normal method
+
+## Item 46: Use Descriptors for Reusable @property Methods
+```python
+class Grade:
+    """Reusable validation descriptor."""
+    def __init__(self):
+        self._values = {}
+
+    def __get__(self, instance, instance_type):
+        if instance is None:
+            return self
+        return self._values.get(instance, 0)
+
+    def __set__(self, instance, value):
+        if not (0 <= value <= 100):
+            raise ValueError('Grade must be between 0 and 100')
+        self._values[instance] = value
+
+class Exam:
+    math_grade = Grade()
+    writing_grade = Grade()
+    science_grade = Grade()
+
+exam = Exam()
+exam.math_grade = 95  # calls Grade.__set__
+print(exam.math_grade)  # calls Grade.__get__
+```
+
+- Use descriptors when you'd copy-paste `@property` logic
+- Store per-instance data using `WeakKeyDictionary` to avoid memory leaks:
+```python
+from weakref import WeakKeyDictionary
+class Grade:
+    def __init__(self):
+        self._values = WeakKeyDictionary()
+```
+
+## Item 47: Use __getattr__, __getattribute__, and __setattr__ for Lazy Attributes
+```python
+# __getattr__ — called only when attribute not found normally
+class LazyRecord:
+    def __init__(self):
+        self.exists = 5
+
+    def __getattr__(self, name):
+        value = f'Value for {name}'
+        setattr(self, name, value)  # cache it
+        return value
+
+# __getattribute__ — called for EVERY attribute access
+class ValidatingRecord:
+    def __getattribute__(self, name):
+        value = super().__getattribute__(name)
+        # validate or log every access
+        return value
+
+# __setattr__ — called for EVERY attribute assignment
+class SavingRecord:
+    def __setattr__(self, name, value):
+        super().__setattr__(name, value)
+        # save to database, etc.
+```
+
+- `__getattr__` is for lazy/dynamic attributes (called only on missing)
+- `__getattribute__` intercepts ALL attribute access (use carefully)
+- Always use `super()` in these methods to avoid infinite recursion
+- `hasattr` and `getattr` also trigger `__getattribute__`
+
+## Item 48: Validate Subclasses with __init_subclass__
+```python
+class Polygon:
+    sides = None
+
+    def __init_subclass__(cls, **kwargs):
+        super().__init_subclass__(**kwargs)
+        if cls.sides is None or cls.sides < 3:
+            raise ValueError('Polygons need 3+ sides')
+
+class Triangle(Polygon):
+    sides = 3  # OK
+
+class Line(Polygon):
+    sides = 2  # Raises ValueError at class definition time!
+```
+
+- `__init_subclass__` is called when a class is subclassed
+- Use it for validation, registration, or class setup
+- Much simpler than metaclasses for most use cases
+- Works with multiple inheritance (use `**kwargs` to pass through)
+
+## Item 49: Register Class Existence with __init_subclass__
+```python
+registry = {}
+
+class Serializable:
+    def __init_subclass__(cls, **kwargs):
+        super().__init_subclass__(**kwargs)
+        registry[cls.__name__] = cls
+
+class Point(Serializable):
+    def __init__(self, x, y):
+        self.x = x
+        self.y = y
+
+# Point is automatically registered
+assert registry['Point'] is Point
+```
+
+- Auto-registration pattern: base class registers all subclasses
+- Useful for serialization, plugin systems, ORM models
+- Replaces the need for explicit registration decorators or metaclasses
+
+## Item 50: Annotate Class Attributes with __set_name__
+```python
+class Field:
+    def __set_name__(self, owner, name):
+        self.name = name          # attribute name on the class
+        self.internal_name = '_' + name  # storage name
+
+    def __get__(self, instance, instance_type):
+        if instance is None:
+            return self
+        return getattr(instance, self.internal_name, '')
+
+    def __set__(self, instance, value):
+        setattr(instance, self.internal_name, value)
+
+class Customer:
+    first_name = Field()  # __set_name__ called with name='first_name'
+    last_name = Field()
+```
+
+- `__set_name__` is called automatically when a descriptor is assigned to a class attribute
+- Eliminates the need to repeat the attribute name
+- Works with descriptors to provide clean, DRY class definitions
+
+## Item 51: Prefer Class Decorators Over Metaclasses for Composable Class Extensions
+```python
+# Class decorator — simple and composable
+def my_class_decorator(cls):
+    # modify or wrap cls
+    original_init = cls.__init__
+
+    def new_init(self, *args, **kwargs):
+        print(f'Creating {cls.__name__}')
+        original_init(self, *args, **kwargs)
+
+    cls.__init__ = new_init
+    return cls
+
+@my_class_decorator
+class MyClass:
+    def __init__(self, value):
+        self.value = value
+```
+
+- Class decorators are simpler than metaclasses
+- They compose easily (stack multiple decorators)
+- Use metaclasses only when you need to control the class creation process itself
+- Prefer: `__init_subclass__` > class decorators > metaclasses

package/effective-python-skill/ref-07-concurrency.md

@@ -0,0 +1,213 @@
+# Chapter 7: Concurrency and Parallelism (Items 52-64)
+
+## Item 52: Use subprocess to Manage Child Processes
+```python
+import subprocess
+
+# Run a command and capture output
+result = subprocess.run(
+    ['echo', 'Hello from subprocess'],
+    capture_output=True,
+    text=True
+)
+print(result.stdout)
+
+# Set timeout
+result = subprocess.run(
+    ['sleep', '10'],
+    timeout=5  # raises TimeoutExpired after 5 seconds
+)
+
+# Pipe data to child process
+result = subprocess.run(
+    ['openssl', 'enc', '-aes-256-cbc', '-pass', 'pass:key'],
+    input=b'data to encrypt',
+    capture_output=True
+)
+
+# Run parallel child processes
+procs = [subprocess.Popen(['cmd', arg]) for arg in args]
+for proc in procs:
+    proc.communicate()  # wait for each
+```
+
+- Use `subprocess.run` for simple command execution
+- Use `subprocess.Popen` for parallel or streaming processes
+- Always set timeouts to prevent hanging
+
+## Item 53: Use Threads for Blocking I/O, Avoid for Parallelism
+```python
+import threading
+
+# Threads for I/O parallelism — GOOD
+def download(url):
+    resp = urllib.request.urlopen(url)
+    return resp.read()
+
+threads = [threading.Thread(target=download, args=(url,)) for url in urls]
+for t in threads:
+    t.start()
+for t in threads:
+    t.join()
+```
+
+- The GIL prevents true CPU parallelism with threads
+- Threads ARE useful for blocking I/O (network, file system, etc.)
+- For CPU-bound work, use `multiprocessing` or `concurrent.futures.ProcessPoolExecutor`
+- Never use threads for CPU-intensive computation
+
+## Item 54: Use Lock to Prevent Data Races in Threads
+```python
+from threading import Lock
+
+class Counter:
+    def __init__(self):
+        self.count = 0
+        self.lock = Lock()
+
+    def increment(self):
+        with self.lock:  # context manager is cleanest
+            self.count += 1
+```
+
+- The GIL does NOT prevent data races on Python objects
+- Operations like `+=` are not atomic — they involve read + modify + write
+- Always use `Lock` when multiple threads modify shared state
+- Use `with lock:` context manager for clean acquire/release
+
+## Item 55: Use Queue to Coordinate Work Between Threads
+```python
+from queue import Queue
+from threading import Thread
+
+def producer(queue):
+    for item in generate_items():
+        queue.put(item)
+    queue.put(None)  # sentinel to signal done
+
+def consumer(queue):
+    while True:
+        item = queue.get()
+        if item is None:
+            break
+        process(item)
+        queue.task_done()
+
+queue = Queue(maxsize=10)  # bounded for backpressure
+Thread(target=producer, args=(queue,)).start()
+Thread(target=consumer, args=(queue,)).start()
+queue.join()  # wait for all items to be processed
+```
+
+- `Queue` provides thread-safe FIFO communication
+- Use `maxsize` for backpressure (producer blocks when full)
+- Use `task_done()` + `join()` for completion tracking
+- Use sentinel values (None) to signal shutdown
+
+## Item 56: Know How to Recognize When Concurrency Is Necessary
+- Concurrency is needed when you have fan-out (one task spawning many) and fan-in (collecting results)
+- Signs you need concurrency: I/O-bound waits, independent tasks, pipeline processing
+- Start simple (sequential), then add concurrency only when needed
+
+## Item 57: Avoid Creating New Thread Instances for On-demand Fan-out
+- Creating a thread per task doesn't scale (thread creation overhead, memory)
+- Use thread pools instead (Item 58/59)
+
+## Item 58: Understand How Using Queue for Concurrency Requires Refactoring
+- Queue-based pipelines require significant refactoring
+- Consider `concurrent.futures` for simpler patterns
+
+## Item 59: Consider ThreadPoolExecutor When Threads Are Necessary for Concurrency
+```python
+from concurrent.futures import ThreadPoolExecutor
+
+def fetch_url(url):
+    return urllib.request.urlopen(url).read()
+
+with ThreadPoolExecutor(max_workers=5) as executor:
+    # Submit individual tasks
+    future = executor.submit(fetch_url, 'https://example.com')
+    result = future.result()
+
+    # Map over multiple inputs
+    results = list(executor.map(fetch_url, urls))
+```
+
+- Simpler than manual thread + Queue management
+- Automatically manages thread lifecycle
+- `max_workers` controls parallelism
+- Use `ProcessPoolExecutor` for CPU-bound tasks
+
+## Item 60: Achieve Highly Concurrent I/O with Coroutines
+```python
+import asyncio
+
+async def fetch_data(url):
+    # async I/O operation
+    reader, writer = await asyncio.open_connection(host, port)
+    writer.write(request)
+    data = await reader.read()
+    return data
+
+async def main():
+    # Run multiple coroutines concurrently
+    results = await asyncio.gather(
+        fetch_data('url1'),
+        fetch_data('url2'),
+        fetch_data('url3'),
+    )
+
+asyncio.run(main())
+```
+
+- Coroutines enable thousands of concurrent I/O operations
+- Use `async def` and `await` keywords
+- `asyncio.gather` runs multiple coroutines concurrently
+- Far more efficient than threads for I/O-heavy workloads
+
+## Item 61: Know How to Port Threaded I/O to asyncio
+- Replace `threading.Thread` with `async def` coroutines
+- Replace blocking I/O calls with `await async_version`
+- Replace `Lock` with `asyncio.Lock`
+- Replace `Queue` with `asyncio.Queue`
+- Use `asyncio.run()` as the entry point
+
+## Item 62: Mix Threads and Coroutines to Ease the Transition to asyncio
+```python
+# Run blocking code in a thread from async context
+import asyncio
+
+async def main():
+    loop = asyncio.get_event_loop()
+    result = await loop.run_in_executor(None, blocking_function, arg)
+
+# Run async code from synchronous context
+def sync_function():
+    loop = asyncio.new_event_loop()
+    result = loop.run_until_complete(async_function())
+```
+
+- Use `run_in_executor` to call blocking code from async code
+- Allows gradual migration from threads to asyncio
+- Never call blocking functions directly in async code (it blocks the event loop)
+
+## Item 63: Avoid Blocking the asyncio Event Loop to Maximize Responsiveness
+- Never use `time.sleep()` in async code — use `await asyncio.sleep()`
+- Never do CPU-heavy work in coroutines — use `run_in_executor`
+- Never use blocking I/O calls — use async equivalents (aiohttp, aiofiles, etc.)
+- Profile with `asyncio.get_event_loop().slow_callback_duration`
+
+## Item 64: Consider concurrent.futures for True Parallelism
+```python
+from concurrent.futures import ProcessPoolExecutor
+
+def cpu_heavy(data):
+    return complex_computation(data)
+
+with ProcessPoolExecutor() as executor:
+    results = list(executor.map(cpu_heavy, data_chunks))
+```
+
+- `ProcessPoolExecutor` bypasses the GIL for true CPU parallelism
+- Data is serialized between processes (use for independent tasks)
+- Same API as `ThreadPoolExecutor` — easy to switch

package/effective-python-skill/ref-08-robustness-performance.md

@@ -0,0 +1,248 @@
+# Chapter 8: Robustness and Performance (Items 65-76)
+
+## Item 65: Take Advantage of Each Block in try/except/else/finally
+```python
+# Full structure
+try:
+    # Code that might raise
+    result = dangerous_operation()
+except SomeError as e:
+    # Handle specific error
+    log_error(e)
+except (TypeError, ValueError):
+    # Handle multiple error types
+    handle_bad_input()
+else:
+    # Runs ONLY if no exception was raised
+    # Use for code that depends on try succeeding
+    process(result)
+finally:
+    # ALWAYS runs, even if exception was raised
+    # Use for cleanup (closing files, releasing locks)
+    cleanup()
+```
+
+- `else` block: reduces code in `try`, makes it clear what you're protecting
+- `finally` block: guaranteed cleanup
+- Don't put too much in `try` — only the code that can raise the expected exception
+
+## Item 66: Consider contextlib and with Statements for Reusable try/finally Behavior
+```python
+from contextlib import contextmanager
+
+@contextmanager
+def log_level(level, name):
+    logger = logging.getLogger(name)
+    old_level = logger.level
+    logger.setLevel(level)
+    try:
+        yield logger
+    finally:
+        logger.setLevel(old_level)
+
+with log_level(logging.DEBUG, 'my-log') as logger:
+    logger.debug('Debug message')
+    # Level is automatically restored after the block
+```
+
+- Use `contextlib.contextmanager` for simple context managers
+- Use `with` statements instead of manual try/finally
+- The `yield` in a context manager is where the `with` block executes
+
+## Item 67: Use datetime Instead of time for Local Clocks
+```python
+from datetime import datetime, timezone
+import pytz  # or zoneinfo (Python 3.9+)
+
+# BAD — time module is unreliable for timezones
+import time
+time.localtime()  # platform-dependent behavior
+
+# GOOD — datetime with explicit timezone
+now = datetime.now(tz=timezone.utc)
+
+# Convert between timezones
+eastern = pytz.timezone('US/Eastern')
+local_time = now.astimezone(eastern)
+
+# Python 3.9+ — use zoneinfo
+from zoneinfo import ZoneInfo
+eastern = ZoneInfo('America/New_York')
+local_time = now.astimezone(eastern)
+```
+
+- Always store/transmit times in UTC
+- Convert to local time only for display
+- Use `pytz` or `zoneinfo` for timezone handling
+- Never use the `time` module for timezone conversions
+
+## Item 68: Make pickle Reliable with copyreg
+```python
+import copyreg
+import pickle
+
+class GameState:
+    def __init__(self, level=0, lives=4, points=0):
+        self.level = level
+        self.lives = lives
+        self.points = points
+
+def pickle_game_state(game_state):
+    kwargs = game_state.__dict__
+    return unpickle_game_state, (kwargs,)
+
+def unpickle_game_state(kwargs):
+    return GameState(**kwargs)
+
+copyreg.pickle(GameState, pickle_game_state)
+```
+
+- `copyreg` makes pickle forward-compatible when classes change
+- Register custom serialization functions for your classes
+- Always provide default values for new attributes
+
+## Item 69: Use decimal When Precision Matters
+```python
+from decimal import Decimal, ROUND_UP
+
+# BAD — float precision issues
+rate = 1.45
+seconds = 222
+cost = rate * seconds / 60  # 5.364999999999999
+
+# GOOD — Decimal for exact arithmetic
+rate = Decimal('1.45')
+seconds = Decimal('222')
+cost = rate * seconds / Decimal('60')
+rounded = cost.quantize(Decimal('0.01'), rounding=ROUND_UP)
+```
+
+- Use `Decimal` for financial calculations, exact fractions
+- Always construct from strings (`Decimal('1.45')`) not floats (`Decimal(1.45)`)
+- Use `quantize` for rounding control
+
+## Item 70: Profile Before Optimizing
+```python
+from cProfile import Profile
+from pstats import Stats
+
+profiler = Profile()
+profiler.runcall(my_function, arg1, arg2)
+
+stats = Stats(profiler)
+stats.strip_dirs()
+stats.sort_stats('cumulative')
+stats.print_stats()
+```
+
+- Never guess where bottlenecks are — profile first
+- Use `cProfile` for C-extension speed profiling
+- `cumulative` time shows total time including sub-calls
+- `tottime` shows time in the function itself (excluding sub-calls)
+- Focus optimization on the top functions by cumulative time
+
+## Item 71: Prefer deque for Producer-Consumer Queues
+```python
+from collections import deque
+
+# FIFO queue operations
+queue = deque()
+queue.append('item')      # O(1) add to right
+item = queue.popleft()    # O(1) remove from left
+
+# BAD — list as queue
+queue = []
+queue.append('item')      # O(1)
+item = queue.pop(0)       # O(n)! shifts all elements
+```
+
+- `list.pop(0)` is O(n); `deque.popleft()` is O(1)
+- `deque` also supports `maxlen` for bounded buffers
+- Use `deque` for any FIFO pattern
+
+## Item 72: Consider Searching Sorted Sequences with bisect
+```python
+import bisect
+
+sorted_list = [2, 5, 8, 12, 16, 23, 38, 56, 72, 91]
+
+# Find insertion point
+index = bisect.bisect_left(sorted_list, 12)  # 3
+index = bisect.bisect_right(sorted_list, 12) # 4
+
+# Insert while maintaining sort order
+bisect.insort(sorted_list, 15)  # inserts 15 in correct position
+```
+
+- Binary search is O(log n) vs O(n) for linear search
+- Use `bisect_left` for leftmost position, `bisect_right` for rightmost
+- `insort` keeps list sorted after insertion
+- Requires the sequence to already be sorted
+
+## Item 73: Know How to Use heapq for Priority Queues
+```python
+import heapq
+
+# Create a min-heap
+heap = []
+heapq.heappush(heap, 5)
+heapq.heappush(heap, 1)
+heapq.heappush(heap, 3)
+
+# Pop smallest
+smallest = heapq.heappop(heap)  # 1
+
+# Get n smallest/largest
+heapq.nsmallest(3, data)
+heapq.nlargest(3, data)
+
+# Priority queue with tuples
+heapq.heappush(heap, (priority, item))
+```
+
+- heapq provides O(log n) push and pop operations
+- Always a min-heap (smallest first)
+- For max-heap, negate the values
+- Use for priority queues, top-K problems, merge sorted streams
+
+## Item 74: Consider memoryview and bytearray for Zero-Copy Interactions with bytes
+```python
+# BAD — copying bytes on every slice
+data = b'large data...'
+chunk = data[10:20]  # creates a new bytes object
+
+# GOOD — zero-copy with memoryview
+data = bytearray(b'large data...')
+view = memoryview(data)
+chunk = view[10:20]  # no copy, just a view
+chunk[:5] = b'hello'  # writes directly to original data
+```
+
+- `memoryview` provides zero-copy slicing of bytes-like objects
+- Essential for high-performance I/O and data processing
+- Works with `bytearray`, `array.array`, NumPy arrays
+- Use for socket I/O, file I/O, binary protocol parsing
+
+## Item 75: Use repr Strings for Debugging Output
+```python
+class MyClass:
+    def __init__(self, value):
+        self.value = value
+
+    def __repr__(self):
+        return f'{self.__class__.__name__}({self.value!r})'
+
+    def __str__(self):
+        return f'MyClass with value {self.value}'
+```
+
+- `repr()` gives an unambiguous string for debugging
+- `str()` gives a human-readable string
+- Always implement `__repr__` on your classes
+- Use `!r` in f-strings for repr formatting: `f'{obj!r}'`
+
+## Item 76: Verify Related Behaviors in TestCase Subclasses
+(Cross-reference with Chapter 9 Testing)
+- Group related tests in TestCase subclasses
+- Use descriptive test method names
+- Test both success and failure cases