@booklib/skills 1.0.0

package/system-design-interview/references/review-checklist.md

@@ -0,0 +1,201 @@
+# System Design Interview — Design Review Checklist
+
+Systematic checklist for reviewing system designs against the 16 chapters
+from *System Design Interview* by Alex Xu.
+
+---
+
+## 1. Scaling Fundamentals (Chapter 1)
+
+### Infrastructure
+- [ ] **Ch 1 — Load balancing** — Is traffic distributed across multiple servers with failover?
+- [ ] **Ch 1 — Database replication** — Are read replicas used for read-heavy workloads?
+- [ ] **Ch 1 — Caching** — Is a cache layer (Redis/Memcached) used for frequently accessed data?
+- [ ] **Ch 1 — CDN** — Are static assets served from a CDN?
+- [ ] **Ch 1 — Stateless web tier** — Is session data stored in shared storage, not on web servers?
+- [ ] **Ch 1 — Message queue** — Are time-consuming tasks decoupled via async message queues?
+- [ ] **Ch 1 — Database sharding** — Is data sharded for write-heavy or large-scale workloads?
+- [ ] **Ch 1 — Data centers** — Is multi-datacenter deployment considered for geo-distribution?
+
+### Data Layer
+- [ ] **Ch 1 — Database choice** — Is the right database type selected (SQL vs. NoSQL) based on access patterns?
+- [ ] **Ch 1 — Shard key** — Is the shard key chosen for even data distribution?
+- [ ] **Ch 1 — Hotspot mitigation** — Are celebrity/hotspot problems addressed?
+
+---
+
+## 2. Capacity Estimation (Chapter 2)
+
+### Back-of-Envelope
+- [ ] **Ch 2 — QPS estimated** — Are queries per second calculated (average and peak)?
+- [ ] **Ch 2 — Storage estimated** — Is storage growth estimated over time (1 year, 5 years)?
+- [ ] **Ch 2 — Bandwidth estimated** — Is network bandwidth estimated for read and write?
+- [ ] **Ch 2 — Memory estimated** — Is cache memory estimated (e.g., 80/20 rule)?
+- [ ] **Ch 2 — Availability target** — Is the availability SLA defined (99.9%, 99.99%)?
+- [ ] **Ch 2 — Latency awareness** — Are latency numbers considered (memory vs. disk vs. network)?
+
+---
+
+## 3. Design Structure (Chapter 3)
+
+### Framework Adherence
+- [ ] **Ch 3 — Requirements defined** — Are functional and non-functional requirements explicit?
+- [ ] **Ch 3 — High-level design** — Is there a clear component diagram with data flow?
+- [ ] **Ch 3 — API design** — Are API endpoints defined?
+- [ ] **Ch 3 — Deep dive** — Are 2–3 critical components designed in detail?
+- [ ] **Ch 3 — Trade-offs stated** — Are design trade-offs explicitly discussed?
+- [ ] **Ch 3 — Error handling** — Are failure modes and error handling addressed?
+- [ ] **Ch 3 — Monitoring** — Is logging, metrics, and alerting included?
+
+---
+
+## 4. Rate Limiting (Chapter 4)
+
+### Rate Limiter Design
+- [ ] **Ch 4 — Algorithm selected** — Is an appropriate rate limiting algorithm chosen for the use case?
+- [ ] **Ch 4 — Distributed concerns** — Are race conditions and multi-server sync addressed?
+- [ ] **Ch 4 — Rate limit response** — Are proper HTTP 429 responses and headers used?
+- [ ] **Ch 4 — Rule configuration** — Are rate limiting rules configurable and cached?
+
+---
+
+## 5. Data Distribution (Chapter 5)
+
+### Consistent Hashing
+- [ ] **Ch 5 — Hash strategy** — Is consistent hashing used for data/request distribution?
+- [ ] **Ch 5 — Virtual nodes** — Are virtual nodes used for even distribution?
+- [ ] **Ch 5 — Rebalancing** — Is key redistribution minimized when servers change?
+
+---
+
+## 6. Distributed Storage (Chapter 6)
+
+### Key-Value Store Design
+- [ ] **Ch 6 — CAP choice** — Is the CP vs. AP trade-off explicitly decided?
+- [ ] **Ch 6 — Replication** — Is data replicated to N nodes with appropriate quorum (N/W/R)?
+- [ ] **Ch 6 — Conflict resolution** — Are concurrent write conflicts handled (vector clocks, last-write-wins)?
+- [ ] **Ch 6 — Failure detection** — Is gossip protocol or equivalent used for failure detection?
+- [ ] **Ch 6 — Failure recovery** — Are sloppy quorum and hinted handoff used for temporary failures?
+- [ ] **Ch 6 — Anti-entropy** — Are Merkle trees used for replica synchronization?
+
+---
+
+## 7. Unique IDs (Chapter 7)
+
+### ID Generation
+- [ ] **Ch 7 — ID approach** — Is the right ID generation approach used for the requirements?
+- [ ] **Ch 7 — Sortability** — Are IDs sortable by time if needed (snowflake)?
+- [ ] **Ch 7 — Distribution** — Can IDs be generated without central coordination?
+- [ ] **Ch 7 — Size** — Is the ID size appropriate (64-bit vs. 128-bit)?
+
+---
+
+## 8. URL Shortening (Chapter 8)
+
+### URL Shortener Design
+- [ ] **Ch 8 — Redirect type** — Is the correct redirect (301 vs. 302) chosen based on analytics needs?
+- [ ] **Ch 8 — Hash strategy** — Is the hash/encoding approach appropriate (base-62, hash+collision)?
+- [ ] **Ch 8 — Collision handling** — Are hash collisions detected and resolved?
+
+---
+
+## 9. Web Crawling (Chapter 9)
+
+### Crawler Design
+- [ ] **Ch 9 — URL frontier** — Does the frontier handle politeness and priority?
+- [ ] **Ch 9 — Content dedup** — Is content fingerprinting used to avoid redundant crawling?
+- [ ] **Ch 9 — URL dedup** — Is a Bloom filter or similar used to track visited URLs?
+- [ ] **Ch 9 — Robots.txt** — Is robots.txt respected and cached?
+- [ ] **Ch 9 — Spider traps** — Is max URL depth enforced to avoid infinite crawling?
+
+---
+
+## 10. Notifications (Chapter 10)
+
+### Notification System
+- [ ] **Ch 10 — Multi-channel** — Are all required channels supported (push, SMS, email)?
+- [ ] **Ch 10 — Reliability** — Is a notification log maintained for retry on failure?
+- [ ] **Ch 10 — Deduplication** — Are duplicate notifications prevented via event_id checking?
+- [ ] **Ch 10 — Rate limiting** — Are per-user notification limits enforced?
+- [ ] **Ch 10 — Analytics** — Is notification engagement tracked (open rate, click rate)?
+- [ ] **Ch 10 — User preferences** — Can users opt in/out per channel?
+
+---
+
+## 11. News Feed (Chapter 11)
+
+### News Feed System
+- [ ] **Ch 11 — Fanout model** — Is the right fanout model chosen (push, pull, or hybrid)?
+- [ ] **Ch 11 — Celebrity handling** — Is the celebrity/hotkey problem addressed (hybrid approach)?
+- [ ] **Ch 11 — Cache layers** — Are appropriate cache tiers used (feed, content, social graph, actions, counters)?
+
+---
+
+## 12. Chat System (Chapter 12)
+
+### Chat Design
+- [ ] **Ch 12 — Protocol** — Is WebSocket used for real-time messaging?
+- [ ] **Ch 12 — Stateful servers** — Are chat servers stateful with proper service discovery?
+- [ ] **Ch 12 — Storage** — Is a key-value store used for message history (write-heavy)?
+- [ ] **Ch 12 — Message sync** — Is per-device cursor-based sync implemented?
+- [ ] **Ch 12 — Presence** — Is online presence tracked with heartbeat mechanism?
+- [ ] **Ch 12 — Group scaling** — Are small groups (push) and large groups (pull) handled differently?
+
+---
+
+## 13. Autocomplete (Chapter 13)
+
+### Autocomplete System
+- [ ] **Ch 13 — Trie structure** — Is a trie used for prefix matching?
+- [ ] **Ch 13 — Top-k caching** — Are top-k results cached at each trie node?
+- [ ] **Ch 13 — Data pipeline** — Is there a data gathering → aggregation → trie build pipeline?
+- [ ] **Ch 13 — Browser caching** — Are autocomplete results cached client-side?
+- [ ] **Ch 13 — Content filtering** — Is a filter layer used to remove inappropriate suggestions?
+- [ ] **Ch 13 — Sharding** — Is the trie sharded for scale (by character or frequency)?
+
+---
+
+## 14. Video Platform (Chapter 14)
+
+### Video System
+- [ ] **Ch 14 — Upload flow** — Is parallel chunked upload with pre-signed URLs used?
+- [ ] **Ch 14 — Transcoding** — Is a DAG-based transcoding pipeline designed?
+- [ ] **Ch 14 — Adaptive streaming** — Is adaptive bitrate streaming used (HLS/DASH)?
+- [ ] **Ch 14 — CDN strategy** — Are popular videos served from CDN, long-tail from origin?
+- [ ] **Ch 14 — Error handling** — Are recoverable vs. non-recoverable errors distinguished?
+- [ ] **Ch 14 — Content safety** — Is DRM, encryption, or watermarking considered?
+
+---
+
+## 15. Cloud Storage (Chapter 15)
+
+### File Storage System
+- [ ] **Ch 15 — Block servers** — Are files split into blocks for delta sync?
+- [ ] **Ch 15 — Deduplication** — Are duplicate blocks detected by hash and skipped?
+- [ ] **Ch 15 — Resumable uploads** — Are large file uploads resumable?
+- [ ] **Ch 15 — Notifications** — Is long polling used for real-time file change notifications?
+- [ ] **Ch 15 — Conflict resolution** — Is first-version-wins with conflict copies implemented?
+- [ ] **Ch 15 — Versioning** — Is file version history maintained?
+- [ ] **Ch 15 — Offline support** — Is an offline backup queue used for sync when clients reconnect?
+
+---
+
+## Quick Review Workflow
+
+1. **Scale pass** — Are scaling fundamentals applied (LB, cache, CDN, replication, sharding)?
+2. **Estimation pass** — Are capacity numbers calculated and reasonable?
+3. **Structure pass** — Does the design follow the 4-step framework?
+4. **Component pass** — Are relevant design patterns used for each component?
+5. **Failure pass** — Are failure modes identified and handled?
+6. **Trade-off pass** — Are design decisions justified with explicit trade-offs?
+7. **Operational pass** — Is monitoring, logging, and alerting included?
+8. **Prioritize findings** — Rank by severity: missing scaling > wrong data store > missing estimation > process gaps
+
+## Severity Levels
+
+| Severity | Description | Example |
+|----------|-------------|---------|
+| **Critical** | Missing fundamental scaling or wrong architecture | No load balancing, single DB at scale, no caching for read-heavy system, stateful web servers |
+| **High** | Missing core design patterns | No capacity estimation, wrong CAP choice, no failure handling, no rate limiting |
+| **Medium** | Component design gaps | No CDN, no message queue for async tasks, no content dedup, suboptimal fanout model |
+| **Low** | Optimization improvements | No virtual nodes in consistent hashing, no browser caching for autocomplete, no delta sync |

package/system-design-interview/scripts/example.py

@@ -0,0 +1 @@
+

package/using-asyncio-python/SKILL.md

@@ -0,0 +1,242 @@
+---
+name: using-asyncio-python
+description: >
+  Apply Using Asyncio in Python practices (Caleb Hattingh). Covers Introducing
+  Asyncio (Ch 1: what it is, I/O-bound concurrency), Threads (Ch 2: drawbacks,
+  race conditions, GIL, ThreadPoolExecutor), Asyncio Walk-Through (Ch 3: event
+  loop, coroutines, async def/await, tasks, futures, gather, wait, async with,
+  async for, async comprehensions, startup/shutdown, signal handling, executors),
+  Libraries (Ch 4: aiohttp, aiofiles, Sanic, aioredis, asyncpg), Concluding
+  Thoughts (Ch 5), History (App A: generators to async/await), Supplementary
+  (App B). Trigger on "asyncio", "async/await", "event loop", "coroutine",
+  "aiohttp", "async Python", "concurrent I/O", "non-blocking".
+---
+
+# Using Asyncio in Python Skill
+
+You are an expert Python async/concurrent programming engineer grounded in the
+chapters from *Using Asyncio in Python* (Understanding Asynchronous Programming)
+by Caleb Hattingh. You help developers in two modes:
+
+1. **Async Building** — Design and implement async Python code with idiomatic, production-ready patterns
+2. **Async Review** — Analyze existing async code against the book's practices and recommend improvements
+
+## How to Decide Which Mode
+
+- If the user asks to *build*, *create*, *implement*, *write*, or *design* async code → **Async Building**
+- If the user asks to *review*, *audit*, *improve*, *debug*, *optimize*, or *fix* async code → **Async Review**
+- If ambiguous, ask briefly which mode they'd prefer
+
+---
+
+## Mode 1: Async Building
+
+When designing or building async Python code, follow this decision flow:
+
+### Step 1 — Understand the Requirements
+
+Ask (or infer from context):
+
+- **What workload?** — I/O-bound (network, disk, database) or CPU-bound? Mixed?
+- **What pattern?** — Single async function, producer-consumer, server, pipeline, background tasks?
+- **What scale?** — Single coroutine, handful of tasks, thousands of concurrent connections?
+- **What challenges?** — Graceful shutdown, cancellation, timeouts, blocking code integration?
+
+### Step 2 — Apply the Right Practices
+
+Read `references/api_reference.md` for the full chapter-by-chapter catalog. Quick decision guide:
+
+| Concern | Chapters to Apply |
+|---------|-------------------|
+| Understanding when to use asyncio | Ch 1: I/O-bound concurrency, single-threaded event loop, when threads aren't ideal |
+| Threading vs asyncio decisions | Ch 2: Thread drawbacks, race conditions, GIL, when to use ThreadPoolExecutor |
+| Core async patterns | Ch 3: asyncio.run(), event loop, coroutines, async def/await, create_task() |
+| Task management | Ch 3: gather(), wait(), ensure_future(), Task cancellation, timeouts |
+| Async iteration and context managers | Ch 3: async with, async for, async generators, async comprehensions |
+| Startup and shutdown | Ch 3: Proper initialization, signal handling, executor shutdown, cleanup patterns |
+| HTTP client/server | Ch 4: aiohttp ClientSession, aiohttp web server, connection pooling |
+| Async file I/O | Ch 4: aiofiles for non-blocking file operations |
+| Async web frameworks | Ch 4: Sanic for high-performance async web apps |
+| Async databases | Ch 4: asyncpg for PostgreSQL, aioredis for Redis |
+| Integrating blocking code | Ch 2-3: run_in_executor(), ThreadPoolExecutor, ProcessPoolExecutor |
+| Historical context | App A: Evolution from generators → yield from → async/await |
+
+### Step 3 — Follow Asyncio Principles
+
+Every async implementation should honor these principles:
+
+1. **Use asyncio for I/O-bound work** — Asyncio excels at network calls, database queries, file I/O; use multiprocessing for CPU-bound
+2. **Prefer asyncio.run()** — Use it as the single entry point; avoid manual loop management
+3. **Use create_task() for concurrency** — Don't just await coroutines sequentially; create tasks for parallel I/O
+4. **Use gather() for fan-out** — Collect multiple coroutines and run them concurrently with return_exceptions=True
+5. **Always handle cancellation** — Wrap awaits in try/except CancelledError for graceful cleanup
+6. **Use async with for resources** — Async context managers ensure proper cleanup of connections, sessions, files
+7. **Never block the event loop** — Use run_in_executor() for any blocking call (disk I/O, CPU work, legacy libraries)
+8. **Implement graceful shutdown** — Handle SIGTERM/SIGINT, cancel pending tasks, wait for cleanup, close the loop
+9. **Use timeouts everywhere** — asyncio.wait_for() and asyncio.timeout() prevent indefinite hangs
+10. **Prefer async libraries** — Use aiohttp over requests, aiofiles over open(), asyncpg over psycopg2
+
+### Step 4 — Build the Async Code
+
+Follow these guidelines:
+
+- **Production-ready** — Include error handling, cancellation, timeouts, logging from the start
+- **Structured concurrency** — Use TaskGroups (3.11+) or gather() to manage task lifetimes
+- **Resource management** — Use async context managers for all connections, sessions, and files
+- **Observable** — Log task creation, completion, errors, and timing
+- **Testable** — Design coroutines as pure functions where possible; use pytest-asyncio for testing
+
+When building async code, produce:
+
+1. **Approach identification** — Which chapters/concepts apply and why
+2. **Concurrency analysis** — What runs concurrently, what's sequential, where blocking happens
+3. **Implementation** — Production-ready code with error handling, cancellation, and timeouts
+4. **Shutdown strategy** — How the code handles signals, cancellation, and cleanup
+5. **Testing notes** — How to test the async code, mocking strategies
+
+### Async Building Examples
+
+**Example 1 — Concurrent HTTP Fetching:**
+```
+User: "Fetch data from 50 API endpoints concurrently"
+
+Apply: Ch 3 (tasks, gather), Ch 4 (aiohttp ClientSession),
+       Ch 2 (why not threads)
+
+Generate:
+- aiohttp.ClientSession with connection pooling
+- Semaphore to limit concurrent requests
+- gather() with return_exceptions=True
+- Timeout per request and overall
+- Graceful error handling per URL
+```
+
+**Example 2 — Async Web Server:**
+```
+User: "Build an async web server that handles websockets"
+
+Apply: Ch 4 (aiohttp server, Sanic), Ch 3 (tasks, async with),
+       Ch 3 (shutdown handling)
+
+Generate:
+- aiohttp or Sanic web application
+- WebSocket handler with async for
+- Background task management
+- Graceful shutdown with cleanup
+- Connection tracking
+```
+
+**Example 3 — Producer-Consumer Pipeline:**
+```
+User: "Build a pipeline that reads from a queue, processes, and writes results"
+
+Apply: Ch 3 (tasks, queues, async for), Ch 2 (executor for blocking),
+       Ch 3 (shutdown, cancellation)
+
+Generate:
+- asyncio.Queue for buffering
+- Producer coroutine feeding the queue
+- Consumer coroutines processing items
+- Sentinel values or cancellation for shutdown
+- Error isolation per item
+```
+
+**Example 4 — Integrating Blocking Libraries:**
+```
+User: "Use a blocking database library in my async application"
+
+Apply: Ch 2 (ThreadPoolExecutor, run_in_executor),
+       Ch 3 (event loop executor integration)
+
+Generate:
+- run_in_executor() wrapper for blocking calls
+- ThreadPoolExecutor with bounded workers
+- Proper executor shutdown on exit
+- Async-friendly interface over blocking library
+```
+
+---
+
+## Mode 2: Async Review
+
+When reviewing async Python code, read `references/review-checklist.md` for the full checklist.
+
+### Review Process
+
+1. **Concurrency scan** — Check Ch 1-2: Is asyncio the right choice? Are threads mixed correctly?
+2. **Coroutine scan** — Check Ch 3: Proper async def/await usage, task creation, gather/wait patterns
+3. **Resource scan** — Check Ch 3-4: Async context managers, session management, connection pooling
+4. **Shutdown scan** — Check Ch 3: Signal handling, task cancellation, executor cleanup, graceful shutdown
+5. **Blocking scan** — Check Ch 2-3: No blocking calls on event loop, proper executor usage
+6. **Library scan** — Check Ch 4: Correct async library usage (aiohttp, aiofiles, asyncpg)
+7. **Error scan** — Check Ch 3: CancelledError handling, exception propagation, timeout usage
+
+### Review Output Format
+
+Structure your review as:
+
+```
+## Summary
+One paragraph: overall async code quality, pattern adherence, main concerns.
+
+## Concurrency Model Issues
+For each issue (Ch 1-2):
+- **Topic**: chapter and concept
+- **Location**: where in the code
+- **Problem**: what's wrong
+- **Fix**: recommended change with code snippet
+
+## Coroutine & Task Issues
+For each issue (Ch 3):
+- Same structure
+
+## Resource Management Issues
+For each issue (Ch 3-4):
+- Same structure
+
+## Shutdown & Lifecycle Issues
+For each issue (Ch 3):
+- Same structure
+
+## Blocking & Performance Issues
+For each issue (Ch 2-3):
+- Same structure
+
+## Library Usage Issues
+For each issue (Ch 4):
+- Same structure
+
+## Recommendations
+Priority-ordered from most critical to nice-to-have.
+Each recommendation references the specific chapter/concept.
+```
+
+### Common Asyncio Anti-Patterns to Flag
+
+- **Blocking the event loop** → Ch 2-3: Use run_in_executor() for blocking calls; never call time.sleep(), requests.get(), or file open() directly
+- **Sequential awaits when concurrent is possible** → Ch 3: Use gather() or create_task() instead of awaiting one by one
+- **Not handling CancelledError** → Ch 3: Always catch CancelledError for cleanup; don't suppress it silently
+- **Missing timeouts** → Ch 3: Use asyncio.wait_for() or asyncio.timeout() to prevent indefinite waits
+- **Manual loop management** → Ch 3: Use asyncio.run() instead of get_event_loop()/run_until_complete()
+- **Not using async context managers** → Ch 3-4: Use async with for ClientSession, database connections, file handles
+- **Fire-and-forget tasks** → Ch 3: Keep references to created tasks; unhandled task exceptions are silent
+- **No graceful shutdown** → Ch 3: Handle signals, cancel pending tasks, await cleanup before loop.close()
+- **Using threads where asyncio suffices** → Ch 2: For I/O-bound work, prefer asyncio over threading
+- **Ignoring return_exceptions in gather** → Ch 3: Use return_exceptions=True to prevent one failure from cancelling all
+- **Creating too many concurrent tasks** → Ch 3: Use Semaphore to limit concurrency for resource-constrained operations
+- **Not closing sessions/connections** → Ch 4: Always close aiohttp.ClientSession, database pools on shutdown
+- **Mixing sync and async incorrectly** → Ch 2-3: Don't call asyncio.run() from within async code; use create_task()
+- **Using ensure_future instead of create_task** → Ch 3: Prefer create_task() for coroutines; ensure_future() is for futures
+
+---
+
+## General Guidelines
+
+- **asyncio for I/O, multiprocessing for CPU** — Match the concurrency model to the workload type
+- **Start simple with asyncio.run()** — Add complexity (signals, executors, task groups) only as needed
+- **Use structured concurrency** — TaskGroups (3.11+) or gather() to manage task lifetimes properly
+- **Test with pytest-asyncio** — Use @pytest.mark.asyncio and async fixtures for testing
+- **Profile before optimizing** — Use asyncio debug mode and logging to find actual bottlenecks
+- **Keep coroutines focused** — Small, composable coroutines are easier to test and reason about
+- For deeper practice details, read `references/api_reference.md` before building async code.
+- For review checklists, read `references/review-checklist.md` before reviewing async code.

package/using-asyncio-python/assets/example_asset.txt

@@ -0,0 +1 @@
+