forgecommandline 0.8.0__py3-none-any.whl

forgecommandline-0.8.0.data/data/skills/library/system-design/references/database-scaling.md

@@ -0,0 +1,295 @@
+# Database Design and Scaling
+
+How to choose, design, and scale databases for distributed systems. The database is usually the first bottleneck -- understanding scaling strategies lets you plan growth deliberately.
+
+## SQL vs NoSQL Decision Framework
+
+### When to Choose SQL (Relational)
+
+Choose relational databases (PostgreSQL, MySQL, Oracle) when:
+
+- **ACID transactions** are required (financial data, inventory, bookings)
+- **Complex joins** across multiple tables are common
+- **Schema is well-defined** and unlikely to change dramatically
+- **Data integrity** via foreign keys and constraints is critical
+- **Reporting and ad-hoc queries** are needed (SQL is incredibly expressive)
+
+### When to Choose NoSQL
+
+Choose NoSQL when the access pattern matches a specific NoSQL model:
+
+| NoSQL Type | Best For | Examples | When to Use |
+|-----------|----------|---------|-------------|
+| **Key-Value** | Simple lookups by key | Redis, DynamoDB, Riak | Session storage, caching, user preferences |
+| **Document** | Flexible schema, nested data | MongoDB, CouchDB, Firestore | Product catalogs, content management, user profiles |
+| **Wide-Column** | High write throughput, time-series | Cassandra, HBase, ScyllaDB | IoT data, event logs, messaging history |
+| **Graph** | Highly connected data | Neo4j, Amazon Neptune, JanusGraph | Social networks, recommendation engines, fraud detection |
+
+### The Decision Checklist
+
+| Question | If Yes: SQL | If Yes: NoSQL |
+|----------|-------------|---------------|
+| Need ACID transactions? | SQL | |
+| Need complex joins? | SQL | |
+| Schema is stable? | SQL | |
+| Schema evolves rapidly? | | Document DB |
+| Need horizontal write scaling? | | Wide-column or Key-value |
+| Access pattern is simple key lookup? | | Key-value |
+| Data is highly connected (graph)? | | Graph DB |
+| Need full-text search? | | Consider Elasticsearch alongside primary DB |
+
+**Important:** Many real systems use multiple databases (polyglot persistence). A social app might use PostgreSQL for user accounts, Redis for sessions, Cassandra for the activity feed, and Elasticsearch for search.
+
+---
+
+## Vertical Scaling (Scale Up)
+
+### What It Means
+
+Make a single server more powerful: more CPU, more RAM, faster disks, more network bandwidth.
+
+### Advantages
+
+- Simple -- no code changes required
+- No distributed systems complexity
+- Strong consistency is easy (single node)
+- No need for sharding logic
+
+### Limitations
+
+- Hard ceiling -- you can't buy an infinitely powerful machine
+- Single point of failure -- one server, one failure domain
+- Cost increases non-linearly -- a 2x more powerful server costs more than 2x the price
+- Downtime during upgrade -- usually requires restart
+
+### When to Use
+
+- Early-stage products where simplicity matters more than scale
+- When vertical limits haven't been reached (modern servers can handle significant load)
+- When strong consistency requirements make horizontal scaling very difficult
+
+**Rule of thumb:** Start vertical. A single PostgreSQL server with 64 cores and 256 GB RAM handles more than most applications need. Move to horizontal when you actually hit the limits.
+
+---
+
+## Replication
+
+### Leader-Follower (Master-Slave)
+
+The most common replication pattern:
+
+```
+Writes -> Leader (Primary)
+Reads  -> Leader or Followers (Replicas)
+```
+
+**How it works:**
+1. All writes go to the leader
+2. Leader writes to its write-ahead log (WAL)
+3. Followers replicate the WAL and apply changes
+4. Reads can go to any node (leader or follower)
+
+**Advantages:**
+- Scales reads horizontally (add more followers)
+- Provides redundancy (promote a follower if leader fails)
+- Simple mental model
+
+**Challenges:**
+
+| Challenge | Description | Mitigation |
+|-----------|------------|------------|
+| **Replication lag** | Followers may be seconds behind leader | Read-your-writes: route user's reads to leader after they write |
+| **Failover complexity** | Promoting a follower requires coordination | Automated failover with consensus (e.g., Patroni for PostgreSQL) |
+| **Write bottleneck** | All writes still go to one node | See sharding section below |
+
+### Multi-Leader (Master-Master)
+
+Multiple nodes accept writes:
+
+```
+Region A writes -> Leader A -> replicates to -> Leader B
+Region B writes -> Leader B -> replicates to -> Leader A
+```
+
+**When to use:**
+- Multi-region deployments where each region needs local writes
+- Offline-capable applications (each device is a "leader")
+
+**The hard problem: write conflicts.** When two leaders modify the same row:
+
+| Strategy | How It Works | Tradeoff |
+|----------|-------------|----------|
+| **Last-write-wins (LWW)** | Timestamp-based, latest wins | Simple but can lose data |
+| **Application-level merge** | App defines custom merge logic | Correct but complex |
+| **CRDTs** | Conflict-free data structures | Automatic merge but limited data types |
+| **Conflict avoidance** | Route same data to same leader | Simple but limits flexibility |
+
+### Leaderless Replication
+
+All nodes are equal; reads and writes go to multiple nodes:
+
+- Write to W nodes, read from R nodes
+- If W + R > N (total nodes), reads are guaranteed to see latest write
+- Used by Cassandra and DynamoDB
+
+**Advantages:** No single point of failure, high availability
+**Disadvantages:** Complex consistency model, conflict resolution needed
+
+---
+
+## Sharding (Horizontal Partitioning)
+
+### What It Means
+
+Split data across multiple database servers (shards), each holding a subset of the data. Each shard is a fully independent database.
+
+### When to Shard
+
+Shard only when you have exhausted simpler options:
+1. Vertical scaling (bigger server)
+2. Read replicas (for read-heavy workloads)
+3. Caching (for hot data)
+4. Query optimization (indexes, query rewriting)
+5. **Then** shard (for write-heavy workloads or data too large for one server)
+
+### Sharding Strategies
+
+#### Hash-Based Sharding
+
+```
+shard_number = hash(shard_key) % number_of_shards
+```
+
+| Pros | Cons |
+|------|------|
+| Even data distribution | Adding/removing shards requires data redistribution |
+| Simple implementation | Range queries across shards are difficult |
+| No hotspots (if hash is good) | Re-sharding is expensive |
+
+**Mitigation for redistribution:** Use consistent hashing instead of simple modulo.
+
+#### Range-Based Sharding
+
+```
+Shard 1: user_id 1 - 1,000,000
+Shard 2: user_id 1,000,001 - 2,000,000
+```
+
+| Pros | Cons |
+|------|------|
+| Efficient range queries | Uneven distribution (hotspots) |
+| Natural data locality | New data may overwhelm one shard |
+| Simple to understand | Requires manual range management |
+
+**Best for:** Time-series data (shard by month), geographic data (shard by region).
+
+#### Directory-Based Sharding
+
+A lookup service maps each key to its shard:
+
+```
+Lookup table:
+  user_123 -> shard_3
+  user_456 -> shard_1
+```
+
+| Pros | Cons |
+|------|------|
+| Flexible, any key-to-shard mapping | Lookup service is a single point of failure |
+| Easy rebalancing | Extra network hop for every query |
+| Can handle hotspots by moving individual keys | Directory itself needs to be highly available |
+
+### Choosing a Shard Key
+
+The shard key determines everything about your sharding strategy. Choose carefully:
+
+| Criteria | Why It Matters |
+|----------|---------------|
+| **High cardinality** | More distinct values = more even distribution |
+| **Even distribution** | Avoid hotspots where one shard gets disproportionate traffic |
+| **Query patterns** | Most queries should hit a single shard (avoid scatter-gather) |
+| **Growth stability** | Key shouldn't cause imbalance as data grows |
+
+**Common shard keys:**
+- `user_id` -- good for user-centric applications (each user's data on one shard)
+- `tenant_id` -- good for multi-tenant SaaS (each tenant isolated)
+- `geo_region` -- good for location-based services
+- `created_at` -- good for time-series (but recent shard gets all writes)
+
+### The Celebrity/Hotspot Problem
+
+Even with good hash-based sharding, a single popular entity can overwhelm one shard.
+
+**Example:** A celebrity with 100 million followers. Any action on their account generates massive read/write traffic on the shard that holds their data.
+
+**Solutions:**
+1. **Dedicated shard:** Move the hot entity to its own shard
+2. **Secondary partition:** Split the hot entity's data further (e.g., partition followers into sub-groups)
+3. **Caching:** Put a cache in front of the hot shard
+4. **Application-level routing:** Detect hot keys and handle them differently
+
+### Cross-Shard Operations
+
+The biggest challenge with sharding: operations that span multiple shards.
+
+| Operation | Challenge | Solution |
+|-----------|----------|----------|
+| **Joins** | Can't join across shards efficiently | Denormalize, or join at application level |
+| **Aggregations** | Must scatter query to all shards and gather results | Pre-compute aggregates, use a separate analytics DB |
+| **Transactions** | Distributed transactions are slow and complex | Design for single-shard transactions, use sagas for cross-shard |
+| **Unique constraints** | Can't enforce uniqueness across shards | Use a global sequence service, or accept probabilistic uniqueness |
+
+---
+
+## Denormalization
+
+### What It Means
+
+Duplicate data across tables or documents to avoid expensive joins at read time.
+
+### When to Denormalize
+
+| Indicator | Example |
+|-----------|---------|
+| Read-heavy workload (100:1 read:write ratio) | News feed, product catalog |
+| Joins are too slow | Dashboard aggregating data from 5+ tables |
+| Data changes infrequently | User profile data embedded in every post |
+| Scale requires sharding (joins across shards are impractical) | Any sharded system |
+
+### Denormalization Patterns
+
+| Pattern | How It Works | Example |
+|---------|-------------|---------|
+| **Embed related data** | Store related data in same row/document | Store author name in each post (not just author_id) |
+| **Materialized view** | Pre-compute and store query results | Nightly job that joins orders + products into a summary table |
+| **Counter cache** | Store computed counts alongside parent | `posts_count` column on user table, incremented on new post |
+| **Summary table** | Aggregate data into a summary | Daily revenue summary table updated by background job |
+
+### Tradeoffs
+
+| Benefit | Cost |
+|---------|------|
+| Faster reads (no joins) | Slower writes (update multiple places) |
+| Simpler queries | Data inconsistency risk |
+| Works with sharding | More storage used |
+| Reduced database load | Application must maintain consistency |
+
+---
+
+## Database Selection Guide
+
+Quick reference for choosing the right database:
+
+| Requirement | Recommended | Examples |
+|-------------|------------|---------|
+| General-purpose web app | Relational | PostgreSQL, MySQL |
+| Simple key-value lookups at massive scale | Key-value | Redis, DynamoDB |
+| Flexible schema, document-oriented | Document | MongoDB, Firestore |
+| High write throughput, time-series | Wide-column | Cassandra, ScyllaDB |
+| Graph relationships | Graph | Neo4j, Neptune |
+| Full-text search | Search engine | Elasticsearch, OpenSearch |
+| Analytics and OLAP | Columnar | ClickHouse, BigQuery, Redshift |
+| Caching layer | In-memory | Redis, Memcached |
+| Message/event storage | Distributed log | Apache Kafka |
+
+**Remember:** Most systems use 2-3 databases for different access patterns. This is polyglot persistence and it is normal at scale.

forgecommandline-0.8.0.data/data/skills/library/system-design/references/estimation-numbers.md

@@ -0,0 +1,250 @@
+# Back-of-the-Envelope Estimation
+
+Essential numbers, formulas, and worked examples for estimating system capacity. The goal is order of magnitude, not precision -- being within 10x is success; being within 2x is excellent.
+
+## Powers of Two
+
+Every engineer should know these instantly:
+
+| Power | Exact Value | Approximate |
+|-------|-------------|-------------|
+| 2^10 | 1,024 | 1 thousand (1 KB) |
+| 2^20 | 1,048,576 | 1 million (1 MB) |
+| 2^30 | 1,073,741,824 | 1 billion (1 GB) |
+| 2^40 | 1,099,511,627,776 | 1 trillion (1 TB) |
+| 2^50 | ~1.13 x 10^15 | 1 quadrillion (1 PB) |
+
+**Quick conversions:**
+- 1 KB = 10^3 bytes
+- 1 MB = 10^6 bytes
+- 1 GB = 10^9 bytes
+- 1 TB = 10^12 bytes
+- 1 PB = 10^15 bytes
+
+**Useful round numbers:**
+- 1 million seconds = ~11.5 days
+- 1 billion seconds = ~31.7 years
+- 86,400 seconds in a day (~10^5)
+- 2.5 million seconds in a month (~2.5 x 10^6)
+- 31.5 million seconds in a year (~3 x 10^7)
+
+---
+
+## Latency Numbers Every Programmer Should Know
+
+These numbers, originally compiled by Jeff Dean, define the performance landscape:
+
+| Operation | Latency | Notes |
+|-----------|---------|-------|
+| L1 cache reference | 0.5 ns | |
+| Branch mispredict | 5 ns | |
+| L2 cache reference | 7 ns | |
+| Mutex lock/unlock | 25 ns | |
+| Main memory reference | 100 ns | |
+| Compress 1 KB with Snappy | 3 us | |
+| Send 1 KB over 1 Gbps network | 10 us | |
+| Read 4 KB randomly from SSD | 150 us | |
+| Read 1 MB sequentially from memory | 250 us | |
+| Round trip within same datacenter | 500 us | |
+| Read 1 MB sequentially from SSD | 1 ms | |
+| HDD disk seek | 10 ms | |
+| Read 1 MB sequentially from HDD | 20 ms | |
+| Send packet CA -> Netherlands -> CA | 150 ms | |
+
+### Key Takeaways
+
+1. **Memory is fast, disk is slow.** Memory access (~100 ns) is 100,000x faster than disk seek (~10 ms).
+2. **SSD is much faster than HDD.** SSD random read (~150 us) is ~67x faster than HDD seek (~10 ms).
+3. **Sequential reads are fast everywhere.** Sequential 1 MB from memory (250 us) vs SSD (1 ms) vs HDD (20 ms).
+4. **Network within a datacenter is fast.** Round trip ~0.5 ms. Cross-continent is ~150 ms (300x slower).
+5. **Compression is cheap.** Compressing 1 KB takes ~3 us -- almost always worth doing for network transfer.
+
+### Design Implications
+
+| Latency Requirement | Design Strategy |
+|--------------------|-----------------|
+| < 1 ms | Must be in memory (cache, in-process) |
+| < 10 ms | Can hit SSD or local cache |
+| < 100 ms | Can hit database (with indexes) or remote cache |
+| < 500 ms | Can make 1-2 network calls within datacenter |
+| < 1 second | Can make multiple datacenter calls or one cross-region call |
+| > 1 second | Must be async, show loading state to user |
+
+---
+
+## Availability Nines
+
+| Availability | Downtime/Year | Downtime/Month | Downtime/Week |
+|-------------|---------------|----------------|---------------|
+| 99% (two nines) | 3.65 days | 7.31 hours | 1.68 hours |
+| 99.9% (three nines) | 8.77 hours | 43.83 minutes | 10.08 minutes |
+| 99.95% | 4.38 hours | 21.92 minutes | 5.04 minutes |
+| 99.99% (four nines) | 52.60 minutes | 4.38 minutes | 1.01 minutes |
+| 99.999% (five nines) | 5.26 minutes | 26.30 seconds | 6.05 seconds |
+
+### SLA Composition
+
+When services depend on each other, availabilities multiply:
+
+- Service A: 99.9%, Service B: 99.9%
+- Combined (A depends on B): 99.9% x 99.9% = 99.8%
+
+This means:
+- A chain of 3 services at 99.9% each = 99.7% combined
+- A chain of 5 services at 99.9% each = 99.5% combined
+
+**Implication:** To achieve 99.99% end-to-end, each component must be significantly better than 99.99% individually, or you need redundancy (parallel paths) to improve overall availability.
+
+**Parallel redundancy:**
+- Two instances of a service, each at 99%: 1 - (0.01 x 0.01) = 99.99%
+- Both must fail simultaneously for the system to be unavailable
+
+---
+
+## QPS Estimation
+
+### Formula
+
+```
+Average QPS = DAU x average_actions_per_user / 86,400
+Peak QPS    = Average QPS x peak_factor (typically 2-5x)
+```
+
+### Worked Examples
+
+**Twitter-like service:**
+- DAU: 300 million
+- Tweets per user per day: 2 (average)
+- Tweet reads per user per day: 100
+- Write QPS: 300M x 2 / 86,400 = ~7,000 QPS
+- Peak write QPS: 7,000 x 3 = ~21,000 QPS
+- Read QPS: 300M x 100 / 86,400 = ~350,000 QPS
+- Peak read QPS: 350,000 x 3 = ~1,050,000 QPS
+- Read:write ratio = 50:1
+
+**URL shortener:**
+- DAU: 100 million
+- URLs created per user per day: 0.1 (most users only read)
+- URL redirects per user per day: 5
+- Write QPS: 100M x 0.1 / 86,400 = ~116 QPS
+- Read QPS: 100M x 5 / 86,400 = ~5,800 QPS
+- Peak read QPS: 5,800 x 5 = ~29,000 QPS
+
+**Chat application:**
+- DAU: 50 million
+- Messages per user per day: 40
+- Message QPS: 50M x 40 / 86,400 = ~23,000 QPS
+- Peak QPS: 23,000 x 5 = ~115,000 QPS
+
+---
+
+## Storage Estimation
+
+### Formula
+
+```
+Daily storage  = records_per_day x average_record_size
+Yearly storage = daily_storage x 365
+Total storage  = yearly_storage x retention_years
+```
+
+### Worked Examples
+
+**Twitter-like service (5-year retention):**
+- Tweets per day: 300M users x 2 tweets = 600M tweets
+- Average tweet size: tweet_id (8 bytes) + user_id (8 bytes) + text (280 chars = 280 bytes) + timestamp (8 bytes) + metadata (100 bytes) = ~400 bytes
+- Daily storage: 600M x 400 bytes = 240 GB/day
+- Yearly storage: 240 GB x 365 = ~88 TB/year
+- 5-year storage: 88 TB x 5 = ~440 TB
+- With media (images, videos): multiply by 10-50x
+
+**URL shortener (10-year retention):**
+- New URLs per day: 100M x 0.1 = 10M URLs
+- Average record size: short_code (7 bytes) + original_url (200 bytes) + metadata (50 bytes) = ~257 bytes, round to 300 bytes
+- Daily storage: 10M x 300 bytes = 3 GB/day
+- Yearly storage: 3 GB x 365 = ~1.1 TB/year
+- 10-year storage: 1.1 TB x 10 = ~11 TB
+
+**Chat application (indefinite retention):**
+- Messages per day: 50M users x 40 messages = 2 billion messages
+- Average message size: 200 bytes (text) + 100 bytes (metadata) = 300 bytes
+- Daily storage: 2B x 300 bytes = 600 GB/day
+- Yearly storage: 600 GB x 365 = ~219 TB/year
+
+---
+
+## Bandwidth Estimation
+
+### Formula
+
+```
+Incoming bandwidth = write_QPS x average_request_size
+Outgoing bandwidth = read_QPS x average_response_size
+```
+
+### Worked Examples
+
+**URL shortener:**
+- Write: 116 QPS x 300 bytes = ~35 KB/s (negligible)
+- Read: 5,800 QPS x 500 bytes (redirect response) = ~2.9 MB/s
+- Peak read: 29,000 QPS x 500 bytes = ~14.5 MB/s
+
+**Image hosting service:**
+- Upload: 100 QPS x 2 MB = 200 MB/s incoming
+- Download: 10,000 QPS x 500 KB (average image) = 5 GB/s outgoing
+- This immediately tells you: CDN is mandatory
+
+---
+
+## Server Count Estimation
+
+### Rule of Thumb
+
+- A single web server handles ~1,000-10,000 QPS (depends on workload complexity)
+- A single database server handles ~5,000-10,000 QPS for simple queries
+- A single cache server (Redis) handles ~100,000 QPS for simple GET/SET
+- A single message queue node handles ~10,000-100,000 messages/second
+
+### Formula
+
+```
+Server count = Peak QPS / QPS per server
+Add redundancy factor (typically 2-3x for fault tolerance)
+```
+
+### Worked Example
+
+**URL shortener:**
+- Peak read QPS: 29,000
+- QPS per web server: ~5,000 (lightweight redirect)
+- Minimum web servers: 29,000 / 5,000 = ~6
+- With redundancy: 6 x 2 = 12 web servers
+- Cache QPS: 29,000 (cache all reads)
+- Redis nodes: 29,000 / 100,000 = 1 (with a replica for failover)
+
+---
+
+## Estimation Quick Reference
+
+| What to Estimate | Formula | Typical Inputs |
+|-----------------|---------|----------------|
+| Average QPS | DAU x actions / 86,400 | DAU, actions per user |
+| Peak QPS | Avg QPS x 2-5 | Traffic pattern |
+| Daily storage | Records/day x record size | Record count, schema |
+| Yearly storage | Daily x 365 | Daily storage |
+| Bandwidth | QPS x response size | QPS, payload size |
+| Server count | Peak QPS / capacity per server x 2 | Peak QPS |
+| Cache size | Working set x record size | Hot records, record size |
+
+---
+
+## Common Estimation Mistakes
+
+| Mistake | Why It Matters | Fix |
+|---------|---------------|-----|
+| Forgetting peak vs average | Average is fine, but peak causes outages | Always multiply by 2-5x for peak |
+| Ignoring media storage | Text is tiny; images and video dominate | Account for media separately |
+| Precise calculation | 3,472 QPS is false precision | Round to 3,500 or "about 3K-4K" |
+| Forgetting replication | Storage and bandwidth multiply with replicas | Multiply by replication factor (typically 3x) |
+| Ignoring metadata | Indexes, logs, and overhead add up | Add 30-50% overhead to raw data size |
+| Using current numbers only | Systems grow | Apply expected growth rate (typically 2-3x/year) |