codymaster 4.1.4 → 4.4.1

package/skills/release-it/SKILL.md

@@ -0,0 +1,235 @@
+---
+name: release-it
+description: 'Build production-ready systems with stability patterns: circuit breakers, bulkheads, timeouts, and retry logic. Use when the user mentions "production outage", "circuit breaker", "timeout strategy", "deployment pipeline", or "chaos engineering". Covers capacity planning, health checks, and anti-fragility patterns. For data systems, see ddia-systems. For system architecture, see system-design.'
+license: MIT
+metadata:
+  author: wondelai
+  version: "1.0.1"
+---
+
+# Release It! Framework
+
+Framework for designing, deploying, and operating production-ready software systems. Based on a fundamental truth: the software that passes QA is not the software that survives production. Production is a hostile environment -- and your system must be built to expect and handle failure at every level.
+
+## Core Principle
+
+**Every system will eventually be pushed beyond its design limits.** The question is not whether failures will happen, but whether your system degrades gracefully or collapses catastrophically. Production-ready software is not just correct -- it is resilient, observable, and designed to operate through partial failures without human intervention.
+
+## Scoring
+
+**Goal: 10/10.** When reviewing or creating production systems, rate them 0-10 based on adherence to the principles below. A 10/10 means full alignment with all guidelines; lower scores indicate gaps to address. Always provide the current score and specific improvements needed to reach 10/10.
+
+## The Release It! Framework
+
+Six areas that determine whether software survives contact with production:
+
+### 1. Stability Anti-Patterns
+
+**Core concept:** Failures propagate through integration points, cascading across system boundaries. The most dangerous patterns are not bugs in your code -- they are emergent behaviors that arise when systems interact under stress.
+
+**Why it works:** Recognizing anti-patterns lets you identify and eliminate the cracks before production traffic finds them. Every production outage traces back to one or more of these patterns. They are predictable, recurring, and preventable.
+
+**Key insights:**
+- Integration points are the number-one killer of production systems -- every socket, HTTP call, or queue is a risk
+- Cascading failures spread when one system's failure causes its callers to fail, which causes their callers to fail
+- Slow responses are worse than no response -- they tie up threads, exhaust pools, and propagate delays across the entire call chain
+- Unbounded result sets turn a harmless query into an out-of-memory crash when data grows beyond test assumptions
+- Users generate load patterns that no test suite can predict -- bots, retry storms, and flash crowds
+- Self-denial attacks occur when your own marketing, coupons, or viral features overwhelm your infrastructure
+- Blocked threads are the silent killer -- deadlocks and resource contention show no errors until everything stops
+
+**Code applications:**
+
+| Context | Pattern | Example |
+|---------|---------|---------|
+| **HTTP calls** | Assume every remote call can fail, hang, or return garbage | Wrap all external calls with timeout + circuit breaker |
+| **Database queries** | Enforce result set limits on every query | Add `LIMIT` clause; paginate all list endpoints |
+| **Thread pools** | Isolate pools per dependency to prevent cross-contamination | Separate thread pool for payment gateway vs. search |
+| **Load testing** | Simulate realistic traffic including spikes and abuse patterns | Use production traffic replays, not synthetic happy-path scripts |
+| **Marketing events** | Coordinate launches with capacity planning | Pre-scale before Black Friday; add queue for coupon redemption |
+
+See: [references/anti-patterns.md](references/anti-patterns.md) for detailed analysis of each anti-pattern with failure scenarios and detection strategies.
+
+### 2. Stability Patterns
+
+**Core concept:** Counter each anti-pattern with a stability pattern. Circuit breakers stop cascading failures. Bulkheads isolate blast radius. Timeouts reclaim stuck resources. Together they create a system that bends under load but does not break.
+
+**Why it works:** These patterns work because they accept failure as inevitable and design the system's response to failure, rather than trying to prevent all failures. A circuit breaker that trips is the system working correctly -- it is protecting itself from a downstream failure.
+
+**Key insights:**
+- Circuit Breaker: three states (closed, open, half-open) -- trips after threshold failures, periodically tests recovery
+- Bulkheads: partition resources so one failing component cannot drain the entire system
+- Timeouts: every outbound call needs both a connect timeout and a read timeout -- and timeouts must propagate up the call chain
+- Retry with backoff: exponential backoff + jitter prevents thundering herd on recovery
+- Fail Fast: if you know a request will fail, reject it immediately -- do not waste resources attempting it
+- Steady State: systems accumulate cruft (logs, sessions, temp files) -- design for automatic cleanup
+- Let It Crash: sometimes the safest recovery is to restart the process cleanly rather than limping along in an unknown state
+- Handshaking: let the server tell the client whether it can accept work before the client sends it
+
+**Code applications:**
+
+| Context | Pattern | Example |
+|---------|---------|---------|
+| **Service calls** | Circuit Breaker with threshold and recovery timeout | Open after 5 failures in 60s; half-open after 30s |
+| **Resource isolation** | Bulkhead with dedicated pools per dependency | Separate connection pools for critical vs. non-critical services |
+| **Network calls** | Timeout with propagation | Connect: 1s, read: 5s; propagate deadline to downstream calls |
+| **Retries** | Exponential backoff + jitter + retry budget | Base 100ms, max 3 retries, 20% retry budget across fleet |
+| **Data cleanup** | Steady State with automated purging | Delete sessions older than 24h; rotate logs at 500MB |
+
+See: [references/stability-patterns.md](references/stability-patterns.md) for implementation details, state machines, threshold tuning, and pattern combinations.
+
+### 3. Capacity and Availability
+
+**Core concept:** Capacity is not a single number -- it is a multi-dimensional function of CPU, memory, network, disk I/O, connection pools, and thread counts. Capacity planning means understanding which resource becomes the bottleneck first and at what load level.
+
+**Why it works:** Systems that are not capacity-tested fail in production at the worst possible moment -- during peak load. Understanding your system's actual limits (not theoretical limits) lets you set realistic SLAs and plan scaling before users hit the wall.
+
+**Key insights:**
+- Performance testing taxonomy: load test (expected traffic), stress test (beyond limits), soak test (sustained load over time), spike test (sudden bursts)
+- The Universal Scalability Law: throughput does not scale linearly -- contention and coherence costs cause diminishing returns
+- Connection pools are finite and precious -- a pool exhaustion looks identical to a database outage from the application's perspective
+- Thread pools must be sized based on measured throughput, not guesses -- too few starve the system, too many cause context-switching overhead
+- Myths: "The cloud is infinitely scalable" -- auto-scaling has lag time, cold-start costs, and hard limits
+- Resource pools need health checks, eviction policies, and maximum lifetime limits
+
+**Code applications:**
+
+| Context | Pattern | Example |
+|---------|---------|---------|
+| **Load testing** | Ramp to expected peak, then 2x, observe degradation curve | Gradually increase RPS until latency exceeds SLO |
+| **Connection pools** | Size based on measured concurrency, not defaults | Measure active connections under load; set pool to P99 + 20% headroom |
+| **Auto-scaling** | Define scaling triggers with appropriate cooldown | Scale on CPU > 70% sustained 3 min; cooldown 5 min |
+| **Soak testing** | Run at 80% capacity for 24-72 hours | Catch memory leaks, connection leaks, file handle exhaustion |
+| **Capacity model** | Document resource bottleneck per service | "Service X is memory-bound at 2000 RPS; needs 4GB per instance" |
+
+See: [references/capacity-planning.md](references/capacity-planning.md) for testing methodologies, resource pool management, and scalability modeling.
+
+### 4. Deployment and Release
+
+**Core concept:** Deployment (putting code on servers) and release (exposing code to users) are separate operations that should be decoupled. Separating them gives you the ability to deploy without risk and release with confidence.
+
+**Why it works:** Most outages are caused by changes -- deployments, configuration updates, database migrations. Decoupling deployment from release means you can deploy code to production, verify it works, and only then route traffic to it. If something goes wrong, you roll back the release, not the deployment.
+
+**Key insights:**
+- Zero-downtime deployment is non-negotiable for any system with users -- rolling deploys, blue-green, or canary
+- Feature flags decouple deployment from release -- dark-launch code and enable it independently
+- Database migrations must be backward-compatible -- the old code and new code will run simultaneously during deployment
+- Immutable infrastructure: never patch a running server -- build a new image, deploy it, destroy the old one
+- Canary releases limit blast radius by routing a small percentage of traffic to the new version first
+- Rollback must be faster than roll-forward -- if rollback takes 30 minutes, you will avoid deploying
+
+**Code applications:**
+
+| Context | Pattern | Example |
+|---------|---------|---------|
+| **Deploys** | Blue-green with health check gate | Deploy to green; run smoke tests; swap router |
+| **Progressive rollout** | Canary with automated rollback | Route 5% traffic to canary; auto-rollback if error rate > 1% |
+| **Feature launch** | Feature flags with emergency off switch | Ship code behind flag; enable for 10% of users; monitor; ramp |
+| **Schema changes** | Expand-contract migration pattern | Add new column; deploy code that writes both; backfill; drop old column |
+| **Rollback** | Instant rollback via traffic routing | Keep previous version running; rollback = switch load balancer target |
+
+See: [references/deployment-strategies.md](references/deployment-strategies.md) for deployment patterns, migration strategies, and infrastructure-as-code practices.
+
+### 5. Health Checks and Observability
+
+**Core concept:** You cannot operate what you cannot observe. Observability is not an afterthought -- it is a first-class design concern. Health checks, metrics, logs, and traces are the sensory organs of your system in production.
+
+**Why it works:** Production systems fail in ways that are invisible without proper instrumentation. A health check that only returns "OK" tells you nothing. Metrics without context are noise. Observability done right gives you the ability to answer questions about your system that you did not anticipate at design time.
+
+**Key insights:**
+- Health checks come in two flavors: shallow (process alive) and deep (dependencies reachable, resources available)
+- The three pillars of observability: structured logs (what happened), metrics (how much), distributed traces (where and how long)
+- RED method for services: Rate (requests/sec), Errors (error rate), Duration (latency distribution)
+- USE method for resources: Utilization (%), Saturation (queue depth), Errors (error count)
+- SLIs measure user experience; SLOs set targets; SLAs create contractual obligations -- define them in that order
+- Alerting on symptoms (user-facing errors) beats alerting on causes (CPU usage) -- alert on what users feel
+- Dashboards should answer "Is the system healthy right now?" within 5 seconds of looking
+
+**Code applications:**
+
+| Context | Pattern | Example |
+|---------|---------|---------|
+| **Health endpoints** | Deep health check with dependency status | `/health` returns status of DB, cache, queue, and disk space |
+| **Service metrics** | RED method instrumentation | Track request rate, error rate, and p50/p95/p99 latency per endpoint |
+| **Resource metrics** | USE method for infrastructure | Track CPU utilization, request queue depth, and error counts per host |
+| **Distributed tracing** | Propagate trace context across service boundaries | Inject trace ID in headers; correlate logs across services |
+| **Alerting** | Alert on SLO burn rate, not raw thresholds | "Error budget burning 10x normal rate" vs. "CPU > 80%" |
+
+See: [references/observability.md](references/observability.md) for health check design, metrics instrumentation, SLO frameworks, and alerting strategies.
+
+### 6. Adaptation and Chaos Engineering
+
+> **Safety note:** Chaos engineering experiments are design-time planning activities. The patterns below describe *what to test* and *what to verify*, not actions for an AI agent to execute autonomously. All failure injection must be performed by authorized engineers using dedicated tooling (e.g., Gremlin, Litmus, AWS FIS) with proper approvals, rollback plans, and blast radius controls in place.
+
+**Core concept:** Confidence in your system's resilience comes from testing it under realistic failure conditions. Chaos engineering is the discipline of experimenting on a system in a controlled environment to build confidence in its ability to withstand turbulent conditions.
+
+**Why it works:** You cannot know how your system handles failure until it actually fails. Waiting for production incidents to discover weaknesses is reactive and expensive. Chaos engineering proactively injects failures in a controlled way, turning unknown-unknowns into known-knowns before they cause real outages.
+
+**Key insights:**
+- Define steady state first -- you need a measurable baseline to detect when behavior deviates
+- Start small in non-production environments: terminate a single process, add latency to one call -- then escalate gradually with approvals
+- Minimize blast radius: use canary populations, feature flags, and emergency stop mechanisms for experiments
+- Production experiments require explicit authorization, monitoring, and immediate rollback capability
+- Automate recurring experiments so resilience is continuously verified, not a one-time event
+- GameDay exercises combine chaos engineering with incident response practice -- test both the system and the team
+- Every experiment should have a hypothesis: "We believe that when X fails, the system will Y"
+- Build a culture where finding weaknesses is celebrated, not punished
+
+**Code applications:**
+
+| Context | Pattern | Example |
+|---------|---------|---------|
+| **Process failure** | Controlled instance termination (via chaos tooling) | Terminate one pod using Gremlin/Litmus; verify service recovers within SLO |
+| **Network failure** | Inject latency or partition between services (via chaos tooling) | Add 500ms latency to DB calls; verify circuit breaker trips |
+| **Dependency failure** | Simulate downstream service outage (via chaos tooling) | Return 503 from payment API; verify graceful degradation |
+| **Resource exhaustion** | Simulate resource pressure (via chaos tooling) | Stress-test memory limits; verify process restarts cleanly |
+| **GameDay** | Scheduled team exercise with realistic failure scenario | "Primary database goes read-only at 2pm" -- practice response |
+
+See: [references/chaos-engineering.md](references/chaos-engineering.md) for experiment design, blast radius management, and building a chaos engineering practice.
+
+## Common Mistakes
+
+| Mistake | Why It Fails | Fix |
+|---------|-------------|------|
+| **No timeouts on outbound calls** | One slow dependency freezes the entire system | Set connect and read timeouts on every external call |
+| **Unbounded retries** | Retry storms amplify failures instead of recovering from them | Use exponential backoff, jitter, and fleet-wide retry budgets |
+| **Shared thread/connection pools** | One failing dependency drains resources from all features | Bulkhead: isolate pools per dependency or feature |
+| **Shallow health checks only** | Load balancer routes traffic to instances with broken dependencies | Implement deep health checks that verify downstream connectivity |
+| **Testing only the happy path** | System works perfectly until the first real failure | Load test, soak test, and chaos test before every major release |
+| **Coupling deploy and release** | Every deployment is a high-risk event with all-or-nothing rollout | Use feature flags, canary releases, and blue-green deployments |
+| **Alerting on causes, not symptoms** | High CPU alerts fire but users are fine; errors spike but no alert fires | Alert on user-facing SLIs: error rate, latency, availability |
+| **No capacity model** | System falls over at 2x load during an event nobody planned for | Model bottleneck resources; load test to 3x expected peak |
+
+## Quick Diagnostic
+
+Audit any production system:
+
+| Question | If No | Action |
+|----------|-------|--------|
+| Does every outbound call have a timeout? | Calls can hang indefinitely, blocking threads | Add connect and read timeouts to all external calls |
+| Are circuit breakers in place for critical dependencies? | One dependency failure takes down the whole system | Add circuit breakers with appropriate thresholds |
+| Are thread/connection pools isolated per dependency? | Shared pools allow cross-contamination of failures | Implement bulkhead pattern with dedicated pools |
+| Can you deploy without downtime? | Deployments cause user-visible outages | Implement rolling, blue-green, or canary deployment |
+| Do health checks verify dependency connectivity? | Dead instances receive traffic; partial failures go undetected | Add deep health checks that test DB, cache, queue |
+| Are logs, metrics, and traces correlated? | Debugging requires manual log searching across services | Implement distributed tracing with correlated IDs |
+| Have you load-tested beyond expected peak? | Unknown failure mode under real load | Load test to 2-3x expected peak; document breaking point |
+| Do you practice failure injection? | Resilience is theoretical, not verified | Start chaos engineering with low-risk experiments |
+
+## Reference Files
+
+- [anti-patterns.md](references/anti-patterns.md): Integration point failures, cascading failures, blocked threads, unbounded result sets, self-denial attacks, slow responses
+- [stability-patterns.md](references/stability-patterns.md): Circuit Breaker, Bulkhead, Timeout, Retry, Fail Fast, Steady State, Let It Crash, Handshaking
+- [capacity-planning.md](references/capacity-planning.md): Load/stress/soak testing, connection pool sizing, thread pool tuning, Universal Scalability Law
+- [deployment-strategies.md](references/deployment-strategies.md): Blue-green, canary, rolling deploys, feature flags, database migrations, immutable infrastructure
+- [observability.md](references/observability.md): Health checks, RED/USE methods, SLIs/SLOs/SLAs, distributed tracing, alerting strategy
+- [chaos-engineering.md](references/chaos-engineering.md): Steady state hypothesis, failure injection, GameDay exercises, blast radius management
+
+## Further Reading
+
+This skill is based on Michael Nygard's essential guide to building production-ready software. For the complete methodology, war stories, and implementation details:
+
+- [*"Release It! Design and Deploy Production-Ready Software"* (2nd Edition)](https://www.amazon.com/Release-Design-Deploy-Production-Ready-Software/dp/1680502395?tag=wondelai00-20) by Michael T. Nygard
+
+## About the Author
+
+**Michael T. Nygard** is a software architect and author with over 30 years of experience building and operating large-scale production systems. He has worked across industries including finance, retail, and government, and has been responsible for systems handling millions of transactions per day. Nygard is known for bridging the gap between development and operations, advocating that architects must be responsible for the systems they design long after the code is written. The first edition of *Release It!* (2007) became a foundational text in the DevOps and site reliability engineering movements. The second edition (2018) expands coverage to cloud-native architectures, containerization, and modern deployment practices. Nygard is a frequent conference speaker and has contributed to the broader conversation about resilience engineering, sociotechnical systems, and the human factors that influence production stability.

package/skills/release-it/references/anti-patterns.md

@@ -0,0 +1,279 @@
+# Stability Anti-Patterns
+
+Production systems do not fail because of bugs found in unit tests. They fail because of emergent behaviors that arise when systems interact under real-world conditions -- load spikes, network partitions, slow dependencies, and data growth that exceeds every assumption made during development.
+
+Michael Nygard identifies a recurring set of anti-patterns that cause the vast majority of production outages. Recognizing these patterns is the first step toward building resilient systems.
+
+## 1. Integration Points: The Number-One Killer
+
+Every integration point -- every socket connection, HTTP call, database query, message queue interaction, or third-party API call -- is a potential failure point. Integration points are the number-one killer of production systems because they introduce failure modes that do not exist in isolated testing.
+
+### How Integration Points Fail
+
+| Failure Mode | Description | Consequence |
+|-------------|-------------|-------------|
+| **Connection refused** | Remote host rejects the connection | Fast failure; relatively benign |
+| **Connection timeout** | Remote host does not respond to SYN | Thread blocked for 30-120 seconds (OS default) |
+| **Read timeout** | Connection established but response never arrives | Thread blocked indefinitely without explicit timeout |
+| **Partial response** | Connection drops mid-response | Corrupted data; parser exceptions |
+| **Slow response** | Response eventually arrives but takes minutes | Threads accumulate; pool exhaustion; cascading failure |
+| **Protocol violation** | Remote returns unexpected content type or format | Unhandled exceptions; crash |
+
+### Defense Strategy
+
+Every integration point needs a defense-in-depth strategy:
+
+1. **Timeouts** on every connection and read operation -- never use defaults
+2. **Circuit breakers** to stop calling a failing dependency
+3. **Bulkheads** to isolate the failure from other parts of the system
+4. **Fallbacks** to degrade gracefully when the dependency is unavailable
+5. **Monitoring** to detect degradation before users notice
+
+### Real-World Example
+
+A retail application calls an inventory service to check stock levels. The inventory service's database enters a long garbage collection pause. The inventory service stops responding but does not close connections. The retail application's thread pool fills with threads waiting for inventory responses. The retail application can no longer serve any requests -- including requests that do not need inventory data. The entire site goes down because one dependency slowed down.
+
+**Root cause:** No read timeout on the inventory service call. No bulkhead isolating inventory calls from other request handling.
+
+---
+
+## 2. Cascading Failures
+
+A cascading failure occurs when a failure in one system causes failures in the systems that depend on it, which in turn cause failures in the systems that depend on them, and so on. The defining characteristic of a cascading failure is that the damage spreads far beyond the original failure.
+
+### Cascade Mechanics
+
+```
+Service A (database overloaded)
+    → Service B (calls A, threads block waiting)
+        → Service C (calls B, threads block waiting)
+            → Service D (calls C, threads block waiting)
+                → User-facing application (completely unresponsive)
+```
+
+### Why Cascading Failures Are Devastating
+
+- The original failure may be minor (one database query slow)
+- The cascade amplifies the failure geometrically
+- Each layer adds more blocked threads, more resource consumption
+- By the time the cascade is visible, every system in the chain is degraded
+- Recovery requires coordinated action across all affected systems
+
+### Breaking the Cascade
+
+| Pattern | How It Breaks the Cascade |
+|---------|--------------------------|
+| **Circuit Breaker** | Stops calling the failing service; returns error immediately |
+| **Timeout** | Limits how long a caller waits; frees threads to handle other work |
+| **Bulkhead** | Isolates the failing dependency's impact to a limited set of resources |
+| **Fallback** | Provides degraded but functional response when dependency fails |
+| **Fail Fast** | Rejects requests immediately when system knows it cannot fulfill them |
+
+### Prevention Checklist
+
+- [ ] Every service-to-service call has a timeout
+- [ ] Circuit breakers protect against sustained failures
+- [ ] Failure in one dependency does not affect unrelated features
+- [ ] Fallback behavior is defined for every critical dependency
+- [ ] Cascading failure scenarios are included in chaos experiments
+
+---
+
+## 3. Users as a Source of Load
+
+Users are not gentle with your system. They do not arrive in an orderly queue at predictable intervals. Real user behavior generates load patterns that are fundamentally different from what synthetic tests produce.
+
+### Unexpected User Behaviors
+
+| Behavior | Load Impact | Example |
+|----------|------------|---------|
+| **Refresh storms** | Multiplies load when pages are slow | Users hit F5 repeatedly when checkout is slow |
+| **Bot traffic** | Can exceed human traffic by 10-100x | Scrapers, search engines, monitoring tools |
+| **Flash crowds** | Sudden, massive traffic spikes | Hacker News front page, TV mention, viral tweet |
+| **Abandoned sessions** | Resource consumption without completion | Users open carts, leave; sessions consume server memory |
+| **Power users** | Disproportionate resource consumption | One user with 50,000 items in a list; API consumers with no rate limit |
+
+### Defense Strategy
+
+- Rate limiting per user, per IP, and per API key
+- Session limits and timeouts to reclaim abandoned resources
+- Bot detection and separate handling (different rate limits, caching strategies)
+- Graceful degradation under load (serve cached content, disable non-essential features)
+- Load shedding: deliberately reject some requests to preserve service for others
+
+---
+
+## 4. Blocked Threads
+
+Blocked threads are the silent killer. Unlike a crash (which is loud and obvious), blocked threads produce no errors, no exceptions, and no log entries. The system simply stops processing requests.
+
+### Common Causes of Blocked Threads
+
+| Cause | How It Blocks | Detection |
+|-------|--------------|-----------|
+| **Missing timeouts** | Thread waits indefinitely for a response | Thread dump shows threads in WAITING/TIMED_WAITING |
+| **Deadlocks** | Two threads each hold a lock the other needs | Thread dump shows circular lock dependencies |
+| **Synchronized access** | All threads queue for a single lock | Throughput drops to single-threaded speed |
+| **DNS resolution** | DNS lookup blocks the calling thread | Threads stuck in `InetAddress.getByName()` |
+| **Log file I/O** | Synchronous logging blocks application threads | Threads stuck in file write; especially during disk pressure |
+
+### Detection and Prevention
+
+**Detection:**
+- Thread dump analysis (scheduled periodic dumps, not just during incidents)
+- Thread pool utilization metrics (active/idle/max)
+- Request latency distribution (sudden latency spike = possible thread starvation)
+- Health checks that verify thread pool availability
+
+**Prevention:**
+- Explicit timeouts on all blocking operations
+- Asynchronous I/O where possible
+- Bounded queues with rejection policies (not unbounded queues that grow forever)
+- Thread pool sizing based on measured workload, not defaults
+
+---
+
+## 5. Self-Denial Attacks
+
+A self-denial attack is when your own system, marketing, or business operations generate load that overwhelms your infrastructure. The irony is that these are "success disasters" -- everything is working as intended, but the system cannot handle its own success.
+
+### Common Self-Denial Scenarios
+
+| Scenario | Mechanism | Prevention |
+|----------|-----------|------------|
+| **Marketing email blast** | 500,000 emails sent simultaneously, 10% click through in 5 minutes | Stagger sends over hours; pre-scale infrastructure |
+| **Coupon code launch** | Viral sharing of limited coupon creates stampede | Queue-based redemption; rate limit per user |
+| **Product launch countdown** | Users refresh at exactly midnight | Serve static page at launch time; queue for access |
+| **Social media viral moment** | CEO's tweet goes viral, floods landing page | CDN caching; static page fallback |
+| **Cron job stampede** | Every server runs cleanup job at midnight | Randomize cron schedules; use distributed job scheduling |
+
+### Defense Strategy
+
+- Coordinate marketing events with engineering capacity planning
+- Use CDN and static page caching for high-traffic landing pages
+- Implement queue-based access for limited-resource events
+- Stagger scheduled jobs with random jitter
+- Pre-scale infrastructure before planned events
+
+---
+
+## 6. Scaling Effects
+
+Patterns that work at small scale break at large scale. A design that performs well with 10 servers may collapse at 100 servers. Scaling effects are the emergent behaviors that appear only when the system grows.
+
+### Examples of Scaling Effects
+
+| Pattern | Works at Small Scale | Breaks at Large Scale |
+|---------|---------------------|----------------------|
+| **Point-to-point connections** | 5 services = 20 connections | 50 services = 2,450 connections |
+| **Broadcast messages** | 10 subscribers = manageable | 1,000 subscribers = message storm |
+| **Shared database** | 5 services share one DB | 50 services = connection pool exhaustion |
+| **Health check polling** | Load balancer checks 5 servers | Load balancer checks 500 servers = significant traffic |
+| **Distributed locks** | Low contention with few nodes | High contention with many nodes |
+
+### Mitigation
+
+- Replace point-to-point with message buses or service meshes
+- Use fan-out patterns with controlled concurrency
+- Give each service its own data store (or at minimum, its own connection pool limits)
+- Use push-based health checks or sampling strategies at scale
+- Avoid distributed locks; use optimistic concurrency or partitioning instead
+
+---
+
+## 7. Unbalanced Capacities
+
+When upstream and downstream systems have different capacity limits, the faster system can overwhelm the slower one. This is particularly dangerous when a frontend tier can generate more requests than a backend tier can handle.
+
+### Common Imbalances
+
+- Web tier: 100 servers; backend API: 10 servers
+- Batch job produces 10,000 records/sec; downstream consumer processes 500/sec
+- Marketing campaign drives 10x normal traffic to a service scaled for 1x
+
+### Defense Strategy
+
+- Back-pressure: slow producers when consumers are overwhelmed
+- Queue-based buffering between tiers with different throughput
+- Rate limiting at the boundary between tiers
+- Capacity modeling that considers the full request chain, not individual services in isolation
+- Autoscaling policies that scale the entire chain, not just the entry point
+
+---
+
+## 8. Dogpile / Thundering Herd
+
+A dogpile (also called thundering herd) occurs when many threads or processes simultaneously attempt the same expensive operation, typically after a cache expires or a service recovers.
+
+### Common Dogpile Scenarios
+
+| Scenario | Mechanism | Impact |
+|----------|-----------|--------|
+| **Cache expiration** | Popular cache key expires; 1,000 threads hit the database simultaneously | Database overwhelmed; slow response; more cache misses |
+| **Service recovery** | Circuit breaker half-opens; all waiting requests flood the recovering service | Service fails again immediately |
+| **Cron overlap** | Slow job still running when next execution triggers | Double resource consumption; data corruption |
+| **Lock release** | Mutex released; all waiting threads resume simultaneously | Resource spike; possible re-contention |
+
+### Prevention
+
+- **Cache stampede prevention:** Use probabilistic early expiration, lock-based recomputation (only one thread refreshes), or serve stale + refresh in background
+- **Circuit breaker recovery:** Half-open state allows only a small number of test requests through
+- **Cron jobs:** Use distributed locks or leader election to ensure only one instance runs
+- **Gradual ramp-up:** When recovering, slowly increase traffic rather than allowing full load immediately
+
+---
+
+## 9. Slow Responses (Worse Than No Response)
+
+A fast failure is annoying. A slow failure is catastrophic. When a system responds slowly instead of failing fast, it creates a chain reaction of blocked resources throughout the calling stack.
+
+### Why Slow Is Worse Than Down
+
+| Fast Failure | Slow Failure |
+|-------------|-------------|
+| Circuit breaker trips immediately | Circuit breaker does not trip (still getting responses) |
+| Thread released after milliseconds | Thread blocked for seconds or minutes |
+| Error is visible and actionable | Problem is invisible until pool exhaustion |
+| Users see error, retry once, move on | Users see spinner, refresh, multiply load |
+| Affects one request | Blocks a thread, affecting all subsequent requests |
+
+### Detection
+
+- Monitor latency at p99 and p999, not just p50
+- Alert on latency shifts, not just absolute thresholds
+- Track thread pool utilization -- rising active threads with flat throughput indicates blocking
+- Trace slow requests across service boundaries to find the bottleneck
+
+### Prevention
+
+- Set aggressive read timeouts (seconds, not minutes)
+- Implement deadline propagation: if the user's request has 5 seconds left, do not start a 10-second operation
+- Use asynchronous processing for long operations
+- Implement load shedding: reject requests when response time exceeds SLO
+
+---
+
+## 10. Unbounded Result Sets
+
+A query that returns 10 rows in development returns 10 million rows in production. Unbounded result sets are a time bomb that detonates when data grows beyond test assumptions.
+
+### Common Manifestations
+
+| Query Pattern | Development | Production |
+|--------------|-------------|------------|
+| `SELECT * FROM orders WHERE user_id = ?` | 3 orders | 50,000 orders (power user) |
+| `SELECT * FROM events WHERE date > ?` | 100 events | 2 million events (6 months of data) |
+| `SELECT * FROM logs` | 500 rows | Out of memory |
+| API response with nested objects | Small JSON | 200MB JSON response |
+
+### Prevention
+
+- **Always paginate:** Every list query must have a `LIMIT` and `OFFSET` (or cursor-based pagination)
+- **Cap result sizes:** Enforce maximum result count at the API layer, even if the caller does not request it
+- **Stream large results:** Use cursors or streaming for batch operations rather than loading everything into memory
+- **Monitor query performance:** Track query execution time and result set sizes in production
+- **Test with production-scale data:** Use anonymized production data volumes in performance testing
+
+### Rule of Thumb
+
+If your code ever does `results = query.getAll()` without a limit, it is a production incident waiting to happen. Every query, every API response, every list operation must have an upper bound.