aigroup-workflow 2.2.1 → 2.2.2

package/skills/backend/microservices-architect/references/patterns.md

@@ -1,603 +1,603 @@
-# Resilience and Reliability Patterns
-
-Essential patterns for building fault-tolerant distributed systems.
-
-## Resilience Patterns
-
-### Circuit Breaker
-
-**Purpose:** Prevent cascading failures by failing fast when a dependency is unhealthy.
-
-**How It Works:**
-```
-States:
-1. CLOSED (normal operation)
-   - Requests pass through
-   - Track failure rate
-   - If failures exceed threshold → OPEN
-
-2. OPEN (failing fast)
-   - Immediately reject requests
-   - Return fallback response
-   - After timeout period → HALF_OPEN
-
-3. HALF_OPEN (testing recovery)
-   - Allow limited test requests
-   - If successful → CLOSED
-   - If failed → OPEN
-
-Configuration:
-- Failure threshold: 50% failures in 10 requests
-- Timeout: 30 seconds in OPEN state
-- Success threshold: 2 consecutive successes in HALF_OPEN
-```
-
-**Implementation Example:**
-```python
-# Using resilience4j-like pattern
-@CircuitBreaker(
-    name="payment-service",
-    fallbackMethod="paymentFallback",
-    failureThreshold=50,
-    waitDurationInOpenState=30000,  # 30s
-    permittedNumberOfCallsInHalfOpenState=3
-)
-async def process_payment(order_id: str, amount: float):
-    async with httpx.AsyncClient() as client:
-        response = await client.post(
-            f"{PAYMENT_SERVICE_URL}/payments",
-            json={"orderId": order_id, "amount": amount},
-            timeout=5.0
-        )
-        return response.json()
-
-async def paymentFallback(order_id: str, amount: float, exception):
-    # Log the failure
-    logger.error(f"Payment service unavailable: {exception}")
-    # Return graceful degradation
-    return {
-        "status": "pending",
-        "message": "Payment processing delayed, will retry"
-    }
-```
-
-**When to Use:**
-```
-Apply circuit breakers to:
-✓ External service calls
-✓ Database queries
-✓ Third-party APIs
-✓ Microservice-to-microservice calls
-
-Configuration Guidelines:
-- Fast services (p99 < 100ms): 5s timeout, 10s circuit open
-- Medium services (p99 < 1s): 10s timeout, 30s circuit open
-- Slow services (p99 > 1s): 30s timeout, 60s circuit open
-```
-
-### Retry Pattern
-
-**Purpose:** Handle transient failures by retrying operations.
-
-**Strategies:**
-
-**1. Exponential Backoff:**
-```
-Retry delays: 100ms, 200ms, 400ms, 800ms, 1600ms
-
-Benefits:
-- Reduces load during incidents
-- Gives service time to recover
-- Prevents thundering herd
-
-Implementation:
-attempts = 0
-max_attempts = 5
-base_delay = 0.1  # 100ms
-
-while attempts < max_attempts:
-    try:
-        return await make_request()
-    except TransientError as e:
-        attempts += 1
-        if attempts == max_attempts:
-            raise
-        delay = base_delay * (2 ** attempts) + random.uniform(0, 0.1)
-        await asyncio.sleep(delay)
-```
-
-**2. Retry with Jitter:**
-```
-Why: Prevents synchronized retries (thundering herd)
-
-Full Jitter:
-delay = random.uniform(0, base_delay * (2 ** attempt))
-
-Decorrelated Jitter:
-delay = min(cap, random.uniform(base, previous_delay * 3))
-
-Recommended: Decorrelated jitter for production systems
-```
-
-**3. Idempotency Keys:**
-```
-Problem: Retries can cause duplicate operations
-
-Solution: Idempotency keys
-POST /api/v1/payments
-Headers:
-  Idempotency-Key: uuid-12345
-
-Server Logic:
-1. Check if operation with this key already processed
-2. If yes, return cached response
-3. If no, process and cache result
-4. Cache for 24 hours
-
-Ensures safe retries even for non-idempotent operations
-```
-
-**Retry Best Practices:**
-```
-DO:
-✓ Only retry transient errors (timeout, 503, 429)
-✓ Use exponential backoff with jitter
-✓ Set maximum retry attempts (3-5)
-✓ Implement overall timeout
-✓ Use idempotency keys for writes
-✓ Log each retry attempt
-
-DON'T:
-✗ Retry client errors (400, 401, 404)
-✗ Retry without backoff (causes load spikes)
-✗ Infinite retries
-✗ Retry non-idempotent operations without safeguards
-```
-
-### Bulkhead Pattern
-
-**Purpose:** Isolate resources to prevent total system failure.
-
-**Thread Pool Isolation:**
-```
-Concept: Separate thread pools for different operations
-
-Example:
-- Payment Service Thread Pool: 20 threads
-- Inventory Service Thread Pool: 20 threads
-- Notification Service Thread Pool: 10 threads
-
-If payment service becomes slow:
-- Only payment thread pool exhausted
-- Inventory and notification still work
-- System partially degraded, not completely down
-```
-
-**Connection Pool Isolation:**
-```
-Database Connection Pools:
-- Read-only queries: 50 connections
-- Write queries: 20 connections
-- Reporting queries: 10 connections
-
-Heavy reporting query won't starve transactional operations
-```
-
-**Rate Limiting per Tenant:**
-```
-Multi-tenant SaaS application:
-
-tenant-a: 1000 requests/minute
-tenant-b: 1000 requests/minute
-tenant-c: 1000 requests/minute
-
-If tenant-a floods the system:
-- Only tenant-a throttled
-- tenant-b and tenant-c unaffected
-```
-
-**Implementation:**
-```python
-# Using semaphores for concurrency limits
-class BulkheadExecutor:
-    def __init__(self):
-        self.payment_semaphore = asyncio.Semaphore(20)
-        self.inventory_semaphore = asyncio.Semaphore(20)
-        self.notification_semaphore = asyncio.Semaphore(10)
-
-    async def call_payment_service(self, data):
-        async with self.payment_semaphore:
-            return await payment_service.call(data)
-
-    async def call_inventory_service(self, data):
-        async with self.inventory_semaphore:
-            return await inventory_service.call(data)
-```
-
-### Timeout Pattern
-
-**Purpose:** Prevent indefinite waiting for responses.
-
-**Timeout Types:**
-
-**1. Connection Timeout:**
-```
-Time allowed to establish connection
-
-Recommended: 2-5 seconds
-If takes longer, network likely has issues
-
-httpx.AsyncClient(timeout=httpx.Timeout(connect=3.0))
-```
-
-**2. Read Timeout:**
-```
-Time allowed to receive response after connection
-
-Varies by service:
-- Fast APIs: 5 seconds
-- Database queries: 10 seconds
-- Complex processing: 30 seconds
-
-httpx.AsyncClient(timeout=httpx.Timeout(read=10.0))
-```
-
-**3. Total Timeout:**
-```
-Overall time budget for entire operation
-
-Example: User checkout flow
-- Total budget: 30 seconds
-- Payment service: 10 seconds
-- Inventory check: 5 seconds
-- Order creation: 5 seconds
-- Buffer: 10 seconds
-
-async with asyncio.timeout(30):
-    result = await complete_checkout()
-```
-
-**Timeout Best Practices:**
-```
-Timeouts Hierarchy:
-Parent timeout > sum of child timeouts
-
-Request → API Gateway (30s timeout)
-  → Service A (10s timeout)
-    → Service B (5s timeout)
-      → Database (2s timeout)
-
-Set timeouts everywhere:
-✓ HTTP clients
-✓ Database connections
-✓ Message consumers
-✓ gRPC calls
-✓ Cache operations
-```
-
-## Distributed Transaction Patterns
-
-### Saga Pattern
-
-**Purpose:** Manage distributed transactions across services.
-
-**Choreography-Based Saga:**
-```
-Example: Order Creation Saga
-
-Events:
-1. OrderService: order.created
-2. PaymentService: payment.completed OR payment.failed
-3. InventoryService: inventory.reserved OR inventory.reservation.failed
-4. ShippingService: shipment.created
-
-Compensating Transactions:
-If inventory.reservation.failed:
-  → PaymentService listens → refund.initiated
-  → OrderService listens → order.cancelled
-
-Pros:
-- Decentralized
-- No single point of failure
-- Services autonomous
-
-Cons:
-- Difficult to track saga state
-- Complex debugging
-- No saga-wide timeout
-```
-
-**Orchestration-Based Saga:**
-```
-Example: Order Saga Orchestrator
-
-Saga Steps:
-1. Create Order (OrderService)
-2. Charge Payment (PaymentService)
-3. Reserve Inventory (InventoryService)
-4. Create Shipment (ShippingService)
-
-Orchestrator Logic:
-step1_result = await order_service.create_order()
-if not step1_result.success:
-    return failure("Order creation failed")
-
-step2_result = await payment_service.charge(amount)
-if not step2_result.success:
-    await order_service.cancel_order(step1_result.order_id)
-    return failure("Payment failed")
-
-step3_result = await inventory_service.reserve(items)
-if not step3_result.success:
-    await payment_service.refund(step2_result.payment_id)
-    await order_service.cancel_order(step1_result.order_id)
-    return failure("Inventory unavailable")
-
-# Continue saga...
-
-Pros:
-- Clear workflow
-- Centralized monitoring
-- Easy to understand
-
-Cons:
-- Orchestrator complexity
-- Potential bottleneck
-- Coupling to orchestrator
-```
-
-**Saga State Management:**
-```
-Persist saga state to handle failures:
-
-CREATE TABLE saga_instances (
-    saga_id UUID PRIMARY KEY,
-    saga_type VARCHAR(50),
-    current_step VARCHAR(50),
-    status VARCHAR(20),
-    payload JSONB,
-    created_at TIMESTAMP,
-    updated_at TIMESTAMP
-);
-
-On orchestrator restart:
-- Load incomplete sagas
-- Resume from last completed step
-- Execute remaining steps or compensations
-```
-
-### Event Sourcing
-
-**Purpose:** Store all state changes as events, derive current state by replaying.
-
-**Implementation:**
-```
-Traditional Approach:
-UPDATE orders SET status = 'shipped' WHERE id = 123;
-(Lost: when shipped, by whom, from where)
-
-Event Sourcing Approach:
-Events:
-1. OrderPlaced { orderId, customerId, items, timestamp }
-2. PaymentReceived { orderId, amount, paymentId, timestamp }
-3. OrderShipped { orderId, trackingNumber, carrier, timestamp }
-
-Current state = replay all events
-```
-
-**Event Store:**
-```
-CREATE TABLE events (
-    event_id UUID PRIMARY KEY,
-    aggregate_id UUID,
-    aggregate_type VARCHAR(50),
-    event_type VARCHAR(100),
-    event_data JSONB,
-    version INTEGER,
-    timestamp TIMESTAMP,
-    correlation_id UUID
-);
-
-CREATE INDEX idx_aggregate ON events(aggregate_id, version);
-
-Guarantees:
-- Events immutable
-- Events ordered by version
-- Optimistic locking prevents conflicts
-```
-
-**Benefits:**
-```
-✓ Full audit trail
-✓ Time travel (replay to any point)
-✓ Event replay for debugging
-✓ Multiple read models from same events
-✓ Temporal queries ("show orders as of yesterday")
-
-Challenges:
-✗ Eventual consistency
-✗ Event schema evolution
-✗ Snapshot strategy needed
-✗ Increased storage
-```
-
-### CQRS (Command Query Responsibility Segregation)
-
-**Purpose:** Separate read and write models for different optimization strategies.
-
-**Architecture:**
-```
-Write Side (Command):
-- Receives commands (CreateOrder, UpdateInventory)
-- Validates business rules
-- Stores events in event store
-- Optimized for consistency and writes
-
-Read Side (Query):
-- Listens to events
-- Updates denormalized read models
-- Optimized for queries
-- Eventual consistency
-
-Example:
-Command: CreateOrder
-  → Order aggregate validates
-  → Publishes OrderCreated event
-  → Event stored in event store
-
-Query Side:
-  → Listens to OrderCreated
-  → Updates order_summary table (denormalized)
-  → Updates customer_order_history (different view)
-  → Updates order_analytics (aggregated metrics)
-```
-
-**Read Models:**
-```
-Multiple specialized views from same events:
-
-1. Order Detail View (for customer):
-   { orderId, items, status, total, estimatedDelivery }
-
-2. Order List View (for admin):
-   { orderId, customerName, orderDate, status, total }
-
-3. Analytics View:
-   { date, totalOrders, totalRevenue, averageOrderValue }
-
-Each optimized for specific query patterns
-```
-
-## Fault Tolerance Patterns
-
-### Health Checks
-
-**Types:**
-
-**1. Liveness Probe:**
-```
-Purpose: Is the service alive?
-
-Endpoint: GET /health/live
-
-Returns 200 if:
-- Application process running
-- Not deadlocked
-
-Kubernetes Action:
-- If fails: Restart container
-```
-
-**2. Readiness Probe:**
-```
-Purpose: Is the service ready to receive traffic?
-
-Endpoint: GET /health/ready
-
-Returns 200 if:
-- Database connection pool healthy
-- Cache accessible
-- Downstream dependencies responsive
-
-Kubernetes Action:
-- If fails: Remove from load balancer
-- Don't send traffic until ready
-```
-
-**3. Startup Probe:**
-```
-Purpose: Has the service finished initialization?
-
-Endpoint: GET /health/startup
-
-For slow-starting applications:
-- Prevents premature liveness checks
-- Allows longer startup time
-```
-
-**Implementation:**
-```python
-@app.get("/health/live")
-async def liveness():
-    return {"status": "alive"}
-
-@app.get("/health/ready")
-async def readiness():
-    checks = {
-        "database": await check_database(),
-        "cache": await check_cache(),
-        "payment_service": await check_payment_service()
-    }
-
-    all_healthy = all(checks.values())
-    status_code = 200 if all_healthy else 503
-
-    return JSONResponse(
-        status_code=status_code,
-        content={"status": "ready" if all_healthy else "not ready", "checks": checks}
-    )
-```
-
-### Graceful Degradation
-
-**Purpose:** Provide reduced functionality when dependencies fail.
-
-**Strategies:**
-
-**1. Cached Responses:**
-```
-async def get_product_recommendations(user_id):
-    try:
-        async with circuit_breaker:
-            return await ml_service.get_recommendations(user_id)
-    except ServiceUnavailable:
-        # Fallback to cached popular products
-        return await cache.get_popular_products()
-```
-
-**2. Default Values:**
-```
-async def get_user_preferences(user_id):
-    try:
-        return await preferences_service.get(user_id)
-    except ServiceUnavailable:
-        # Return sensible defaults
-        return {
-            "language": "en",
-            "currency": "USD",
-            "theme": "light"
-        }
-```
-
-**3. Feature Toggles:**
-```
-if feature_flags.is_enabled("personalized_recommendations"):
-    recommendations = await ml_service.get_recommendations()
-else:
-    # Fallback to simple algorithm
-    recommendations = await get_popular_products()
-```
-
-## Summary
-
-Resilience patterns are mandatory in distributed systems. Layer multiple patterns for defense in depth:
-
-**Essential Stack:**
-1. Timeouts (prevent hanging)
-2. Retries with backoff (handle transient errors)
-3. Circuit breakers (prevent cascading failures)
-4. Bulkheads (isolate failures)
-5. Health checks (enable auto-healing)
-6. Graceful degradation (maintain partial functionality)
-
-**Choose Saga Pattern When:**
-- Distributed transaction needed
-- Strong consistency not required
-- Compensating transactions possible
-
-**Choose Event Sourcing When:**
-- Full audit trail required
-- Temporal queries needed
-- Multiple read models beneficial
-
-Always test failure scenarios. Use chaos engineering to validate resilience.
+# Resilience and Reliability Patterns
+
+Essential patterns for building fault-tolerant distributed systems.
+
+## Resilience Patterns
+
+### Circuit Breaker
+
+**Purpose:** Prevent cascading failures by failing fast when a dependency is unhealthy.
+
+**How It Works:**
+```
+States:
+1. CLOSED (normal operation)
+   - Requests pass through
+   - Track failure rate
+   - If failures exceed threshold → OPEN
+
+2. OPEN (failing fast)
+   - Immediately reject requests
+   - Return fallback response
+   - After timeout period → HALF_OPEN
+
+3. HALF_OPEN (testing recovery)
+   - Allow limited test requests
+   - If successful → CLOSED
+   - If failed → OPEN
+
+Configuration:
+- Failure threshold: 50% failures in 10 requests
+- Timeout: 30 seconds in OPEN state
+- Success threshold: 2 consecutive successes in HALF_OPEN
+```
+
+**Implementation Example:**
+```python
+# Using resilience4j-like pattern
+@CircuitBreaker(
+    name="payment-service",
+    fallbackMethod="paymentFallback",
+    failureThreshold=50,
+    waitDurationInOpenState=30000,  # 30s
+    permittedNumberOfCallsInHalfOpenState=3
+)
+async def process_payment(order_id: str, amount: float):
+    async with httpx.AsyncClient() as client:
+        response = await client.post(
+            f"{PAYMENT_SERVICE_URL}/payments",
+            json={"orderId": order_id, "amount": amount},
+            timeout=5.0
+        )
+        return response.json()
+
+async def paymentFallback(order_id: str, amount: float, exception):
+    # Log the failure
+    logger.error(f"Payment service unavailable: {exception}")
+    # Return graceful degradation
+    return {
+        "status": "pending",
+        "message": "Payment processing delayed, will retry"
+    }
+```
+
+**When to Use:**
+```
+Apply circuit breakers to:
+✓ External service calls
+✓ Database queries
+✓ Third-party APIs
+✓ Microservice-to-microservice calls
+
+Configuration Guidelines:
+- Fast services (p99 < 100ms): 5s timeout, 10s circuit open
+- Medium services (p99 < 1s): 10s timeout, 30s circuit open
+- Slow services (p99 > 1s): 30s timeout, 60s circuit open
+```
+
+### Retry Pattern
+
+**Purpose:** Handle transient failures by retrying operations.
+
+**Strategies:**
+
+**1. Exponential Backoff:**
+```
+Retry delays: 100ms, 200ms, 400ms, 800ms, 1600ms
+
+Benefits:
+- Reduces load during incidents
+- Gives service time to recover
+- Prevents thundering herd
+
+Implementation:
+attempts = 0
+max_attempts = 5
+base_delay = 0.1  # 100ms
+
+while attempts < max_attempts:
+    try:
+        return await make_request()
+    except TransientError as e:
+        attempts += 1
+        if attempts == max_attempts:
+            raise
+        delay = base_delay * (2 ** attempts) + random.uniform(0, 0.1)
+        await asyncio.sleep(delay)
+```
+
+**2. Retry with Jitter:**
+```
+Why: Prevents synchronized retries (thundering herd)
+
+Full Jitter:
+delay = random.uniform(0, base_delay * (2 ** attempt))
+
+Decorrelated Jitter:
+delay = min(cap, random.uniform(base, previous_delay * 3))
+
+Recommended: Decorrelated jitter for production systems
+```
+
+**3. Idempotency Keys:**
+```
+Problem: Retries can cause duplicate operations
+
+Solution: Idempotency keys
+POST /api/v1/payments
+Headers:
+  Idempotency-Key: uuid-12345
+
+Server Logic:
+1. Check if operation with this key already processed
+2. If yes, return cached response
+3. If no, process and cache result
+4. Cache for 24 hours
+
+Ensures safe retries even for non-idempotent operations
+```
+
+**Retry Best Practices:**
+```
+DO:
+✓ Only retry transient errors (timeout, 503, 429)
+✓ Use exponential backoff with jitter
+✓ Set maximum retry attempts (3-5)
+✓ Implement overall timeout
+✓ Use idempotency keys for writes
+✓ Log each retry attempt
+
+DON'T:
+✗ Retry client errors (400, 401, 404)
+✗ Retry without backoff (causes load spikes)
+✗ Infinite retries
+✗ Retry non-idempotent operations without safeguards
+```
+
+### Bulkhead Pattern
+
+**Purpose:** Isolate resources to prevent total system failure.
+
+**Thread Pool Isolation:**
+```
+Concept: Separate thread pools for different operations
+
+Example:
+- Payment Service Thread Pool: 20 threads
+- Inventory Service Thread Pool: 20 threads
+- Notification Service Thread Pool: 10 threads
+
+If payment service becomes slow:
+- Only payment thread pool exhausted
+- Inventory and notification still work
+- System partially degraded, not completely down
+```
+
+**Connection Pool Isolation:**
+```
+Database Connection Pools:
+- Read-only queries: 50 connections
+- Write queries: 20 connections
+- Reporting queries: 10 connections
+
+Heavy reporting query won't starve transactional operations
+```
+
+**Rate Limiting per Tenant:**
+```
+Multi-tenant SaaS application:
+
+tenant-a: 1000 requests/minute
+tenant-b: 1000 requests/minute
+tenant-c: 1000 requests/minute
+
+If tenant-a floods the system:
+- Only tenant-a throttled
+- tenant-b and tenant-c unaffected
+```
+
+**Implementation:**
+```python
+# Using semaphores for concurrency limits
+class BulkheadExecutor:
+    def __init__(self):
+        self.payment_semaphore = asyncio.Semaphore(20)
+        self.inventory_semaphore = asyncio.Semaphore(20)
+        self.notification_semaphore = asyncio.Semaphore(10)
+
+    async def call_payment_service(self, data):
+        async with self.payment_semaphore:
+            return await payment_service.call(data)
+
+    async def call_inventory_service(self, data):
+        async with self.inventory_semaphore:
+            return await inventory_service.call(data)
+```
+
+### Timeout Pattern
+
+**Purpose:** Prevent indefinite waiting for responses.
+
+**Timeout Types:**
+
+**1. Connection Timeout:**
+```
+Time allowed to establish connection
+
+Recommended: 2-5 seconds
+If takes longer, network likely has issues
+
+httpx.AsyncClient(timeout=httpx.Timeout(connect=3.0))
+```
+
+**2. Read Timeout:**
+```
+Time allowed to receive response after connection
+
+Varies by service:
+- Fast APIs: 5 seconds
+- Database queries: 10 seconds
+- Complex processing: 30 seconds
+
+httpx.AsyncClient(timeout=httpx.Timeout(read=10.0))
+```
+
+**3. Total Timeout:**
+```
+Overall time budget for entire operation
+
+Example: User checkout flow
+- Total budget: 30 seconds
+- Payment service: 10 seconds
+- Inventory check: 5 seconds
+- Order creation: 5 seconds
+- Buffer: 10 seconds
+
+async with asyncio.timeout(30):
+    result = await complete_checkout()
+```
+
+**Timeout Best Practices:**
+```
+Timeouts Hierarchy:
+Parent timeout > sum of child timeouts
+
+Request → API Gateway (30s timeout)
+  → Service A (10s timeout)
+    → Service B (5s timeout)
+      → Database (2s timeout)
+
+Set timeouts everywhere:
+✓ HTTP clients
+✓ Database connections
+✓ Message consumers
+✓ gRPC calls
+✓ Cache operations
+```
+
+## Distributed Transaction Patterns
+
+### Saga Pattern
+
+**Purpose:** Manage distributed transactions across services.
+
+**Choreography-Based Saga:**
+```
+Example: Order Creation Saga
+
+Events:
+1. OrderService: order.created
+2. PaymentService: payment.completed OR payment.failed
+3. InventoryService: inventory.reserved OR inventory.reservation.failed
+4. ShippingService: shipment.created
+
+Compensating Transactions:
+If inventory.reservation.failed:
+  → PaymentService listens → refund.initiated
+  → OrderService listens → order.cancelled
+
+Pros:
+- Decentralized
+- No single point of failure
+- Services autonomous
+
+Cons:
+- Difficult to track saga state
+- Complex debugging
+- No saga-wide timeout
+```
+
+**Orchestration-Based Saga:**
+```
+Example: Order Saga Orchestrator
+
+Saga Steps:
+1. Create Order (OrderService)
+2. Charge Payment (PaymentService)
+3. Reserve Inventory (InventoryService)
+4. Create Shipment (ShippingService)
+
+Orchestrator Logic:
+step1_result = await order_service.create_order()
+if not step1_result.success:
+    return failure("Order creation failed")
+
+step2_result = await payment_service.charge(amount)
+if not step2_result.success:
+    await order_service.cancel_order(step1_result.order_id)
+    return failure("Payment failed")
+
+step3_result = await inventory_service.reserve(items)
+if not step3_result.success:
+    await payment_service.refund(step2_result.payment_id)
+    await order_service.cancel_order(step1_result.order_id)
+    return failure("Inventory unavailable")
+
+# Continue saga...
+
+Pros:
+- Clear workflow
+- Centralized monitoring
+- Easy to understand
+
+Cons:
+- Orchestrator complexity
+- Potential bottleneck
+- Coupling to orchestrator
+```
+
+**Saga State Management:**
+```
+Persist saga state to handle failures:
+
+CREATE TABLE saga_instances (
+    saga_id UUID PRIMARY KEY,
+    saga_type VARCHAR(50),
+    current_step VARCHAR(50),
+    status VARCHAR(20),
+    payload JSONB,
+    created_at TIMESTAMP,
+    updated_at TIMESTAMP
+);
+
+On orchestrator restart:
+- Load incomplete sagas
+- Resume from last completed step
+- Execute remaining steps or compensations
+```
+
+### Event Sourcing
+
+**Purpose:** Store all state changes as events, derive current state by replaying.
+
+**Implementation:**
+```
+Traditional Approach:
+UPDATE orders SET status = 'shipped' WHERE id = 123;
+(Lost: when shipped, by whom, from where)
+
+Event Sourcing Approach:
+Events:
+1. OrderPlaced { orderId, customerId, items, timestamp }
+2. PaymentReceived { orderId, amount, paymentId, timestamp }
+3. OrderShipped { orderId, trackingNumber, carrier, timestamp }
+
+Current state = replay all events
+```
+
+**Event Store:**
+```
+CREATE TABLE events (
+    event_id UUID PRIMARY KEY,
+    aggregate_id UUID,
+    aggregate_type VARCHAR(50),
+    event_type VARCHAR(100),
+    event_data JSONB,
+    version INTEGER,
+    timestamp TIMESTAMP,
+    correlation_id UUID
+);
+
+CREATE INDEX idx_aggregate ON events(aggregate_id, version);
+
+Guarantees:
+- Events immutable
+- Events ordered by version
+- Optimistic locking prevents conflicts
+```
+
+**Benefits:**
+```
+✓ Full audit trail
+✓ Time travel (replay to any point)
+✓ Event replay for debugging
+✓ Multiple read models from same events
+✓ Temporal queries ("show orders as of yesterday")
+
+Challenges:
+✗ Eventual consistency
+✗ Event schema evolution
+✗ Snapshot strategy needed
+✗ Increased storage
+```
+
+### CQRS (Command Query Responsibility Segregation)
+
+**Purpose:** Separate read and write models for different optimization strategies.
+
+**Architecture:**
+```
+Write Side (Command):
+- Receives commands (CreateOrder, UpdateInventory)
+- Validates business rules
+- Stores events in event store
+- Optimized for consistency and writes
+
+Read Side (Query):
+- Listens to events
+- Updates denormalized read models
+- Optimized for queries
+- Eventual consistency
+
+Example:
+Command: CreateOrder
+  → Order aggregate validates
+  → Publishes OrderCreated event
+  → Event stored in event store
+
+Query Side:
+  → Listens to OrderCreated
+  → Updates order_summary table (denormalized)
+  → Updates customer_order_history (different view)
+  → Updates order_analytics (aggregated metrics)
+```
+
+**Read Models:**
+```
+Multiple specialized views from same events:
+
+1. Order Detail View (for customer):
+   { orderId, items, status, total, estimatedDelivery }
+
+2. Order List View (for admin):
+   { orderId, customerName, orderDate, status, total }
+
+3. Analytics View:
+   { date, totalOrders, totalRevenue, averageOrderValue }
+
+Each optimized for specific query patterns
+```
+
+## Fault Tolerance Patterns
+
+### Health Checks
+
+**Types:**
+
+**1. Liveness Probe:**
+```
+Purpose: Is the service alive?
+
+Endpoint: GET /health/live
+
+Returns 200 if:
+- Application process running
+- Not deadlocked
+
+Kubernetes Action:
+- If fails: Restart container
+```
+
+**2. Readiness Probe:**
+```
+Purpose: Is the service ready to receive traffic?
+
+Endpoint: GET /health/ready
+
+Returns 200 if:
+- Database connection pool healthy
+- Cache accessible
+- Downstream dependencies responsive
+
+Kubernetes Action:
+- If fails: Remove from load balancer
+- Don't send traffic until ready
+```
+
+**3. Startup Probe:**
+```
+Purpose: Has the service finished initialization?
+
+Endpoint: GET /health/startup
+
+For slow-starting applications:
+- Prevents premature liveness checks
+- Allows longer startup time
+```
+
+**Implementation:**
+```python
+@app.get("/health/live")
+async def liveness():
+    return {"status": "alive"}
+
+@app.get("/health/ready")
+async def readiness():
+    checks = {
+        "database": await check_database(),
+        "cache": await check_cache(),
+        "payment_service": await check_payment_service()
+    }
+
+    all_healthy = all(checks.values())
+    status_code = 200 if all_healthy else 503
+
+    return JSONResponse(
+        status_code=status_code,
+        content={"status": "ready" if all_healthy else "not ready", "checks": checks}
+    )
+```
+
+### Graceful Degradation
+
+**Purpose:** Provide reduced functionality when dependencies fail.
+
+**Strategies:**
+
+**1. Cached Responses:**
+```
+async def get_product_recommendations(user_id):
+    try:
+        async with circuit_breaker:
+            return await ml_service.get_recommendations(user_id)
+    except ServiceUnavailable:
+        # Fallback to cached popular products
+        return await cache.get_popular_products()
+```
+
+**2. Default Values:**
+```
+async def get_user_preferences(user_id):
+    try:
+        return await preferences_service.get(user_id)
+    except ServiceUnavailable:
+        # Return sensible defaults
+        return {
+            "language": "en",
+            "currency": "USD",
+            "theme": "light"
+        }
+```
+
+**3. Feature Toggles:**
+```
+if feature_flags.is_enabled("personalized_recommendations"):
+    recommendations = await ml_service.get_recommendations()
+else:
+    # Fallback to simple algorithm
+    recommendations = await get_popular_products()
+```
+
+## Summary
+
+Resilience patterns are mandatory in distributed systems. Layer multiple patterns for defense in depth:
+
+**Essential Stack:**
+1. Timeouts (prevent hanging)
+2. Retries with backoff (handle transient errors)
+3. Circuit breakers (prevent cascading failures)
+4. Bulkheads (isolate failures)
+5. Health checks (enable auto-healing)
+6. Graceful degradation (maintain partial functionality)
+
+**Choose Saga Pattern When:**
+- Distributed transaction needed
+- Strong consistency not required
+- Compensating transactions possible
+
+**Choose Event Sourcing When:**
+- Full audit trail required
+- Temporal queries needed
+- Multiple read models beneficial
+
+Always test failure scenarios. Use chaos engineering to validate resilience.