aigroup-workflow 2.1.2 → 2.2.1

package/skills/microservices-architect/references/data.md

@@ -1,721 +0,0 @@
-# Data Management in Microservices
-
-Comprehensive guide for managing data across distributed services.
-
-## Fundamental Principles
-
-### Database per Service
-
-**Core Principle:** Each microservice owns its data exclusively.
-
-**Rules:**
-```
-✓ DO:
-- Each service has its own database/schema
-- Service owns all CRUD operations on its data
-- Other services access data via APIs only
-- Service can choose its own database technology
-
-✗ DON'T:
-- Share database between services
-- Direct database queries across services
-- Shared tables or schemas
-- Database-level joins across services
-```
-
-**Implementation Options:**
-
-**1. Separate Database Instances:**
-```
-UserService → PostgreSQL instance 1
-OrderService → PostgreSQL instance 2
-InventoryService → PostgreSQL instance 3
-
-Pros:
-- Complete isolation
-- Independent scaling
-- No shared resource contention
-
-Cons:
-- Higher infrastructure cost
-- More operational overhead
-```
-
-**2. Separate Schemas:**
-```
-Same PostgreSQL instance:
-- Schema: user_service
-- Schema: order_service
-- Schema: inventory_service
-
-Pros:
-- Lower cost
-- Easier local development
-
-Cons:
-- Shared resource (CPU, memory)
-- Not true isolation
-- Scaling limitations
-
-Recommendation: Use separate schemas for dev/test, separate instances for production
-```
-
-**3. Polyglot Persistence:**
-```
-Each service chooses optimal database:
-
-UserService → PostgreSQL
-  (Relational data, ACID transactions)
-
-ProductCatalog → Elasticsearch
-  (Full-text search, faceted navigation)
-
-SessionStore → Redis
-  (Fast key-value, TTL support)
-
-EventLog → Kafka
-  (Event streaming, replay)
-
-RecommendationEngine → MongoDB
-  (Flexible schema, denormalized data)
-
-Benefits: Right tool for the job
-Challenges: Multiple technologies to manage
-```
-
-## Data Consistency Patterns
-
-### Strong Consistency vs Eventual Consistency
-
-**Strong Consistency:**
-```
-Definition: Read after write returns latest value
-
-Requires:
-- Distributed transaction (2PC, 3PC)
-- Coordination across services
-- Blocking operations
-
-Cost:
-- Higher latency
-- Reduced availability (CAP theorem)
-- Complexity
-
-When to Use:
-- Financial transactions
-- Inventory reservations
-- Critical business operations
-- Regulatory requirements
-```
-
-**Eventual Consistency:**
-```
-Definition: System converges to consistent state over time
-
-Characteristics:
-- Temporary inconsistencies acceptable
-- Non-blocking operations
-- Higher availability
-- Lower latency
-
-Example:
-1. Order placed (OrderService)
-2. Immediately return success to user
-3. Event published: order.created
-4. InventoryService eventually processes event
-5. Stock count updated (few milliseconds later)
-
-When to Use:
-- Social media feeds
-- Analytics dashboards
-- Recommendation systems
-- Non-critical updates
-```
-
-### Managing Cross-Service Data
-
-**Problem:** Order service needs customer data owned by User service.
-
-**Anti-Pattern Solutions:**
-```
-✗ Direct database access
-✗ Shared database
-✗ Database replication between services
-```
-
-**Proper Solutions:**
-
-**1. API Composition:**
-```
-Client Query: Get order with customer details
-
-API Gateway:
-1. GET /orders/123 from OrderService
-   Response: { orderId: 123, customerId: 456, items: [...] }
-
-2. GET /customers/456 from UserService
-   Response: { customerId: 456, name: "John", email: "john@example.com" }
-
-3. Combine responses and return to client
-
-Pros:
-- Maintains service boundaries
-- Real-time data
-
-Cons:
-- Multiple network calls (latency)
-- Partial failure handling complex
-- N+1 query problem
-```
-
-**2. Data Replication via Events:**
-```
-OrderService maintains denormalized customer data:
-
-CREATE TABLE orders (
-    order_id UUID PRIMARY KEY,
-    customer_id UUID,
-    customer_name VARCHAR(255),  -- Denormalized
-    customer_email VARCHAR(255), -- Denormalized
-    order_total DECIMAL,
-    created_at TIMESTAMP
-);
-
-UserService publishes events:
-- customer.created
-- customer.updated
-- customer.deleted
-
-OrderService subscribes and updates local copy:
-
-async def on_customer_updated(event):
-    await db.execute(
-        "UPDATE orders SET customer_name = $1, customer_email = $2 WHERE customer_id = $3",
-        event.name, event.email, event.customer_id
-    )
-
-Pros:
-- Fast queries (no joins across services)
-- Resilient to UserService downtime
-
-Cons:
-- Eventual consistency
-- Storage duplication
-- Keeping data in sync
-```
-
-**3. CQRS with Shared Read Model:**
-```
-Write Models (Command Side):
-- UserService writes to user_db
-- OrderService writes to order_db
-
-Read Model (Query Side):
-- Dedicated database for queries
-- Subscribes to events from both services
-- Denormalized view for efficient queries
-
-Example Read Model:
-CREATE TABLE order_details_view (
-    order_id UUID,
-    customer_id UUID,
-    customer_name VARCHAR(255),
-    customer_email VARCHAR(255),
-    items JSONB,
-    order_total DECIMAL,
-    order_status VARCHAR(50)
-);
-
-Pros:
-- Optimized for queries
-- No cross-service calls
-- Can rebuild from events
-
-Cons:
-- Eventual consistency
-- Additional infrastructure
-- Event replay mechanism needed
-```
-
-## Distributed Transactions
-
-### Two-Phase Commit (2PC)
-
-**How It Works:**
-```
-Phase 1: Prepare
-Coordinator asks all participants: "Can you commit?"
-- Service A: YES
-- Service B: YES
-- Service C: YES
-
-Phase 2: Commit
-If all YES:
-  Coordinator tells all: "Commit"
-If any NO:
-  Coordinator tells all: "Rollback"
-
-Example:
-Transfer $100 from Account A to Account B
-
-Prepare:
-- AccountService A: Can deduct $100? YES (balance sufficient)
-- AccountService B: Can add $100? YES (account active)
-
-Commit:
-- AccountService A: Deduct $100 (committed)
-- AccountService B: Add $100 (committed)
-```
-
-**Problems with 2PC:**
-```
-✗ Blocking protocol (participants wait for coordinator)
-✗ Single point of failure (coordinator down = all blocked)
-✗ Reduced availability
-✗ Poor performance (synchronous coordination)
-✗ Doesn't scale well
-
-Recommendation: Avoid 2PC in microservices, use Saga pattern instead
-```
-
-### Saga Pattern (Recommended)
-
-**Orchestration-Based Saga:**
-```
-Transfer Money Saga:
-
-Steps:
-1. Debit Account A
-2. Credit Account B
-
-Compensations:
-1. Credit Account A (reverse debit)
-
-Saga Orchestrator:
-saga_state = {
-    "saga_id": "saga-123",
-    "status": "in_progress",
-    "steps_completed": []
-}
-
-# Step 1
-result1 = await account_service.debit(account_a, 100)
-if not result1.success:
-    return fail_saga("Insufficient funds")
-
-saga_state["steps_completed"].append("debit_a")
-
-# Step 2
-result2 = await account_service.credit(account_b, 100)
-if not result2.success:
-    # Compensate step 1
-    await account_service.credit(account_a, 100)
-    return fail_saga("Account B invalid")
-
-saga_state["status"] = "completed"
-return success_saga()
-```
-
-**Saga State Persistence:**
-```
-CREATE TABLE saga_state (
-    saga_id UUID PRIMARY KEY,
-    saga_type VARCHAR(50),
-    current_step INTEGER,
-    max_steps INTEGER,
-    status VARCHAR(20),
-    payload JSONB,
-    steps_completed JSONB,
-    created_at TIMESTAMP,
-    updated_at TIMESTAMP
-);
-
-After each step:
-UPDATE saga_state
-SET
-    current_step = current_step + 1,
-    steps_completed = jsonb_array_append(steps_completed, 'step_name'),
-    updated_at = NOW()
-WHERE saga_id = $1;
-
-On failure, load saga state and execute compensations
-```
-
-**Idempotency for Saga Steps:**
-```
-Each saga step must be idempotent:
-
-Debit Operation:
-async def debit_account(account_id, amount, saga_id):
-    # Check if already processed
-    existing = await db.fetchone(
-        "SELECT * FROM transactions WHERE saga_id = $1 AND operation = 'debit'",
-        saga_id
-    )
-    if existing:
-        return {"success": True, "transaction_id": existing.id}
-
-    # Process debit
-    result = await db.execute(
-        "UPDATE accounts SET balance = balance - $1 WHERE id = $2 AND balance >= $1",
-        amount, account_id
-    )
-
-    if result.rowcount == 0:
-        return {"success": False, "error": "Insufficient funds"}
-
-    # Record transaction
-    await db.execute(
-        "INSERT INTO transactions (saga_id, account_id, amount, operation) VALUES ($1, $2, $3, 'debit')",
-        saga_id, account_id, amount
-    )
-
-    return {"success": True}
-
-Compensating Operation:
-async def compensate_debit(account_id, amount, saga_id):
-    await credit_account(account_id, amount, saga_id)
-```
-
-## Event Sourcing
-
-### Core Concepts
-
-**Event Store:**
-```
-All state changes stored as immutable events
-
-Example: Bank Account
-
-Events:
-1. AccountOpened { accountId: "acc-123", customerId: "cust-456", initialBalance: 0 }
-2. MoneyDeposited { accountId: "acc-123", amount: 1000, timestamp: "2025-01-15T10:00:00Z" }
-3. MoneyWithdrawn { accountId: "acc-123", amount: 200, timestamp: "2025-01-16T14:30:00Z" }
-4. MoneyDeposited { accountId: "acc-123", amount: 500, timestamp: "2025-01-17T09:15:00Z" }
-
-Current Balance = 0 + 1000 - 200 + 500 = 1300
-
-Replay all events to reconstruct current state
-```
-
-**Event Schema:**
-```json
-{
-  "eventId": "evt-789",
-  "aggregateId": "acc-123",
-  "aggregateType": "BankAccount",
-  "eventType": "MoneyDeposited",
-  "eventVersion": "1.0",
-  "timestamp": "2025-01-15T10:00:00Z",
-  "correlationId": "corr-456",
-  "causationId": "cmd-123",
-  "payload": {
-    "amount": 1000,
-    "currency": "USD",
-    "source": "wire_transfer"
-  },
-  "metadata": {
-    "userId": "user-789",
-    "ipAddress": "192.168.1.1"
-  }
-}
-```
-
-### Snapshots
-
-**Problem:** Replaying thousands of events is slow.
-
-**Solution:** Periodic snapshots.
-
-```
-Event Stream:
-1. AccountOpened (version 1)
-2. MoneyDeposited (version 2)
-...
-1000. MoneyDeposited (version 1000)
-[SNAPSHOT at version 1000: balance = $50,000]
-1001. MoneyWithdrawn (version 1001)
-...
-1500. MoneyDeposited (version 1500)
-
-To get current state:
-1. Load snapshot at version 1000 (balance = $50,000)
-2. Replay events 1001-1500 (only 500 events)
-
-Much faster than replaying all 1500 events
-
-Snapshot Strategy:
-- Every 100 events
-- Or every 24 hours
-- Async background process
-```
-
-**Snapshot Table:**
-```sql
-CREATE TABLE snapshots (
-    aggregate_id UUID,
-    aggregate_type VARCHAR(50),
-    version INTEGER,
-    state JSONB,
-    created_at TIMESTAMP,
-    PRIMARY KEY (aggregate_id, version)
-);
-
-CREATE INDEX idx_latest_snapshot ON snapshots(aggregate_id, version DESC);
-```
-
-### Event Schema Evolution
-
-**Challenge:** Events are immutable, but requirements change.
-
-**Strategies:**
-
-**1. Event Versioning:**
-```
-Version 1:
-{
-  "eventType": "OrderPlaced",
-  "eventVersion": "1.0",
-  "payload": {
-    "orderId": "123",
-    "amount": 99.99
-  }
-}
-
-Version 2 (added customer email):
-{
-  "eventType": "OrderPlaced",
-  "eventVersion": "2.0",
-  "payload": {
-    "orderId": "123",
-    "amount": 99.99,
-    "customerEmail": "customer@example.com"
-  }
-}
-
-Event Handler:
-def handle_order_placed(event):
-    if event.eventVersion == "1.0":
-        # Handle old format
-        process_order_v1(event.payload)
-    elif event.eventVersion == "2.0":
-        # Handle new format
-        process_order_v2(event.payload)
-```
-
-**2. Event Upcasting:**
-```
-Transform old events to new format during replay:
-
-def upcast_event(event):
-    if event.eventType == "OrderPlaced" and event.eventVersion == "1.0":
-        # Transform to v2.0
-        return {
-            "eventType": "OrderPlaced",
-            "eventVersion": "2.0",
-            "payload": {
-                **event.payload,
-                "customerEmail": "unknown@example.com"  # Default value
-            }
-        }
-    return event
-```
-
-**3. Event Transformation:**
-```
-Create new event types, keep old ones for historical accuracy:
-
-Old: OrderPlaced
-New: OrderPlacedV2
-
-Projections handle both:
-- Old events for historical data
-- New events for current processing
-```
-
-## Data Synchronization
-
-### Change Data Capture (CDC)
-
-**Purpose:** Capture database changes and publish as events.
-
-**How It Works:**
-```
-Database transaction log → CDC Tool → Event Stream
-
-Example with Debezium:
-
-PostgreSQL:
-INSERT INTO orders (id, customer_id, total) VALUES (123, 456, 99.99);
-
-Debezium captures:
-{
-  "before": null,
-  "after": {
-    "id": 123,
-    "customer_id": 456,
-    "total": 99.99,
-    "created_at": "2025-01-15T10:00:00Z"
-  },
-  "op": "c",  // create
-  "ts_ms": 1705314000000
-}
-
-Published to Kafka topic: postgres.public.orders
-
-Other services subscribe and update their read models
-```
-
-**Benefits:**
-```
-✓ No application code changes
-✓ Guaranteed delivery (based on database transaction log)
-✓ Captures all changes (even from direct DB access)
-✓ Low latency
-✓ Ordering preserved
-
-Use Cases:
-- Keep search index synchronized
-- Update cache automatically
-- Replicate to data warehouse
-- Trigger workflows on database changes
-```
-
-### Materialized Views
-
-**Purpose:** Pre-computed denormalized views for fast queries.
-
-**Pattern:**
-```
-Event-Driven Materialized View:
-
-1. Services publish domain events
-2. View service subscribes to events
-3. Updates materialized view in real-time
-
-Example: Order Summary View
-
-Events:
-- order.created
-- order.payment_received
-- order.shipped
-- order.delivered
-
-Materialized View:
-CREATE TABLE order_summary (
-    order_id UUID PRIMARY KEY,
-    customer_id UUID,
-    customer_name VARCHAR(255),
-    order_date TIMESTAMP,
-    total_amount DECIMAL,
-    status VARCHAR(50),
-    items_count INTEGER,
-    last_updated TIMESTAMP
-);
-
-View Service:
-async def on_order_created(event):
-    await db.execute(
-        "INSERT INTO order_summary (order_id, customer_id, status, ...) VALUES (...)",
-        event.data
-    )
-
-async def on_order_shipped(event):
-    await db.execute(
-        "UPDATE order_summary SET status = 'shipped', last_updated = NOW() WHERE order_id = $1",
-        event.order_id
-    )
-```
-
-## Data Partitioning
-
-### Horizontal Partitioning (Sharding)
-
-**When to Use:**
-```
-- Single database can't handle load
-- Data size exceeds single server capacity
-- Want to distribute geographically
-```
-
-**Sharding Strategies:**
-
-**1. Hash-Based Sharding:**
-```
-Shard = hash(customer_id) % num_shards
-
-customer_id: cust-123 → hash → 7234 → mod 4 → Shard 2
-customer_id: cust-456 → hash → 9812 → mod 4 → Shard 0
-
-Pros:
-- Even distribution
-- Simple to implement
-
-Cons:
-- Adding shards requires re-sharding
-- Range queries difficult
-```
-
-**2. Range-Based Sharding:**
-```
-Shard 0: customer_id 0-999
-Shard 1: customer_id 1000-1999
-Shard 2: customer_id 2000-2999
-
-Pros:
-- Range queries efficient
-- Easy to add shards
-
-Cons:
-- Uneven distribution (hotspots)
-- Requires shard map
-```
-
-**3. Geography-Based Sharding:**
-```
-Shard US: customers in USA
-Shard EU: customers in Europe
-Shard APAC: customers in Asia-Pacific
-
-Pros:
-- Data locality (GDPR compliance)
-- Lower latency
-
-Cons:
-- Uneven distribution
-- Cross-shard queries complex
-```
-
-**Shard Management:**
-```
-Shard Map Service:
-
-GET /shard-location?customer_id=cust-123
-Response: { "shard": "shard-2", "endpoint": "db2.example.com" }
-
-Application logic:
-customer_id = request.customer_id
-shard_info = await shard_map.get_shard(customer_id)
-db_connection = connection_pool.get(shard_info.endpoint)
-result = await db_connection.query("SELECT * FROM customers WHERE id = $1", customer_id)
-```
-
-## Summary
-
-Data management in microservices requires careful design:
-
-**Key Principles:**
-- Database per service (non-negotiable)
-- Embrace eventual consistency where possible
-- Use Saga pattern for distributed transactions
-- Event sourcing for audit trail and temporal queries
-- CQRS for read/write optimization
-- CDC for data synchronization
-
-**Decision Framework:**
-- Strong consistency → Saga with careful compensation logic
-- Audit trail → Event sourcing
-- Complex queries → CQRS with read models
-- Large scale → Sharding with appropriate strategy
-
-Always design for failure: compensating transactions, idempotent operations, and proper monitoring are essential.